I have 500 files to plot and I want to do this automatically. I have the gnuplot script
that does the plotting with the file name hard coded. I would like to have a loop that calls gnuplot every iteration with a different file name, but it does not seem that gnuplot support command line arguments.
Is there an easy way? I also installed the gnuplot-python package in case I can do it via a python script.However, I couldn't find the api so it's a bit difficult to figure out.
Thank you!
You can transform your gnuplot script to a shell script by prepending the lines
#!/bin/sh
gnuplot << EOF
appending the line
EOF
and substituting every $ by \$. Then, you can substitute every occurence of the filename by $1 and call the shell script with the filename as parameter.
Regarding the $'s in Sven Marnach's solution (the lines between EOF are called a "here script" in shell parlance): in my experience, one uses shell variables as usual, but $s that are meant for gnuplot itself must be escaped.
Here is an example:
for distrib in "uniform" "correlated" "clustered"
do
gnuplot << EOF
# gnuplot preamble omitted for brevity
set output "../plots/$distrib-$eps-$points.pdf"
set title "$distrib distribution, eps = $eps, #points = $points"
plot "../plots/dat/$distrib-$eps-$points.dat" using 1:(\$2/$points) title "exact NN"
EOF
done
Note the backslash escaping the dollar so that gnuplot sees it.
A simple way is to generate 500 gnuplot scripts like so:
for filename in list_of_files:
outfile = open(filename + '-script', 'w')
outfile.write(script_text % (filename,))
where script_text is the text of your gnuplot script with the filename replaced with a %s.
I've done this some times. But don't use EOF, because you cannot write on bash inside the << EOF and EOF tags. Depending on the names of the files you can do it in diferent ways.
a) If the filenames are loopable (sort of 1.dat 2.dat 3.dat, etc.)
#!/bin/bash
for((i=0;i<1;i++)) do
echo "plot '-' u 1:2"
for((j=1;j<=3;j++)) do
cat $i.dat
echo "e"
done
done | gnuplot -persist
The first loop is a kind of buffer to feed it all to gnuplot.
b) If the filenames aren't loopable (such as ñlasjkd.dat ajñljd.mov añlsjkd.gif) you first need to move them to a new folder. Then do
#!/bin/bash
ffiles=$(ls | xargs) # a list of the folder's files
# Use the list to pipe all the data to gnuplot using cat
for((i=0;i<1;i++)) do
echo "plot '-' u 1:2 w lp";
cat $ffiles;
echo "e";
done | gnuplot -persist
c) If you want some more, that is: to keep the information of the separated files just on one file... but maintaning the datasheets alive use "index" of gnuplot (if gnuplot reads two black lines guesses that is another datasheet)
#!/bin/bash
ffiles=$(ls|xargs)
ls $ffiles > ffiles.list # A file with the folder's files
while read line
do
cat $line;
echo -e; echo -e;
done < ffiles.list > alldata.dat
# ^-feeding ffiles.list to the while loop
# and writing the file alldata.dat
now you can go to gnuplot and acces to one datasheet
plot 'alldata.dat' index 1 u 1:2
and you will see the first file appearing on the list "ffiles.list". If you whant to see more than one, say 4
plot 'alldata.dat' index 1:4 u 1:2
tricky but easy.
Unless I'm misunderstanding your question, there's an easier way.
In the gnuplot script, replace all occurrences of your filename with $0.
Then in bash, loop over your plot files (let's say they are .txt files in $plotsDir) and call gnuplot:
for f in "$plotsDir/*.txt"; do
echo "call '$f'" | gnuplot
done
Here is one way to do this. Your gnuplot input file, say plot.gp, contains a line
plot "data"
or something like that. Save the lines before this line in plot1.gp and the lines after this line in plot2.gp. Now call gnuplot as
gnuplot plot1.gp -e `plot "data"` plot2.gp
which facilitates passing the name of data on the command line.
I create a file named test.txt containing plot [0:20] x;
I run gnuplot test.txt and I see that gnuplot has indeed read contents of my file, so it does support arguments at runtime.
Different methods of solving this problem are also covered in How to pass command line argument to gnuplot? :
Using -e
Using -c or call with ARG0 like variables (gnuplot >= 5)
Using call with $0 like variables (gnuplot <= 4.6.6)
Using environment variables and system()
Piping the program into gnuplot
Related
I have a number of xml files (which can be considered as text files in this situation) that I wish to concatenate. Normally I think I could do something like this from a Linux command prompt or bash script:
cat somefile.xml someotherfile.xml adifferentfile.xml > out.txt
Except that in this case, I need to copy the first file in its entirety EXCEPT for the very last line, but in all subsequent files omit exactly the first four lines and the very last line (technically, I do need the last line from the last file but it is always the same, so I can easily add it with a separate statement).
In all these files the first four lines and the last line are always the same, but the contents in between varies. The names of the xml files can be hardcoded into the script or read from a separate data file, and the number of them may vary from time to time but always will number somewhere around 10-12.
I'm wondering what would be the easiest and most understandable way to do this. I think I would prefer either a bash script or maybe a python script, though I generally understand bash scripts a little better. What I can't get my head around is how to trim off just those first four lines (on all but the first file) and the last line of every file. My suspicion is there's some Linux command that can do this, but I have no idea what it would be. Any suggestions?
sed '$d' firstfile > out.txt
sed --separate '1,4d; $d' file2 file3 file4 >> out.txt
sed '1,4d' lastfile >> out.txt
It's important to use the --separate (or shorter -s) option so that the range statements 1,4 and $ apply to each file individually.
From GNU sed manual:
-s, --separate
By default, sed will consider the files specified on the command line as a single continuous long stream. This GNU sed
extension allows the user to consider them as separate files.
Do it in two steps:
use the head command (to get the lines you want)
Use cat to combine
You could use temp files or bash trickery.
I am using a bash script to call google-api's upload_video.py (https://developers.google.com/youtube/v3/guides/uploading_a_video )
I have a mp4 called output.mp4 which I would like to upload.
The problem is I cannot get my array to work how I would like.
This new line character is "required" because my arguments to python script contain spaces.
Here is a simplified version of my bash script:
# Operator may change these
hold=100
location="Foo, Montana "
declare -a file_array=("unique_ID_0" "unique_ID_1")
upload_file=upload_file.txt
upload_movie=output.mp4
# Hit enter at end b/c \n not recognized
upload_title=$location' - '${file_array[0]}' - Hold '$hold' Sweeps
'
upload_description='The spectrum recording was made in at '$location'.
'
# Overwrite with 1st call > else apppend >>
echo "$upload_title" > $upload_file
echo "$upload_description" >> $upload_file
# Load each line of text file into array
IFS=$'\n'
cmd_google=$(<$upload_file)
unset IFS
nn=1
for i in "${cmd_google[#]}"
do
echo "$i"
# Delete last character: \n
#i=${i[-nn]%?}
#i=${i: : -nn}
#i=${i::${#i}-nn}
i=${i%?}
#i=${i#"\n"}
#i=${i%"\n"}
echo "$i"
done
python upload_video.py --file=$upload_movie --title="${cmd_google[0]}" --description="${cmd_google[1]}"
At first I attempted to remove the new line character, but it appears that the enter or \n is not working how I would like, each line is not separate. It writes the title and description as one line.
How do I modify my bash script to recognize a newline character?
This is much simpler than you are making it.
# Operator may change these
hold=100
location="Foo, Montana"
declare -a file_array=("unique_ID_0" "unique_ID_1")
upload_file=upload_file.txt
upload_movie=output.mp4
upload_title="$location - ${file_array[0]} - Hold $hold Sweeps"
upload_description="The spectrum recording was made in at $location."
cat <<EOF > "$upload_file"
$upload_title
$upload_description
EOF
# ...
readarray -t cmd_google < "$upload_file"
python upload_video.py --file="$upload_movie" --title="${cmd_google[0]}" --description="${cmd_google[1]}"
I suspect the readarray command is all you are really looking for, since much of the above code is simply creating a file that I assume you are receiving already created.
I figured it out with help from chepner's answer. My question hid the fact that I wanted to write new line characters into the video's description.
Instead of adding a new line character in the bash script, it is much easier to have a text file which contains the correctly formatted script and read it in, then concatenate it with run-time specific variable.
In my case the correctly formatted text is called description.txt:
Here is a snip of my description.txt which contains newline characters
Here is my final version of the script:
# Operator may change these
hold=100
location="Foo, Montana"
declare -a file_array=("unique_ID_0" "unique_ID_1")
upload_title="$location - ${file_array[0]} - Hold $hold Sweeps"
upload_description="The spectrum recording was made in at $location. "
# Read in script which contains newline
temp=$(<description.txt)
# Concatenate them
upload_description="$upload_description$temp"
upload_movie=output.mp4
python upload_video.py --file="$upload_movie" --title="$upload_title" --description="$upload_description"
Trying to print filename of files that don't have 12 columns.
This works at the command line:
for i in *dim*; do awk -F',' '{if (NR==1 && NF!=12)print FILENAME}' $i; done;
When I try to embed this in subprocess.call in a python script, it doesn't work:
subprocess.call("""for %i in (*dim*.csv) do (awk -F, '{if ("NR==1 && NF!=12"^) {print FILENAME}}' %i)""", shell=True)
The first error I received was "Print is unexpected at this time" so I googled and added ^ within parentheses. Next error was "unexpected newline or end of string" so googled again and added the quotes around NR==1 && NF!=12. With the current code it's printing many lines in each file so I suspect something is wrong with the if statement. I've used awk and for looped before in this style in subprocess.call but not combined and with an if statement.
Multiple input files in AWK
In the string you are passing to subprocess.call(), your if statement is evaluating a string (probably not the comparison you want). It might be easier to just simplify the shell command by doing everything in AWK. You are executing AWK for every $i in the shell's for loop. Since you can give multiple input files to AWK, there is really no need for this loop.
You might want to scan through the entire files until you find any line that has other than 12 fields, and not only check the first line (NR==1). In this case, the condition would be only NF!=12.
If you want to check only the first line of each file, then NR==1 becomes FNR==1 when using multiple files. NR is the "number of records" (across all input files) and FNR is "file number of records" for the current input file only. These are special built-in variables in AWK.
Also, the syntax of AWK allows for the blocks to be executed only if the line matches some condition. Giving no condition (as you did) runs the block for every line. For example, to scan through all files given to AWK and print the name of a file with other than 12 fields on the first line, try:
awk -F, 'FNR==1 && NF!=12{print FILENAME; nextfile}' *dim*.csv
I have added the .csv to your wildcard *dim* as you had in the Python version. The -F, of course changes the field separator to a comma from the default space. For every line in each file, AWK checks if the number of fields NF is 12, if it's not, it executes the block of code, otherwise it goes on to the next line. This block prints the FILENAME of the current file AWK is processing, then skips to the beginning of the next file with nextfile.
Try running this AWK version with your subprocess module in Python:
subprocess.call("""awk -F, 'FNR==1 && NF!=12{print FILENAME; nextfile}' *dim*.csv""", shell=True)
The triple quotes makes it a literal string. The output of AWK goes to stdout and I'm assuming you know how to use this in Python with the subprocess module.
Using only Python
Don't forget that Python is itself an expressive and powerful language. If you are already using Python, it may be simpler, easier, and more portable to use only Python instead of a mixture of Python, bash, and AWK.
You can find the names of files (selected from *dim*.csv) with the first line of each file having other than 12 comma-separated fields with:
import glob
files_found = []
for filename in glob.glob('*dim*.csv'):
with open(filename, 'r') as f:
firstline = f.readline()
if len(firstline.split(',')) != 12:
files_found.append(filename)
f.close()
print(files_found)
The glob module gives the listing of files matching the wildcard pattern *dim*.csv. The first line of each of these files is read and split into fields separated by commas. If the number of these fields is not 12, it is added to the list files_found.
I have a python script that can be called in Windows as:
python.exe do_my_work.py param > output.csv
param is an input parameter.
I also have a txt file called params.txt that contains many lines, each line is a value for the parameter of the python script:
hello
world
this
is
test
Because params.txt consists of many many lines, so I would like to write a .batch file that read params.txt line by line then calls python script with the read line as the parameter. The pseudo code as:
open params.txt;
while !eof do:
read a line;
call "python.exe do_my_work.py $line >> output.csv";
end while;
close params.txt;
Could you please show me how to solve this.
P/S: I don't want to change do_my_work.py source code due to some special reasons.
#echo off
set "params=C:\params.txt"
set "output=output.csv"
for /f "usebackq tokens=* delims=" %%# in ("%params%") do (
python do_my_work.py %%# 1>>"%output%"
)
I would like to call a Python script from within a Bash while loop. However, I do not understand very well how to use appropriately the while loop (and maybe variable) syntax of the Bash. The behaviour I am looking for is that, while a file still contains lines (DNA sequences), I am calling a Python script to extract groups of sequences so that another program (dialign2) can align them. Finally, I add the alignments to a result file. Note: I am not trying to iterate over the file. What should I change in order for the Bash while loop to work? I also want to be sure that the while loop will re-check the changing file.txt on each loop. Here is my attempt:
#!/bin/bash
# Call a python script as many times as needed to treat a text file
c=1
while [ `wc -l file.txt` > 0 ] ; # Stop when file.txt has no more lines
do
echo "Python script called $c times"
python script.py # Uses file.txt and removes lines from it
# The Python script also returns a temp.txt file containing DNA sequences
c=$c + 1
dialign -f temp.txt # aligns DNA sequences
cat temp.fa >>results.txt # append DNA alignements to result file
done
Thanks!
No idea why you want to do this.
c=1
while [[ -s file.txt ]] ; # Stop when file.txt has no more lines
do
echo "Python script called $c times"
python script.py # Uses file.txt and removes lines from it
c=$(($c + 1))
done
try -gt to eliminate the shell metacharacter >
while [ `wc -l file.txt` -gt 0 ]
do
...
c=$[c + 1]
done
#OP if you want to loop through a file , just use while read loop. Also, you are not using the variables $c as well as the line. Are you passing each line to your Python script? Or you just calling your Python script whenever a line is encountered? (your script going to be slow if you do that)
while true
do
while read -r line
do
# if you are taking STDIN in myscript.py, then something must be passed to
# myscript.py, if not i really don't understand what you are doing.
echo "$line" | python myscript.py > temp.txt
dialign -f temp.txt # aligns DNA sequences
cat temp.txt >>results.txt
done <"file.txt"
if [ ! -s "file.txt" ]; break ;fi
done
Lastly, you could have done everything in Python. the way to iterate "file.txt" in Python is simply
f=open("file.txt"):
for line in f:
print "do something with line"
print "or bring what you have in myscript.py here"
f.close()
The following should do what you say you want:
#!/bin/bash
c=1
while read line;
do
echo "Python script called $c times"
# $line contains a line of text from file.txt
python script.py
c=$((c + 1))
done < file.txt
However, there is no need to use bash, to iterate over the lines in a file. You can do that quite easily without ever leaving python:
myfile = open('file.txt', 'r')
for count, line in enumerate(myfile):
print '%i lines in file' % (count + 1,)
# the variable "line" contains the line of text from the file.txt
# Do your thing here.