I am using a bash script to call google-api's upload_video.py (https://developers.google.com/youtube/v3/guides/uploading_a_video )
I have a mp4 called output.mp4 which I would like to upload.
The problem is I cannot get my array to work how I would like.
This new line character is "required" because my arguments to python script contain spaces.
Here is a simplified version of my bash script:
# Operator may change these
hold=100
location="Foo, Montana "
declare -a file_array=("unique_ID_0" "unique_ID_1")
upload_file=upload_file.txt
upload_movie=output.mp4
# Hit enter at end b/c \n not recognized
upload_title=$location' - '${file_array[0]}' - Hold '$hold' Sweeps
'
upload_description='The spectrum recording was made in at '$location'.
'
# Overwrite with 1st call > else apppend >>
echo "$upload_title" > $upload_file
echo "$upload_description" >> $upload_file
# Load each line of text file into array
IFS=$'\n'
cmd_google=$(<$upload_file)
unset IFS
nn=1
for i in "${cmd_google[#]}"
do
echo "$i"
# Delete last character: \n
#i=${i[-nn]%?}
#i=${i: : -nn}
#i=${i::${#i}-nn}
i=${i%?}
#i=${i#"\n"}
#i=${i%"\n"}
echo "$i"
done
python upload_video.py --file=$upload_movie --title="${cmd_google[0]}" --description="${cmd_google[1]}"
At first I attempted to remove the new line character, but it appears that the enter or \n is not working how I would like, each line is not separate. It writes the title and description as one line.
How do I modify my bash script to recognize a newline character?
This is much simpler than you are making it.
# Operator may change these
hold=100
location="Foo, Montana"
declare -a file_array=("unique_ID_0" "unique_ID_1")
upload_file=upload_file.txt
upload_movie=output.mp4
upload_title="$location - ${file_array[0]} - Hold $hold Sweeps"
upload_description="The spectrum recording was made in at $location."
cat <<EOF > "$upload_file"
$upload_title
$upload_description
EOF
# ...
readarray -t cmd_google < "$upload_file"
python upload_video.py --file="$upload_movie" --title="${cmd_google[0]}" --description="${cmd_google[1]}"
I suspect the readarray command is all you are really looking for, since much of the above code is simply creating a file that I assume you are receiving already created.
I figured it out with help from chepner's answer. My question hid the fact that I wanted to write new line characters into the video's description.
Instead of adding a new line character in the bash script, it is much easier to have a text file which contains the correctly formatted script and read it in, then concatenate it with run-time specific variable.
In my case the correctly formatted text is called description.txt:
Here is a snip of my description.txt which contains newline characters
Here is my final version of the script:
# Operator may change these
hold=100
location="Foo, Montana"
declare -a file_array=("unique_ID_0" "unique_ID_1")
upload_title="$location - ${file_array[0]} - Hold $hold Sweeps"
upload_description="The spectrum recording was made in at $location. "
# Read in script which contains newline
temp=$(<description.txt)
# Concatenate them
upload_description="$upload_description$temp"
upload_movie=output.mp4
python upload_video.py --file="$upload_movie" --title="$upload_title" --description="$upload_description"
Related
I have a long list of unformatted data say data.txt where each set is started with a header and ends with a blank line, like:
TypeA/Price:20$
alexmob
moblexto
unkntom
TypeB/Price:25$
moblexto2
unkntom0
alexmob3
poptop9
tyloret
TypeC/Price:30$
rtyuoper0
kunlohpe6
mobryhox
Now, i want to add the header of each set with it's content side by side with comma separated. Like:
alexmob,TypeA/Price:20$
moblexto,TypeA/Price:20$
unkntom,TypeA/Price:20$
moblexto2,TypeB/Price:25$
unkntom0,TypeB/Price:25$
alexmob3,TypeB/Price:25$
poptop9,TypeB/Price:25$
tyloret,TypeB/Price:25$
rtyuoper0,TypeC/Price:30$
kunlohpe6,TypeC/Price:30$
mobryhox,TypeC/Price:30$
so that whenever i will grep with one keyword, relevant content along with the header comes together. Like:
$grep mob data.txt
alexmob,TypeA/Price:20$
moblexto,TypeA/Price:20$
moblexto2,TypeB/Price:25$
alexmob3,TypeB/Price:25$
mobryhox,TypeC/Price:30$
I am newbie on bash scripting as well as python and recently started learning these, so would really appreciate any simple bash scipting (using sed/awk) or python scripting.
Using sed
$ sed '/Type/{h;d;};/[a-z]/{G;s/\n/,/}' input_file
alexmob,TypeA/Price:20$
moblexto,TypeA/Price:20$
unkntom,TypeA/Price:20$
moblexto2,TypeB/Price:25$
unkntom0,TypeB/Price:25$
alexmob3,TypeB/Price:25$
poptop9,TypeB/Price:25$
tyloret,TypeB/Price:25$
rtyuoper0,TypeC/Price:30$
kunlohpe6,TypeC/Price:30$
mobryhox,TypeC/Price:30$
Match lines containing Type, hold it in memory and delete it.
Match lines with alphabetic characters, append G the contents of the hold space. Finally, sub new line for a comma.
I would use GNU AWK for this task following way, let file.txt content be
TypeA/Price:20$
alexmob
moblexto
unkntom
TypeB/Price:25$
moblexto2
unkntom0
alexmob3
poptop9
tyloret
TypeC/Price:30$
rtyuoper0
kunlohpe6
mobryhox
then
awk '/^Type/{header=$0;next}{print /./?$0 ";" header:$0}' file.txt
output
alexmob;TypeA/Price:20$
moblexto;TypeA/Price:20$
unkntom;TypeA/Price:20$
moblexto2;TypeB/Price:25$
unkntom0;TypeB/Price:25$
alexmob3;TypeB/Price:25$
poptop9;TypeB/Price:25$
tyloret;TypeB/Price:25$
rtyuoper0;TypeC/Price:30$
kunlohpe6;TypeC/Price:30$
mobryhox;TypeC/Price:30$
Explanation: If line starts with (^) Type set header value to that line ($0) and go to next line. For every line print if it does contain at least one character (/./) line ($0) concatenated with ; and header, otherwise print line ($0) as is.
(tested in GNU Awk 5.0.1)
Using any awk in any shell on every Unix box regardless of which characters are in your data:
$ awk -v RS= -F'\n' -v OFS=',' '{for (i=2;i<=NF;i++) print $i, $1; print ""}' file
alexmob,TypeA/Price:20$
moblexto,TypeA/Price:20$
unkntom,TypeA/Price:20$
moblexto2,TypeB/Price:25$
unkntom0,TypeB/Price:25$
alexmob3,TypeB/Price:25$
poptop9,TypeB/Price:25$
tyloret,TypeB/Price:25$
rtyuoper0,TypeC/Price:30$
kunlohpe6,TypeC/Price:30$
mobryhox,TypeC/Price:30$
So I have a python file with a ton of lines in it that I want to read into python then echo into another file over a socket.
Assuming I have file foo.py
import os
os.popen('some command blah')
print("some other commands, doesn't matter")
Then I try and open the file, read all the lines, and echo each line into a new file.
Something along the lines of
scriptCode = open(os.path.realpath(__file__)).readlines()
for line in scriptCode:
connection.send("echo " + line + " >> newfile.py")
print("file transfered!")
However, when I do this, the command is executed in the remote shell.
So my question:
How do I safely echo text into a file without executing any keywords in it?
What have I tried?
Adding single quotes around line
Adding single quotes around line and then a backslash to single quotes in line
Things I've considered but haven't tried yet:
Base64 encoding the line until on the remote machine then decoding it (I don't want to do this because there's no guarentee it'll have this command)
I know this is odd. Why am I doing this?
I'm building a pentesting reverse shell handler.
shlex.quote will:
Return a shell-escaped version of the string s. The returned value is a string that can safely be used as one token in a shell command line, for cases where you cannot use a list.
Much safer than trying to quote a string by yourself.
I'm coding a bash script in order to generate .spec file (building RPM) automatically. I read all the files in the directory (which I hope to convert it into rpm package) and write all the paths of files needed to install in .spec file, I realize that I need to shorten them. An example:
/tmp/a/1.jpg
/tmp/a/2.conf
/tmp/a/b/srf.cfg
/tmp/a/b/ssp.cfg
/tmp/a/conf_web_16.2/c/.htaccess
/tmp/a/conf_web_16.2/c/.htaccess.WebProv
/tmp/a/conf_web_16.2/c/.htprofiles
=> What I want to get:
/tmp/a/*.jpg
/tmp/a/*.conf
/tmp/a/b/*.cfg
/tmp/a/conf_web_16.2/c/*
/tmp/a/conf_web_16.2/c/*.WebProv
You guys please give me some advice about my problem. I hope you guys can suggest your idea in bash shell, python or C. Thank you in advance.
To convert any file name which contains a dot in a character other than the first into a wildcard covering the part up to just before the dot, and any remaining files to just a wildcard,
sed -e 's%/[^/][^/]*\(\.[^./]*\)$%/*\1%' -e t -e 's%/[^/]*$%/*%'
The behavior of sed is to read its input one line at a time, and execute the script of commands on each in turn. The s%foo%bar% substitution command replaces a regex match with a string, and the t command causes the script to skip further substitutions if one was already performed on the current line. (I'm simplifying somewhat.) The first regex matches file names which contain a dot in a position other than the first, and captures the match from the dot through the end in a back reference which is used in the substitution as well (that's the \1). The second is applied to any remaining file names, because of the t command in between.
The result will probably need to be piped to sort -u to remove any duplicates.
If you don't have a list of the file names, you can use find to pipe in a listing.
find . -type f | sed ... | sort -u
Trying to print filename of files that don't have 12 columns.
This works at the command line:
for i in *dim*; do awk -F',' '{if (NR==1 && NF!=12)print FILENAME}' $i; done;
When I try to embed this in subprocess.call in a python script, it doesn't work:
subprocess.call("""for %i in (*dim*.csv) do (awk -F, '{if ("NR==1 && NF!=12"^) {print FILENAME}}' %i)""", shell=True)
The first error I received was "Print is unexpected at this time" so I googled and added ^ within parentheses. Next error was "unexpected newline or end of string" so googled again and added the quotes around NR==1 && NF!=12. With the current code it's printing many lines in each file so I suspect something is wrong with the if statement. I've used awk and for looped before in this style in subprocess.call but not combined and with an if statement.
Multiple input files in AWK
In the string you are passing to subprocess.call(), your if statement is evaluating a string (probably not the comparison you want). It might be easier to just simplify the shell command by doing everything in AWK. You are executing AWK for every $i in the shell's for loop. Since you can give multiple input files to AWK, there is really no need for this loop.
You might want to scan through the entire files until you find any line that has other than 12 fields, and not only check the first line (NR==1). In this case, the condition would be only NF!=12.
If you want to check only the first line of each file, then NR==1 becomes FNR==1 when using multiple files. NR is the "number of records" (across all input files) and FNR is "file number of records" for the current input file only. These are special built-in variables in AWK.
Also, the syntax of AWK allows for the blocks to be executed only if the line matches some condition. Giving no condition (as you did) runs the block for every line. For example, to scan through all files given to AWK and print the name of a file with other than 12 fields on the first line, try:
awk -F, 'FNR==1 && NF!=12{print FILENAME; nextfile}' *dim*.csv
I have added the .csv to your wildcard *dim* as you had in the Python version. The -F, of course changes the field separator to a comma from the default space. For every line in each file, AWK checks if the number of fields NF is 12, if it's not, it executes the block of code, otherwise it goes on to the next line. This block prints the FILENAME of the current file AWK is processing, then skips to the beginning of the next file with nextfile.
Try running this AWK version with your subprocess module in Python:
subprocess.call("""awk -F, 'FNR==1 && NF!=12{print FILENAME; nextfile}' *dim*.csv""", shell=True)
The triple quotes makes it a literal string. The output of AWK goes to stdout and I'm assuming you know how to use this in Python with the subprocess module.
Using only Python
Don't forget that Python is itself an expressive and powerful language. If you are already using Python, it may be simpler, easier, and more portable to use only Python instead of a mixture of Python, bash, and AWK.
You can find the names of files (selected from *dim*.csv) with the first line of each file having other than 12 comma-separated fields with:
import glob
files_found = []
for filename in glob.glob('*dim*.csv'):
with open(filename, 'r') as f:
firstline = f.readline()
if len(firstline.split(',')) != 12:
files_found.append(filename)
f.close()
print(files_found)
The glob module gives the listing of files matching the wildcard pattern *dim*.csv. The first line of each of these files is read and split into fields separated by commas. If the number of these fields is not 12, it is added to the list files_found.
I have 500 files to plot and I want to do this automatically. I have the gnuplot script
that does the plotting with the file name hard coded. I would like to have a loop that calls gnuplot every iteration with a different file name, but it does not seem that gnuplot support command line arguments.
Is there an easy way? I also installed the gnuplot-python package in case I can do it via a python script.However, I couldn't find the api so it's a bit difficult to figure out.
Thank you!
You can transform your gnuplot script to a shell script by prepending the lines
#!/bin/sh
gnuplot << EOF
appending the line
EOF
and substituting every $ by \$. Then, you can substitute every occurence of the filename by $1 and call the shell script with the filename as parameter.
Regarding the $'s in Sven Marnach's solution (the lines between EOF are called a "here script" in shell parlance): in my experience, one uses shell variables as usual, but $s that are meant for gnuplot itself must be escaped.
Here is an example:
for distrib in "uniform" "correlated" "clustered"
do
gnuplot << EOF
# gnuplot preamble omitted for brevity
set output "../plots/$distrib-$eps-$points.pdf"
set title "$distrib distribution, eps = $eps, #points = $points"
plot "../plots/dat/$distrib-$eps-$points.dat" using 1:(\$2/$points) title "exact NN"
EOF
done
Note the backslash escaping the dollar so that gnuplot sees it.
A simple way is to generate 500 gnuplot scripts like so:
for filename in list_of_files:
outfile = open(filename + '-script', 'w')
outfile.write(script_text % (filename,))
where script_text is the text of your gnuplot script with the filename replaced with a %s.
I've done this some times. But don't use EOF, because you cannot write on bash inside the << EOF and EOF tags. Depending on the names of the files you can do it in diferent ways.
a) If the filenames are loopable (sort of 1.dat 2.dat 3.dat, etc.)
#!/bin/bash
for((i=0;i<1;i++)) do
echo "plot '-' u 1:2"
for((j=1;j<=3;j++)) do
cat $i.dat
echo "e"
done
done | gnuplot -persist
The first loop is a kind of buffer to feed it all to gnuplot.
b) If the filenames aren't loopable (such as ñlasjkd.dat ajñljd.mov añlsjkd.gif) you first need to move them to a new folder. Then do
#!/bin/bash
ffiles=$(ls | xargs) # a list of the folder's files
# Use the list to pipe all the data to gnuplot using cat
for((i=0;i<1;i++)) do
echo "plot '-' u 1:2 w lp";
cat $ffiles;
echo "e";
done | gnuplot -persist
c) If you want some more, that is: to keep the information of the separated files just on one file... but maintaning the datasheets alive use "index" of gnuplot (if gnuplot reads two black lines guesses that is another datasheet)
#!/bin/bash
ffiles=$(ls|xargs)
ls $ffiles > ffiles.list # A file with the folder's files
while read line
do
cat $line;
echo -e; echo -e;
done < ffiles.list > alldata.dat
# ^-feeding ffiles.list to the while loop
# and writing the file alldata.dat
now you can go to gnuplot and acces to one datasheet
plot 'alldata.dat' index 1 u 1:2
and you will see the first file appearing on the list "ffiles.list". If you whant to see more than one, say 4
plot 'alldata.dat' index 1:4 u 1:2
tricky but easy.
Unless I'm misunderstanding your question, there's an easier way.
In the gnuplot script, replace all occurrences of your filename with $0.
Then in bash, loop over your plot files (let's say they are .txt files in $plotsDir) and call gnuplot:
for f in "$plotsDir/*.txt"; do
echo "call '$f'" | gnuplot
done
Here is one way to do this. Your gnuplot input file, say plot.gp, contains a line
plot "data"
or something like that. Save the lines before this line in plot1.gp and the lines after this line in plot2.gp. Now call gnuplot as
gnuplot plot1.gp -e `plot "data"` plot2.gp
which facilitates passing the name of data on the command line.
I create a file named test.txt containing plot [0:20] x;
I run gnuplot test.txt and I see that gnuplot has indeed read contents of my file, so it does support arguments at runtime.
Different methods of solving this problem are also covered in How to pass command line argument to gnuplot? :
Using -e
Using -c or call with ARG0 like variables (gnuplot >= 5)
Using call with $0 like variables (gnuplot <= 4.6.6)
Using environment variables and system()
Piping the program into gnuplot