Develop Reference

Extracting all JavaScript filenames from a log file using bash script

I have 4 different named log files, all with txt extensions. I need to write a bash script file that extracts JavaScript file names from any of these log files regardless of their names. The output of the script should not include the path, have to be unique, and sorted
After some research I came up with this:
cat logfile1.txt | grep '[^.(]*\.js' | awk -F " " '{print $7}' | sort | uniq -c| sort -nr
This code does only haft the job;
PRO: It does extract any JS, sorts it, and gives unique results.
CON: I need this in a file.sh not a command line as, it is now. Also, I'm getting the entire path to the JS file. I only need the file name jquery.js
I tried adding grep -v "*/name-of-path-before-JS" to block the result from giving me the full path but that isn't working.
I found someone who made something kind of similar using python;
source
filenames = set()
with open(r"/home/filelog.txt") as f:
for line in f:
end = line.rfind(".js") + 3 # 3 = len(".js")
start = line.rfind("/", 0, end) + 1 # 1 = len("/")
filename = line[start:end]
if filename.endswith(".js"):
filenames.add(filename)
for filename in sorted(filenames, key=str.lower):
print(filename)
Although is missing the sort and uniq options when giving the output it does give the results by only putting out filename.js and not the whole path as the command line I made. Also, I to add the path to the log.txt file while running the script and not just appended it as in the python script below.
Example;
$./LogReaderScript.sh File-log.txt

Would you please try the shell script LogReaderScript.sh:
#!/bin/bash
if [[ $# -eq 0 ]]; then # if no filenames are given
echo "usage: $0 logfile .." # then show the usage and abort
exit 1
fi
grep -hoE "[^/]+\.js" "$#" | sort | uniq -c | sort -nr
By setting the file as executable with chmod +x LogReaderScript.sh,
you can invoke:
./LogReaderScript.sh File-log.txt
If you want to process multiple files at a time, you can also say something
like:
./LogReaderScript.sh *.txt
-o option to grep tells grep to print the matched substrings only,
instead of printing the matched line.
-E option specifies extended regex as a pettern.
-h option suppresses the prefixed filenames on the output if multiple
files are given.
The pattern (regex) [^/]+\.js matches a sequence of any characters
other than a slash, and followed by a extention .js. It will match
the target filenames.
"$#" is expanded to the filename(s) passed as arguments to the script.

There's really no need to have a script as you can do the job with the oneliner, since you've mentioned you have multiple log files to parse i'm assuming this is a task you're doing on a regular basis.
In this case just define an alias in your .bashrc file with this oneliner:
cat $1 | awk '{print $7}' | grep '.js' | awk -F\/ '{print $NF}' | sort | uniq
Let's say you've created the alias parser then you'd just have to invoke parser /path/to/logfile.log
With the example logfile you've provided above, the output is:
➜ ~ cat logfile.txt | awk '{print $7}' | grep '.js' | awk -F\/ '{print $NF}' | sort | uniq
jquery.js
jquery.jshowoff.min.js
jshowoff.css
Explanation:
cat is used to parse the file and then pipe the content into..
awk which is extracting the 7th space separated field from the file, since those are apache access logs and you're searching for the requested file, the seventh field is what you need
grep is extracting only the javascript files, ie. those ending with the .js extension
awk is used again to print only the file name, we're defining a custom field separator this time with the -F flag, and executing the print command using the $NF argument which instructs awk to print only the last field
sort and uniq are self explanatory, we're sorting the output then printing only the first occurrence for each repeated value.
jquery.jshowoff.min.js looked like bogus to me and i suspected i did something wrong with my commands, but it's an actual line (280) in your logfile
75.75.112.64 - - [21/Apr/2013:17:32:23 -0700] "GET /include/jquery.jshowoff.min.js HTTP/1.1" 200 2553 "http://random-site.com/" "Mozilla/5.0 (iPod; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25" "random-site.com"

Bash script executing curl json query piped to python inline script for parsing json, proper execution

The long form of this command works but is not as readable as I would like:
curl -X GET http://somewhere.ext/json | python -c "import sys, json; print json.load(sys.stdin)['key']['valueIWant']
The result of that query is just a single value contained in key.valueIWant.
To try and make it more readable, I tried to do this but can't come up with the right syntax or there is something I'm missing fundamentally:
CURLCMD="curl -X GET http://somewhere.ext/json"
PYTHONCMD=$"python -c \"import sys, json; print json.load(sys.stdin)['key']['valueIWant']\""
$VALUE=$( $CURLCMD | $PYTHONCMD )
echo $CURLCMD yields:
curl -X GET http://somewhere.ext/json
This is correct.
echo $PYTHONCMD yields:
python -c "import sys, json; print json.load(sys.stdin)['key']['valueIWant']"
This is correct.
When I copy the output of those echos and pipe them together, the command works. When I try to find the short form expanding the variables, none of the variations I can think of works.
Most of the variations of $CURLCMD | $PYTHONCMD complain about an EOL character after the import. I think the curl output has a EOL with no new line character at the end of it, so maybe that is giving it a problem. I don't understand why the long version with the pipe works, but the expanded variable version doesn't.
Just a few of the other variations I've tried:
VALUE=$( $CURLCMD | $PYTHONCMD )
VALUE=$( CURLCMD ) | ( $PYTHONCMD )
VALUE=$( ${CURLCMD} | ${PYTHONCMD} )
(really I thought the last one was the closest to the long form, but I guess not.)
Is there a form of $CURLCMD | $PYTHONCMD that will work like how the long form works?
This works and is better readable, but seems like there ought to be a way to use variable expansion to do the same thing.
VALUE=$( $CURLCMD | python -c "import sys, json; print json.load(sys.stdin)['key']['valueIWant']")
(This is from memory as I'm not anywhere near the script right now.)

embed python in bash script

I'm pretty new to python and I'm trying to embed it in a bash script. I have the following bash shell script:
#!/bin/bash
while read line
do
ORD=`echo $line | cut -c 1-7`
if [[ -r ../FASTA_SEC/COMBI_RAW/${ORD}_COMBI_RAW.fa ]]
then
touch ../Results/Log_Files/Log_${ORD}.txt
for (( win = 2; win < 20; win += 2 )); do
printf 'The value of win is %d, now entereing inner loop\n' "$win"
for (( per = 50; per < 100; per += 2 )); do
printf 'Value of per is %d\n' "$per"
WCNO=`grep -o "n\|N" ../FASTA_SEC/COMBI_RAW/${ORD}_COMBI_RAW.fa | wc -l`
WCNF=`grep -o "n\|N" ../Results/COMBI/${ORD}_COMBI/${ORD}_w${win}_p${per}_COMBI.fa | wc -l`
WCAO=`grep -o "a\|A\|g\|G\|t\|T\|c\|C" ../FASTA_SEC/COMBI_RAW/${ORD}_COMBI_RAW.fa | wc -l`
WCAF=`grep -o "a\|A\|g\|G\|t\|T\|c\|C" ../Results/COMBI/${ORD}_COMBI/${ORD}_w${win}_p${per}_COMBI.fa | wc -l`
PERN=`echo print ($WCNO-$WCNF)/$WCNO. | python`
echo $ORD $PERN
done
done
else
echo ${ORD}"_COMBI_RAW.fa does not exist"
fi
done < ./Data_ORD.txt
I would like to carry out this percentage calculation stored in a variable calling python
PERN=`echo print ($WCNO-$WCNF)/$WCNO. | python`
Parenthesis are giving trouble so i get the following output when i run the bash script:
command substitution: line 17: syntax error near unexpected token `('
command substitution: line 17: `echo print ($WCNO-$WCNF)/$WCNO. | python'
Is it something wrong with bash? operation order are defined with parenthesis in python as far as I know.
Thank you,

bash is interpreting the ( when you don't want it to. You need to enclose the expression you're sending to python in quotes.
PERN=`echo "print ($WCNO-$WCNF)/$WCNO." | python`
If you're curious what bash thought you were doing...
The construct (...) indicates something to run inside a subshell. First, bash would interpolate the variables $WCNO and $WCNF. Then it would try to run the thing in the parens as a command, in a subshell. Unless that string coincidentally exists as a command, bash will complain because the command it tried to find does not exist.
See also this answer on Unix.SE.

Realize padding and rounding up values via awk out of a python script with subprocess

I use this command string to get the percentage value of the CPU utilization.
top -d 0.5 -b -n2 | grep 'Cpu(s)'|tail -n 1 | awk '{result = $2 + $4} END {printf "%3.0f\n", result'}
In the shell it works, but I need to execute it out of a python script.
This does not work because of the "%3.0f\n" part.
p = subprocess.Popen("top -d 0.5 -b -n2 | grep 'Cpu(s)'|tail -n 1 | awk '{result = $2 + $4} END {printf "%3.0f\n", result'}", stdout=subprocess.PIPE, shell=True)
How can I realize the padding to 3 characters and rounding up in this scenario?
I think the masking is the problem.
But maybe there is another way, which is better suited?
Thanks for any help!
Update: The solution was using tripple quotes:
p = subprocess.Popen('''top -d 0.5 -b -n2 | grep 'Cpu(s)'|tail -n 1 | awk '{result = $2 + $4} END {printf "%3.0f", result'}''', stdout=subprocess.PIPE, shell=True)

Either escape the " quotes inside your command string, i.e., replace them with \" or quote the command string with triple quotes `'''.
Using Jotne's version
cmd = '''top -d 0.5 -b -n2 | awk '/Cpu\(s\)/ {result=$2+$4} END {printf "%3.0f\n",result'}'''

In my opinion and as you suggested you may try other solutions, there already exists some tools which will fit your needs. For instance the psutil library.
Here is a very simple example which I think does what you want:
import psutil
print psutil.cpu_percent(interval=1)
If you still want to use your solution, you should read the python subprocess documentations about how to replace shell pipelines, and/or the existing stackoverflow topics about that (for instance this one)

Escaping escape sequence in Python

I am kind of new to python. Goal is to execute a shell command using subprocess parse & retrive the printed output from shell. The execution errors out as shown in the sample output msg below. Also shown below is the sample code snippet
Code snippet:
testStr = "cat tst.txt | grep Location | sed -e '/.*Location: //g' "
print "testStr = "+testStr
testStrOut = subprocess.Popen([testStr],shell=True,stdout=subprocess.PIPE).communicate()[0]
Output:
testStr = cat tst.txt | grep Location | sed -e '/.*Location: //g'
cat: tst.txt: No such file or directory
sed: -e expression #1, char 15: unknown command: `/'
Is there a workaround or a function that could be used ?
Appreciate your help
Thanks

I suppose your main error is not python related. To be more precise, there are 3 of them:
You forgot to import subprocess.
It should be sed -e 's/.*Location: //g'. You wrote ///g instead of s///g.
tst.txt does not exist.

You should be passing testStr directly as the first argument, rather than enclosing it in a list. See subprocess.Popen, the paragraph that starts "On Unix, with shell=True: ...".