Develop Reference

How to use os.system() to run `> >(tee -a log)`?

I run the following command in the terminal.
sh -c "echo out; echo err 2>&1" >>(tee -a stdout.log) 2>>(tee -a stdout.log >&2)
output:
out
err
Using os.system in Python will report an error.
import os
cmd = """
sh -c "echo out; echo err 2>&1" > >(tee -a stdout.log) 2> >(tee -a stdout.log >&2)
"""
os.system(cmd)
sh: -c: line 1: syntax error near unexpected token `>'
sh: -c: line 1: `sh -c "echo out" > >(tee -a stdout.log) 2> >(tee -a stdout.log >&2)'

>(...) is bash-specific syntax. Make that bash -c instead of sh -c.
Also you should enclose the entire command in quotes since -c expects a single argument.
cmd = """
bash -c 'echo out > >(tee -a stdout.log) 2> >(tee -a stdout.log >&2)'
"""
To test writing to both stdout and stderr like your original example, try like this with curly braces:
cmd = """
bash -c '{ echo out; echo err 2>&1; } > >(tee -a stdout.log) 2> >(tee -a stdout.log >&2)'
"""

Bash Script - Python can't find a path

Please I need help!
This is my first bash script, and it's calling a python script at some points.
But I always get this output at line 28 and 40:
./startTests.sh: line 28: 'logs/tp1/load-test1-workloada.txt': No such file or directory
The directory logs/tp1 do exist!
I think it's something with the char '>' that I have to use there, but I don't know how to fix it.
This is my script:
#! /bin/bash
echo -e "\033[01;34m------------------------------"
echo -e "| Testes YCSB 0.17 |"
echo -e "------------------------------\033[01;37m"
echo -e 'Está rodando o script da raiz do YCSB? (y/n)'
read yesno
if [ $yesno = 'y' ]; then
echo -e 'Número de throughputs:'
read numOfThroughputs
echo -e 'Número de testes:'
read numOfTests
echo -e 'Iniciando...'
for ((throughput = 1; throughput <= $numOfThroughputs; throughput++)); do
echo -e '\033[01;32m------------------------------'
echo -e "Throughput $throughput "
echo -e '------------------------------\033[01;37m'
for ((test = 1; test <= $numOfTests; test++)); do
echo -e '\033[01;32m------------------------------'
echo -e "Iciando teste $test"
echo -e '------------------------------\033[01;37m'
echo -e 'Loading Workload A'
python ./bin/ycsb load mongodb-async -P workloads/workloada -p mongodb.url=mongodb://172.18.0.3:27017/ycsb?w=1 -P config/load.dat -s > "'logs/tp${throughput}/load-test${test}-workloada.txt'"
for workload in {a,b,c,,f,d,e}; do
if [ $workload = 'e' ]; then
echo -e "\033[01;33mPreparação para workload $workload"
echo -e "Apague o banco e precione QUALQUER TECLA 2X"
read enter
echo -e "Precione QUALQUER TECLA\033[01;37m"
read enter2
python ./bin/ycsb load mongodb-async -P workloads/workloade -p mongodb.url=mongodb://172.18.0.3:27017/ycsb?w=1 -P config/load.dat -s > "'logs/tp${throughput}/load-test${test}-workloade.txt'"
fi
echo -e "Workload $workload"
python ./bin/ycsb run mongodb-async -P workloads/workload"${workload}" -p mongodb.url=mongodb://172.18.0.3:27017/ycsb?w=1 -P config/runtp"${throughput}".dat -s > ""'logs/tp${throughput}/run-test${test}-workload${workload}.txt'"
done
echo -e "\033[01;32mTeste $test finalizado!"
echo -e "\033[01;33mApague o banco e precione QUALQUER TECLA 2X"
read enter
echo -e "Precione QUALQUER TECLA\033[01;37m"
read enter2
done
done
fi

Get rid of the single quotes inside double quotes.
> "'logs/tp${throughput}/load-test${test}-workloada.txt'"
is putting literal single quotes in the pathname, since single quotes have no special meaning inside double quotes. That should be
> "logs/tp${throughput}/load-test${test}-workloada.txt"
You also have this on a later line:
> ""'logs/tp${throughput}/run-test${test}-workload${workload}.txt'"
This has an extra " at the beginning, which makes the quotes unbalanced. It should be:
> "logs/tp${throughput}/run-test${test}-workload${workload}.txt"

ogs/tp1 may be there but this is not same as 'logs/tp1/'.
You should remove the single quotes.

running a os.system / subprocess.call returns Syntax error [duplicate]

This question already has answers here:
Process substitution not allowed by Python's subprocess with shell=True?
(3 answers)
Closed 2 years ago.
I have this bash command I want to run from Python2.7:
time ( s=172.20.16 ; for i in $(seq 1 254) ; do ( ping -n -c 1 -w 1 $s.$i 1>/dev/null 2>&1 && printf "%-16s %s\n" $s.$i responded ) & done ; wait ; echo )
I tried running it like this:
cmd = 'time ( s=172.20.16 ; for i in $(seq 1 254) ; do ( ping -n -c 1 -w 1 $s.$i 1>/dev/null 2>&1 && printf "%-16s %s\n" $s.$i responded ) & done ; wait ; echo )'
#1. subprocess.call(cmd.split())
#2. subprocess.call(cmd, shell=True)
#3. os.system(cmd)
But all returned /bin/sh: 1: Syntax error: word unexpected (expecting ")"), while running it from bash worked prefectly. I also tried adding a /bin/bash to the head of the command, but that didn't work.
When using os.system('bash "{}"'.format(cmd)) it didn't crash with the previous error, but the loop unfolded incorecctly (it printed 1..254 instead of using them as the IP suffix)
I managed to make it work by saving the command in a bash script and then calling the script from python, but I would rather do that directly. What is the problem here?

shell=True uses /bin/sh. /bin/sh is not bash.
Leaving all the problems with the shell script in place, but invoking it with bash, would look like the following:
cmd = 'time ( s=172.20.16 ; for i in $(seq 1 254) ; do ( ping -n -c 1 -w 1 $s.$i 1>/dev/null 2>&1 && printf "%-16s %s\n" $s.$i responded ) & done ; wait ; echo )'
subprocess.call(['bash', '-c', cmd])
Rewriting it to actually be a better shell script might instead look like:
cmd = r'''
time {
s=172.20.16
for ((i=1; i<=254; i++)); do
{ ping -n -c 1 -w 1 "$s.$i" >/dev/null 2>&1 && \
printf "%-16s %s\n" "$s.$i" "responded"
} &
done
wait
echo
}
'''
subprocess.call(['bash', '-c', cmd])
Note that we're using { ...; }, not ( ... ), for grouping (thus avoiding more subshell creations than necessary); and that we're always quoting substitutions.

you are splitting by space to construct the array of commands/parameters for the suprocess call method; but notice that there are parameters that include spaces, so it should count as a single parameter, not two (ie this one: "%-16s %s\n")

Try using subprocess as per this link Running Bash commands in Python
import subprocess
subprocess.call("{}".format(cmd).split())

Why doesn't Python run my bash code?

I'm trying to debug some unit tests that have been provided for testing an integration.
I'm sure this worked last time I tested it on my local machine, but that seems to have changed - the file hasn't been altered, so I don't know what's changed since then.
I have stripped out the identifying comments and changed some names from the original unit tests because it's proprietary software.
The syntax error is:
File "unitTests.sh", line 39
gLastFullPath=`python -c "import os; print os.path.realpath('${1}')"`
^
SyntaxError: invalid syntax
The full script is here:
#!/bin/bash
# If non-zero, then run in debug mode, outputting debug information
debug=0
# Set the following to 1 to force an error for testing purposes
forceError=0
separator="===================================================================================================="
#-------------------------------------------------------------------------------
# Convert the specified path to a full path and return it in the gLastFullPath
# global variable.
#
# Input params:
# $1 - Path to convert to full
#
# Output params:
# $gLastFullPath - Set to the converted full path
gLastFullPath=""
getFullPath()
{
# Use Python (because it's easier than Bash) to convert the passed path to
# a full path.
gLastFullPath=`python -c "import os; print os.path.realpath('${1}')"`
}
#-------------------------------------------------------------------------------
fatalError()
{
echo "${separator}"
echo "Fatal Error: $1"
echo "${separator}"
exit 1
}
#-------------------------------------------------------------------------------
# If a file or folder exists at the specified path, then delete it. If it's a
# directory, then its entire contents is deleted.
#-------------------------------------------------------------------------------
deleteIfExists()
{
if [[ 0 -ne $debug ]]; then
echo "deleteIfExists called..."
fi
if [[ -e "$1" ]]; then
# If it's a directory, then make sure it contains no locked files
if [[ -d "$1" ]]; then
chflags -R nouchg "$1"
fi
if [[ 0 -ne $debug ]]; then
echo " Deleting the existing file or directory:"
echo " $1"
fi
# Do the remove and check for an error.
/bin/rm -rf "$1"
if [[ $? -ne 0 ]]; then
fatalError "Unable to delete $1."
fi
fi
if [[ 0 -ne $debug ]]; then
echo
fi
}
#-------------------------------------------------------------------------------
# Script starts here
#-------------------------------------------------------------------------------
# Get the full path to this script
scriptPath=`which "$0"`
getFullPath "${scriptPath}"
scriptFullPath="${gLastFullPath}"
scriptDir=`dirname "${scriptFullPath}"`
scriptName=`basename "${scriptFullPath}"`
if [[ 0 -ne $debug ]]; then
echo "$scriptName: Debug tracing is on."
echo
fi
# Get the SDK project root path
getFullPath "${scriptDir}/.."
projRoot="${gLastFullPath}"
# Get the top of the server tree
getFullPath "${projRoot}/SUBSYS_TOP"
subsysTop="${gLastFullPath}"
libPythonBase="${projRoot}/src/lib/py/devilsoftPy"
devilsoftPython="${libPythonBase}/devilsoftpy"
if [[ 0 -ne $debug ]]; then
echo "$scriptName: Project root dir: \"${projRoot}\""
echo "$scriptName: SUBSYS_TOP: \"${subsysTop}\""
echo "$scriptName: Lib python base: \"${libPythonBase}\""
echo "$scriptName: devilsoft python: \"${devilsoftPython}\""
echo
fi
# First we have to launch the test python server. This is used by some of the other client tests to
# run against.
testServer="${devilsoftPython}/test/TestServer.py"
if [[ ! -f "${testServer}" ]]; then
fatalError "Could not find the expected test server: \"${testServer}\""
fi
# Carve out a place for our test server log file
tempFolder="/tmp/devilsoft"
mkdir -p "${tempFolder}"
testServerLogFile="${tempFolder}/TestServer.log"
echo "Starting the test server: \"${testServer}\""
echo " Logging to this file: \"${testServerLogFile}\""
export PYTHONPATH="${libPythonBase}:${PYTHONPATH}"; "${testServer}" > "${testServerLogFile}" 2>&1 &
testServerPid=$!
echo " Server started with pid ${testServerPid}..."
echo
echo " Taking a little snooze to let the test server initialize..."
sleep 2
# If we're forcing errors for testing, then kill the test server. This will cause downstream scripts
# to fail because there will be no server to talk to.
if [[ $forceError -ne 0 ]]; then
echo "Forcing downstream errors by killing the test server..."
kill ${testServerPid}
wait ${testServerPid}
testServerPid=0
echo
fi
testResultsLogFile="${tempFolder}/TestResults.log"
echo "Testing each python script in the library..."
echo " Test results will be written to this log file: \"${testResultsLogFile}\""
echo
deleteIfExists "${testResultsLogFile}"
# Save and set the field separator so that we can handle spaces in paths
SAVEIFS=$IFS
IFS=$'\n'
failedScripts=()
lastError=0
pythonSources=($(find "${devilsoftPython}" -name '*.py' ! -name '*.svn*' ! -name '__init__.py' ! -name 'TestServer.py' ! -name 'ServerClient.py'))
for pythonSourceFile in ${pythonSources[*]}; do
echo " Testing python source \"${pythonSourceFile}\""
export PYTHONPATH="${libPythonBase}:${PYTHONPATH}"; "${pythonSourceFile}" >> "${testResultsLogFile}" 2>&1
result=$?
if [[ $result -ne 0 ]]; then
pythonSourceName=`basename "${pythonSourceFile}"`
echo " Error ${result} returned from the above script ${pythonSourceName}!"
lastError=${result}
failedScripts+=("${pythonSourceFile}")
fi
done
echo
# Restore the original field separator
IFS=$SAVEIFS
if [[ ${testServerPid} -ne 0 ]]; then
echo "Telling the test server to quit..."
kill ${testServerPid}
wait ${testServerPid}
echo
fi
# If we got an error, tell the user
if [[ $lastError -ne 0 ]]; then
echo "IMPORTANT! The following scripts failed with errors:"
for failedScript in "${failedScripts[#]}"; do
echo " \"${failedScript}\""
done
echo
fatalError "Review the log files to figure out why the above scripts failed."
fi
echo "${separator}"
echo " Hurray! All tests passed!"
echo "${separator}"
echo
exit 0
This is all being run in Python 2.7

This is a bash script, not a Python script. Run it using ./script_name.sh or bash script_name.sh instead of python script_name.sh.

How do i run nikto.pl file from wapiti.py file?

I am working on a project which involves wapiti and nikto web tools. i have managed to produce one report for both these tool with this command
python wapiti.py www.kca.ac.ke ;perl nikto.pl -h www.kca.ac.ke -Display V -F htm -output /root/.wapiti/generated_report/index.html.
But i would like to run a command like
python wapiti.py www.kca.ac.ke
and get both the wapiti and nikto web scan report. How do i achieve this guys?

A shell script would work. Save the following as 'run_wapiti_and_nikto_scans', then run it as:
bash run_wapiti_and_nikto_scans www.my.site.com
Here is the script:
#!/bin/bash
SITE=$1
if [ -n "$SITE" ]; then # -n tests to see if the argument is non empty
echo "Looking to scan $SITE"
echo "Running 'python wapiti.py $SITE'"
python wapiti.py $SITE || echo "Failed to run wapiti!" && exit 1;
echo "Running 'perl nikto.pl -h $SITE -Display V -F htm -output /root/.wapiti/generated_report/index.html'"
perl nikto.pl -h $SITE -Display V -F htm -output /root/.wapiti/generated_report/index.html || echo "Failed to run nikto!" && exit 1;
echo "Done!"
exit 0; # Success
fi
echo "usage: run_wapiti_and_nikto_scans www.my.site.com";
exit 1; # Failure