Develop Reference

executing a command from a string and timing it

I need to write a script that will receive several parameters, from which the most important one
Is a string that contains a command (in linux).
I need to be able to run it, keep the output in STDOUT (the usual), but also time it, and later output some .csv file.
Say it looks something like this:
timing_script.py param "echo hello world; cat /tmp/foo_bar"
The command will output stuff to STDOUT every couple of milliseconds, which I need it to stay there. I'm saying this because my previous attempt at this script was in bash and I had to cut from the time command to actually time that, which also meant having to disregard the output of the command.
I'll also have to append something like param,0.345 to a csv file.
How do I execute a command from a string and also time it?

You can use subprocess to run linux command from string and time to calculate execution time:
import time
from subprocess import Popen, PIPE
start = time.time()
p1 = Popen(["my_linux_cmd"], stdout=PIPE)
print(p1.communicate()) # sdout
end = time.time()
exec_time = end - start
print(exec_time) # exeution time
Check subprocess.Popen fro more details about the available options
Warning: to print the stdout you can also use Popen.stdout.read but use communicate() rather to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

A simpler way which stays in the shell uses the formatting option -f of the time command. You can use it like that :
$ param="foo"
$ command="echo bar ; cat /tmp/foobar"
$ /usr/bin/time -f "$param,%e" bash -c "$command"
bar
#Beginning of foobar file
#End of foobar file
foo,0.00
Please have a look at man time for further examples about formatting the output of time
Of course, you can also directly run the following command (i.e. without using variables) :
/usr/bin/time -f "myparam,%e" bash -c "echo bar ; cat /tmp/foobar"
Have fun

getting python script to print to terminal without returning as part of stdout

I'm trying to write a python script that returns a value which I can then pass in to a bash script. Thing is that I want a singe value returned in bash, but I want a few things printed to the terminal along the way.
Here is an example script. Let's call it return5.py:
#! /usr/bin/env python
print "hi"
sys.stdout.write(str(5))
what I want is to have this perform this way when I run it from the command line:
~:five=`./return5.py`
hi
~:echo $five
5
but what I get is:
~:five=`./return5.py`
~:echo $five
hi 5
In other words I don't know how to have a python script print and clear the stdout, then assign it to the specific value I want.

Not sure why #yorodm suggests not to use stderr. That's the best option I can think of in this case.
Notice that print will add a newline automatically, but when you use sys.stderr.write, you need to include one yourself with a "\n".
#! /usr/bin/env python
import sys
sys.stderr.write("This is an important message,")
sys.stderr.write(" but I dont want it to be considered")
sys.stderr.write(" part of the output. \n")
sys.stderr.write("It will be printed to the screen.\n")
# The following will be output.
print 5
Using this script looks like this:
bash$ five=`./return5.py`
This is an important message, but I dont want it to be considered part of the output.
It will be printed to the screen.
bash$ echo $five
5
This works because the terminal is really showing you three streams of information : stdout, stdin and stderr. The `cmd` syntax says "capture the stdout from this process", but it doesn't affect what happens to stderr. This was designed exactly for the purpose you're using it for -- communicating information about errors, warnings or what's going on inside the process.
You may not have realized that stdin is also displayed in the terminal, because it's just what shows up when you type. But it wouldn't have to be that way. You could imagine typing into the terminal and having nothing show up. In fact, this is exactly what happens when you type in a password. You're still sending data to stdin, but the terminal is not displaying it.

from my comment..
#!/usr/bin/env python
#foo.py
import sys
print "hi"
sys.exit(5)
then the output
[~] ./foo.py
hi
[~] FIVE=$?
[~] echo $FIVE
5

You can use stdout to output your messages and stderr to capture the values in bash. Unfortunately this is some weird behaviour as stderr is intended for programs to communicate error messages so I strongly advice you against it.
OTOH you can always process your script output in bash

why is call to ssh with Python subprocess breaking outer bash loop

I have a bash script myscript.sh:
#!/bin/bash
while read line; do
myprog.py
done
calling a python program myprog.py
#!/usr/bin/env python
import subprocess
output = subprocess.check_output(['ssh', 'user#host', 'cmd'])
The ssh command that is called by subprocess executes without error, the output is correct. But when called like this the loop in myscript.sh only runs through the first line of input and then exits with status 0. If I replace the subprocess.check_output(...) call with a subprocess.Popen(...) and don't subsequently call Popen.wait() then the outer loop works as expected and the output from the ssh command is dumped to standard out some time after any output from the bash script. With the Popen.wait() behavior is the same as with check_output: bash loop only goes through one iteration before exiting without error.
If instead of ssh another command, e.g. ls, is called with check_output then the bash loop works as expected.
Can anyone help me understand why the code as shown isn't working as expected?
Note: this is a simplified version of what I am trying to do, though I do experience the same behavior with this code. In reality I am doing something with "$line" in the bash script and the subprocess call is wrapped in a try/except block.

As #larsmans guessed the ssh call was consuming stdin, breaking the outer bash loop. Adding the -n option to the ssh command resolved the issue:
output = subprocess.check_output(['ssh', '-n', 'user#host', 'cmd'])

The problem is that ssh reads from standard input, therefore it "eats" all the remaining lines in the loop. You can just connect its standard input to nowhere using the -n flag:
output = subprocess.check_output(['ssh', '-n', 'user#host', 'cmd'])
Look for the details on the man pages of ssh here https://linux.die.net/man/1/ssh and https://man.openbsd.org/ssh

how can I silence all of the output from a particular python command?

Autodesk Maya 2012 provides "mayapy" - a modded build of python filled with the necessary packages to load Maya files and act as a headless 3D editor for batch work. I'm calling it from a bash script. If that script opens a scene file in it with cmds.file(filepath, open=True), it spews pages of warnings, errors, and other info I don't want. I want to turn all of that off only while the cmds.file command is running.
I've tried redirecting from inside of the Python commands I'm sending into mayapy inside the shell script, but that doesn't work. I can silence everything by redirecting stdout/err to /dev/null in the call to the bash script. Is there any way to silence it in the call to the shell, but still allow my passed-in command inside the script to print out information?
test.sh:
#!/bin/bash
/usr/autodesk/maya/bin/mayapy -c "
cmds.file('filepath', open=True);
print 'hello'
"
calling it:
$ ./test.sh # spews info, then prints 'hello'
$ ./test.sh > /dev/null 2>&1 # completely silent

Basically, I think the best way to solve this is to implement a wrapper that will execute test.sh and sanitize the output to the shell. To sanitize the output, I would simply prepend some string to notify your wrapper that this text is good for output. My inspiration for the wrapper file came from this: https://stackoverflow.com/a/4760274/2030274
The contents are as follows:
import subprocess
def runProcess(exe):
p = subprocess.Popen(exe, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
retcode = p.poll() #returns None while subprocess is running
line = p.stdout.readline()
yield line
if(retcode is not None):
break
for line in runProcess(['./test.sh']):
if line.startswith('GARYFIXLER:'):
print line,
Now you could imagine test.sh being something along the lines of
#!/bin/bash
/usr/autodesk/maya/bin/mayapy -c "
cmds.file('filepath', open=True);
print 'GARYFIXLER:hello'
"
and this will only print the hello line. Since we are wrapping the python call in a subprocess, all output typically displayed to the shell should get captured and you should intercept the lines that you don't want.
Of course, to call test.sh from a python script, you need to make sure you have the correct permissions.

I knew I was just getting twisted around with pipes. Maya is indeed sending all batch output to stderror. This frees stdout entirely once you properly pipe stderr away. Here's an all-bash one-liner that works.
# load file in batch; divert Maya's output to /dev/null
# then print listing of things in file with cmds.ls()
/usr/autodesk/maya/bin/mayapy -c "import maya.standalone;maya.standalone.initialize(name='python');cmds.file('mayafile.ma', open=True);print cmds.ls()" 2>/dev/null

How to pass a string as stdin to python script from command line interface

Currently, I am using the following command to do this
$ python scriptName.py <filePath
This command uses "<" to stdin the file to script.
and it works fine, I can use sys.stdin.read to get the file data.
But, what if I want to pass file data as a string,
I don't want to pass file path in operator "<".
Is there is anyway, where I can pass String as stdin to a python script?
Thanks,
Kamal

The way I read your question, you currently have some file abc.txt with content
Input to my program
And you execute it this way:
python scriptName.py <abc.txt
Now you no longer want to go by way of this file, and instead type the input as part of the command, while still reading from stdin. Working on the windows command line you may do it like this:
echo Input to my program | python scriptName.py
while on Linux/Mac you'd better quote it to avoid shell expansion:
echo "Input to my program" | python scriptName.py
This only works for single-line input on windows (AFAIK), while on linux (and probably Mac) you can use the -e switch to insert newlines:
echo -e "first line\nsecond line" | python scriptName.py

There is raw_input which you can use make the program prompt for input and you can send in a string. And yes, it is mentioned in the first few pages of the tutorial at http://www.python.org.
>>> x = raw_input()
Something # you type
>>> x
'Something'
And sending the input via < the shell redirection operation is the property of shell and not python.

I could be wrong, but the way that I read the OP's question, I think he may currently be calling an os command to run a shell script inside of his python script, and then using a < operator to pass a file's contents into this shell script, and he is just hard coding the < and filename.
What he really desires to do is a more dynamic approach where he can pass a string defined in Python to this shell script.
If this is the case, the method I would suggest is this:
import subprocess;
script_child = subprocess.Popen(['/path/to/script/myScript.sh'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
stdout, stderr = clone_child.communicate("String to pass to the script.")
print "Stdout: ", stdout
print "Stderr: ", stderr
Alternatively, you can pass arguments to the script in the initial Popen like so:
script_child = subprocess.Popen(['/path/to/script/myScript.sh', '-v', 'value', '-fs'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)