Develop Reference

Problems with nohup of a python code which calls another nohup?

The problem is like this.
I have a python code "test.py", which calls a command line:
os.system('nohup some_command > b.txt &')
print(some_results)
and it works well, which redirects the output info of "some_command" to "b.txt", and only printing the "some_results" to the terminal.
I want to run the code in background (let it keep running when I exit the terminal), so I use this command:
nohup python test.py > a.txt &
Now everything, including the "some_results" and the output of "some_command", is redirected into "a.txt", which causes the program to run not properly.
Is there a way to redirect only the "some_results", instead of everything, to "a.txt"? What should I do?
PS: I don't know what keywords should I search, and by searching "nohup in nohup" I have not found relevant solution.
============================= Some unsuccessful attempts =========================
After reading a recommended question: Redirect stdout to a file in Python?
I had an idea:
import sys
sys.stdout=open('c.txt','w')
os.system('nohup some_command > b.txt &')
print(some_results)
But the problem is still not solved.
nohup python test.py > a.txt & redirects the python outputs to "c.txt" (as expected), but everything else to "a.txt" instead of "b.txt", and causes a failure.
python test.py > a.txt & works temporarily: It redirects the python outputs to "c.txt", the command outputs to "b.txt", and leaving "a.txt" blank, as expected (sort of).
However, the terminal would be poped up with "nohup: redirecting stderr to stdout" messages each time the "os.system" command is called. After restarting the terminal, the messages no longer pops and the program is still running, but the redirection becomes the same as nohup python test.py > a.txt &.
============== Some additional information =====================
The os.system(blabla) is executed multiple times.
The "some_command" is actually "pocketsphinx", which outputs a lot of logs including the time alignment of phonemes, and finally a line describing the phoneme sequence without time alignment. What I need is the "time alignment" section.
In normal conditions, the last line always follows the info section, no matter where they are printed.
In my problem, the last line is always in the "b.txt" correctly. The info (including the time alignments which I want them to be in "b.txt") are redirected to somewhere else.

In your script, just redirect both stdout and stderr to a file, no nohup, no background:
os.system('some_command > b.txt 2>&1')
print(some_results)
In the terminal:
nohup python my_script.py > a.txt &

Capturing the output of a program started via cmd script

I am trying to do a complexity analysis of the stanford parser. To do so, I am starting the program via a cmd file, so therefore if I use subprocess.check_output, my python program will give me the commandline arguments I am using. The parser prints its own runtime on the commandline, so therefore I have to actually come up with something which reads out what the program I have started printed on the commandline.
subprocess.check_output("path-to-cmd", shell=True
tldr: This gives me the cmd-files output, I want what the started program printed in the terminal.
As my question was marked as a duplicate, I want the output of a program that I have started via the cmd, if I use subproces.check_output, it will simply give me the content of my cmd, and not the output of the java program I have run. I want to capture what the java program wrote to the terminal.

import subprocess
# Create test file.
with open('test.cmd', 'w') as w:
w.write(
'#echo off\n'
'echo 1 stdout\n'
'>&2 echo 2 stderr\n'
'>&3 echo 3 program output\n')
output = subprocess.check_output(
'test.cmd 1>nul',
universal_newlines=True,
shell=True)
print('check_output:', repr(output))
The example will get the programs output from handle 3. The program
here is just an echo to mimic a program though the redirection
is the goal.
CMD supports up to 9 output handles as quoted from the SS64 site:
STDIN = 0 Keyboard input
STDOUT = 1 Text output
STDERR = 2 Error text output
UNDEFINED = 3-9
You can output programs to handle 3 in the batch file.
Then you can redirect handle 1 to nul i.e. 1>nul or
just >nul in the Python file.
Thus, check_output will only output handle 3 as the stdout.
Output:
2 stderr
check_output: '3 program output\n'
Output uses repr() to show output in 1 line for testing.
No output of the line 1 stdout as handle 1 was redirected to nul.
Stderr will still print out to console as it is not redirected.
You can choose how to handle stderr.
If the Stanford Parser outputs the data as stderr (handle 2)
instead of stdout (handle 1), then you may use 2>&3
in the batch file command to redirect to handle 3. i.e.
2>&3 java -cp stanford-parser.jar ...
I have no experience with the Stanford Parser so the command
example is a guess from online examples from stanford.edu.
If you want all of the output instead of just program output
and the program outputs to handle 2. Then use in the
check_output with 2>&1 or the recommended argument
stderr=subprocess.STDOUT and omit the 1>nul. This may
include batch file script errors which may be undesirable.
If possible, rewrite the batch file to Python as you avoid
complication and 1 script gets all the control.

Is there any way to have output piped line-by-line from a currently executing python program?

When piping printed output from a python script to a command like grep, the output from the script seems to only be piped to the follow-up command after completion of the entire script.
For example, in a script test_grep.py like the following:
#!/usr/bin/env python
from time import sleep
print "message1"
sleep(5)
print "message2"
sleep(5)
print "message3"
when called with ./test_grep.py | grep message, nothing will appear for 10 seconds, at which time all three lines will appear.
Compare this to a script test_grep.sh:
#!/usr/bin/env bash
echo "message1"
sleep 5
echo "message2"
sleep 5
echo "message3"
./test_grep.sh | grep message will immediately output message1, followed at 5 second intervals by message2 and message3.
I expect this is because only once the python interpreter finishes executing is the output available for the next command. Is there any way to alter this behavior?

You can do it:
By flushing every print in python
By setting stdout to be unbuffered
By setting stdout to be line-buffered
You can even call python -u to disable buffering.
I would go for the line-buffering option as it seems most natural.
open(file, mode='r', buffering=-1 ....)
buffering is an optional integer used to set the buffering policy.
Pass 0 to switch buffering off (only allowed in binary mode), 1 to
select line buffering (only usable in text mode), and an integer > 1
to indicate the size of a fixed-size chunk buffer.
When you don't specify buffering (the typical "open") it will use line-buffering if it detects the output is going directly do a TTY, i.e. to your screen console. If you pipe output or redirect it to a file it will switch back to a large (4K / 8K) buffer.
How do you "set stdout to be line-buffered"?
You can reopen stdout via sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1).

how can I silence all of the output from a particular python command?

Autodesk Maya 2012 provides "mayapy" - a modded build of python filled with the necessary packages to load Maya files and act as a headless 3D editor for batch work. I'm calling it from a bash script. If that script opens a scene file in it with cmds.file(filepath, open=True), it spews pages of warnings, errors, and other info I don't want. I want to turn all of that off only while the cmds.file command is running.
I've tried redirecting from inside of the Python commands I'm sending into mayapy inside the shell script, but that doesn't work. I can silence everything by redirecting stdout/err to /dev/null in the call to the bash script. Is there any way to silence it in the call to the shell, but still allow my passed-in command inside the script to print out information?
test.sh:
#!/bin/bash
/usr/autodesk/maya/bin/mayapy -c "
cmds.file('filepath', open=True);
print 'hello'
"
calling it:
$ ./test.sh # spews info, then prints 'hello'
$ ./test.sh > /dev/null 2>&1 # completely silent

Basically, I think the best way to solve this is to implement a wrapper that will execute test.sh and sanitize the output to the shell. To sanitize the output, I would simply prepend some string to notify your wrapper that this text is good for output. My inspiration for the wrapper file came from this: https://stackoverflow.com/a/4760274/2030274
The contents are as follows:
import subprocess
def runProcess(exe):
p = subprocess.Popen(exe, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
retcode = p.poll() #returns None while subprocess is running
line = p.stdout.readline()
yield line
if(retcode is not None):
break
for line in runProcess(['./test.sh']):
if line.startswith('GARYFIXLER:'):
print line,
Now you could imagine test.sh being something along the lines of
#!/bin/bash
/usr/autodesk/maya/bin/mayapy -c "
cmds.file('filepath', open=True);
print 'GARYFIXLER:hello'
"
and this will only print the hello line. Since we are wrapping the python call in a subprocess, all output typically displayed to the shell should get captured and you should intercept the lines that you don't want.
Of course, to call test.sh from a python script, you need to make sure you have the correct permissions.

I knew I was just getting twisted around with pipes. Maya is indeed sending all batch output to stderror. This frees stdout entirely once you properly pipe stderr away. Here's an all-bash one-liner that works.
# load file in batch; divert Maya's output to /dev/null
# then print listing of things in file with cmds.ls()
/usr/autodesk/maya/bin/mayapy -c "import maya.standalone;maya.standalone.initialize(name='python');cmds.file('mayafile.ma', open=True);print cmds.ls()" 2>/dev/null

How do I redirect stdin/stdout when I have a sequence of commands in Bash?

I've currently got a Bash command being executed (via Python's subprocess.Popen) which is reading from stdin, doing something and outputing to stdout. Something along the lines of:
pid = subprocess.Popen( ["-c", "cmd1 | cmd2"],
stdin = subprocess.PIPE,
stdout = subprocess.PIPE,
shell =True )
output_data = pid.communicate( "input data\n" )
Now, what I want to do is to change that to execute another command in that same subshell that will alter the state before the next commands execute, so my shell command line will now (conceptually) be:
cmd0; cmd1 | cmd2
Is there any way to have the input sent to cmd1 instead of cmd0 in this scenario? I'm assuming the output will include cmd0's output (which will be empty) followed by cmd2's output.
cmd0 shouldn't actually read anything from stdin, does that make a difference in this situation?
I know this is probably just a dumb way of doing this, I'm trying to patch in cmd0 without altering the other code too significantly. That said, I'm open to suggestions if there's a much cleaner way to approach this.

execute cmd0 and cmd1 in a subshell and redirect /dev/null as stdin for cmd0:
(cmd0 </dev/null; cmd1) | cmd2

I don't think you should have to do anything special. If cmd0 doesn't touch stdin, it'll be intact for cmd1. Try for yourself:
ls | ( echo "foo"; sed 's/^/input: /')
(Using ls as an arbitrary command to produce a few lines of input for the pipeline)
And the additional pipe to cmd2 doesn't affect the input either, of course.

Ok, I think I may be able to duplicate the stdin file descriptor to a temporary one, close it, run cmd0, then restore it before running cmd1:
exec 0>&3; exec 0<&-; cmd0 ; exec 3>&0 ; cmd1 | cmd2
Not sure if it's possible to redirect stdin in this way though, and can't test this at the moment.
http://tldp.org/LDP/abs/html/io-redirection.html
http://tldp.org/LDP/abs/html/x17601.html