Currently I am redirecting a script to a log file with the following command:
python /usr/home/scripts/myscript.py 2>&1 | tee /usr/home/logs/mylogfile.log
This seems to work but it does not write to the file as soon as there is a print command. Rather it waits until there is a group of lines that it can print. I want the console and the log file to be written to simultaneously. How can this be done with output redirection. Note that running the script on the console prints everything when it should. Though doing a tail -f on the logfile is not smooth since it writes about 50 lines at a time. Any suggestions?
It sounds like the shell is actually what's doing the buffering, since you say it outputs as expected to the console when not tee'd.
You could look at this post for potential solutions to undo that shell buffering: https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe
But I would recommend doing it entirely within Python, so you have more direct control, and instead of printing to stdout, use the logging module.
This would allow additional flexibility in terms of multiple logging levels, the ability to add multiple sources to the logging object centrally (i.e. stdout and a file -- and one which rotates with size if you'd like with logging.handlers.RotatingFileHandler) and you wouldn't be subject to the external buffering of the shell.
More info: https://docs.python.org/2/howto/logging.html
Related
I'm a programming newbie and I'm trying to understand how stdin, stdout, and stderr work. As I understand it, stdout and stderr are two different places where we can direct output from programs. I guess I don't understand what's the point of having a second "stream" of output just for errors with stderr? Why not have errors on the regular stdout? What does having errors on stderr allow me to do (basically why is stderr useful)?
There are two "points" to supporting distinct stout and stderr streams:
When you are writing applications that can be chained together (e.g. using pipelines) you don't want the "normal" output to get mixed up with errors, warnings, debug info and other "chit chat". Mixing them in the same stream would make life difficult for the next program in the chain / pipeline.
Example:
$ cat some-file | grep not
$ echo $?
If the cat command did not write its error messages to stderr, then the grep command would see a "file not found" message if "some-file" did not exist. It would then (incorrectly) match on the "not", and set the return code for the pipeline incorrectly. Constructing pipelines that coped with this sort of thing would be hellishly difficult.
Separate stdout and stderr streams have been support in (at least) UNIX and UNIX-like system since ... umm ... the 1970's. And they are part of the POSIX standard. If a new programming language's runtime libraries did not support this, then it would be considered to be crippled; i.e. unsuitable for writing production quality applications.
(In the history of programming languages, Python is still relatively new.)
However, nobody is forcing to write your applications to use stderr for its intended purpose. (Well ... maybe your future co-workers will :-) )
In UNIX (and Linux, and other Posix-compatible systems) programs are often combined with pipes, so that one program takes the output of another one as input. If you would mix normal output and error information, every program would need to know how to treat diagnostic info from its pipe data producer differently from normal data. In practice, that is impossible due to the large number of program combinations.
By writing error information to stderr, each program makes it possible for the user to get this info without needing to filter it out of the data stream intended to be read by the next program in the pipe.
I have a python application which i want to purpose as a multi as a multi terminal handler, i want each object to have it's own terminal separated from the rest each running it's own instance, exactly like when i run two or more separate terminals in Linux (/bin/sh or /bin/bash)
sample: (just logic not code)
first_terminal = terminalInstance()
second_terminal = terminalInstance()
first_result = first_terminal.doSomething("command")
second_result = second_terminal.doSomething("command")
i actually need to have each terminal to grab a stdin & stdout in a virtual environment and control them, this is why they must be seperate, is this possible in python range? i've seen alot of codes handling a single terminal but how do you do it with multiple terminals.
PS i don't want to include while loops (if possible) since i want to add scalability from dealing with 2 or more terminals to as much as my system can handle? is it possible to control them by reference giving each terminal a reference and then calling on that object and issuing a command?
The pexpect module (https://pypi.python.org/pypi/pexpect/), among others, allows you to launch programs via a pseudo-tty, which "allows your script to spawn a child application and control it as if a human were typing commands."
You can easily spawn multiple commands, each running in a separate pseudo-tty and represented by a separate object, and you can interact with each object separately. There is a lot of flexibility as to when/how you interact. You can send input to them, and read their output, either blocking or non-blocking, and incorporating timeouts and alternative outputs.
Here's a trivial session example (run bash, have it execute an "ls" command, gather the first line of output).
import pexpect
x = pexpect.spawn("/bin/bash")
x.sendline("ls")
x.expect("\n") # End of echoed command
x.expect("\n") # End of first line of output
print x.before # Print first line of output
Note that you'll receive all the output from the terminal, typically including an echoed copy of every character you send to it. If running something like a shell, you might also need to set the shell prompt (or determine the shell prompt in use) and use that in parsing the output (i.e. in finding the end of each command's output).
I am trying to dump the output of curses window into a file without displaying it on stdout. Currently I am using addstr() function to print on stdout and at the end I am calling instr() function to dump the entire screen into a file. In certain cases, the ncurses does not work properly on xterm and hence I need to redirect the output to a file without actually printing it on stdout. I thought of using logger module but I lose color coding which addstr() provides. What is the best method to achieve this?
For example:
If I run the following command
$ python get_stats.py
it should display on stdout and when I run the command
$ python get_stats.py --dump-to-file
it should dump to a file without displaying on stdout.
Does addstr() takes additional parameters to determine whether the output should go to a file or stdout?
No, addstr does not take additional parameters. By default (using initscr, that is), curses writes to the standard output. You could simply do
python get_stats.py >myfile
But for making a program option, you would have to tell Python (or curses) to write to the given file. Python's binding for ncurses does not include newterm, which is the usual way of controlling the output stream. But in Python, you can redirect the standard output as discussed in these questions:
Temporarily Redirect stdout/stderr
Redirect stdout to a file in Python?
how to redirect stdout to file and console with scripting
Because curses saves a copy of the output stream locally, you must do the redirection before calling initscr. Also, keep in mind that curses may use the file descriptor rather than the buffered output stream.
I write a python script in which there are several print statement. The printed information can help me to monitor the progress of the script. But when I qsub the bash script, which contains python my_script &> output, onto computing nodes, the output file contains nothing even when the script is running and printing something. The output file will contains the output when the script is done. So how can I get the output in real time through the output file when the script is running.
Actually write to the file rather than piping and flush after each write or after each write call sys.stdout.flush() but you are better off using a logger function and replacing the prints with logs.
From Comments:
A logger function is one that you call instead of print that will output to somewhere the text, possibly timestamped and with other information, they usually let you output various amounts of information to various destinations including stdout and files. See python 2 or 3 documents for information on pythons built in logging function.
I like to write data to sys.stderr sometimes for this sort of thing. It obviates the need to flush so much. But if you're generating output for piping sometimes, you remain better off with sys.stdout.
I have a simple command-line utility which produces output both on the console and the filesystem. While I know very well how to capture the console output, I am not aware how can I also intercept the file - for which I know the filename in advance.
I would like to keep the execution "in memory" without touching the filesystem as I immediately parse and delete the file created and this creates an unnecessary bottleneck (especially when I need to run the little tool millions of times).
So, to sum up, I am trying to achieve following:
Run a binary using python's subprocess
Capture both the tool's output AND contents of a file it creates (in current working directory with in-advance known name)
Ideally, run it all without touching the filesystem.
Since you only need to support Linux, one possibility is to use named pipes. The idea is to pre-create the output file as a named pipe, and have your process read the tool's output from the pipe.
See, for example, Introduction to Named Pipes.
The Python API is os.mkfifo().