Okay, so, I just have a quick question regarding python and linux.
I have a program that collects and outputs data to stdout indefinitely. I need to parse this data, and I have a python program I wrote that will do just that. However, I cannot save this data to a file first, as it produces far too much output to save to disk. Is there any way to use redirects to somehow pipe this output into the program?
Example:
python parser.py < ./dataCollector.sh
Close, but you want an actual pipe not a shell redirect:
./dataCollector.sh | python parser.py
Related
When I execute a python program, the results starts to appear quickly and I can't read it all. It just flushes over my screen.
When the execution ends, I can no longer see the first displays, because the terminal display space is limited.
How save the output, so I can read all of it?
You have a few options here.
Add a breakpoint and learn how to use the debugger. Once you add this command (import pdb;pdb.set_trace() # this will take some learning so look up what pdb is online. actually, i prefer 'ipdb' instead.), the code will stop at that specific point when you execute it.
Save it to a file (python file.py > filename.txt) and then read it afterwards. Bonus: Before you ask yourself, where are my outputs? https://askubuntu.com/questions/625224/how-to-redirect-stderr-to-a-file
(More advanced) Your code is spitting out too much garbage output. You can remove some of the code or use python logging filters.
May be platform dependant.
On Linux you can also pipe your program output into your favorite pager (less for example) if you don't want to write it to a file.
python file.py | less
I write a python script in which there are several print statement. The printed information can help me to monitor the progress of the script. But when I qsub the bash script, which contains python my_script &> output, onto computing nodes, the output file contains nothing even when the script is running and printing something. The output file will contains the output when the script is done. So how can I get the output in real time through the output file when the script is running.
Actually write to the file rather than piping and flush after each write or after each write call sys.stdout.flush() but you are better off using a logger function and replacing the prints with logs.
From Comments:
A logger function is one that you call instead of print that will output to somewhere the text, possibly timestamped and with other information, they usually let you output various amounts of information to various destinations including stdout and files. See python 2 or 3 documents for information on pythons built in logging function.
I like to write data to sys.stderr sometimes for this sort of thing. It obviates the need to flush so much. But if you're generating output for piping sometimes, you remain better off with sys.stdout.
I am writing a check.py file which reads a file that maps a python file to an output. That file looks something like this
001.py 233168
002.py 4613732
This means 001.py when ran should print out 233168. What is the best way to capture stdout from 00*.py? Overriding stdout and using execfile? or using a subprocess?
I have never done anything like this before, but it seems like
subprocess.check_output
does exactly what I want, is there a more appropriate way of doing this?
A subprocess. That way the script being executed cannot disrupt the calling script too badly.
i have tests that i ran which can take up to 15m at a time. during these 15m, a log file is periodically written to. however, most of the content is useless.
in response to this i have a python script that parses out the useless text and displays the relevant data.
what i'm trying to achieve is similar to what tail -f log_file, constantly updating the terminal with the newest additions to a file. i was thinking that if a python script ran as a process, it could parse the log file whenever the tests write to it, then the python script can go to sleep until interrupted again once the log file is written to.
any ideas how one can achieve this?
i already have a script that does the parsing, i just don't know how to make it do it continually and efficiently.
You could just have the script filter standard input, and pipe tail -f through it. When you're waiting on stdin, your script will sleep, so it's plenty efficient.
Eg.
python long_running_script.py && tail -f log_file | python filter_logs.py
Your script can be something like
while true:
line = sys.stdin.readline()
if filter_line(line): print line
looks like you need something like "pytailer":
http://code.google.com/p/pytailer/
While I never used it myself, last example looks like what you want.
any ideas how one can achieve this?
This should be pretty easy to do. Most of what you want is already part of your OS.
python test.py | python log_parser.py
Be sure your tests write their log to stdout instead of some other file. This is often easy to do with small changes to the logging configuration.
Having implemented almost this exact tool, I had great success using the inotify capability in twisted
I have one script in Perl and the other in Python. I need to get the results of Perl in Python and then give the final report. The results from Perl can be scalar variable, hash variable, or an array.
Please let me know as soon as possible regarding this.
Use the subprocess module to run your Perl script to capture its output:
You can format the output however you choose in either script, and use Python to print the final report. For example: your Perl script can output XML which can be parsed by the Python script and then printed using a different format.
Take a look at PyYAML in Python and YAML in Perl.
You could serialize the results to some sort of a string format, print this to standard output in the Perl script. Then, from python call the perl script and redirect the results of stdout to a variable in python.