I am running a script remotely on a server via SSH and Python. This script looks through log files and returns some information based on each log entry it encounters. The problem I am running into is that although my script spits out the information as soon as it hits each log entry, my local machine needs to wait for the entire process to finish before it can read the lines from the ssh connection's stdout.
ssh = subprocess.Popen(cmd.split(' '), stdout=subprocess.PIPE)
with open(remote_ip+'.hits', 'w') as f:
for line in ssh.stdout:
print line
Essentially this code prints all of the results all at once, at the end. I was wondering if there was a way for me to print out the contents of stdout as it was being produced on the server. Sorry if this is unclear, if something is ambiguous I will do my best to clarify it. Thanks!
EDIT: I should also add that I would preferably like to do this without any external packages, just built-in modules from 2.6 if possible.
Related
I have a Python script running on a server through SSH with the following command:
nohup python3 python_script.py >> output.txt
It was running for a very long time and (probably) created useful output, so I want to force it to stop now. How do I make it write the output it has so far, into the output.txt file?
The file was automatically created when I started running the script, but the size is zero (nothing has been written in it so far).
as Robert said in his comment, check that the output you are expecting to go to the file is actually making it there and not stderr. If the process is already running and has been for a long time without any response or writes into your output file, I think there are 3 options:
It is generating outout but it's not going where you are expecting (Roberts response)
It is generating output but it's buffered and for some reason has yet to be flushed
It's hasn't generated any output
Option 3 is easy: wait longer. Options 1 & 2 are a little bit tricky. If you are expecting a significant amount of output from this process, you could check the memory consumption of the python instance running your script and see if it's growing or very large. Also you could use lsof to see if it has the file open and to get some idea what it's doing with it.
If you find that your output appears to be going somewhere like /dev/null, take a look at this answer on redirecting output for an existing process.
In the unlikely event that you have a huge buffer that hasn't been flushed, you could try using ps to get the PID and then use kill -STOP [PID] to pause the process and see where you can get using GDB.
Unless it would be extremely painful, I would probably just restart the whole thing, but add periodic flushing to the script, and maybe some extra status reporting so you can tell what is going on. It might help too if you could post your code, since there may be other options available in your situation depending on how the program is written.
I want to write a python script, that opens an *.exe-file (it is a CMD-console application)
communicates with it by sending input and reading output (for example via stdin, stdout) many times.
I tried it with communicate(), but it closes the pipe after I send the first input (communicate(input='\n')),
so it does work for me only once.
Then I tried it again via p.stdin.readline(), but I only can read line by line. When I read an newline, the process
terminates (that is not what I need).
I just want to start a program, read the output and send an input to it, then wait until the next output and send
a new input to it, and so on....
Is there a good way to do it? Does anybody have an example or a similar problem that is solved?
I need the same code as you, actually i am trying using:
https://pexpect.readthedocs.io/en/stable/index.html
After no sucess with subprocess..
I know there are similar questions posted already, but non of the methods I have seen seems to work. I want to launch the application xfoil, on mac, with python subprocess, and send xfoil a bunch of commands with a script (xfoil is an application that runs in a terminal window and you interact with it through text commands). I am able to launch xfoil with the script, but I can't seem to find out how to send commands to it. This is the code I am currently trying:
import subprocess as sp
xfoil = sp.Popen(['open', '-a', '/Applications/Xfoil.app/Contents/MacOS/Xfoil'], stdin=sp.PIPE, stdout=sp.PIPE)
stdout_data = xfoil.communicate(input='NACA 0012')
I have also tried
xfoil.stdin.write('NACA 0012\n')
in order to send commands to xfoil.
As the man page says,
The open command opens a file (or a directory or URL), just as if you had double-clicked the file's icon.
Ultimately, the application gets started by LaunchServices, but that's not important—what's important is that it's not a child of your shell, or Python script.
Also, the whole point of open is to open the app itself, so you don't have to dig into it and find the Unix executable file. If you already have that, and want to run it as a Unix executable… just run it:
xfoil = sp.Popen(['/Applications/Xfoil.app/Contents/MacOS/Xfoil'], stdin=sp.PIPE, stdout=sp.PIPE)
As it turns out, in this case, MacOS/Xfoil isn't even the right program; it's apparently some kind of wrapper around Resources/xfoil, which is the actual equivalent to what you get as /usr/local/bin/xfoil on linux. So you want to do this:
xfoil = sp.Popen(['/Applications/Xfoil.app/Contents/Resouces/xfoil'], stdin=sp.PIPE, stdout=sp.PIPE)
(Also, technically, your command line shouldn't even work at all; the -a specifies an application, not a Unix executable, and you're supposed to pass at least one file to open. But because LaunchServices can launch Unix executables as if they were applications, and open doesn't check that the arguments are valid, open -a /Applications/Xfoil.app/Contents/MacOS/Xfoil ends up doing effectively the same thing as open /Applications/Xfoil.app/Contents/MacOS/Xfoil.)
For the benefit of future readers, I'll include this information from the comments:
If you just write a line to stdin and then return from the function/fall off the end of the main script/etc., the Popen object will get garbage collected, closing both of its pipes. If xfoil hasn't finished running yet, it will get an error the next time it tries to write any output, and apparently it handles this by printing Fortran runtime error: end of file (to stderr?) and bailing. You need to call xfoil.wait() (or something else that implicitly waits) to prevent this from happening.
I am using a cluster of computers to do some parallel computation. My home directory is shared across the cluster. In one machine, I have a ruby code that creates bash script containing computation command and write the script to, say, ~/q/ directory. The scripts are named *.worker1.sh, *.worker2.sh, etc.
On other 20 machines, I have 20 python code running ( one at each machine ) that (constantly) check the ~/q/ directory and look for jobs that belong to that machine, using a python code like this:
jobs = glob.glob('q/*.worker1.sh')
[os.system('sh ' + job + ' &') for job in jobs]
For some additional control, the ruby code will create a empty file like workeri.start (i = 1..20) at q directory after it write the bash script to q directory, the python code will check for that 'start' file before it runs the above code. And in the bash script, if the command finishes successfully, the bash script will create an empty file like 'workeri.sccuess', the python code checks this file after it runs the above code to make sure the computation finishs successfully. If python finds out that the computation finishs successfully, it will remove the 'start' file in q directory, so the ruby code knows that job finishs successfully. After the 20 bash script all finished, the ruby code will create new bash script and python read and executes new scripts and so on.
I know this is not a elegant way to coordinate the computation, but I haven't figured out a better to communicate between different machines.
Now the question is: I expect that the 20 jobs will run somewhat in parallel. The total time to finish the 20 jobs will not be much longer than the time to finish one job. However, it seems that these jobs runs sequentially and time is much longer than I expected.
I suspect that part of the reason is that multiple codes are reading and writing the same directory at once but the linux system or python locks the directory and only allow one process to oprate the directory. This makes the code execute one at a time.
I am not sure if this is the case. If I split the bash scripts to different directories, and let the python code on different machines read and write different directories, will that solve the problem? Or is there any other reasons that cause the problem?
Thanks a lot for any suggestions! Let me know if I didn't explain anything clearly.
Some additional info:
my home directory is at /home/my_group/my_home, here is the mount info for it
:/vol/my_group on /home/my_group type nfs (rw,nosuid,nodev,noatime,tcp,timeo=600,retrans=2,rsize=65536,wsize=65536,addr=...)
I say constantly check the q directory, meaning a python loop like this:
While True:
if 'start' file exists:
find the scripts and execute them as I mentioned above
I know this is not a elegant way to coordinate the computation, but I
haven't figured out a better to communicate between different
machines.
While this isn't directly what you asked, you should really, really consider fixing your problem at this level, using some sort of shared message queue is likely to be a lot simpler to manage and debug than relying on the locking semantics of a particular networked filesystem.
The simplest solution to set up and run in my experience is redis on the machine currently running the Ruby script that creates the jobs. It should literally be as simple as downloading the source, compiling it and starting it up. Once the redis server is up and running, you change your code to append your the computation commands to one or more Redis lists. In ruby you would use the redis-rb library like this:
require "redis"
redis = Redis.new
# Your other code to build up command lists...
redis.lpush 'commands', command1, command2...
If the computations need to be handled by certain machines, use a list per-machine like this:
redis.lpush 'jobs:machine1', command1
# etc.
Then in your Python code, you can use redis-py to connect to the Redis server and pull jobs off the list like so:
from redis import Redis
r = Redis(host="hostname-of-machine-running-redis")
while r.llen('jobs:machine1'):
job = r.lpop('commands:machine1')
os.system('sh ' + job + ' &')
Of course, you could just as easily pull jobs off the queue and execute them in Ruby:
require 'redis'
redis = Redis.new(:host => 'hostname-of-machine-running-redis')
while redis.llen('jobs:machine1')
job = redis.lpop('commands:machine1')
`sh #{job} &`
end
With some more details about the needs of the computation and the environment it's running in, it would be possible to recommend even simpler approaches to managing it.
Try a while loop? If that doesn't work, on the python side try using a TRY statement like so:
Try:
with open("myfile.whatever", "r") as f:
f.read()
except:
(do something if it doesnt work, perhaps a PASS? (must be in a loop to constantly check this)
else:
execute your code if successful
I am writing a Python program which runs a virtual terminal. Currently I am launching it like so:
import pexpect, thread
def create_input(child, scrollers, textlength=80, height=12):
while 1:
newtext = child.readline()
print newtext
child = pexpect.spawn("bash", timeout=30000)
thread.start_new_thread(create_input,(child))
This works, and I can send commands to it via child.send(command). However, I only get entire lines as output. This means that if I launch something like Nano or Links, I don't receive any output until the process has completed. I also can't see what I'm typing until I press enter. Is there any way to read the individual characters as bash outputs them?
You would need to change the output of whatever program bash is running to be unbuffered instead of line buffering. A good amount of programs have a command line option for unbuffered output.
The expect project has a tool called unbuffer that looks like it can give you all bash output unbuffered. I have personally never used it, but there are other answers here on SO that also recommend it: bash: force exec'd process to have unbuffered stdout
The problem is lies in something else. If you open an interactive shell normally a terminal window is opened that runs bash, sh, csh or whatever. See the word terminal!
In the old days, we connected a terminal to a serial port (telnet does the same but over ip), again the word terminal.
Even a dumb terminal respond to ESC codes, to report its type and to set the cursor position, colors, clear screen etc.
So you are starting a subprocess with interactive output, but there is no way in telling that shell and subprocesses are to a terminal in this setup other than with bash startup parameters if there are any.
I suggest you enable telnetd but only on localhost (127.0.0.1)
Within your program, make a socket and connect to localhost:telnet and look up how to emulate a proper terminal. If a program is in line mode you are fine but if you go to full screen editing, somewhere you will need an array of 80x24 or 132x24 or whatever size you want to store its characters, color. You also need to be able to shift lines up in that array.
I have not looked but I cannot imagine there is no telnet client example in python, and a terminal emu must be there too!
Another great thing is that telnet sessions clean up if the the ip connection is lost, eliminating ghost processes.
Martijn