Develop Reference

Call sub-process in Python with args and read the stdout

I am try to call one script from another script in Python. I'm doing this with subprocess OK with one call, but I don't know how make more than one call to the same sub-process.
I have a GUI, then I call my Python script called simulator.py.
simulator.py:
while 1:
try:
comando = raw_input('>> ')
cmds = comando.split(' ')
cmds = filter(None, cmds)
except EOFError:
break
if not comando: continue
elif cmds[0] == 'c':
print "works first cmd"
elif cmds[0] == 'm':
print "works second cmd"
In the script where I implement the GUI, I want to start simulator.py in a sub-process when I press the button "load simulator". With other buttons, I want to send different messages to the same sub-process previously opened.
#load simulator method
term = subp.Popen('python simulator.py', shell=True, stdout=subp.PIPE, stdin=subp.PIPE, stderr=subp.PIPE)
resp, error = term.communicate("c some_argument")
print resp
self.spT
#click other button method
term = self.spT
resp, error = term.communicate("m some_argument")
print resp
That code causes an error. Then I tried with this:
#load simulator button method
term = subp.Popen('python simulator.py', shell=True, stdout=subp.PIPE, stdin=subp.PIPE, stderr=subp.PIPE)
term.stdin.write("c argument")
term.stdin.close()
print term.stdout.read()
self.spT = term
And when I press the other button, it calls the same sub-process and sends it the other "command" and other argument:
#button X to send other arg to the same subprocess
term = self.spT
term.stdin.write("m argument")
term.stdin.close()
print term.stdout.read()
But that causes an error too. I have also tried with term.stdin.flush, wait, etc. I have tried some examples on the Internet and here as well. I can't find one that will do the thing that I want.
Do I need to use threads? How can I implement this functionality? Can the sub-process still be running without hanging my GUI? Do I have to use another module?

Is there a way to have a python program run an action when it's about to crash?

I have a python script with a loop that crashes every so often with various exceptions, and needs to be restarted. Is there a way to run an action when this happens, so that I can be notified?

You could install an exception hook, by assigning a custom function to the sys.excepthook handler. The function is called whenever there is a unhandled exception (so one that exits the interpreter).
import sys
def myexcepthook(type, value, tb):
import traceback
from email.mime.text import MIMEText
from subprocess import Popen, PIPE
tbtext = ''.join(traceback.format_exception(type, value, tb))
msg = MIMEText("There was a problem with your program:\n\n" + tbtext)
msg["From"] = "me#example.com"
msg["To"] = "you#example.com"
msg["Subject"] = "Program exited with a traceback."
p = Popen(["/usr/sbin/sendmail", "-t"], stdin=PIPE)
p.communicate(msg.as_string())
sys.excepthook = myexcepthook
This exception hook emails you the traceback whenever the program exits, provided you have a working sendmail command on your system.

You can surrond your whole program with a try/except block, which is not very beautiful I find. Another way is to run your python script in a .sh file and execute this:
#!/bin/bash
while true
do
python your_script.py
if $? != 0 then
sendmail "blabla" "see the doc" "for arguments"
fi
done
This will execute the Python script and when it stops, it'll send you a mail and restarted it (since it's an infinite loop). The mail is sent only if there's an error in the Python program and the exit code is different of 0. To be more efficient, you could get the stdout, and put it in the mail to know what fails and how resolve this.

You should try like this:
while True:
try:
x = int(raw_input("Please enter a number: "))
break
except ValueError:
print "Oops! That was no valid number. Try again..."

Python / Pexpect before output out of sync

I'm using Python/Pexpect to spawn an SSH session to multiple routers. The code will work for one router but then the output of session.before will get out of sync with some routers so that it will return the output from a previous sendline. This seems particularly the case when sending a blank line (sendline()). Anyone got any ideas? Any insight would be really appreciated.
Below is a sample of what I'm seeing:
ssh_session.sendline('sh version')
while (iresult==2):
iresult = ssh_session.expect(['>','#','--More--'],timeout=SESSION_TIMEOUT)
debug_print("execute_1 " + str(iresult))
debug_print("execute_bef " + ssh_session.before)
debug_print("execute_af " + ssh_session.after)
thisoutput = ssh_session.before
output += thisoutput
if(iresult==2):
debug_print("exec MORE")
ssh_session.send(" ")
else:
debug_print("exec: end loop")
for cmd in config_commands:
debug_print("------------------------------------------------\n")
debug_print ("running command " + cmd.strip() + "\n")
iresult=2
ssh_session.sendline(cmd.strip())
while (iresult==2):
iresult = ssh_session.expect([prompt+">",prompt+"#"," --More-- "],timeout=SESSION_TIMEOUT)
thisoutput = ssh_session.before
debug_print("execute_1 " + str(iresult))
debug_print("execute_af " + ssh_session.after)
debug_print("execute_bef " + thisoutput)
thisoutput = ssh_session.before
output += thisoutput
if(iresult==2):
debug_print("exec MORE")
ssh_session.send(" ")
else:
debug_print("exec: end loop")
I get this:
logged in
exec: sh version
execute_1 1
execute_bef
R9
execute_af #
exec: end loop
------------------------------------------------
running command config t
execute_1 1
execute_af #
execute_bef sh version
Cisco IOS Software, 1841 Software (C1841-IPBASEK9-M), Version 15.1(4)M4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport...

I've run into this before with pexpect (and I'm trying to remember how I worked around it).
You can re-synchronize with the terminal session by sending a return and then expecting for the prompt in a loop. When the expect times out then you know that you are synchronized.
The root cause is probably that you are either:
Calling send without a match expect (because you don't care about the output)
Running a command that produces output but expecting for a pattern in the middle of that output and then not to next prompt that is at end of the output. One way to deal with this is to change your expect pattern to "(.+)PROMPT" - this will expect until the next prompt and capture all the output of the command sent (which you can parse in the next step).

I faced a similar problem. I tried waiting for the command to be printed on the screen and the sending enter.
I you want to execute say command 'cmd', then you do:
session.send(cmd)
index = session.expect([cmd, pexpect.TIMEOUT], 1)
session.send('\n')
index = session.expect([whatever you expect])
Worked for me.

I'm not sure this is the root of your problem, but it may be worth a try.
Something I've run into is that when you spawn a session that starts with or lands you in a shell, you have to deal with quirks of the TERM type (vt220, color-xterm, etc.). You will see characters used to move the cursor or change colors. The problem is almost guaranteed to show up with the prompt; the string you are looking for to identify the prompt appears twice because of how color changes are handled (the prompt is sent, then codes to backspace, change the color, then the prompt is sent again... but expect sees both instances of the prompt).
Here's something that handles this, guaranteed to be ugly, hacky, not very Pythonic, and functional:
import pexpect
# wait_for_prompt: handle terminal prompt craziness
# returns either the pexpect.before contents that occurred before the
# first sighting of the prompt, or returns False if we had a timeout
#
def wait_for_prompt(session, wait_for_this, wait_timeout=30):
status = session.expect([wait_for_this, pexpect.TIMEOUT, pexpect.EOF], timeout=wait_timeout)
if status != 0:
print 'ERROR : timeout waiting for "' + wait_for_this + '"'
return False
before = session.before # this is what we will want to return
# now look for and handle any additional sightings of the prompt
while True:
try:
session.expect(wait_for_this, timeout=0.1)
except:
# we expect a timeout here. All is normal. Move along, Citizen.
break # get out of the while loop
return before
s = pexpect.spawn('ssh me#myserver.local')
s.expect('password') # yes, we assume that the SSH key is already there
# and that we will successfully connect. I'm bad.
s.sendline('mypasswordisverysecure') # Also assuming the right password
prompt = 'me$'
wait_for_prompt(s, prompt)
s.sendline('df -h') # how full are my disks?
results = wait_for_prompt(s, prompt)
if results:
print results
sys.exit(0)
else:
print 'Misery. You lose.'
sys.exit(1)

I know this is an old thread, but I didn't find much about this online and I just got through making my own quick-and-dirty workaround for this. I'm also using pexpect to run through a list of network devices and record statistics and so forth, and my pexpect.spawn.before will also get out of sync sometimes. This happens very often on the faster, more modern devices for some reason.
My solution was to write an empty carriage return between each command, and check the len() of the .before variable. If it's too small, it means it only captured the prompt, which means it must be at least one command behind the actual ssh session. If that's the case, the program sends another empty line to move the actual data that I want into the .before variable:
def new_line(this, iteration):
if iteration > 4:
return data
else:
iteration+=1
this.expect(":")
this.sendline(" \r")
data = this.before
if len(data) < 50:
# The numer 50 was chosen because it should be longer than just the hostname and prompt of the device, but shorter than any actual output
data = new_line(this, iteration)
return data
def login(hostname):
this = pexpect.spawn("ssh %s" % hostname)
stop = this.expect([pexpect.TIMEOUT,pexpect.EOF,":"], timeout=20)
if stop == 2:
try:
this.sendline("\r")
this.expect(":")
this.sendline("show version\r")
version = new_line(this,0)
this.expect(":")
this.sendline("quit\r")
return version
except:
print 'failed to execute commands'
this.kill(0)
else:
print 'failed to login'
this.kill(0)
I accomplish this by a recursive command that will call itself until the .before variable finally captures the command's output, or until it calls itself 5 times, in which case it simply gives up.

Open a Python port from Erlang: no reply messages

Based on Chapter 12 of the OTP in Action book and Cesarini's book I wrote this Erlang code:
Erlang:
p(Param) ->
?DBG("Starting~n", []),
Cmd = "python test.py",
Port = open_port({spawn,Cmd}, [stream,{line, 1024}, exit_status]),
?DBG("Opened the port: ~w~n", [Port]),
Payload = term_to_binary(list_to_binary(integer_to_list(Param))),
erlang:port_command(Port, Payload),
?DBG("Sent command to port: ~w~n", [Payload]),
?DBG("Ready to receive results for command: ~w~n", [Payload]),
receive
{Port, {data, Data}} ->
?DBG("Received data: ~w~n", [Data]),
{result, Text} = binary_to_term(Data),
Blah = binary_to_list(Text),
io:format("~p~n", [Blah]);
Other ->
io:format("Unexpected data: ~p~n", [Other])
end.
Python:
import sys
def main():
while True:
line = sys.stdin.readline().strip()
if line == "stop-good":
return 0
elif line == "stop-bad":
return 1
sys.stdout.write("Python got ")
sys.stdout.write(line)
sys.stdout.write("\n")
sys.stdout.flush()
if __name__ == "__main__":
sys.exit(main())
The Erlang code suspends at the recieve clause - it never gets any message.
I have also checked Python from a regular Linux shell - it prints out every user input (1 - "Python got 1").
Where is the mistake here? Why doesn't my Erlang code get anything back?

There are two points:
make sure that Python does not buffer your output, try running python -u in open_port
using term_to_binary/1 and binary_to_term/1 won't work, since they assume that Python is able to encode/decode Erlang External Term Format, which does not seem to be the case. If you want to go this route, check out ErlPort

Does your Param contain the command limiter for Python? (in this case I assume newline, "\n"). Also, list_to_binary/1 and then a term_to_binary/1 feels kinda wrong. term_to_binary/1 directly (including the newline) should be sufficient.

Call the Python interactive interpreter from within a Python script

Is there any way to start up the Python interpreter from within a script , in a manner similar to just using python -i so that the objects/namespace, etc. from the current script are retained? The reason for not using python -i is that the script initializes a connection to an XML-RPC server, and I need to be able to stop the entire program if there's an error. I can't loop until there's valid input because apparently, I can't do something like this:
#!/usr/bin/python -i
# -*- coding: utf-8 -*-
import xmlrpclib
# Create an object to represent our server.
server_url = str(raw_input("Server: "))
while not server = xmlrpclib.Server(server_url):
print 'Unable to connect to server. Please try again'
else:
print 'Xmlrpclib.Server object `__main__.server\' of URL `', server_url, "' created"
break
# Python interpreter starts...
because:
% chmod u+x ./rpcclient.py
% ./rpclient.py
Traceback (most recent call last):
File "./rpcclient.py", line 8
while not server = xmlrpclib.Server(server_url):
^
SyntaxError: invalid syntax
>>>
Unfortunately, python -i starts the interpreter just after it prints out the traceback, so I somehow have to call the interactive interpreter - replacing the execution of the script so it retains the server connection - from within the script

Have you tried reading the error message? :)
= is assignment, you want the comparison operator == instead.

Well, I finally got it to work.
Basically, I put the entire try/except/else clause in a while True: loop, with the else suite being a break statement and the end of the except suite being a continue statement. The result is that it now continually loops if the user puts in an address that doesn't have a fully compliant XML-RPC2 server listening. Here's how it turned out:
#!/usr/bin/python -i
# -*- coding: utf-8 -*-
import xmlrpclib, socket
from sys import exit
# Create an object to represent our server.
#server = xmlrpclib.Server(server_url) and print 'Xmlrpclib.Server object `__main__.server\' of URL `', server_url, "' created"
server_url = str(raw_input("Server: "))
server = xmlrpclib.ServerProxy(server_url)
while True:
try:
server.system.listMethods()
except xmlrpclib.ProtocolError, socket.error:
print 'Unable to connect to server. Please try again'
server_url = str(raw_input("Server: "))
server = xmlrpclib.ServerProxy(server_url)
continue
except EOFError:
exit(1)
else:
break
print 'Xmlrpclib.Server object `__main__.server\' of URL `', server_url, "' created"
# Python interpreter starts...
Thank you very much!
...and I have to wait another day to accept this...