Develop Reference

Python multiprocessing without locking the parent process

OS:- Mac OSX
Python
I'm new to Multiprocessing with python. For my application, I want to open a new process from main and run it without locking the main process. for eg. I'm running process A and now i need to open a new application from A, lets call it process B. I want to open B in such a way that it does not blocks process A and still from Process i should be able to stop the process whenever i wish to.
Till now whatever code i have tried are basic and they lock the process A. and hence i'm unable to achieve it. Is there any workaround to do this ?
I read about fork and spawn but couldn't understand how can i use it to open an application. And i have tried threading also. But with no success. Can anyone tell me how can i do that ?
Currently I'm using subprocess.call() to open Mac Applications through Python.
It would be really helpful.
EDIT :-
I tried the accepted answer of this link
but to no avail. Because it would block the terminal and once we close the app manually it exits with output 0.
Also I have tried this solution. It would do the same. I want to make the the calling process not to be blocked by the called process.
while doing the same task in windows with os.system() gives me exactly what i want. But i don't know how can i do this in Mac.
EDIT 2: CODE
module 1:
import subprocess
def openCmd(name):
subprocess.call(["/usr/bin/open", "-W", "-n", "-a", "/Applications/"+name+".app"])
def closeCmd(name):
subprocess.call(['osascript', '-e', 'tell "'+name+'" to quit'])
main module:
import speech_recognition as sr
import pyttsx
import opCl
speech_engine = pyttsx.init('nsss')
speech_engine.setProperty('rate', 150)
OPEN_COGNATES=['open']
CLOSE_COGNATES=['close']
def speak(text):
speech_engine.say(text)
speech_engine.runAndWait()
re = sr.Recognizer()
def listen():
with sr.Microphone() as source:
re.adjust_for_ambient_noise(source)
while True:
speak("Say something!")
print ">>",
audio = re.listen(source)
try:
speak("now to recognise it,")
value=re.recognize_google(audio)
print (value)
speak("I heard you say {}".format(value))
value=value.strip().split()
name=" ".join(value[1:])
if value[0] in OPEN_COGNATES:
speak("opening "+name)
opCl.openCmd(name)
pass
elif value[0] in CLOSE_COGNATES:
speak("opening "+name)
opCl.closeCmd(name)
pass
else:
pass
except sr.UnknownValueError as e:
speak("Could not understand audio")
print ('Could not understand audio')
except sr.RequestError as e:
speak("can't recognise what you said.")
print ("can't recognise what you said")
if __name__=='__main__':
listen()

Comment: it gave a traceback. FileNotFoundError: [Errno 2] No such file or directory: 'leafpad'
As i wrote, I can't us "/usr/bin/open" and osascript, so my example uses 'leafpad'.
Have you tried replacing Popen([name]) with your
Popen(["/usr/bin/open", "-W", "-n", "-a", "/Applications/"+name+".app"])?
You must pass the same command args as you start it from the command line.
Read this: launch-an-app-on-os-x-with-command-line
Reread From Python » 3.6.1 Documentation subprocess.Popen
Note
shlex.split() can be useful when determining the correct tokenization for args, especially in complex cases:
Python » 3.6.1 Documentation:
subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None)
Run the command described by args. Wait for command to complete, then return the returncode attribute
Your given "/usr/bin/open" and osascript didn't work for me.
From Python » 3.6.1 Documentation subprocess.Popen
NOWAIT example, for instance:
import subprocess
def openCmd(name):
subprocess.Popen([name])
def closeCmd(name):
subprocess.Popen(['killall', name])
if __name__ == '__main__':
while True:
key = input('input 1=open, 0=cloes, q=quit:')
if key == '1':
openCmd(('leafpad'))
if key == '0':
closeCmd('leafpad')
if key == 'q':
break
Note: Killing a process can lead to data loos and or other problems.
Tested with Python:3.4.2
I'm

A Python Wrapper or Handler for A Minecraft Server

I am using Windows and am looking for a handler or wrapper using Python for a Minecraft server so that I can automatically enter commands without user input. I have searched through many questions on the website and only found half answers (in my case at least). I believe I will need to use the subprocess module but cannot decide which to use at the moment I am experimenting with the Popen functions. I have found an answer which I modified for my case:
server = Popen("java -jar minecraft_server.jar nogui", stdin=PIPE, stdout=PIPE, stderr=STDOUT)
while True:
print(server.stdout.readline())
server.stdout.flush()
command = input("> ")
if command:
server.stdin.write(bytes(command + "\r\n", "ascii"))
server.stdin.flush()
This does work in some way but only prints a line for every time you enter a command, which cannot work and all my efforts to change this end up with the program unable to execute anything else and instead just read. This is not a duplicate question because none of the answers in similar questions could help me enough.

As you already know, your server.stdout.readline() and input("> ") are blocking your code execution.
You need to make your code non-blocking, by not waiting to actually return what you want, but by checking, if there is anything to read and ignore it, if there isn't and continue to do other things.
On Linux systems you might be able to use select module, but on Windows it only works on sockets.
I was able to make it work on Windows by using threads and queues. (note: it's Python 2 code)
import subprocess, sys
from Queue import Queue, Empty
from threading import Thread
def process_line(line):
if line == "stop\n": # lines have trailing new line characters
print "SERVER SHUTDOWN PREVENTED"
return None
elif line == "quit\n":
return "stop\n"
elif line == "l\n":
return "list\n"
return line
s = subprocess.Popen("java -jar minecraft_server.jar nogui", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
def read_lines(stream, queue):
while True:
queue.put(stream.readline())
# terminal reading thread
q = Queue()
t = Thread(target=read_lines, args=(sys.stdin, q))
t.daemon = True
t.start()
# server reading thread
qs = Queue()
ts = Thread(target=read_lines, args=(s.stdout, qs))
ts.daemon = True
ts.start()
while s.poll() == None: # loop while the server process is running
# get a user entered line and send it to the server
try:
line = q.get_nowait()
except Empty:
pass
else:
line = process_line(line) # do something with the user entered line
if line != None:
s.stdin.write(line)
s.stdin.flush()
# just pass-through data from the server to the terminal output
try:
line = qs.get_nowait()
except Empty:
pass
else:
sys.stdout.write(line)
sys.stdout.flush()

Python Non-Blocking Reading of Commands

I'm trying to read command outputs from hcitools in Linux (it scans for bluetooth devices).
I just need to read the first line that it returns, as sometimes this tool has an error. The issue is that this tool continues to run in a infinite loop, which locks up the rest of my Python script. The script is run with sudo so that it has root privileges to use the hcitool command.
I have created a class to try to pipe the data in asynchronously:
class ASyncThread(threading.Thread): #pOpen read and readline are blocking. So we must use an async thread to read from hciTool
def __init__(self, command, parameters = []):
self.stdout = None
self.stderr = None
self.command = command
self.parameters = parameters
self.process = None
threading.Thread.__init__(self)
def run(self):
if len(self.command) >= 1:
self.process = subprocess.Popen([self.command] + self.parameters, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
self.stdout, self.stderr = self.process.communicate()
else:
print "[ASyncThread::run()] Error: Empty command given."
def terminate(self):
try:
self.process.terminate()
except Exception, ex:
print "[ASyncThread::terminate()] Error: ", ex
And I'm calling it with:
print "Checking HCI Tool Status..."
hciThread = ASyncThread("/usr/local/bin/hciconfig", ["lescan"])
hciThread.start()
time.sleep(1) #Give the program time to run.
hciThread.terminate() #If terminate is not placed here, it locks up my Python script when the thread is joined.
hciThread.join()
outputText = hciThread.stdout + " | " + hciThread.stderr
When this is run, the output is just " | ".
If I run this command:
sudo /usr/local/bin/hcitool lescan
It instantly starts working immediately:
slyke#ubuntu ~ $ sudo hcitool lescan
Set scan parameters failed: Input/output error
I've been working on this for a few hours now. I originally tried to do this with pOpen, but read() and readline() are both blocking. This is not normally a problem, except that there may not be an error, or any data produced by this command, so my Python script hangs. This is why I moved to threading, so it can wait for a second before stopping it, and continuing on.

It seems to me you cannot possibly join a thread, after you have just terminated it on the line above.
Your particular issue about doing an lescan is probably better solved with the solution from mikerr/btle-scan.py - https://gist.github.com/mikerr/372911c955e2a94b96089fbc300c2b5d

reading output from pexpect sendline

I have pexpect working, but I am having problems printing the output back from it. In my test script below, it creates the ssh connection, and then sends a sudo su -, then my password, and then sends a line that would require sudo access to do (I have also added p.interact() a few times to make sure it is at root). The problem I am having, is with returning the output of the commands I run. In the end I am wanting to run some top commands, and some du -h, and other(much more complex) space commands. But currently when it tries to print p.before, I get:
Traceback (most recent call last):
File "./ssh.py", line 37, in <module>
print p.before()
TypeError: 'str' object is not callable
Here is the script I am working from(edited to remove my pass and such)
#!/usr/bin/env python
import pexpect
import struct, fcntl, os, sys, signal
def sigwinch_passthrough (sig, data):
# Check for buggy platforms (see pexpect.setwinsize()).
if 'TIOCGWINSZ' in dir(termios):
TIOCGWINSZ = termios.TIOCGWINSZ
else:
TIOCGWINSZ = 1074295912 # assume
s = struct.pack ("HHHH", 0, 0, 0, 0)
a = struct.unpack ('HHHH', fcntl.ioctl(sys.stdout.fileno(), TIOCGWINSZ , s))
global global_pexpect_instance
global_pexpect_instance.setwinsize(a[0],a[1])
ssh_newkey = 'Are you sure you want to continue connecting'
p=pexpect.spawn('ssh user#localhost')
i=p.expect([ssh_newkey,'password:',pexpect.EOF,pexpect.TIMEOUT],1)
if i==0:
print "I say yes"
p.sendline('yes')
i=p.expect([ssh_newkey,'password:',pexpect.EOF])
if i==1:
print "I give password",
p.sendline("mypassword")
elif i==2:
print "I either got key or connection timeout"
pass
elif i==3: #timeout
pass
global global_pexpect_instance
global_pexpect_instance = p
p.sendline("sudo su -")
p.sendline("mypasswd")
p.sendline("mkdir /home/user/test")
print p.before
I am working off of this link: http://linux.byexamples.com/archives/346/python-how-to-access-ssh-with-pexpect/
Any help is much appreciated.
EDIT: As Armin Rigo pointed out below. I was calling to p.before as a function like p.before(). Stupid mistake on my part, as this explains why I was getting this error today, and not yesterday when I was trying this. After making that change to my script, and modifying the command being sent, print p.before, and no output is returned. Any other ways to return output from a sendline() command?

Use logfile, that logfile is store all output in terminal.use that example code:-
child = pexpect.spawn("ssh user#localhost")
child.logfile = open("/tmp/mylog", "w")
child.expect(".*assword:")
child.send("guest\r")
child.expect(".*\$ ")
child.sendline("python -V\r")
open the log file and see everything in terminals event

To fetch the complete output after sendline use child.read()
e.g.
cmd_resp = pexpect.spawnu(cmd) # for execution of the command
str_to_search = 'Please Enter The Password'
cmd_resp.sendline('yes') # for sending the input 'yes'
resp = cmd_resp.expect([str_to_search, 'password:', EOF], timeout=30) # fetch the output status
if resp == 1:
cmd_resp.sendline(password)
resp = cmd_resp.expect([str_to_search, 'outputString:', EOF], timeout=30)
print(cmd_resp.read()) # to fetch the complete output log

p.before is a string - not a function. To see the output you have to write
print p.before.
Hope this might help you

Running an interactive command from within Python

I have a script that I want to run from within Python (2.6.5) that follows the logic below:
Prompts the user for a password. It looks like ("Enter password: ") (*Note: Input does not echo to screen)
Output irrelevant information
Prompt the user for a response ("Blah Blah filename.txt blah blah (Y/N)?: ")
The last prompt line contains text which I need to parse (filename.txt). The response provided doesn't matter (the program could actually exit here without providing one, as long as I can parse the line).
My requirements are somewhat similar to Wrapping an interactive command line application in a Python script, but the responses there seem a bit confusing, and mine still hangs even when the OP mentions that it doesn't for him.
Through looking around, I've come to the conclusion that subprocess is the best way of doing this, but I'm having a few issues. Here is my Popen line:
p = subprocess.Popen("cmd", shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
When I call a read() or readline() on stdout, the prompt is printer to the screen and it hangs.
If I call a write("password\n") for stdin, the prompt is written to the screen and it hangs. The text in write() is not written (I don't the cursor move the a new line).
If I call p.communicate("password\n"), same behavior as write()
I was looking for a few ideas here on the best way to input to stdin and possibly how to parse the last line in the output if your feeling generous, though I could probably figure that out eventually.

If you are communicating with a program that subprocess spawns, you should check out A non-blocking read on a subprocess.PIPE in Python. I had a similar problem with my application and found using queues to be the best way to do ongoing communication with a subprocess.
As for getting values from the user, you can always use the raw_input() builtin to get responses, and for passwords, try using the getpass module to get non-echoing passwords from your user. You can then parse those responses and write them to your subprocess' stdin.
I ended up doing something akin to the following:
import sys
import subprocess
from threading import Thread
try:
from Queue import Queue, Empty
except ImportError:
from queue import Queue, Empty # Python 3.x
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
def getOutput(outQueue):
outStr = ''
try:
while True: # Adds output from the Queue until it is empty
outStr+=outQueue.get_nowait()
except Empty:
return outStr
p = subprocess.Popen("cmd", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False, universal_newlines=True)
outQueue = Queue()
errQueue = Queue()
outThread = Thread(target=enqueue_output, args=(p.stdout, outQueue))
errThread = Thread(target=enqueue_output, args=(p.stderr, errQueue))
outThread.daemon = True
errThread.daemon = True
outThread.start()
errThread.start()
try:
someInput = raw_input("Input: ")
except NameError:
someInput = input("Input: ")
p.stdin.write(someInput)
errors = getOutput(errQueue)
output = getOutput(outQueue)
Once you have the queues made and the threads started, you can loop through getting input from the user, getting errors and output from the process, and processing and displaying them to the user.

Using threading it might be slightly overkill for simple tasks.
Instead os.spawnvpe can be used. It will spawn script shell as a process. You will be able to communicate interactively with the script.
In this example I passed password as an argument, obviously that is not a good idea.
import os
import sys
from getpass import unix_getpass
def cmd(cmd):
cmd = cmd.split()
code = os.spawnvpe(os.P_WAIT, cmd[0], cmd, os.environ)
if code == 127:
sys.stderr.write('{0}: command not found\n'.format(cmd[0]))
return code
password = unix_getpass('Password: ')
cmd_run = './run.sh --password {0}'.format(password)
cmd(cmd_run)
pattern = raw_input('Pattern: ')
lines = []
with open('filename.txt', 'r') as fd:
for line in fd:
if pattern in line:
lines.append(line)
# manipulate lines

If you just want a user to enter a password without it being echoed to the screen just use the standard library's getpass module:
import getpass
print("You entered:", getpass.getpass())
NOTE:The prompt for this function defaults to "Password: " also this will only work on command lines where echoing can be controlled. So if it doesn't work try running it from terminal.