Develop Reference

Correct way to implement auto-update feature for PyQt

I'm nearly finished with a PyQt application for Windows, which I would like to have automatically update from a ZIP archive hosted on a remote server.
I have most of the update script complete, with it correctly downloading the new source and extracting it. My final step is to stop the software, replace the old source and restart the app.
My question is about which of the following is more appropriate:
to run the updater script through a system call to python and killing the software using python.
run the updater script using a system call to a batch file that kills the main software before overwriting it's source.
import the updater as a module and perform all of it's actions in the same process as the main software.
I can provide the scripts if necessary.
UPDATE:
So I've been exploring all the ways to do this including using multiprocessing (which produced a child process that was killed along with the parent) and subprocess.
The latter apparently can run the child process separately, which will allow me to shutdown the main app before extracting the new source. Here's what I have working:
#staticmethod
def install(folder):
# stop Pierre, unpack newest version, then restart Pierre.
try:
with open('pierre.pid', mode='r') as pid:
os.kill(int(pid.read()), signal.SIGINT)
with zipfile.ZipFile(file=folder) as zipped:
zipped.extractall(path='src')
try:
pierre = os.path.join(os.path.abspath(os.getcwd()), 'src/pierre.py')
exec(pierre)
except OSError as exc:
logging.error("Restarting Pierre failed. " + str(exc))
try:
os.remove('src.zip')
except OSError as exc:
logging.error("Deletion of zip failed. " + str(exc))
except zipfile.BadZipFile:
logging.error("Pierre update file is corrupt.")
except Exception as exc:
logging.error("Pierre update install failed. " + str(exc))
What's not working:
#staticmethod
def update_process():
# Begin the update process by spawning the updater script.
script = 'python ' + os.getcwd() + '\\updater.py'
subprocess.Popen([script])
The subprocess is producing a FileNotFoundError despite the path working when run manually in command prompt. (The second method is what starts the script, leading to the first method.)

I've got it. Here's what the process spawner looks like now:
#staticmethod
def update_process():
# Begin the update process by spawning the updater script.
script = os.path.join(os.getcwd() + '/updater.py')
script = script.replace('\\', '/')
subprocess.Popen([sys.executable, script], shell=True)
This launches the separate updater script.

python-daemon to start an independent process but let the main application continue?

Okay I'm officially out of ideas after running each and every sample I could find on google up to 19th page. I have a "provider" script. The goal of this python script is to start up other services that run indefinitely even after this "provider" stopped running. Basically, start the process then forget about it but continue the script and not stopping it...
My problem: python-daemon... I have actions (web-service calls to start/stop/get status from the started services). I create the start commands on the fly and perform variable substitution on the config files as required.
Let's start from this point: I have a command to run (A bash script that executes a java process - a long running service that will be stopped sometime later).
def start(command, working_directory):
pidfile = os.path.join(working_directory, 'application.pid')
# I expect the pid of the started application to be here. The file is not created. Nothing is there.
context = daemon.DaemonContext(working_directory=working_directory,
pidfile=daemon.pidfile.PIDLockFile(pidfile))
with context:
psutil.Popen(command)
# This part never runs. Even if I put a simple print statement at this point, that never appears. Debugging in pycharms shows that my script returns with 0 on with context
with open(pidfile, 'r') as pf:
pid = pf.read()
return pid
From here on in my caller to this method I prepare a json object to return to the client which essentially contains an instance_id (don't mind it) and a pid (that'll be used to stop this process in another request.
What happens? After with context my application exits with status 0, nothing is returned, no json response gets created, no pidfile gets created only the executed psutil.Popen command runs. How can I achieve what I need? I need an independently running process and need to know its PID in order to stop it later on. The executed process must run even if the current python script stops for some reason. I can't get around the shell script as that application is not mine I have to use what I have.
Thanks for any tip!
#Edit:
I tried using simply the Popen from psutil/subprocess with somewhat more promising result.
def start(self, command):
import psutil/subprocess
proc = psutil.Popen(command)
return str(proc.pid)
Now If I debug the application and wait some undefined time on the return statement everything is working great! The service is running the pid is there, I can stop later on. Then I simply ran the provider without debugging. It returns the pid but the process is not running. Seems like Popen has no time to start the service because the whole provider stops faster.
#Update:
Using os.fork:
#staticmethod
def __start_process(command, working_directory):
pid = os.fork()
if pid == 0:
os.chdir(working_directory)
proc = psutil.Popen(command)
with open('application.pid', 'w') as pf:
pf.write(proc.pid)
def start(self):
...
__start_process(command, working_directory)
with open(os.path.join(working_directory, 'application.pid'), 'r') as pf:
pid = int(pf.read())
proc = psutil.Process(pid)
print("RUNNING" if proc.status() == psutil.STATUS_RUNNING else "...")
After running the above sample, RUNNING is written on console. After the main script exits because I'm not fast enough:
ps auxf | grep
No instances are running...
Checking the pidfile; sure it's there it was created
cat /application.pid
EMPTY 0bytes

From multiple partial tips i got, finally managed to get it working...
def start(command, working_directory):
pid = os.fork()
if pid == 0:
os.setsid()
os.umask(0) # I'm not sure about this, not on my notebook at the moment
os.execv(command[0], command) # This was strange as i needed to use the name of the shell script twice: command argv[0] [args]. Upon using ksh as command i got a nice error...
else:
with open(os.path.join(working_directory, 'application.pid'), 'w') as pf:
pf.write(str(pid))
return pid
That together solved the issue. The started process is not a child process of the running python script and won't stop when the script terminates.

Have you tried with os.fork()?
In a nutshell, os.fork() spawns a new process and returns the PID of that new process.
You could do something like this:
#!/usr/bin/env python
import os
import subprocess
import sys
import time
command = 'ls' # YOUR COMMAND
working_directory = '/etc' # YOUR WORKING DIRECTORY
def child(command, directory):
print "I'm the child process, will execute '%s' in '%s'" % (command, directory)
# Change working directory
os.chdir(directory)
# Execute command
cmd = subprocess.Popen(command
, shell=True
, stdout=subprocess.PIPE
, stderr=subprocess.PIPE
, stdin=subprocess.PIPE
)
# Retrieve output and error(s), if any
output = cmd.stdout.read() + cmd.stderr.read()
print output
# Exiting
print 'Child process ending now'
sys.exit(0)
def main():
print "I'm the main process"
pid = os.fork()
if pid == 0:
child(command, working_directory)
else:
print 'A subprocess was created with PID: %s' % pid
# Do stuff here ...
time.sleep(5)
print 'Main process ending now.'
sys.exit(0)
if __name__ == '__main__':
main()
Further info:
Documentation: https://docs.python.org/2/library/os.html#os.fork
Examples: http://www.python-course.eu/forking.php
Another related-question: Regarding The os.fork() Function In Python

Python multiprocessing without locking the parent process

OS:- Mac OSX
Python
I'm new to Multiprocessing with python. For my application, I want to open a new process from main and run it without locking the main process. for eg. I'm running process A and now i need to open a new application from A, lets call it process B. I want to open B in such a way that it does not blocks process A and still from Process i should be able to stop the process whenever i wish to.
Till now whatever code i have tried are basic and they lock the process A. and hence i'm unable to achieve it. Is there any workaround to do this ?
I read about fork and spawn but couldn't understand how can i use it to open an application. And i have tried threading also. But with no success. Can anyone tell me how can i do that ?
Currently I'm using subprocess.call() to open Mac Applications through Python.
It would be really helpful.
EDIT :-
I tried the accepted answer of this link
but to no avail. Because it would block the terminal and once we close the app manually it exits with output 0.
Also I have tried this solution. It would do the same. I want to make the the calling process not to be blocked by the called process.
while doing the same task in windows with os.system() gives me exactly what i want. But i don't know how can i do this in Mac.
EDIT 2: CODE
module 1:
import subprocess
def openCmd(name):
subprocess.call(["/usr/bin/open", "-W", "-n", "-a", "/Applications/"+name+".app"])
def closeCmd(name):
subprocess.call(['osascript', '-e', 'tell "'+name+'" to quit'])
main module:
import speech_recognition as sr
import pyttsx
import opCl
speech_engine = pyttsx.init('nsss')
speech_engine.setProperty('rate', 150)
OPEN_COGNATES=['open']
CLOSE_COGNATES=['close']
def speak(text):
speech_engine.say(text)
speech_engine.runAndWait()
re = sr.Recognizer()
def listen():
with sr.Microphone() as source:
re.adjust_for_ambient_noise(source)
while True:
speak("Say something!")
print ">>",
audio = re.listen(source)
try:
speak("now to recognise it,")
value=re.recognize_google(audio)
print (value)
speak("I heard you say {}".format(value))
value=value.strip().split()
name=" ".join(value[1:])
if value[0] in OPEN_COGNATES:
speak("opening "+name)
opCl.openCmd(name)
pass
elif value[0] in CLOSE_COGNATES:
speak("opening "+name)
opCl.closeCmd(name)
pass
else:
pass
except sr.UnknownValueError as e:
speak("Could not understand audio")
print ('Could not understand audio')
except sr.RequestError as e:
speak("can't recognise what you said.")
print ("can't recognise what you said")
if __name__=='__main__':
listen()

Comment: it gave a traceback. FileNotFoundError: [Errno 2] No such file or directory: 'leafpad'
As i wrote, I can't us "/usr/bin/open" and osascript, so my example uses 'leafpad'.
Have you tried replacing Popen([name]) with your
Popen(["/usr/bin/open", "-W", "-n", "-a", "/Applications/"+name+".app"])?
You must pass the same command args as you start it from the command line.
Read this: launch-an-app-on-os-x-with-command-line
Reread From Python » 3.6.1 Documentation subprocess.Popen
Note
shlex.split() can be useful when determining the correct tokenization for args, especially in complex cases:
Python » 3.6.1 Documentation:
subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None)
Run the command described by args. Wait for command to complete, then return the returncode attribute
Your given "/usr/bin/open" and osascript didn't work for me.
From Python » 3.6.1 Documentation subprocess.Popen
NOWAIT example, for instance:
import subprocess
def openCmd(name):
subprocess.Popen([name])
def closeCmd(name):
subprocess.Popen(['killall', name])
if __name__ == '__main__':
while True:
key = input('input 1=open, 0=cloes, q=quit:')
if key == '1':
openCmd(('leafpad'))
if key == '0':
closeCmd('leafpad')
if key == 'q':
break
Note: Killing a process can lead to data loos and or other problems.
Tested with Python:3.4.2
I'm

How to terminate process from Python using pid?

I'm trying to write some short script in python which would start another python code in subprocess if is not already started else terminate terminal & app (Linux).
So it looks like:
#!/usr/bin/python
from subprocess import Popen
text_file = open(".proc", "rb")
dat = text_file.read()
text_file.close()
def do(dat):
text_file = open(".proc", "w")
p = None
if dat == "x" :
p = Popen('python StripCore.py', shell=True)
text_file.write( str( p.pid ) )
else :
text_file.write( "x" )
p = # Assign process by pid / pid from int( dat )
p.terminate()
text_file.close()
do( dat )
Have problem of lacking knowledge to name proces by pid which app reads from file ".proc".
The other problem is that interpreter says that string named dat is not equal to "x" ??? What I've missed ?

Using the awesome psutil library it's pretty simple:
p = psutil.Process(pid)
p.terminate() #or p.kill()
If you don't want to install a new library, you can use the os module:
import os
import signal
os.kill(pid, signal.SIGTERM) #or signal.SIGKILL
See also the os.kill documentation.
If you are interested in starting the command python StripCore.py if it is not running, and killing it otherwise, you can use psutil to do this reliably.
Something like:
import psutil
from subprocess import Popen
for process in psutil.process_iter():
if process.cmdline() == ['python', 'StripCore.py']:
print('Process found. Terminating it.')
process.terminate()
break
else:
print('Process not found: starting it.')
Popen(['python', 'StripCore.py'])
Sample run:
$python test_strip.py #test_strip.py contains the code above
Process not found: starting it.
$python test_strip.py
Process found. Terminating it.
$python test_strip.py
Process not found: starting it.
$killall python
$python test_strip.py
Process not found: starting it.
$python test_strip.py
Process found. Terminating it.
$python test_strip.py
Process not found: starting it.
Note: In previous psutil versions cmdline was an attribute instead of a method.

I wanted to do the same thing as, but I wanted to do it in the one file.
So the logic would be:
if a script with my name is running, kill it, then exit
if a script with my name is not running, do stuff
I modified the answer by Bakuriu and came up with this:
from os import getpid
from sys import argv, exit
import psutil ## pip install psutil
myname = argv[0]
mypid = getpid()
for process in psutil.process_iter():
if process.pid != mypid:
for path in process.cmdline():
if myname in path:
print "process found"
process.terminate()
exit()
## your program starts here...
Running the script will do whatever the script does. Running another instance of the script will kill any existing instance of the script.
I use this to display a little PyGTK calendar widget which runs when I click the clock. If I click and the calendar is not up, the calendar displays. If the calendar is running and I click the clock, the calendar disappears.

So, not directly related but this is the first question that appears when you try to find how to terminate a process running from a specific folder using Python.
It also answers the question in a way(even though it is an old one with lots of answers).
While creating a faster way to scrape some government sites for data I had an issue where if any of the processes in the pool got stuck they would be skipped but still take up memory from my computer. This is the solution I reached for killing them, if anyone knows a better way to do it please let me know!
import pandas as pd
import wmi
from re import escape
import os
def kill_process(kill_path, execs):
f = wmi.WMI()
esc = escape(kill_path)
temp = {'id':[], 'path':[], 'name':[]}
for process in f.Win32_Process():
temp['id'].append(process.ProcessId)
temp['path'].append(process.ExecutablePath)
temp['name'].append(process.Name)
temp = pd.DataFrame(temp)
temp = temp.dropna(subset=['path']).reset_index().drop(columns=['index'])
temp = temp.loc[temp['path'].str.contains(esc)].loc[temp.name.isin(execs)].reset_index().drop(columns=['index'])
[os.system('taskkill /PID {} /f'.format(t)) for t in temp['id']]

Run a .bat program in the background on Windows

I am trying to run a .bat file (which acts as a simulator) in a new window, so it must always be running in the background. I think that creating a new process is the only option that I have. Basically, I want my code to do something like this:
def startSim:
# open .bat file in a new window
os.system("startsim.bat")
# continue doing other stuff here
print("Simulator started")
I'm on Windows so I can't do os.fork.

Use subprocess.Popen (not tested on Windows, but should work).
import subprocess
def startSim():
child_process = subprocess.Popen("startsim.bat")
# Do your stuff here.
# You can terminate the child process after done.
child_process.terminate()
# You may want to give it some time to terminate before killing it.
time.sleep(1)
if child_process.returncode is None:
# It has not terminated. Kill it.
child_process.kill()
Edit: you could also use os.startfile (Windows only, not tested too).
import os
def startSim():
os.startfile("startsim.bat")
# Do your stuff here.

Looks like you want "os.spawn*", which seems to equate to os.fork, but for Windows.
Some searching turned up this example:
# File: os-spawn-example-3.py
import os
import string
if os.name in ("nt", "dos"):
exefile = ".exe"
else:
exefile = ""
def spawn(program, *args):
try:
# check if the os module provides a shortcut
return os.spawnvp(program, (program,) + args)
except AttributeError:
pass
try:
spawnv = os.spawnv
except AttributeError:
# assume it's unix
pid = os.fork()
if not pid:
os.execvp(program, (program,) + args)
return os.wait()[0]
else:
# got spawnv but no spawnp: go look for an executable
for path in string.split(os.environ["PATH"], os.pathsep):
file = os.path.join(path, program) + exefile
try:
return spawnv(os.P_WAIT, file, (file,) + args)
except os.error:
pass
raise IOError, "cannot find executable"
#
# try it out!
spawn("python", "hello.py")
print "goodbye"

On Windows, a background process is called a "service". Check this other question about how to create a Windows service with Python: Creating a python win32 service

import subprocess
proc = subprocess.Popen(['/path/script.bat'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
Using subprocess.Popen() will run the given .bat path ( or any other executable).
If you do wish to wait for the process to finish just add proc.wait():
proc.wait()