Develop Reference

How could I stop the script without having to wait for the time set for interval to pass?

In this script I was looking to launch a given program and monitor it as long as the program exists. Thus, I reached the point where I got to use the threading's module Timer method for controlling a loop that writes to a file and prints out to the console a specific stat of the launched process (for this case, mspaint).
The problem arises when I'm hitting CTRL + C in the console or when I close mspaint, with the script capturing any of the 2 events only after the time defined for the interval has completely ran out. These events make the script stop.
For example, if a 20 seconds time is set for the interval, once the script has started, if at second 5 I either hit CTRL + C or close mspaint, the script will stop only after the remaining 15 seconds will have passed.
I would like for the script to stop right away when I either hit CTRL + C or close mspaint (or any other process launched through this script).
The script can be used with the following command, according to the example:
python.exe mon_tool.py -p "C:\Windows\System32\mspaint.exe" -i 20
I'd really appreciate if you could come up with a working example.
I had used python 3.10.4 and psutil 5.9.0 .
This is the code:
# mon_tool.py
import psutil, sys, os, argparse
from subprocess import Popen
from threading import Timer
debug = False
def parse_args(args):
parser = argparse.ArgumentParser()
parser.add_argument("-p", "--path", type=str, required=True)
parser.add_argument("-i", "--interval", type=float, required=True)
return parser.parse_args(args)
def exceptionHandler(exception_type, exception, traceback, debug_hook=sys.excepthook):
'''Print user friendly error messages normally, full traceback if DEBUG on.
Adapted from http://stackoverflow.com/questions/27674602/hide-traceback-unless-a-debug-flag-is-set
'''
if debug:
print('\n*** Error:')
debug_hook(exception_type, exception, traceback)
else:
print("%s: %s" % (exception_type.__name__, exception))
sys.excepthook = exceptionHandler
def validate(data):
try:
if data.interval < 0:
raise ValueError
except ValueError:
raise ValueError(f"Time has a negative value: {data.interval}. Please use a positive value")
def main():
args = parse_args(sys.argv[1:])
validate(args)
# creates the "Process monitor data" folder in the "Documents" folder
# of the current Windows profile
default_path: str = f"{os.path.expanduser('~')}\\Documents\Process monitor data"
if not os.path.exists(default_path):
os.makedirs(default_path)
abs_path: str = f'{default_path}\data_test.txt'
print("data_test.txt can be found in: " + default_path)
# launches the provided process for the path argument, and
# it checks if the process was indeed launched
p: Popen[bytes] = Popen(args.path)
PID = p.pid
isProcess: bool = True
while isProcess:
for proc in psutil.process_iter():
if(proc.pid == PID):
isProcess = False
process_stats = psutil.Process(PID)
# creates the data_test.txt and it erases its content
with open(abs_path, 'w', newline='', encoding='utf-8') as testfile:
testfile.write("")
# loop for writing the handles count to data_test.txt, and
# for printing out the handles count to the console
def process_monitor_loop():
with open(abs_path, 'a', newline='', encoding='utf-8') as testfile:
testfile.write(f"{process_stats.num_handles()}\n")
print(process_stats.num_handles())
Timer(args.interval, process_monitor_loop).start()
process_monitor_loop()
if __name__ == '__main__':
main()
Thank you!

I think you could use python-worker (link) for the alternatives
import time
from datetime import datetime
from worker import worker, enableKeyboardInterrupt
# make sure to execute this before running the worker to enable keyboard interrupt
enableKeyboardInterrupt()
# your codes
...
# block lines with periodic check
def block_next_lines(duration):
t0 = time.time()
while time.time() - t0 <= duration:
time.sleep(0.05) # to reduce resource consumption
def main():
# your codes
...
#worker(keyboard_interrupt=True)
def process_monitor_loop():
while True:
print("hii", datetime.now().isoformat())
block_next_lines(3)
return process_monitor_loop()
if __name__ == '__main__':
main_worker = main()
main_worker.wait()
here your process_monitor_loop will be able to stop even if it's not exactly 20 sec of interval

You can try registering a signal handler for SIGINT, that way whenever the user presses Ctrl+C you can have a custom handler to clean all of your dependencies, like the interval, and exit gracefully.
See this for a simple implementation.

This is the solution for the second part of the problem, which checks if the launched process exists. If it doesn't exist, it stops the script.
This solution comes on top of the solution, for the first part of the problem, provided above by #danangjoyoo, which deals with stopping the script when CTRL + C is used.
Thank you very much once again, #danangjoyoo! :)
This is the code for the second part of the problem:
import time, psutil, sys, os
from datetime import datetime
from worker import worker, enableKeyboardInterrupt, abort_all_thread, ThreadWorkerManager
from threading import Timer
# make sure to execute this before running the worker to enable keyboard interrupt
enableKeyboardInterrupt()
# block lines with periodic check
def block_next_lines(duration):
t0 = time.time()
while time.time() - t0 <= duration:
time.sleep(0.05) # to reduce resource consumption
def main():
# launches mspaint, gets its PID and checks if it was indeed launched
path = f"C:\Windows\System32\mspaint.exe"
p = psutil.Popen(path)
PID = p.pid
isProcess: bool = True
while isProcess:
for proc in psutil.process_iter():
if(proc.pid == PID):
isProcess = False
interval = 5
global counter
counter = 0
#allows for sub_process to run only once
global run_sub_process_once
run_sub_process_once = 1
#worker(keyboard_interrupt=True)
def process_monitor_loop():
while True:
print("hii", datetime.now().isoformat())
def sub_proccess():
'''
Checks every second if the launched process still exists.
If the process doesn't exist anymore, the script will be stopped.
'''
print("Process online:", psutil.pid_exists(PID))
t = Timer(1, sub_proccess)
t.start()
global counter
counter += 1
print(counter)
# Checks if the worker thread is alive.
# If it is not alive, it will kill the thread spawned by sub_process
# hence, stopping the script.
for _, key in enumerate(ThreadWorkerManager.allWorkers):
w = ThreadWorkerManager.allWorkers[key]
if not w.is_alive:
t.cancel()
if not psutil.pid_exists(PID):
abort_all_thread()
t.cancel()
global run_sub_process_once
if run_sub_process_once:
run_sub_process_once = 0
sub_proccess()
block_next_lines(interval)
return process_monitor_loop()
if __name__ == '__main__':
main_worker = main()
main_worker.wait()
Also, I have to note that #danangjoyoo's solution comes as an alternative to signal.pause() for Windows. This only deals with CTRL + C problem part. signal.pause() works only for Unix systems. This is how it was supposed for its usage, for my case, in case it were a Unix system:
import signal, sys
from threading import Timer
def main():
def signal_handler(sig, frame):
print('\nYou pressed Ctrl+C!')
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
print('Press Ctrl+C')
def process_monitor_loop():
try:
print("hi")
except KeyboardInterrupt:
signal.pause()
Timer(10, process_monitor_loop).start()
process_monitor_loop()
if __name__ == '__main__':
main()
The code above is based on this.

Python pool code not running

I'm writing a Windows service with multiproccesing and I'm running into problems with the method that's called in the pool.
Right now I'm able to install the service and run it, it outputs The service started running... to the log file but nothing else.
Looking at the process explorer (see screenshot below), I see that the processes are being created and finishing constantly, but the code within the TestMethod isn't being run, and the service isn't exiting the pool because nothing else is being written to the file.
I'm not able to stop the service since it's stuck in the pool and doesn't reach the check for the stop event.
Why is the code within the TestMethod not running at all?
Service code:
import servicemanager
import win32event
import win32service
import win32serviceutil
import multiprocessing
class TestService(win32serviceutil.ServiceFramework):
_svc_name_ = "TestService"
_svc_display_name_ = "Test Service"
def testMethod(self, testVar):
with open('C:\\Test.log', 'a') as f:
f.write('The method is running: ' + testVar)
f.close()
def __init__(self, args):
win32serviceutil.ServiceFramework.__init__(self, args)
self.hWaitStop = win32event.CreateEvent(None, 0, 0, None)
socket.setdefaulttimeout(60)
def SvcStop(self):
self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
win32event.SetEvent(self.hWaitStop)
def SvcDoRun(self):
with open('C:\\Test.log', 'a') as f:
f.write('The service started running...\n')
f.close()
rc = None
p = multiprocessing.Pool(5)
p.map(TestService.testMethod, range(1,6))
with open('C:\\Test.log', 'a') as f:
f.write('Finished method...\n')
f.close()
while rc != win32event.WAIT_OBJECT_0:
with open('C:\\Test.log', 'a') as f:
f.write('The service is running...\n')
f.close()
rc = win32event.WaitForSingleObject(self.hWaitStop, 5000)
with open('C:\\Test.log', 'a') as f:
f.write('StreamCapture service stopped.\n')
f.close()
if __name__ == '__main__':
if len(sys.argv) == 1:
servicemanager.Initialize()
servicemanager.PrepareToHostSingle(TestService)
servicemanager.StartServiceCtrlDispatcher()
else:
win32serviceutil.HandleCommandLine(TestService)

There are two problems in your code:
you point map to TestService.testMethod, which is an "unbound function" that lives in the Class namespace, but testMethod is defined like a class method. You either need to call it with self.testMethod or you remove the self from the function definition
you try to add an int to a string, use f.write('The method is running: {}'.format(testVar)) instead
Your stripped down program, corrected would look like this:
import multiprocessing
class TestService:
def testMethod(self, testVar):
with open('C:\\Test.log', 'a') as f:
f.write('The method is running: {}'.format(testVar))
f.close()
def SvcDoRun(self):
p = multiprocessing.Pool(5)
p.map(self.testMethod, range(1,6))
if __name__ == "__main__":
d = TestService()
d.SvcDoRun()
P.S. try to post a minimal code example the next time: Strip down the code to the bare minimum which generates the error. The code fragment I posted would have been enough to explain the problem. This way it is easier to understand for readers and you get an answer faster.

The problem was caused by a known issue with pyinstaller and onefile executables on Windows.
Adding the following try block after my imports fixed it for me:
try:
# Python 3.4+
if sys.platform.startswith('win'):
import multiprocessing.popen_spawn_win32 as forking
else:
import multiprocessing.popen_fork as forking
except ImportError:
import multiprocessing.forking as forking
See https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Multiprocessing for more details

gevent + flask seems to block

I want to run a background worker in the same script as flask is running and flask seems to be blocking which I guess is understandable. Pretty much I want a script to check key system metrics every second so I don't want to use something like celery or a big queueing system to do it.
Simple code example
#!/usr/bin/env python
import gevent
from flask import Flask
class Monitor:
def __init__(self, opts):
self.opts = opts
def run(self):
print "do something: %i" % self.opts
gevent.sleep(1)
self.run()
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello World!'
if __name__ == '__main__':
threads = []
for mon in [1,2]:
monitor = Monitor(mon)
threads.append(gevent.spawn(monitor.run))
threads.append(gevent.spawn(app.run))
gevent.joinall(threads)
My output looks like
$ ./so.py
do something: 1
do something: 2
* Running on http://127.0.0.1:5000/
If I remove the theads.append for the app.run it runs fine. Is this possible to do or am I barking up the wrong tree?
Thanks

add in your script next two lines:
from gevent import monkey
monkey.patch_all()
before the line:
from flask import Flask
and all be OK

This is how I ended up handling the issue using apscheduler v2
#!/usr/bin/env python
import gevent
import time
from flask import Flask
from apscheduler.scheduler import Scheduler
sched = Scheduler()
sched.start()
class Monitor:
def __init__(self, opts):
self.opts = opts
def run(self):
#sched.interval_schedule(seconds=1)
def handle_run():
print "do something: %i" % self.opts
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello World!'
if __name__ == '__main__':
for mon in [1,2]:
monitor = Monitor(mon)
monitor.run()
app.run(threaded=True)

try using the below:
class Monitor:
def init(self, opts):
self.opts = opts
def run(self):
While True:
print "do something: %i" % self.opts
gevent.sleep(1)
and then maybe don't joinall, since it doesn't seem like you really want to wait for them to complete before doing something else.
You may also need to put a try/except statement inside the while loop, and respawn if there is an error that kills the greenlet.

Python Multithreading (while and apscheduler)

I am trying to call two functions simultaneously in Python. One is an infinite loop and the other one is started using apscheduler. Like this:
Thread.py
from multiprocessing import Process
import _While
import _Scheduler
if __name__ == '__main__':
p1 = Process(target=_While.main())
p1.start()
p2 = Process(target=_Scheduler.main())
p2.start()
_While.py
import time
def main():
while True:
print "while"
time.sleep(0.5)
_Scheduler.py
import logging
from apscheduler.scheduler import Scheduler
def _scheduler():
print "scheduler"
if __name__ == '__main__':
logging.basicConfig()
scheduler = Scheduler(standalone=True)
scheduler.add_interval_job(lambda: _scheduler(), seconds=2)
scheduler.start()
Since only while is printed it seems that _Scheduler isn’t starting.
Can somone help me?

You've got at least a couple problems here. First, the target keyword should be a function, not the result of a function. e.g.:
p1 = Process(target=_While.main) # Note the lack of function call
Second, I don't see any _Scheduler.main function. Maybe you meant to do something like:
import logging
from apscheduler.scheduler import Scheduler
def _scheduler():
print "scheduler"
def main():
logging.basicConfig()
scheduler = Scheduler(standalone=True)
scheduler.add_interval_job(_scheduler, seconds=2) # I doubt that `lambda` is necessary here ...
scheduler.start()
if __name__ == "__main__":
main()

Python core application

I'm new to Python and I'm writing a script that
includes some timed routines.
My current approach is to instantiate a class
that includes those Timers (from: threading.Timer),
but I don't want the script to return when it gets to the
end of the function:
import mytimer
timer = mytimer()
Suppose I have a imple script like that one. All it
does is instantiate a mytimer object which performs a series
of timed activities.
In order for the application not to exit, I could use Qt like this:
from PyQt4.QtCore import QCoreApplication
import mytimer
import sys
def main():
app = QCoreApplication(sys.argv)
timer = mytimer()
sys.exit(app.exec_())
if __name__ == '__main__':
main()
This way, the sys.exit() call won't return immediately, and the
timer would just keep doing its thing 'forever' in background.
Although this is a solution I've used before, using Qt just for this doesn't
fell right to me.
So my question is, Is there any way to accomplish this using standard Python?
Thanks

Create a function in your script which tests a select or poll object to terminate a loop. Check out serve_forever in SocketServer.py from the standard library as an example.

A Google search for "python timer" finds:
http://docs.python.org/library/sched.html
http://docs.python.org/release/2.5.2/lib/timer-objects.html
The sched module seems to be exactly what you need.
Example:
>>> import sched, time
>>> s = sched.scheduler(time.time, time.sleep)
>>> def print_time(): print "From print_time", time.time()
...
>>> def print_some_times():
... print time.time()
... s.enter(5, 1, print_time, ())
... s.enter(10, 1, print_time, ())
... s.run()
... print time.time()
...
>>> print_some_times()
930343690.257
From print_time 930343695.274
From print_time 930343700.273
930343700.276
Once you have built your queue of times for things to happen, you just call the .run() method on your sched instance, and it will automatically wait until the queue is emptied, then will complete. So you can just put s.run() as the last thing in your script, and it will automatically exit only when the timed tasks are all done.

import mytimer
import sys
from threading import Lock
lock = Lock()
lock.acquire() # put lock into locked state
def main():
timer = mytimer()
lock.acquire() # blocks until someone calls lock.release()
if __name__ == '__main__':
main()
If you want a clean exit, you can just make mytimer() call lock.release() at some point.