Develop Reference

Stop the running instances when max_instances is reached

I'm using apscheduler-django and I created a task that loops every 10 seconds.
This function will make a request to an API and save the content to my database (PostgreSQL).
This is my task:
scheduler.add_job(
SaveAPI,
trigger=CronTrigger(second="*/10"),
id="SaveAPI",
max_instances=1,
replace_existing=True,
)
and my SaveAPI is:
def SaveAPI():
SPORT = 3
print('API Queue Started')
AllMatches = GetAllMatches(SPORT)
for Match in AllMatches:
AddToDatabase(Match, SPORT)
print(f'API Queue Ended')
The GetAllMatches and AddToDatabase are too big and I don't think the implementations are relevant to my question.
My problem is sometimes I will get this error:
Run time of job "SaveAPI (trigger: cron[second='*/10'], next run at: 2022-03-05 23:21:00 +0330)" was missed by 0:00:11.445357
When this happens, it will not get replaced with a new instance because my SaveAPI function doesn't end. And apscheduler will always miss new instances.
I did many tests and function does not have any problem.
How can I make apscheduler stop the last running instance if a new instance is going to be missed?
So if my last instance takes more than 10 seconds, I want to just terminate the instance and create a new one.

apscheduler and apscheduler-django don't directly support that.
You can implement and use a custom executor that tracks the process running a job and kills the process if trying to submit a job that is currently running.
Here's a MaxInstancesCancelEarliestProcessPoolExecutor that uses pebble.ProcessPool.
class MaxInstancesCancelEarliestProcessPoolExecutor(BasePoolExecutor):
def __init__(self):
pool = ProcessPool()
pool.submit = lambda function, *args: pool.schedule(function, args=args)
super().__init__(pool)
self._futures = defaultdict(list)
def submit_job(self, job, run_times):
assert self._lock is not None, 'This executor has not been started yet'
with self._lock:
if self._instances[job.id] >= job.max_instances:
f = self._futures[job.id][0] # +
f.cancel() # +
try: # +
self._pool._pool_manager.update_status() # +
except RuntimeError: # +
pass # +
if self._instances[job.id] >= job.max_instances: # +
raise MaxInstancesReachedError(job)
self._do_submit_job(job, run_times)
self._instances[job.id] += 1
def _do_submit_job(self, job, run_times):
def callback(f):
with self._lock: # +
self._futures[job.id].remove(f) # +
try: # +
exc, tb = (f.exception_info() if hasattr(f, 'exception_info') else
(f.exception(), getattr(f.exception(), '__traceback__', None)))
except CancelledError: # +
exc, tb = TimeoutError(), None # +
if exc:
self._run_job_error(job.id, exc, tb)
else:
self._run_job_success(job.id, f.result())
try:
f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
except BrokenProcessPool:
self._logger.warning('Process pool is broken; replacing pool with a fresh instance')
self._pool = self._pool.__class__(self._pool._max_workers)
f = self._pool.submit(run_job, job, job._jobstore_alias, run_times, self._logger.name)
f.add_done_callback(callback)
self._futures[job.id].append(f) # +
def shutdown(self, wait=True):
if wait:
self._pool.close()
self._pool.join()
else:
self._pool.close()
threading.Thread(target=self._pool.join).start()
Usage:
scheduler.add_executor(MaxInstancesCancelEarliestProcessPoolExecutor(), alias='max_instances_cancel_earliest')
scheduler.add_job(
SaveAPI,
trigger=CronTrigger(second="*/10"),
id="SaveAPI",
max_instances=1,
executor='max_instances_cancel_earliest', # +
replace_existing=True,
)

Monitering time spent on computer w/ Python and xorg-server

I am trying to make a script which will help me track how long I spend on what on my computer. This script should track when I start, stop, and how long I spend on each "task". After some searching I have found a terminal utility called xdotool which will return the current focused window and it's title when ran like so: xdotool getwindowfocus getwindowna me. For example. when focusing on this window it returns:
linux - Monitering time spent on computer w/ Python and xorg-server - Stack Overflow — Firefox Developer Edition
which is exactly what I want. My first idea was to detect when the focused window is changed and then get the time that this happens at, however I could not find any results, so I have resorted to a while loop that runs this command every 5 seconds, but that is quite hack-y and has proven troublesome, I would highly prefer the on-focus-change method, but here is my code as of now:
#!/usr/bin/env python3
from subprocess import run
from time import time, sleep
log = []
prevwindow = ""
while True:
currentwindow = run(['xdotool', 'getwindowfocus', 'getwindowname'],
capture_output=True, text=True).stdout
if currentwindow != prevwindow:
for entry in log:
if currentwindow in entry:
pass # Calculate time spent
print(f"{time()}:\t{currentwindow}")
log.append((time(), currentwindow))
prevwindow = currentwindow
sleep(5)
I am on Arch linux with dwm should that matter

See this gist. Just put your logging mechanism inside the handle_change function and it should work, as tested on an Arch Linux - dwm system.
For archival purposes, here I include the code. All credit goes to Stephan Sokolow (sskolow) on GitHub.
from contextlib import contextmanager
from typing import Any, Dict, Optional, Tuple, Union # noqa
from Xlib import X
from Xlib.display import Display
from Xlib.error import XError
from Xlib.xobject.drawable import Window
from Xlib.protocol.rq import Event
# Connect to the X server and get the root window
disp = Display()
root = disp.screen().root
# Prepare the property names we use so they can be fed into X11 APIs
NET_ACTIVE_WINDOW = disp.intern_atom('_NET_ACTIVE_WINDOW')
NET_WM_NAME = disp.intern_atom('_NET_WM_NAME') # UTF-8
WM_NAME = disp.intern_atom('WM_NAME') # Legacy encoding
last_seen = {'xid': None, 'title': None} # type: Dict[str, Any]
#contextmanager
def window_obj(win_id: Optional[int]) -> Window:
"""Simplify dealing with BadWindow (make it either valid or None)"""
window_obj = None
if win_id:
try:
window_obj = disp.create_resource_object('window', win_id)
except XError:
pass
yield window_obj
def get_active_window() -> Tuple[Optional[int], bool]:
"""Return a (window_obj, focus_has_changed) tuple for the active window."""
response = root.get_full_property(NET_ACTIVE_WINDOW,
X.AnyPropertyType)
if not response:
return None, False
win_id = response.value[0]
focus_changed = (win_id != last_seen['xid'])
if focus_changed:
with window_obj(last_seen['xid']) as old_win:
if old_win:
old_win.change_attributes(event_mask=X.NoEventMask)
last_seen['xid'] = win_id
with window_obj(win_id) as new_win:
if new_win:
new_win.change_attributes(event_mask=X.PropertyChangeMask)
return win_id, focus_changed
def _get_window_name_inner(win_obj: Window) -> str:
"""Simplify dealing with _NET_WM_NAME (UTF-8) vs. WM_NAME (legacy)"""
for atom in (NET_WM_NAME, WM_NAME):
try:
window_name = win_obj.get_full_property(atom, 0)
except UnicodeDecodeError: # Apparently a Debian distro package bug
title = "<could not decode characters>"
else:
if window_name:
win_name = window_name.value # type: Union[str, bytes]
if isinstance(win_name, bytes):
# Apparently COMPOUND_TEXT is so arcane that this is how
# tools like xprop deal with receiving it these days
win_name = win_name.decode('latin1', 'replace')
return win_name
else:
title = "<unnamed window>"
return "{} (XID: {})".format(title, win_obj.id)
def get_window_name(win_id: Optional[int]) -> Tuple[Optional[str], bool]:
"""Look up the window name for a given X11 window ID"""
if not win_id:
last_seen['title'] = None
return last_seen['title'], True
title_changed = False
with window_obj(win_id) as wobj:
if wobj:
try:
win_title = _get_window_name_inner(wobj)
except XError:
pass
else:
title_changed = (win_title != last_seen['title'])
last_seen['title'] = win_title
return last_seen['title'], title_changed
def handle_xevent(event: Event):
"""Handler for X events which ignores anything but focus/title change"""
if event.type != X.PropertyNotify:
return
changed = False
if event.atom == NET_ACTIVE_WINDOW:
if get_active_window()[1]:
get_window_name(last_seen['xid']) # Rely on the side-effects
changed = True
elif event.atom in (NET_WM_NAME, WM_NAME):
changed = changed or get_window_name(last_seen['xid'])[1]
if changed:
handle_change(last_seen)
def handle_change(new_state: dict):
"""Replace this with whatever you want to actually do"""
print(new_state)
if __name__ == '__main__':
# Listen for _NET_ACTIVE_WINDOW changes
root.change_attributes(event_mask=X.PropertyChangeMask)
# Prime last_seen with whatever window was active when we started this
get_window_name(get_active_window()[0])
handle_change(last_seen)
while True: # next_event() sleeps until we get an event
handle_xevent(disp.next_event())

Python QT5 with MultiThreading

Dears,
I am new on python and trying to start python networking with a simple code that have 2 plain text line edits and try to start 2 telnet sessions and get output of each session on a separate plain text line area however i get the below error
QObject::connect: Cannot queue arguments of type 'QTextCursor'
(Make sure 'QTextCursor' is registered using qRegisterMetaType().)
the code as below:
app = QApplication(sys.argv)
def to_str(bytes_or_str):
if isinstance(bytes_or_str, bytes):
value = bytes_or_str.decode() # uses 'utf-8' for encoding
else:
value = bytes_or_str
return value # Instance of str
def testTelnet_start(): # called when button pressed
_thread.start_new_thread(testTelnet,("192.168.3.247",1))
print("Test")
_thread.start_new_thread(testTelnet,("192.168.3.252",2))
def testTelnet(ip,x):
try:
tel_conn = telnetlib.Telnet()
tel_conn.open(host= ip)
data = tel_conn.read_until(b"login:",3)
data = to_str(data)
data = ip + "\n" + data
print(data)
if(x==1):
plain_text_area_Upgrade1.appendPlainText(data)
elif(x==2):
plain_text_area_Upgrade2.appendPlainText(data)
except Exception as e:
print(e)
my_Test_Window = QWidget()
my_Test_Window.setWindowTitle("Telnet Window")
my_Test_Window.resize(490,350)
my_Test_Window.setFixedSize(my_Test_Window.size())
push_test1 = QPushButton("Test#1",my_Test_Window)
push_test1.move(90,280)
plain_text_area_Upgrade1 = QPlainTextEdit(my_Test_Window)
plain_text_area_Upgrade1.resize(160,150)
plain_text_area_Upgrade1.updatesEnabled()
plain_text_area_Upgrade1.move(25,20)
plain_text_area_Upgrade1.setReadOnly(True)
plain_text_area_Upgrade1.insertPlainText("Testing ...")
plain_text_area_Upgrade1.appendPlainText("")
plain_text_area_Upgrade2 = QPlainTextEdit(my_Test_Window)
plain_text_area_Upgrade2.resize(160,150)
plain_text_area_Upgrade2.updatesEnabled()
plain_text_area_Upgrade2.move(250,20)
plain_text_area_Upgrade2.setReadOnly(True)
plain_text_area_Upgrade2.insertPlainText("Testing ...")
plain_text_area_Upgrade2.appendPlainText("")
push_test1.clicked.connect(testTelnet_start)
my_Test_Window.show()
app.exec_()
Any idea why a simple multi threading code cause those errors ?
Thanks.

Python output from Popen command from inside a thread into a GUI form in real time

Being new to Python I have looked around the site and found partial answers but nothing that helps make things clear. This is what I have. The main window through a button activates the thread which runs a command (wash -C -i monX) and a table needs to populate with the results in the GUI in real time. I found the code for the loop here Intercepting stdout of a subprocess while it is running.
the code is here https://github.com/theodhori-dhiamanti/wifern/blob/master
the code in question is:
class WashThread(QtCore.QThread):
def __init__(self, parent=None):
super(WashThread, self).__init__(parent)
def run(self):
global mon_iface
try:
if mon_iface != '':
device = mon_iface
cmd = ['wash', '-C', '-i', device]
wash_cmd = Popen(cmd, stdout=PIPE)
for line in iter(wash_cmd.stdout.readline, b''):
if line.strip() == '' or line.startswith('---'): continue
if line.startswith('Wash') or line.startswith('Copyright') or line.startswith('BSSID'): continue
print line
Split = line.split(' ')
wash_bssid = Split[0]
wash_essid = Split[58]
wash_power = Split[19]
wash_locked = Split[42]
else:
print('No Good')
except OSError:
pass
and the GUI part that calls this method and where the results need to be used is:
def wash(self):
row = 0
col = 0
self.wash_tableWidget.setColumnCount(4)
self.wash_tableWidget.setColumnWidth(1,150)
self.wash_tableWidget.setColumnWidth(4,30)
self.wash_tableWidget.setColumnWidth(3,70)
self.wash_tableWidget.setRowCount(10)
if self.start_wash_Button.text() == 'Start':
self.start_wash_Button.setText('Stop')
self.wash_thread.start()
row_item = QtGui.QTableWidgetItem(wash_bssid)
x = QtGui.QTableWidgetItem(wash_essid)
y = QtGui.QTableWidgetItem(wash_power)
z = QtGui.QTableWidgetItem(wash_locked)
self.wash_tableWidget.setItem(row, col, row_item)
self.wash_tableWidget.setItem(row, 1, x)
self.wash_tableWidget.setItem(row, 2, y)
self.wash_tableWidget.setItem(row, 3, z)
row += 1
else:
try:
os.kill(cmd.pid, SIGTERM)
print('Done')
except OSError:
pass
except UnboundLocalError:
pass
self.wash_thread = WashThread() # method initialized in main body
How can I transfer the output from the thread to the main portion?
After I than can assign the values appropriately to the table fields?
Any help is greatly appreciated.
* Note: out of the thread the method works as intended.

Python multithreading raw_input

I'm currently doing some work with multithreading and i'm trying to figure out why my program isn't working as intended.
def input_watcher():
while True:
input_file = os.path.abspath(raw_input('Input file name: '))
compiler = raw_input('Choose compiler: ')
if os.path.isfile(input_file):
obj = FileObject(input_file, compiler)
with file_lock:
files.append(obj)
print 'Adding %s with %s as compiler' % (obj.file_name, obj.compiler)
else:
print 'File does not exists'
This is running in one thread and it works fine until i start adding adding the second fileobject.
This is the output from the console:
Input file name: C:\Users\Victor\Dropbox\Private\multiFile\main.py
Choose compiler: aImport
Adding main.py with aImport as compiler
Input file name: main.py updated
C:\Users\Victor\Dropbox\Private\multiFile\main.py
Choose compiler: Input file name: Input file name: Input file name: Input file name:
The input filename keeps popping up the second i added the second filename and it ask for a compiler. The program keeps printing input file name until it crashes.'
I have other code running in a different thread, i don't think it has anything to do with the error, but tell me if you think you need to see it and i will post it.
the full code:
import multiprocessing
import threading
import os
import time
file_lock = threading.Lock()
update_interval = 0.1
class FileMethods(object):
def a_import(self):
self.mod_check()
class FileObject(FileMethods):
def __init__(self, full_name, compiler):
self.full_name = os.path.abspath(full_name)
self.file_name = os.path.basename(self.full_name)
self.path_name = os.path.dirname(self.full_name)
name, exstention = os.path.splitext(full_name)
self.concat_name = name + '-concat' + exstention
self.compiler = compiler
self.compiler_methods = {'aImport': self.a_import}
self.last_updated = os.path.getatime(self.full_name)
self.subfiles = []
self.last_subfiles_mod = {}
def exists(self):
return os.path.isfile(self.full_name)
def mod_check(self):
if self.last_updated < os.path.getmtime(self.full_name):
self.last_updated = os.path.getmtime(self.full_name)
print '%s updated' % self.file_name
return True
else:
return False
def sub_mod_check(self):
for s in self.subfiles:
if self.last_subfiles_mod.get(s) < os.path.getmtime(s):
self.last_subfiles_mod[s] = os.path.getmtime(s)
return True
return False
files = []
def input_watcher():
while True:
input_file = os.path.abspath(raw_input('Input file name: '))
compiler = raw_input('Choose compiler: ')
if os.path.isfile(input_file):
obj = FileObject(input_file, compiler)
with file_lock:
files.append(obj)
print 'Adding %s with %s as compiler' % (obj.file_name, obj.compiler)
else:
print 'File does not exists'
def file_manipulation():
if __name__ == '__main__':
for f in files:
p = multiprocessing.Process(target=f.compiler_methods.get(f.compiler)())
p.start()
#f.compiler_methods.get(f.compiler)()
def file_watcher():
while True:
with file_lock:
file_manipulation()
time.sleep(update_interval)
iw = threading.Thread(target=input_watcher)
fw = threading.Thread(target=file_watcher)
iw.start()
fw.start()

This is happening because you're not using an if __name__ == "__main__": guard, while also using multiprocessing.Process on Windows. Windows needs to re-import your module in the child processes it spawns, which means it will keep creating new threads to handle inputs and watch files. This, of course, is a recipe for disaster. Do this to fix the issue:
if __name__ == "__main__":
iw = threading.Thread(target=input_watcher)
fw = threading.Thread(target=file_watcher)
iw.start()
fw.start()
See the "Safe importing of the main module" section in the multiprocessing docs for more info.
I also have a feeling file_watcher isn't really doing what you want it to (it will keep re-spawning processes for files you've already processed), but that's not really related to the original question.