Develop Reference

Python: Continuously running a section of a script

I have written a python script that idles waiting for gmail to push a notification indicating that an email has been received. Then the contents are parsed and a database is searched, then database data is emailed back to the original sender.
Currently once an email is received, the script cannot process another email until the script has emailed back. I am wondering if there is a way for the script to be continually listening for an email. As at this point in time if two emails are received at similar times the second one will not be processed.
I think multi-threading might be a solution but I am not sure. Possibly create a new thread for the processEmail.py section of code below?
Sorry if I have explained badly, I am struggling to explain this adequately, feel free to ask for more information.
EDIT: Instead of down voting me could you help me by commenting telling me what more information I should include?
EDIT 2: Here is a code example, I am trying to have the ability to still listen for an email while an email is being processed in processEmail.py
import imaplib2
import time
import subprocess
from threading import *
from subprocess import call
import processEmail
class Idler(object):
def __init__(self, conn):
self.thread = Thread(target=self.idle)
self.M = conn
self.event = Event()
def start(self):
self.thread.start()
def stop(self):
self.event.set()
def join(self):
self.thread.join()
def idle(self):
while True:
if self.event.isSet():
return
self.needsync = False
def callback(args):
if not self.event.isSet():
self.needsync = True
self.event.set()
self.M.idle(callback=callback)
self.event.wait()
if self.needsync:
self.event.clear()
self.dosync()
def dosync(self):
print "An email has been received, please wait...\n"
self.execute()
def execute(self):
processEmail.main()
M = imaplib2.IMAP4_SSL("imap.gmail.com")
M.login("email_address","email_pass")
M.select("Folder")
idler = Idler(M)
idler.start()
x = False
while not x: time.sleep(0.1)

You can directly inherit from threading.Thread and override its run method:
class SomeTask(threading.Thread):
def run(self):
# Will be executed in separate thread
Start the thread via its start method.
Communication between threads should be handled via queues.

More elegant way to check for an event/trigger periodically?

I have a passive infrared sensor and I wanted to turn off and on my display based on motion. E.g. if there is no motion for 5 minutes, then the display should turn off to save power. However if there is motion don't turn off the display, or turn it back on. (Don't ask why isn't a screensaver good for this. The device I'm making won't have any keyboard or mouse. It only will be a standalone display.)
My idea was to create two threads, a producer, and a consumer. The producer thread (the PIR sensor) puts a message into a queue, which the consumer (which controls the display) reads. This way I can send signals from one to another.
I have a fully functional code below (with some explanation), which completes the previously described. My question is that is there some way to achieve this in a more elegant way? What do you think of my approach, is it okay, is it hackish?
#!/usr/bin/env python
import Queue
from threading import Thread
import RPi.GPIO as gpio
import time
import os
import sys
class PIRSensor:
# PIR sensor's states
current_state = 0
previous_state = 0
def __init__(self, pir_pin, timeout):
# PIR GPIO pin
self.pir_pin = pir_pin
# Timeout between motion detections
self.timeout = timeout
def setup(self):
gpio.setmode(gpio.BCM)
gpio.setup(self.pir_pin, gpio.IN)
# Wait for the PIR sensor to settle
# (loop until PIR output is 0)
while gpio.input(self.pir_pin) == 1:
self.current_state = 0
def report_motion(self, queue):
try:
self.setup()
while True:
self.current_state = gpio.input(self.pir_pin)
if self.current_state == 1 and self.previous_state == 0:
# PIR sensor is triggered
queue.put(True)
# Record previous state
self.previous_state = 1
elif self.current_state == 1 and self.previous_state == 1:
# Feed the queue since there is still motion
queue.put(True)
elif self.current_state == 0 and self.previous_state == 1:
# PIR sensor has returned to ready state
self.previous_state = 0
time.sleep(self.timeout)
except KeyboardInterrupt:
raise
class DisplayControl:
# Display's status
display_on = True
def __init__(self, timeout):
self.timeout = timeout
def turn_off(self):
# Turn off the display
if self.display_on:
os.system("/opt/vc/bin/tvservice -o > /dev/null 2>&1")
self.display_on = False
def turn_on(self):
# Turn on the display
if not self.display_on:
os.system("{ /opt/vc/bin/tvservice -p && chvt 9 && chvt 7 ; } > /dev/null 2>&1")
self.display_on = True
def check_motion(self, queue):
try:
while True:
try:
motion = queue.get(True, self.timeout)
if motion:
self.turn_on()
except Queue.Empty:
self.turn_off()
except KeyboardInterrupt:
raise
if __name__ == "__main__":
try:
pir_sensor = PIRSensor(7, 0.25)
display_control = DisplayControl(300)
queue = Queue.Queue()
producer = Thread(target=pir_sensor.report_motion, args=(queue,))
consumer = Thread(target=display_control.check_motion, args=(queue,))
producer.daemon = True
consumer.daemon = True
producer.start()
consumer.start()
while True:
time.sleep(0.1)
except KeyboardInterrupt:
display_control.turn_on()
# Reset GPIO settings
gpio.cleanup()
sys.exit(0)
The producer thread runs a function (report_motion) of a PIRSensor class instance. The PIRSensor class reads the state of a passive infrared sensor four times per second, and whenever it senses motion puts a message into a queue.
The consumer thread runs a function of (check_motion) of a DisplayControl class instance. It reads the previously mentioned queue in blocking mode with a given timeout. The following can happen:
If the display is on and there is no message in the queue for a given
time, aka the timeout expires, the consumer thread will power off the
display.
If the display is off and a message comes, the thread will
power on the display.

I think the idea is good. The only question I have about your implementation is why have both the consumer and producer in child threads? You could just keep the consumer in the main thread, and then there'd be no need to have this meaningless loop in your main thread.
while True:
time.sleep(0.1)
which is just wasting CPU cycles. Instead you could just call display_motion.check_motion(queue) directly.

I think it is a good solution. The reason being that you have separated the concerns for the different classes. One class handles the PIR sensor. One handles the display. You glue them together by a queue today, that's one approach.
By doing this you can easily test the different classes.
To extend this (read make it extendable) you might introduce a controller. The controller gets events (e.g. from the queue) and acts on the events (e.g. tell the Display Controller to turn off the display). The controller knows about the sensor, and knows about the display. But the sensor should not know about the display or vice versa. (this is very similar to MVC where in this case the data is the model (sensor), the display is the view and the controller sits in between.
This approach makes the design testable, extendable, maintainable. And by that you are not hackish, you are writing real code.

How to call a method from an already running python script

I need to check if the python script is already running then calling a method from the same running python script. But it must be on same process(pid), no new process. Is this possible?
I tried some codes but not worked.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import Tkinter as tk
from Tkinter import *
import socket
class Main():
def mainFunc(self):
self.root = tk.Tk()
self.root.title("Main Window")
self.lbl = Label(self.root, text = "First Text")
self.lbl.pack()
openStngs = Button(self.root, text = "Open Settings", command=self.settingsFunc)
openStngs.pack()
def settingsFunc(self):
stngsRoot = Toplevel()
stngsRoot.title("Settings Window")
changeTextOfLabel = Button(stngsRoot, text = "Change Main Window Text", command=self.change_text)
changeTextOfLabel.pack()
def change_text(self):
self.lbl.config(text="Text changed")
# the get_lock from http://stackoverflow.com/a/7758075/3254912
def get_lock(process_name):
lock_socket = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
try:
print lock_socket
lock_socket.bind('\0' + process_name)
print 'I got the lock'
m.mainFunc()
mainloop()
except socket.error:
print 'lock exists'
m.settingsFunc()
mainloop()
# sys.exit()
if __name__ == '__main__':
m=Main()
get_lock('myPython.py')

You either need:
A proactive check in your running process to look at the environment (for instance, the contents of a file or data coming through a socket) to know when to fire the function,
or for your running process to receive unix signals or some other IPC (possibly one of the user-defined signals) and perform a function when one is received.
Either way you can't just reach into a running process and fire a function inside that process (it MIGHT not be literally impossible if you hook the running process up to a debugger, but I wouldn't recommend it).
Tkinter necessarily has its own event loop system, so I recommend reading up on how that works and how to either run something on a timer in that event loop system, or set up a callback that responds to a signal. You could also wrap a non-event loop based system in a try/except block that will catch an exception generated by a UNIX signal, but it may not be straightforward to resume the operation of the rest of the program after that signal is caught, in that case.

Sockets are a good solution to this kind of interprocess communication problem.
One possible approach would be to set up a socket server in a thread in your original process, this can be used as an entry point for external input. A (rather stupid) example might be:
# main.py
import socket
import SocketServer # socketserver in Python 3+
import time
from Queue import Queue
from threading import Thread
# class for handling requests
class QueueHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server):
self.server = server
server.client_address = client_address
SocketServer.BaseRequestHandler.__init__(self,request, client_address, server)
# receive a block of data
# put it in a Queue instance
# send back the block of data (redundant)
def handle(self):
data = self.request.recv(4096)
self.server.recv_q.put(data)
self.request.send(data)
class TCPServer(SocketServer.TCPServer):
def __init__(self, ip, port, handler_class=QueueHandler):
SocketServer.TCPServer.__init__(self, (ip, port), handler_class, bind_and_activate=False)
self.recv_q = Queue() # a Queue for data received over the socket
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.server_bind()
self.server_activate()
def shutdown(self):
SocketServer.TCPServer.shutdown(self)
def __del__(self):
self.server_close()
# This is the equivalent of the main body of your original code
class TheClassThatLovesToAdd(object):
def __init__(self):
self.value = 1
# create and instance of the server attached to some port
self.server = TCPServer("localhost",9999)
# start it listening in a separate control thread
self.server_thread = Thread(target=self.server.serve_forever)
self.server_thread.start()
self.stop = False
def add_one_to_value(self):
self.value += 1
def run(self):
while not self.stop:
print "Value =",self.value
# if there is stuff in the queue...
while not self.server.recv_q.empty():
# read and parse the message from the queue
msg = self.server.recv_q.get()
# perform some action based on the message
if msg == "add":
self.add_one_to_value()
elif msg == "shutdown":
self.server.shutdown()
self.stop = True
time.sleep(1)
if __name__ == "__main__":
x = TheClassThatLovesToAdd()
x.run()
When you start this running, it should just loop over and over printing to the screen. Output:
Value = 1
Value = 1
Value = 1
...
However the TCPServer instance attached to the TheClassThatLovesToAdd instance now gives us a control path. The simplest looking snippet of control code would be:
# control.py
import socket
import sys
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.settimeout(2)
sock.connect(('localhost',9999))
# send some command line argument through the socket
sock.send(sys.argv[1])
sock.close()
So if I run main.py in one terminal window and call python control.py add from another, the output of main.py will change:
Value = 1
Value = 1
Value = 1
Value = 2
Value = 2
...
Finally to kill it all we can run python control.py shutdown, which will gently bring main.py to a halt.
This is by no means the only solution to your problem, but it is likely to be one of the simplest.

One can try GDB, but not sure how to call a function from within [an idle thread].
Perhaps someone very versed with gdb and debugging/calling Python functions from within GDB can improve this answer.

One solution would be to use a messaging service (such as ActiveMQ or RabbitMQ). Your application subscribes to a queue/topic and whenever you want to send it a command, you write a message to it's queue. I'm not going to go into details because there are thousands of examples on-line. Queues/messaging/MQTT etc. are very simple to implement and are how most business systems (and modern control systems) communicate. Do a search for paho-mqtt.

Can pygame events be handled in select.select input list?

The documentation for python's select.select says:
Note that on Windows, it only works for sockets; on other operating
systems, it also works for other file types (in particular, on Unix,
it works on pipes).
My group is developing a simplistic multiplayer game using pygame and sockets. (We are not using Twisted or zeromq or any similar libraries; this being the only constraint).
Now, for the game design; we want the player to send data to the server when a key event occurs in the pygame screen. The client/player side's socket will otherwise be hooked to server and listen for changes occurring on other players' side. For this task, I'd need the pygame and socket to work parallely. I was recommended to use select module from several users on #python.
Can I do something like:
inp = [self.sock, pygame.event.get]
out = [self.server]
i, o, x = select.select( inp, out, [] )
If not, what should be the way to go?

You could use threads for this task. Is it necessary to process server messages and pygame events in series (not concurrently)? If so, you could do this:
class SocketListener(threading.Thread):
def __init__(self, sock, queue):
threading.Thread.__init__(self)
self.daemon = True
self.socket = sock
self.queue = queue
def run(self):
while True:
msg = self.socket.recv()
self.queue.put(msg)
class PygameHandler(threading.Thread):
def __init__(self, queue):
threading.Thread.__init__(self)
self.queue = queue
self.daemon = True
def run(self):
while True:
self.queue.put(pygame.event.wait())
queue = Queue.Queue()
PygameHandler(queue).start()
SocketListener(queue).start()
while True:
event = queue.get()
"""Process the event()"""
If not, you could process the events inside the run methods of the PygameHandler and SocketListener classes.

Thread doesn't close on exiting tk window

I have an application written in Python, using Tkinter. One of the features allows a serial port to be opened, after which any messages received over the serial port are displayed in a text window. This works fine. The problem comes when I close the window, which doesn't kill the thread monitoring the serial port. It then has to be killed manually (alternatively, unplugging the USB-serial cable causes an exception which kills the process).
I assume I'm missing something simple here, but I would have thought closing the application would close all associated threads. I can't seem to find anything about this in the documentation, but I'm probably looking in the wrong place?
Code for the serial thread in case it's relevant:
class SerialThread(threading.Thread):
def __init__(self, queue, sp):
threading.Thread.__init__(self)
self.queue = queue
self.ser_handle = sp;
def run(self):
while True:
if self.ser_handle.inWaiting():
text = self.ser_handle.readline(self.ser_handle.inWaiting())
self.queue.put(text)
time.sleep(0.2)

You must have a way to ask the thread for stop, you can accomplish that using: threading.Event.
class SerialThread(threading.Thread):
def __init__(self, queue, sp):
threading.Thread.__init__(self)
self.queue = queue
self.event = threading.Event() # An event object.
self.ser_handle = sp;
def stop(self):
self.event.set()
def run(self):
while not self.event.isSet():
if self.ser_handle.inWaiting():
text = self.ser_handle.readline(self.ser_handle.inWaiting())
self.queue.put(text)
time.sleep(0.2)
Then in the on close event of your window, call your_thread.stop()