I'm trying to build a Python application using pyGTK, treads, and sockets. I'm having this weird error, but given all the modules involved, I'm not entirely sure where the error is. I did a little debugging with some print statements to narrow things down a bit and I think the error is somewhere in this snippet of code:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect(("localhost", 5005))
self.collectingThread = threading.Thread(target=self.callCollect)
self.collectingThread.daemon = True
self.collectingThread.start()
def callCollect(self):
gobject.timeout_add(500, self.collectData)
def collectData(self):
print "hello"
try:
print self.sock.recv(1024)
except:
print "except"
print "return"
return True
So basically what I'm trying to do is setup a socket, connect to a "server" script (which is really just another python script running locally), and create a separate thread to collect all incoming data from the server script. This thread is set to run the collectData method every 500 milliseconds.
After inserting the print statements into the collectData method here is what I notice when running the program:
-Initially the GUI is fully functional
-then the following is printed in the terminal:
hello
**all data received from server script and printed here**
return
hello
-after the text is printed in the terminal, the GUI becomes completely nonfunctional (buttons cant be pressed, etc.) and I have to force quit to close the application
What seems to be happening is that the thread prints "hello", prints the data from the server, and prints "return". 500 milliseconds later, it runs the collectData method again, prints "hello", then tries to print data from the server. However, because there is no data left it raises an exception, but for some unknown reason it doesn't execute the code in the exception block and everything just hangs from there.
Any idea on what is going wrong?
timeout_add is scheduling the action to happen on the main thread -- so the recv just blocks the main thread (when it's just waiting for data) and therefore the GUI, so, no exception unless you put a timeout or set the socket to non-blocking.
You need to delegate the receiving to the thread from the scheduled action rather than viceversa to get the effect you're after: have the thread e.g. wait on an event object, and the scheduled action signal that event every 500 milliseconds.
No, obviously the sock.recv call blocks because the socket wasn't closed yet and socket receives are blocking by default. Make sure you close the connection at some point.
It would make more sense to run the receive call in a new thread, or else it might block the GUI because your current implementation runs the recv call in the GUI thread (using timeout_add). The way you're currently doing it only makes sense if the answer is received very fast and/or you have to access widgets.
By the way, creating a new thread for calling gobject.timeout_add is totally unnecessary. timeout_add() and idle_add() register the specified callback function and return immediately. The GTK event loop then automatically executes the callback after the timeout (or on idle status for idle_add).
Related
I have a Python script for automating simple tasks. Its main loop looks like this:
while True:
input = download_task_input()
if input:
output = process_task(input)
upload_task_output(output)
sleep(60)
Some local files are altered during task processing. They are modified when the task is started, and restored back to proper state when the task is done, or if exception is caught. Restoring these files on program exit is very important to me: leaving them in altered state causes some trouble later that I'd like to avoid.
When I want to terminate the script, I hit Ctrl+C. It raises KeyboardInterrupt exception which both stops task processing and triggers files restoration. However, if I hit Ctrl+Break, the program is simply terminated: if a task is being processed at this moment, then local files are left in altered state (which is undesirable).
The question: I'm worried about the situation when Windows OS is shutdown by pressing the Power button. Is it possible to make Python handle it exactly like it handles Ctrl+C? I.e. I'd like to detect OS shutdown in Python script and raise Python exception on the main thread.
I know it is possible to call SetConsoleCtrlHandler function from WinAPI and install own handler for situations like Ctrl+C, Ctrl+Break, Shutdown, etc. However, this handler seems to be executed in additional thread, and raising exception in it does not achieve anything. On the other hand, Python itself supposedly uses the same WinAPI feature to raise KeyboardInterrupt on the main thread on Ctrl+C, so it should be doable.
This is not a serious automation script, so I don't mind if a solution is hacky or not 100% reliable.
Reading through posts of similar questions I strongly suspect there is no way to do what I'm trying to do but figured I'd ask. I have a program using python3 that is designed to run headless, receiving commands from remote users that have logged in. One of the commands of course is a shutdown so that the program can be ended cleanly. This section is working correctly.
However while working on this I realized an option to be able to enter commands directly, without a remote connection, would be useful in the event something unusual happened to prevent remote access. I added a local_control function that runs in it's own thread so that it doesn't interfere with the main loop. This works great for all commands except for the shutdown command.
I have a variable that both loops monitor so that they can end when the shutdown command is sent. Sending the shutdown command from within local_control works fine because the loop ends before getting back to input(). however when sending the shutdown command remotely the program doesn't end until someone presses the enter key locally because that loop remains stuck at input(). As soon as enter is pressed the program continues, successfully breaks the loop and continues with the shutdown as normal. Below is an example of my code.
import threading
self.runserver = True
def local_control(): #system to control server without remote access
while self.runserver:
raw_input = input()
if raw_input == "shutdown":
self.runserver = False
mythread = threading.Thread(target=local_control)
mythread.start()
while self.runserver:
some_input = get_remote_input() #getting command from remote user
if some_input == "shutdown":
self.runserver = False
sys.exit(0) #server is shutdown cleanly
Because the program runs primarily headless GUI options such as pygame aren't an option. Other solutions I've found online involve libraries that are not cross-platform such as msvcrt, termios, and curses. Although it's not as clean an option I'd settle for simply killing the thread to end it if I could however there is no way to do that as well. So is there a cross-platform, non-GUI option to have a non-blocking input? Or is there another way to break a blocked loop from another thread?
Your network-IO thread is blocking the processing of commands while waiting for remote commands, so it will only evaluate the state of runserver after get_remote_input() returns (and it's command is processed).
You will need three threads:
One which loops in local_control(), sending commands to the processing thread.
One which loops on get_remote_input(), also sending commands to the processing thread.
A processing thread (possibly the main thread).
A queue will probably be helpful here, since you need to avoid the race condition caused by unsynchronized access as currently present with regards to runserver.
Not a portable solution, but in *nix, you might be able send yourself an interrupt signal from the local_control function to break the blocking input(). You'll need the pthread ID (pthread_self and save it somewhere readable from local_control) for the network control thread so you can call pthread_kill.
A server who listens and handles connections is required for my program and, as ThreadingTcpServer does the whole job for me I decided to use it.
I noted that ThreadingTcpServer.serve_forever() is a blocking method, so I would make the thread server a thread itself.
My code is:
server = None #a variable that will contain the server
def createAndStartServer():
print "starting..."
server = ThreadingTcpServer((socket.gethostname(),1234),MyRequestHandler)
server.serve_forever() #blocking method
print "stopping..."
myThread = threading.Thread(target=createAndStartServer)
myThread.start()
time.sleep(3)
server.shutdown() #this one should stop the server thread from receiving further requests
The idea is to send a shutdown to the server, this way the thread will at most serve the requests it is already serving and, after that, exits from the serve_forever() loop.
This would cause the thread itself to stop as it exits from the createAndStartServer function. I don't know if it is the best method to do that but it sounds logic to me and in java I often do the same, by modifing the value of a boolean variable which handles the threading server loop...I think the shutdown method does something like that, right?
Anyway I got a:
AttributeError: 'NoneType' object has no attribute 'shutdown'
It seems that the server variable is not populated at all by the thread.
Why? And while we are at it tell me if my logic is correct or you have some better idea to handle my problem.
You are lacking of a "global server" inside your createAndStartServer().
By default, python will consider the server = Threading() as a new variable with a local scope. That explains why calling it from the main fails after with the None value.
I've written a GUI program with PyQt4 that has to send a message string over a serial data link.
I have implemented a GUI interface with two button widgets. I need to send the data over the serial link continuously, once per second when the first button is clicked, and then stop when the second button is clicked.
My current program is able to send data only at the instant a button is clicked. This is the method I wrote to handle the button click signal:
def sendMessage(self):
while 1:
print "Hello........"
if checke == False:
break
Do I need to use threads to solve this problem?
It depends... if the send operation is fast, you can use the QTimer class. It integrates with the Qt event loop so you don't have to worry about threading issues. Serial communications can be slow, depending on how much data you are sending, so I can't say for sure if this is the right solution for you.
Yes. The key to GUI programming is never do any long operation on the main thread, because it'll block the whole program until that operation is complete.
If you want to continuously send data over network, do it in a background thread.
Some example code for you.
class MessageWorker(QtCore.QThread):
def __init__(self):
super(ParserWorker, self).__init__()
self.ok_to_send = False
self.terminated = True
def run(self):
while not self.terminated:
if self.ok_to_send:
self.send_message()
time.sleep(1)
def start_send():
self.ok_to_send = True
def pause_send():
self.ok_to_send = False
def terminated():
self.terminated = False
Then in the main program just call
worker = MessageWorker()
worker.start() # Start the background thread
worker.start_send() # Start sending message
worker.pause_send() # Pause sending message
worker.terminated() # Stop sending message permanently
Yes, you need to use threads. In any GUI-based program, any work that's going to take a non-trivial amount of time should always happen on a separate thread to avoid blocking the UI—whenever you see an "unresponsive" program, that's almost always due to the program failing to process window messages because its UI thread is blocked inside some long operation.
One easy way to startup a background thread is to use the threading module. Here's how you might use it to write data to the serial port once per second:
class MyClass:
# This method will run on a separate thread
def _serial_port_worker(self):
while self._run_worker:
self.send_data_to_serial_port()
time.sleep(1)
# Call this to start the worker thread
def start_worker_thread(self):
self._run_worker = True
worker_thread = threading.Thread(target=self._serial_port_worker,
args=(self,))
worker_thread.start()
# Call this to tell the worker thread to stop
def stop_worker_thread(self):
self._run_worker = False
Basically you have three options:
Use a second thread to do the serial comms. GUI toolkits aren't always thread-safe, so you should only make calls to them from the main thread. Additionally, there is a limitation to threading in Python; only one thread at a time can be executing Python bytecode.
Use the GUI toolkit's timeout function (might be called differently) to create an event every now and then. In the event handler do the serial comms. Make sure that you use non-blocking reads and writes (in pyserial, configure a timeout in the Serial object), otherwise your app might become unresponsive.
Do the serial communications from the second program using the multiprocessing module. Even if the second process blocks, it won't affect the GUI. You can use multiprocessing.Queue to communicate between the QUI and the other process.
I have a Twisted/PyQt application that (among other things) connects to a bunch of remote resources. When the user closes the window I want to shut down all the connections, cleanly if possible, forcibly if not.
The problem is that by the time I go to close the connections, it appears that the reactor is no longer alive to let me do so.
Here's my app code:
# Create app and connect the Twisted/Qt reactors
app = QApplication(sys.argv)
qtreactor.qt4reactor.install()
# Shutdown Twisted when window is closed
#defer.inlineCallbacks
def stop():
print "="*40, "Closing connections..."
yield closeConnections()
print "="*40, "closed."
print "="*40, "Stopping reactor..."
reactor.stop()
print "="*40, "stopped."
app.connect(app, SIGNAL("lastWindowClosed()"), stop)
reactor.runReturn()
rc = app.exec_()
exit(rc)
And here's a stripped down version of my cleanup code:
#defer.inlineCallbacks
def closeConnections():
for connection in connections:
print "Closing connection #%s" % connection
yield threads.deferToThread(popen("/foo/bar/cleanup %s" % connection))
print "Connection closed."
The first print statement is reached, and the command is executed, but I never get the second one, nor do I go through the for loop multiple times.
Is my analysis correct? Is the problem that the reactor is already down, so I never hear back from threads.deferToThread? Or is there some other problem? Furthermore, how do I fix it?
Thanks,
Jonathan
I don't know exactly when that lastWindowClosed() signal fires. However, even if it fires early enough, before the reactor has shut down (preventing you from doing what you want to do), I'm sure that PyQt doesn't know what to do with the Deferred that is returned by your stop function. This means that the shutdown process will proceed merrily onward while your asynchronous cleanup code tries to run. Likely the GUI shutdown will finish before your network shutdown gets anywhere.
So, use reactor.addSystemEventTrigger('before', 'shutdown', stop) instead. I don't know if this will run slightly earlier or slightly later than lastWindowClosed(), but it will run early enough that the reactor will still be usable, and it will pay attention to the Deferred your function returns. Shutdown will be suspended, in fact, until that Deferred fires. This gives you all the time you need to do your cleanup.
Separately from all that, you shouldn't do threads.deferToThread(popen("/foo/bar/cleanup %s" % connection)):
You need to pass a callable to deferToThread, not the result of calling the callable. As written, your code runs popen in the reactor thread and passes a file object to the thread to be called (which makes no sense, of course)
Mixing threads and child processes is iffy. You might get away with it most of the time, I dunno.
reactor.spawnProcess will let you run a child process without blocking, without threads, and without worrying about mixing threads and processes. See also twisted.internet.utils.getProcessOutput if you don't need all the features of spawnProcess (which you appear not to).