I am trying to create a webserver in python which can be started and stopped using a tkinter GUI. In tkinter I have a button which will call start() and a button that will call stop(). Initially everything works fine, the server starts when I click the button and it also stops when I click the stop button. When I try to restart the server again using the start button, I get a runtime error
RuntimeError: threads can only be started once
I believe it has something to do with the fact that I have already initialized threading in my init, and I can not figure out how to get this to work.
I have read through the threading docs multiple times, but I am struggling to understand it entirely. Any assistance would be greatly appreciated.
Thank you!
import threading
import socketserver
import http.server
import os
class WebServer(object):
def __init__(self, host, port):
self.host = host
self.port = port
self.handler = http.server.SimpleHTTPRequestHandler
self.server = socketserver.TCPServer((self.host, self.port), self.handler)
socketserver.TCPServer.allow_reuse_address = True
self.server_thread = threading.Thread(target=self.server.serve_forever, name="Server_Thread")
self.server_thread.setDaemon(True)
def start(self):
web_dir = os.path.join(os.path.dirname(__file__), 'www')
os.chdir(web_dir)
self.server_thread.start()
def stop(self):
os.chdir('..')
self.server.shutdown()
self.server.server_close()
As the python documentation states, the start method of the Thread object can only be called once.
In your case, you can create new instance of the Thread object in the start method:
def start(self):
web_dir = os.path.join(os.path.dirname(__file__), 'www')
os.chdir(web_dir)
self.server_thread = threading.Thread(target=self.server.serve_forever, name="Server_Thread")
self.server_thread.start()
In addition you also may clean the reference to the thread in stop method:
self.server_thread = None
I am using one XBee S2 as coordinator (API mode), 3 XBee S2 as routers (AT mode). The routers are connected to Naze32 board (using MSP).
On the computer side, I have a GUI using wxpython to send out command to request data.
The GUI will send out command to XBee (Coordinator) to request data from the routers every second.
I am using python-xbee library to do the send and receive frame job on computer side. Once new data received, it will notify the GUI to update some labels with the new data.
Currently I am able to send and receive frames outside a thread, but once I move the send and receive functions to a thread, it will never be able to read a frame any more. As I don't want to let the serial stop the GUI or make it not responding. Another thing is if I close the thread, then start new thread with xbee again, it will not work any more.
The communication is controlled by a button on the GUI; once "Turn on" clicked, the "self._serialOn" will set to True, then create new thread; once "Turn off" clicked, "self._serialOn" will set to False and thread is stopped.
How can I fix this problem?
Thanks in advance.
class DataExchange(object):
def __init__(self):
self._observers = []
self._addressList = [['\x00\x13\xA2\x00\x40\xC1\x43\x0F', '\xFF\xFE'],[],[]]
self._serialPort = ''
self._serialOn = False
self.workerSerial = None
# serial switch
def get_serialOn(self):
return self._serialOn
def set_serialOn(self, value):
self._serialOn = value
print(self._serialOn)
if self.serialOn == True:
EVT_ID_VALUE = wx.NewId()
self.workerSerial = WorkerSerialThread(self, EVT_ID_VALUE, self.serialPort, self.addressList)
self.workerSerial.daemon = True
self.workerSerial.start()
elif self.serialOn == False:
self.workerSerial.stop()
del(self.workerSerial)
self.workerSerial = None
serialOn = property(get_serialOn, set_serialOn)
class WorkerSerialThread(threading.Thread):
def __init__(self, notify_window, id, port, addresslist):
threading.Thread.__init__(self)
self.id = id
self.serialPort = port
self.addressList = addresslist
self.counter = 0
self._notify_window = notify_window
self.abort = False
self.sch = SerialCommunication(self.serialPort, self.addressList)
try:
self.sch.PreLoadInfo()
except:
print('failed')
def run(self):
while not self.abort:
self.counter += 1
print('Serial Working on '+str(self.id))
self.sch.RegularLoadInfo()
#wx.PostEvent(self._notify_window, DataEvent(self.counter, self.id))
time.sleep(1)
def stop(self):
self.sch.board.stop()
self.abort = True
This question was finally solved with multiprocessing rather than threading of python.
In the manual of python-xbee, it mentioned that "... Make sure that updates to external state are thread-safe...". Also in the source code, threading was used.
So I guess in this case threading will cause problem.
Anyway, using multiprocessing it finally works.
I have written a python script that idles waiting for gmail to push a notification indicating that an email has been received. Then the contents are parsed and a database is searched, then database data is emailed back to the original sender.
Currently once an email is received, the script cannot process another email until the script has emailed back. I am wondering if there is a way for the script to be continually listening for an email. As at this point in time if two emails are received at similar times the second one will not be processed.
I think multi-threading might be a solution but I am not sure. Possibly create a new thread for the processEmail.py section of code below?
Sorry if I have explained badly, I am struggling to explain this adequately, feel free to ask for more information.
EDIT: Instead of down voting me could you help me by commenting telling me what more information I should include?
EDIT 2: Here is a code example, I am trying to have the ability to still listen for an email while an email is being processed in processEmail.py
import imaplib2
import time
import subprocess
from threading import *
from subprocess import call
import processEmail
class Idler(object):
def __init__(self, conn):
self.thread = Thread(target=self.idle)
self.M = conn
self.event = Event()
def start(self):
self.thread.start()
def stop(self):
self.event.set()
def join(self):
self.thread.join()
def idle(self):
while True:
if self.event.isSet():
return
self.needsync = False
def callback(args):
if not self.event.isSet():
self.needsync = True
self.event.set()
self.M.idle(callback=callback)
self.event.wait()
if self.needsync:
self.event.clear()
self.dosync()
def dosync(self):
print "An email has been received, please wait...\n"
self.execute()
def execute(self):
processEmail.main()
M = imaplib2.IMAP4_SSL("imap.gmail.com")
M.login("email_address","email_pass")
M.select("Folder")
idler = Idler(M)
idler.start()
x = False
while not x: time.sleep(0.1)
You can directly inherit from threading.Thread and override its run method:
class SomeTask(threading.Thread):
def run(self):
# Will be executed in separate thread
Start the thread via its start method.
Communication between threads should be handled via queues.
I have a long running Python script which collect tweets from Twitter, and I would like to know how its doing every once in awhile.
Currently, I am using the signal library to catch interrupts, at which point I call my print function. Something like this:
import signal
def print_info(count):
print "#Tweets:", count
#Print out the process ID so I can interrupt it for info
print 'PID:', os.getpid()
#Start listening for interrupts
signal.signal(signal.SIGUSR1, functools.partial(print_info, tweet_count))
And whenever I want my info, I open up a new terminal and issue my interrupt:
$kill -USR1 <pid>
Is there a better way to do this? I am aware I could have my script something at scheduled intervals, but I am more interested in knowing on demand, and potentially issuing other commands as well.
Sending a signal to process would interrupt the process. Below you will find an approach that uses dedicated thread to emulate python console. The console is exposed as a unix socket.
import traceback
import importlib
from code import InteractiveConsole
import sys
import socket
import os
import threading
from logging import getLogger
# template used to generate file name
SOCK_FILE_TEMPLATE = '%(dir)s/%(prefix)s-%(pid)d.socket'
log = getLogger(__name__)
class SocketConsole(object):
'''
Ported form :eventlet.backdoor.SocketConsole:.
'''
def __init__(self, locals, conn, banner=None): # pylint: diable=W0622
self.locals = locals
self.desc = _fileobject(conn)
self.banner = banner
self.saved = None
def switch(self):
self.saved = sys.stdin, sys.stderr, sys.stdout
sys.stdin = sys.stdout = sys.stderr = self.desc
def switch_out(self):
sys.stdin, sys.stderr, sys.stdout = self.saved
def finalize(self):
self.desc = None
def _run(self):
try:
console = InteractiveConsole(self.locals)
# __builtins__ may either be the __builtin__ module or
# __builtin__.__dict__ in the latter case typing
# locals() at the backdoor prompt spews out lots of
# useless stuff
import __builtin__
console.locals["__builtins__"] = __builtin__
console.interact(banner=self.banner)
except SystemExit: # raised by quit()
sys.exc_clear()
finally:
self.switch_out()
self.finalize()
class _fileobject(socket._fileobject):
def write(self, data):
self._sock.sendall(data)
def isatty(self):
return True
def flush(self):
pass
def readline(self, *a):
return socket._fileobject.readline(self, *a).replace("\r\n", "\n")
def make_threaded_backdoor(prefix=None):
'''
:return: started daemon thread running :main_loop:
'''
socket_file_name = _get_filename(prefix)
db_thread = threading.Thread(target=main_loop, args=(socket_file_name,))
db_thread.setDaemon(True)
db_thread.start()
return db_thread
def _get_filename(prefix):
return SOCK_FILE_TEMPLATE % {
'dir': '/var/run',
'prefix': prefix,
'pid': os.getpid(),
}
def main_loop(socket_filename):
try:
log.debug('Binding backdoor socket to %s', socket_filename)
check_socket(socket_filename)
sockobj = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sockobj.bind(socket_filename)
sockobj.listen(5)
except Exception, e:
log.exception('Failed to init backdoor socket %s', e)
return
while True:
conn = None
try:
conn, _ = sockobj.accept()
console = SocketConsole(locals=None, conn=conn, banner=None)
console.switch()
console._run()
except IOError:
log.debug('IOError closing connection')
finally:
if conn:
conn.close()
def check_socket(socket_filename):
try:
os.unlink(socket_filename)
except OSError:
if os.path.exists(socket_filename):
raise
Example program:
make_threaded_backdoor(prefix='test')
while True:
pass
Example session:
mmatczuk#cactus:~$ rlwrap nc -U /var/run/test-3196.socket
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import os
>>> os.getpid()
3196
>>> quit()
mmatczuk#cactus:~$
This is a pretty robust tool that can be used to:
dump threads,
inspect process memory,
attach debugger on demand, pydev debugger (work for both eclipse and pycharm),
force GC,
monkeypatch function definition on the fly
and even more.
I personally write information to a file so that I have it afterwards, although this has the disadvantage of perhaps being slightly slower because it has to write to a file every time or every few times it retrieves a tweet.
Anyways, if you write it to a file "output.txt", you can open up bash and either type in tail output.txt for the latest 10 lines printed in the file, or you can type tail -f output.txt, which continuously updates the terminal prompt with the lines that you are writing to the file. If you wish to stop, just Ctrl-C.
Here's an example long-running program that also maintains a status socket. When a client connects to the socket, the script writes some status information to the socket.
#!/usr/bin/python
import os
import sys
import argparse
import random
import threading
import socket
import time
import select
val1 = 0
val2 = 0
lastupdate = 0
quit = False
# This function runs in a separate thread. When a client connects,
# we write out some basic status information, close the client socket,
# and wait for the next connection.
def connection_handler(sock):
global val1, val2, lastupdate, quit
while not quit:
# We use select() with a timeout here so that we are able to catch the
# quit flag in a timely manner.
rlist, wlist, xlist = select.select([sock],[],[], 0.5)
if not rlist:
continue
client, clientaddr = sock.accept()
client.send('%s %s %s\n' % (lastupdate, val1, val2))
client.close()
# This function starts the listener thread.
def start_listener():
sock = socket.socket(socket.AF_UNIX)
try:
os.unlink('/var/tmp/myprog.socket')
except OSError:
pass
sock.bind('/var/tmp/myprog.socket')
sock.listen(5)
t = threading.Thread(
target=connection_handler,
args=(sock,))
t.start()
def main():
global val1, val2, lastupdate
start_listener()
# Here is the part of our script that actually does "work".
while True:
print 'updating...'
lastupdate = time.time()
val1 = val1 + random.randint(1,10)
val2 = val2 + random.randint(100,200)
print 'sleeping...'
time.sleep(5)
if __name__ == '__main__':
try:
main()
except (Exception,KeyboardInterrupt,SystemExit):
quit=True
raise
You could write a simple Python client to connect to the socket, or you could use something like socat:
$ socat - unix:/var/tmp/myprog.sock
1403061693.06 6 152
I had write a similar application before.
Here is what I did:
When there are only a few commands needed, I just use signal as you did, just for not making it too complicated. By command, I mean something that you want you application to do, such as print_info in your post.
But when application updated, there are more different commands needed, I began to use a special thread listening on a socket port or reading a local file for accepting commands. Suppose the application need to support prinf_info1 print_info2 print_info3, so you can use a client connect to the target port and write print_info1 to make the application execute command print_info1 (Or just write print_info1 to a local file if you are using the reading local file mechanism).
When using the listening on a socket port mechanism, the disadvantage is it will take a bit more work to write a client to give commands, the advantage is you can give orders anywhere.
When using the reading a local file mechanism, the disadvantage is you have to make the thread check the file in a loop and it will use a bit resource, the advantage is giving orders is very simple (just write a string to a file) and you don't need to write a client and socket listen server.
rpyc is the perfect tool for this task.
In short, you define a rpyc.Service class which exposes the commands you want to expose, and start an rpyc.Server thread.
Your client then connects to your process, and calls the methods which are mapped to the commands your service exposes.
It's as simple and clean as that. No need to worry about sockets, signals, object serialization.
It has other cool features as well, for example the protocol being symmetric.
Your question relates to interprocess communication. You can achieve this by communicating over a unix socket or TCP port, by using a shared memory, or by using a message queue or cache system such as RabbitMQ and Redis.
This post talks about using mmap to achieve shared memory interprocess communication.
Here's how to get started with redis and RabbitMQ, both are rather simple to implement.
I try to write a daemon in python. But I have no idea how can I use a thread to start parallel tcp server in this daemon. And even what type of server I should use : asyncore?SocketServer?socket?
this is part of my code:
import os
def demonized():
child_pid = os.fork()
if child_pid == 0:
child_pid = os.fork()
if child_pid == 0: #fork twice for demonize
file = open('###', "r") # open file
event = file.read()
while event:
#TODO check for changes put changes in list variable
event = file.read()
file.close()
else:
sys.exit(0)
else:
sys.exit(0)
if __name__ == "__main__":
demonized()
So in a loop I have a list variable with some data appended every circle, and I want to start a thread with tcp server that wait for connection in the loop and if client connects send it this data(with zeroing variable). So I do not need to handle multiple clients, the client will be only one at time. What is the optimal way to implement this?
Thank you.
In case you want to avoid repeating boilerplate, Python will soon have a standard module that does the fork() pair and standard-I/O manipulations (which you have not added to your program yet?) that make it a daemon. You can download and use this module right now, from:
http://pypi.python.org/pypi/python-daemon
Running a TCP server in a separate thread is often as simple as:
import threading
def my_tcp_server():
sock = socket.socket(...)
sock.bind(...)
sock.listen()
while True:
conn, address = sock.accept()
...
... talk on the connection ...
...
conn.close()
def main():
...
threading.Thread(target=my_tcp_server).start()
...
I strongly recommend against trying to get your file-reader thread and your socket-answering thread talking with a list and lock of your own devising; such schemes are hard to get working and hard to keep working. Instead, use the standard library's Queue.Queue() class which does all of the locking and appending correctly for you.
Do you want to append items to the list in while event:... loop and serving this list simultaneously? If so then you have two writers and you must somehow protect your list.
In the sample SocketServer.TCPServer and threading.Lock was used:
import threading
import SocketServer
import time
class DataHandler(SocketServer.StreamRequestHandler):
def handle(self):
self.server.list_block.acquire()
self.wfile.write(', '.join(self.server.data))
self.wfile.flush()
self.server.data = []
self.server.list_block.release()
if __name__ == '__main__':
data = []
list_block = threading.Lock()
server = SocketServer.TCPServer(('localhost', 0), DataHandler)
server.list_block = list_block
server.data = data
t = threading.Thread(target=server.serve_forever)
t.start()
while True:
list_block.acquire()
data.append(1)
list_block.release()
time.sleep(1)