I try to write a daemon in python. But I have no idea how can I use a thread to start parallel tcp server in this daemon. And even what type of server I should use : asyncore?SocketServer?socket?
this is part of my code:
import os
def demonized():
child_pid = os.fork()
if child_pid == 0:
child_pid = os.fork()
if child_pid == 0: #fork twice for demonize
file = open('###', "r") # open file
event = file.read()
while event:
#TODO check for changes put changes in list variable
event = file.read()
file.close()
else:
sys.exit(0)
else:
sys.exit(0)
if __name__ == "__main__":
demonized()
So in a loop I have a list variable with some data appended every circle, and I want to start a thread with tcp server that wait for connection in the loop and if client connects send it this data(with zeroing variable). So I do not need to handle multiple clients, the client will be only one at time. What is the optimal way to implement this?
Thank you.
In case you want to avoid repeating boilerplate, Python will soon have a standard module that does the fork() pair and standard-I/O manipulations (which you have not added to your program yet?) that make it a daemon. You can download and use this module right now, from:
http://pypi.python.org/pypi/python-daemon
Running a TCP server in a separate thread is often as simple as:
import threading
def my_tcp_server():
sock = socket.socket(...)
sock.bind(...)
sock.listen()
while True:
conn, address = sock.accept()
...
... talk on the connection ...
...
conn.close()
def main():
...
threading.Thread(target=my_tcp_server).start()
...
I strongly recommend against trying to get your file-reader thread and your socket-answering thread talking with a list and lock of your own devising; such schemes are hard to get working and hard to keep working. Instead, use the standard library's Queue.Queue() class which does all of the locking and appending correctly for you.
Do you want to append items to the list in while event:... loop and serving this list simultaneously? If so then you have two writers and you must somehow protect your list.
In the sample SocketServer.TCPServer and threading.Lock was used:
import threading
import SocketServer
import time
class DataHandler(SocketServer.StreamRequestHandler):
def handle(self):
self.server.list_block.acquire()
self.wfile.write(', '.join(self.server.data))
self.wfile.flush()
self.server.data = []
self.server.list_block.release()
if __name__ == '__main__':
data = []
list_block = threading.Lock()
server = SocketServer.TCPServer(('localhost', 0), DataHandler)
server.list_block = list_block
server.data = data
t = threading.Thread(target=server.serve_forever)
t.start()
while True:
list_block.acquire()
data.append(1)
list_block.release()
time.sleep(1)
Related
I've been trying to write a python program that simulates a radio web stream, but I'm not quite sure how to do it properly. To do so, I would like to have the program continuously "playing" the musics even if there are no clients connected, so it would simulate a "live" radio where you connect and listen to whatever is playing.
What I have now is a server/client relation with TCP basic socket programming, the server side has a producer thread that was supposed to keep reading the music, and on-demand consumer threads that should send the audio frame to the client, that plays it with PyAudio. The problem is probably in the way the data is shared between threads.
First I've tried to do it with a single Queue, but as the client reads data from the queue, this data is removed and if I have multiple clients connected, that will make the music skip some frames.
Then I've tried to create a fixed number (10) of Queue objects that would be used for each client, with the producer thread feeding every queue, but each client would create a consumer thread of its own and read only from the queue "assigned" to it with a control variable. The problem here is: if there are any queues not being consumed (if I have only one client connected, for example), the Queue.put() method will block because these queues are full. How do I keep all queues "running" and synchronized even when they are not being used?
This is where I am now, and any advice is appreciated. I am not an experienced programmer yet, so please be patient with me. I believe Queue is not the recommended IPC method in this case, but if there is a way to use it, let me know.
Below is the code I have for now:
server.py
#TCP config omitted
#Producer Thread
def readTheMusics(queue):
#Control variable to keep looping through 2 music files
i=1
while i < 3:
fname = "music" + str(i) + ".wav"
wf = wave.open(fname, 'rb')
data = wf.readframes(CHUNK)
while data:
for k in range (10):
queue[k].put(data)
data = wf.readframes(CHUNK)
wf.close()
i += 1
if i==3:
i=1
#Consumer Thread
def connection(connectionSocket, addr, queue, index):
while True:
data = queue[index-1].get(True)
connectionSocket.send(data)
connectionSocket.close()
def main():
i = 1
#Queue(1) was used to prevent an infinite queue and therefore a memory leak
queueList = [Queue(1) for j in range(10)]
th2 = threading.Thread(target=musicReading, args=(queueList, ))
th2.start()
while True:
connectionSocket, addr = serverSocket.accept()
print("connected - id {}".format(i))
th = threading.Thread(target=connection, args=(connectionSocket, addr, queueList, i))
th.start()
i = i + 1
if __name__ == '__main__':
main()
Tim Roberts' comments were enough to make it work.
I have a thread pool that handles some tasks concurrently. Now I'd like the tasks (multiply_by_2 here) to print something before exit.
Originally, I created a lock and passed the lock to each worker thread. If a thread wants to print something, it first acquires the lock, prints its message to stdout, then releases the lock.
Now, I want to have a dedicated event-driven server thread to handle the printing. If a thread wants to print something, it just send its message to that server, via a Unix domain socket (AF_UNIX). I hope in this way, each thread's blocking time can be reduced (no need to wait for the lock) and I don't need to share a lock among worker threads. The server thread just prints whatever messages it got from clients (i.e. the worker threads) in order.
I tried for some time with Python's asyncio module (requiring Python 3.7+) but couldn't figure it out. How should I do it?
This cleaned-up template is:
# Python 3.7+
import asyncio
import multiprocessing.dummy as mp # Threading wrapped using multiprocessing API.
import os
import socket
import sys
import threading
import time
server_address = './uds_socket' # UNIX domain socket
def run_multiple_clients_until_complete(input_list):
pool = mp.Pool(8)
result_list = pool.map(multiply_by_2, input_list)
return result_list
def multiply_by_2(n):
time.sleep(0.2) # Simulates some blocking call.
message_str = "client: n = %d" % n
# TODO send message_str.encode() to server
return n * 2
# Server's callback when it gets a client connection
# If you want to change it, please do..
def client_connected_cb(
stream_reader: asyncio.StreamReader,
stream_writer: asyncio.StreamWriter) -> None:
message_str = reader.read().decode()
print(message_str)
def create_server_thread():
pass # TODO
# Let the server finish handling all connections it got, then
# stop the server and join the thread
def stop_server_and_wait_thread(thread):
pass # TODO
def work(input_list):
thread = create_server_thread()
result_list = run_multiple_clients_until_complete(input_list)
stop_server_and_wait_thread(thread)
return result_list
def main():
input_list = list(range(20))
result_list = work(input_list)
print(result_list)
if __name__ == "__main__":
sys.exit(main())
Some extra requirements:
Don't make async: run_multiple_clients_until_complete(), multiply_by_2(), main().
It would be nicer to use the SOCK_DGRAM UDP protocol instead of SOCK_STREAM TCP, but it's unnecessary.
I am using Python3 modules:
requests for HTTP GET calls to a few Particle Photons which are set up as simple HTTP Servers
As a client I am using the Raspberry Pi (which is also an Access Point) as a HTTP Client which uses multiprocessing.dummy.Pool for making HTTP GET resquests to the above mentioned photons
The polling routine is as follows:
def pollURL(url_of_photon):
"""
pollURL: Obtain the IP Address and create a URL for HTTP GET Request
#param: url_of_photon: IP address of the Photon connected to A.P.
"""
create_request = 'http://' + url_of_photon + ':80'
while True:
try:
time.sleep(0.1) # poll every 100ms
response = requests.get(create_request)
if response.status_code == 200:
# if success then dump the data into a temp dump file
with open('temp_data_dump', 'a+') as jFile:
json.dump(response.json(), jFile)
else:
# Currently just break
break
except KeyboardInterrupt as e:
print('KeyboardInterrupt detected ', e)
break
The url_of_photon values are simple IPv4 Addresses obtained from the dnsmasq.leases file available on the Pi.
the main() function:
def main():
# obtain the IP and MAC addresses from the Lease file
IP_addresses = []
MAC_addresses = []
with open('/var/lib/misc/dnsmasq.leases', 'r') as leases_file:
# split lines and words to obtain the useful stuff.
for lines in leases_file:
fields = lines.strip().split()
# use logging in future
print('Photon with MAC: %s has IP address: %s' %(fields[1],fields[2]))
IP_addresses.append(fields[2])
MAC_addresses.append(fields[1])
# Create Thread Pool
pool = ThreadPool(len(IP_addresses))
results = pool.map(pollURL, IP_addresses)
pool.close()
pool.join()
if __name__ == '__main__':
main()
Problem
The program runs well however when I press CTRL + C the program does not terminate. Upon digging I found that the way to do so is using CTRL + \
How do I use this in my pollURL function for a safe way to exit the program, i.e. perform poll.join() so no leftover processes are left?
notes:
the KeyboardInterrupt is never recognized with the function. Hence I am facing trouble trying to detect CTRL + \.
The pollURL is executed in another thread. In Python, signals are handled only in the main thread. Therefore, SIGINT will raise the KeyboardInterrupt only in the main thread.
From the signal documentation:
Signals and threads
Python signal handlers are always executed in the main Python thread, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication. You can use the synchronization primitives from the threading module instead.
Besides, only the main thread is allowed to set a new signal handler.
You can implement your solution in the following way (pseudocode).
event = threading.Event()
def looping_function( ... ):
while event.is_set():
do_your_stuff()
def main():
try:
event.set()
pool = ThreadPool()
pool.map( ... )
except KeyboardInterrupt:
event.clear()
finally:
pool.close()
pool.join()
I need to check if the python script is already running then calling a method from the same running python script. But it must be on same process(pid), no new process. Is this possible?
I tried some codes but not worked.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import Tkinter as tk
from Tkinter import *
import socket
class Main():
def mainFunc(self):
self.root = tk.Tk()
self.root.title("Main Window")
self.lbl = Label(self.root, text = "First Text")
self.lbl.pack()
openStngs = Button(self.root, text = "Open Settings", command=self.settingsFunc)
openStngs.pack()
def settingsFunc(self):
stngsRoot = Toplevel()
stngsRoot.title("Settings Window")
changeTextOfLabel = Button(stngsRoot, text = "Change Main Window Text", command=self.change_text)
changeTextOfLabel.pack()
def change_text(self):
self.lbl.config(text="Text changed")
# the get_lock from http://stackoverflow.com/a/7758075/3254912
def get_lock(process_name):
lock_socket = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
try:
print lock_socket
lock_socket.bind('\0' + process_name)
print 'I got the lock'
m.mainFunc()
mainloop()
except socket.error:
print 'lock exists'
m.settingsFunc()
mainloop()
# sys.exit()
if __name__ == '__main__':
m=Main()
get_lock('myPython.py')
You either need:
A proactive check in your running process to look at the environment (for instance, the contents of a file or data coming through a socket) to know when to fire the function,
or for your running process to receive unix signals or some other IPC (possibly one of the user-defined signals) and perform a function when one is received.
Either way you can't just reach into a running process and fire a function inside that process (it MIGHT not be literally impossible if you hook the running process up to a debugger, but I wouldn't recommend it).
Tkinter necessarily has its own event loop system, so I recommend reading up on how that works and how to either run something on a timer in that event loop system, or set up a callback that responds to a signal. You could also wrap a non-event loop based system in a try/except block that will catch an exception generated by a UNIX signal, but it may not be straightforward to resume the operation of the rest of the program after that signal is caught, in that case.
Sockets are a good solution to this kind of interprocess communication problem.
One possible approach would be to set up a socket server in a thread in your original process, this can be used as an entry point for external input. A (rather stupid) example might be:
# main.py
import socket
import SocketServer # socketserver in Python 3+
import time
from Queue import Queue
from threading import Thread
# class for handling requests
class QueueHandler(SocketServer.BaseRequestHandler):
def __init__(self, request, client_address, server):
self.server = server
server.client_address = client_address
SocketServer.BaseRequestHandler.__init__(self,request, client_address, server)
# receive a block of data
# put it in a Queue instance
# send back the block of data (redundant)
def handle(self):
data = self.request.recv(4096)
self.server.recv_q.put(data)
self.request.send(data)
class TCPServer(SocketServer.TCPServer):
def __init__(self, ip, port, handler_class=QueueHandler):
SocketServer.TCPServer.__init__(self, (ip, port), handler_class, bind_and_activate=False)
self.recv_q = Queue() # a Queue for data received over the socket
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.server_bind()
self.server_activate()
def shutdown(self):
SocketServer.TCPServer.shutdown(self)
def __del__(self):
self.server_close()
# This is the equivalent of the main body of your original code
class TheClassThatLovesToAdd(object):
def __init__(self):
self.value = 1
# create and instance of the server attached to some port
self.server = TCPServer("localhost",9999)
# start it listening in a separate control thread
self.server_thread = Thread(target=self.server.serve_forever)
self.server_thread.start()
self.stop = False
def add_one_to_value(self):
self.value += 1
def run(self):
while not self.stop:
print "Value =",self.value
# if there is stuff in the queue...
while not self.server.recv_q.empty():
# read and parse the message from the queue
msg = self.server.recv_q.get()
# perform some action based on the message
if msg == "add":
self.add_one_to_value()
elif msg == "shutdown":
self.server.shutdown()
self.stop = True
time.sleep(1)
if __name__ == "__main__":
x = TheClassThatLovesToAdd()
x.run()
When you start this running, it should just loop over and over printing to the screen. Output:
Value = 1
Value = 1
Value = 1
...
However the TCPServer instance attached to the TheClassThatLovesToAdd instance now gives us a control path. The simplest looking snippet of control code would be:
# control.py
import socket
import sys
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.settimeout(2)
sock.connect(('localhost',9999))
# send some command line argument through the socket
sock.send(sys.argv[1])
sock.close()
So if I run main.py in one terminal window and call python control.py add from another, the output of main.py will change:
Value = 1
Value = 1
Value = 1
Value = 2
Value = 2
...
Finally to kill it all we can run python control.py shutdown, which will gently bring main.py to a halt.
This is by no means the only solution to your problem, but it is likely to be one of the simplest.
One can try GDB, but not sure how to call a function from within [an idle thread].
Perhaps someone very versed with gdb and debugging/calling Python functions from within GDB can improve this answer.
One solution would be to use a messaging service (such as ActiveMQ or RabbitMQ). Your application subscribes to a queue/topic and whenever you want to send it a command, you write a message to it's queue. I'm not going to go into details because there are thousands of examples on-line. Queues/messaging/MQTT etc. are very simple to implement and are how most business systems (and modern control systems) communicate. Do a search for paho-mqtt.
I am using glade as my gui and creating a process to run my gui in. This app will open a socket when 'on' is clicked. When i press 'send', it will send whatever is in an textfield to the socket. The socket receives this data and sends it back. The problem is after i send data to the socket the thread doesn't terminate. Also after i close my gui it calls a sys.exit() but also leaves a process and doesn't terminate. I believe the error is in how i am implementing my processes or all my processing in general. Can anyone shine some light on this? It also relates to my last post as well. Thanks
main.py
// Main thread that create a new process for my gui and displays it
import socket, thread, gtk, Handler, sys, os, multiprocessing
sys.setrecursionlimit(10000)
if __name__ == '__main__':
builder = gtk.Builder()
#32bit template.glade 64bit template-2.22
# #todo add switching between architectures
#
builder.add_from_file("template/template-2.22.glade")
builder.connect_signals(Handler.Handler(builder))
window = builder.get_object("window1")
window.show_all()
try:
p = multiprocessing.Process(target=gtk.main())
p.start()
except:
print "Error Starting new Thread"
handler.py
// Handler for gtk glade signals, creates new threads and handles button and stuff
import thread, threading, os, server, client,multiprocessing, time
import sys, gtk
class Handler(object):
'''
classdocs
'''
myobject = ''
def __init__(self,object1):
#Getting glade builder
self.myobject = object1
'''
Constructor
'''
def clickme(self,value):
myserver = server.Server()
try:
p = multiprocessing.Process(target=myserver.run)
p.start()
except:
pass
def sendmessage(self,value):
text = self.myobject.get_object('entry1').get_text()
print text
msg = client.MyClass()
p = multiprocessing.Process(target=msg.run,args=([text]))
p.start()
server.py
// Opens a socket and listens for incoming data and sends it back
import socket,multiprocessing, gtk, sys
class Server:
'''
classdocs
'''
def __init__(self):
'''
Constructor
'''
def run(self):
try:
while 1:
HOST = 'localhost' # Symbolic name meaning the local host
PORT = 50006 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(5)
conn, addr = s.accept()
print 'Connected by', addr
while True:
data = conn.recv(1024)
if not data:
conn.close()
sys.exit()
break
elif data != '':
conn.sendall(data)
break
print "Closing"
#conn.close()
finally:
print "End"
pass
client.py
// Sends whatever is inside text area to socket
import time
class MyClass:
'''
classdocs
'''
def __init__(self):
'''
Constructor
'''
def run(self,text):
try:
import socket
HOST = 'localhost' # The localhost
PORT = 50006 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.send(text)
data = s.recv(1024)
while 1:
if data != '':
print 'Received', repr(data)
break
finally:
pass
This is just wrong:
p = multiprocessing.Process(target=gtk.main())
p.start()
First, you can't start the gtk main loop in a subprocess, even if you did it rigth. Fortunately the process never really tries to start main as you call gtk.main(), which will block until the main loop exits and then return None. So what you're actually doing is:
gtk.main()
p = multiprocessing.Process(target=None)
p.start()
Througout the rest of your code you keep creating new processes and then forgetting about them. If you would keep a reference to them, you could at least try to send the TERM signal to them to shut them down (using Process.terminate, or set the daemon flag). If you want to shut down the subprocess cleanly, you either need to handle that signal in the subprocess, or use other IPC mechanisms to get it to shut down cleanly (like mutliprocessing.Event, ...).
Then there is this:
while True:
data = conn.recv(1024)
if not data:
conn.close()
sys.exit()
break
elif data != '':
conn.sendall(data)
break
This while loop will never loop (unless recv magically returns something else then a string). The first execution path ends with sys.exit() (taking the whole server down - the break is unreachable), the second ends with break, so the loop is useless.
A few lines below you have the exact opposite:
data = s.recv(1024)
while 1:
if data != '':
print 'Received', repr(data)
break
Unless data was '' in the first line, this will be an endless loop, as data's value won't change anymore.
Generally you don't really need multiprocessing for most of this. Starting a server in a different process may be ok if if has to do a lot of work, but spawing a subprocess just to send some data is overkill. Sending and receiving using sockets are IO bound, using threading here would be more reasonable.
You have two classes (Server and Handler) which have only two methods, one of which is __init__, and the other one is only used as target for a subprocess:
myserver = server.Server()
try:
p = multiprocessing.Process(target=myserver.run)
and:
msg = client.MyClass()
p = multiprocessing.Process(target=msg.run,args=([text]))
That's a sign that these shouldn't be classes but functions.