I've been trying to write a python program that simulates a radio web stream, but I'm not quite sure how to do it properly. To do so, I would like to have the program continuously "playing" the musics even if there are no clients connected, so it would simulate a "live" radio where you connect and listen to whatever is playing.
What I have now is a server/client relation with TCP basic socket programming, the server side has a producer thread that was supposed to keep reading the music, and on-demand consumer threads that should send the audio frame to the client, that plays it with PyAudio. The problem is probably in the way the data is shared between threads.
First I've tried to do it with a single Queue, but as the client reads data from the queue, this data is removed and if I have multiple clients connected, that will make the music skip some frames.
Then I've tried to create a fixed number (10) of Queue objects that would be used for each client, with the producer thread feeding every queue, but each client would create a consumer thread of its own and read only from the queue "assigned" to it with a control variable. The problem here is: if there are any queues not being consumed (if I have only one client connected, for example), the Queue.put() method will block because these queues are full. How do I keep all queues "running" and synchronized even when they are not being used?
This is where I am now, and any advice is appreciated. I am not an experienced programmer yet, so please be patient with me. I believe Queue is not the recommended IPC method in this case, but if there is a way to use it, let me know.
Below is the code I have for now:
server.py
#TCP config omitted
#Producer Thread
def readTheMusics(queue):
#Control variable to keep looping through 2 music files
i=1
while i < 3:
fname = "music" + str(i) + ".wav"
wf = wave.open(fname, 'rb')
data = wf.readframes(CHUNK)
while data:
for k in range (10):
queue[k].put(data)
data = wf.readframes(CHUNK)
wf.close()
i += 1
if i==3:
i=1
#Consumer Thread
def connection(connectionSocket, addr, queue, index):
while True:
data = queue[index-1].get(True)
connectionSocket.send(data)
connectionSocket.close()
def main():
i = 1
#Queue(1) was used to prevent an infinite queue and therefore a memory leak
queueList = [Queue(1) for j in range(10)]
th2 = threading.Thread(target=musicReading, args=(queueList, ))
th2.start()
while True:
connectionSocket, addr = serverSocket.accept()
print("connected - id {}".format(i))
th = threading.Thread(target=connection, args=(connectionSocket, addr, queueList, i))
th.start()
i = i + 1
if __name__ == '__main__':
main()
Tim Roberts' comments were enough to make it work.
Related
I am developing a server (daemon).
The server has one "worker thread". The worker thread runs a queue of commands. When the queue is empty, the worker thread is paused (but does not exit, because it should preserve certain state in memory). To have exactly one copy of the state in memory, I need to run all time exactly one (not several and not zero) worker thread.
Requests are added to the end of this queue when a client connects to a Unix socket and sends a command.
After the command is issued, it is added to the queue of commands of the worker thread. After it is added to the queue, the server replies something like "OK". There should be not a long pause between server receiving a command and it "OK" reply. However, running commands in the queue may take some time.
The main "work" of the worker thread is split into small (taking relatively little time) chunks. Between chunks, the worker thread inspects ("eats" and empties) the queue and continues to work based on the data extracted from the queue.
How to implement this server/daemon in Python?
This is a sample code with internet sockets, easily replaced with unix domain sockets. It takes whatever you write to the socket, passes it as a "command" to worker, responds OK as soon as it has queued the command. The single worker simulates a lengthy task with sleep(30). You can queue as many tasks as you want, receive OK immediately and every 30 seconds, your worker prints a command from the queue.
import Queue, threading, socket
from time import sleep
class worker(threading.Thread):
def __init__(self,q):
super(worker,self).__init__()
self.qu = q
def run(self):
while True:
new_task=self.qu.get(True)
print new_task
i=0
while i < 10:
print "working ..."
sleep(1)
i += 1
try:
another_task=self.qu.get(False)
print another_task
except Queue.Empty:
pass
task_queue = Queue.Queue()
w = worker(task_queue)
w.daemon = True
w.start()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('localhost', 4200))
sock.listen(1)
try:
while True:
conn, addr = sock.accept()
data = conn.recv(32)
task_queue.put(data)
conn.sendall("OK")
conn.close()
except:
sock.close()
Now I have two threads, thread 1 is main thread and thread 2 is a task thread.I need thread 2 to do all the network issues, so I put all the sockets in thread 2 and set them to no-blocking. Thread 1 is used to push request to thread 2 to do the job.
At first i write something like this:
request_queue = Queue.Queue()
tasks = []
sockets = []
**thread 1:**
while True:
get_user_input()
#...
request_queue.put(request_task)
**thread 2:**
while True:
if have_requests(request_queue):
t = create_task()
tasks.append(t)
sockets.append(t.socket())
select(sockets,timeout=0) #no blocking select
update_tasks()
#...
Obviously,when there are no requests and tasks,thread 2 will waste cpu.I don't want use sleep(),because when thread 2 is sleeping,it can't handle requests in time.Then I think maybe I should change the request_queue to a local host socket,like this:
request_queue = sock.sock()
request_queue.bind(local_host,some_port)
request_queue.listen()
**thread 1**
while True:
get_user_input()
request_queue.send(new_request)
**thread 2**
while True:
select(sockets) # blocking select
if request_queue is active:
t = request_queue.recv()
t = create_task(t)
tasks.append(t)
sockets.append(t.socket())
#check other sockets
#update tasks...
But this looks like a little tricky,I don't know whether this is a good way or not.All I want is thread 2 can handle request in time, don not waste cpu time and process socket events in the same time. Anyone can help?
For async networking look to Tornado, Twisted or Gevent. Also this article may be usefull for you.
Example with Gevent:
def handle_socket(sock):
sock.sendall("payload")
sock.close()
server = socket.socket()
server.bind(('0.0.0.0', 9999))
server.listen(500) # max connections
while True:
try:
new_sock, address = server.accept()
except KeyboardInterrupt:
break
# handle every new connection with a new handler
gevent.spawn(handle_socket, new_sock)
And Celery is most appropriate for background job execution.
I'm developing a Flask/gevent WSGIserver webserver that needs to communicate (in the background) with a hardware device over two sockets using XML.
One socket is initiated by the client (my application) and I can send XML commands to the device. The device answers on a different port and sends back information that my application has to confirm. So my application has to listen to this second port.
Up until now I have issued a command, opened the second port as a server, waited for a response from the device and closed the second port.
The problem is that it's possible that the device sends multiple responses that I have to confirm. So my solution was to keep the port open and keep responding to incoming requests. However, in the end the device is done sending requests, and my application is still listening (I don't know when the device is done), thereby blocking everything else.
This seemed like a perfect use case for a thread, so that my application launches a listening server in a separate thread. Because I'm already using gevent as a WSGI server for Flask, I can use the greenlets.
The problem is, I have looked for a good example of such a thing, but all I can find is examples of multi-threading handlers for a single socket server. I don't need to handle a lot of connections on the socket server, but I need it launched in a separate thread so it can listen for and handle incoming messages while my main program can keep sending messages.
The second problem I'm running into is that in the server, I need to use some methods from my "main" class. Being relatively new to Python I'm unsure how to structure it in a way to make that possible.
class Device(object):
def __init__(self, ...):
self.clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def _connect_to_device(self):
print "OPEN CONNECTION TO DEVICE"
try:
self.clientsocket.connect((self.ip, 5100))
except socket.error as e:
pass
def _disconnect_from_device(self):
print "CLOSE CONNECTION TO DEVICE"
self.clientsocket.close()
def deviceaction1(self, ...):
# the data that is sent is an XML document that depends on the parameters of this method.
self._connect_to_device()
self._send_data(XMLdoc)
self._wait_for_response()
return True
def _send_data(self, data):
print "SEND:"
print(data)
self.clientsocket.send(data)
def _wait_for_response(self):
print "WAITING FOR REQUESTS FROM DEVICE (CHANNEL 1)"
self.serversocket.bind(('10.0.0.16', 5102))
self.serversocket.listen(5) # listen for answer, maximum 5 connections
connection, address = self.serversocket.accept()
# the data is of a specific length I can calculate
if len(data) > 0:
self._process_response(data)
self.serversocket.close()
def _process_response(self, data):
print "RECEIVED:"
print(data)
# here is some code that processes the incoming data and
# responds to the device
# this may or may not result in more incoming data
if __name__ == '__main__':
machine = Device(ip="10.0.0.240")
Device.deviceaction1(...)
This is (globally, I left out sensitive information) what I'm doing now. As you can see everything is sequential.
If anyone can provide an example of a listening server in a separate thread (preferably using greenlets) and a way to communicate from the listening server back to the spawning thread, it would be of great help.
Thanks.
EDIT:
After trying several methods, I decided to use Pythons default select() method to solve this problem. This worked, so my question regarding the use of threads is no longer relevant. Thanks for the people who provided input for your time and effort.
Hope it can provide some help, In example class if we will call tenMessageSender function then it will fire up an async thread without blocking main loop and then _zmqBasedListener will start listening on separate port untill that thread is alive. and whatever message our tenMessageSender function will send, those will be received by client and respond back to zmqBasedListener.
Server Side
import threading
import zmq
import sys
class Example:
def __init__(self):
self.context = zmq.Context()
self.publisher = self.context.socket(zmq.PUB)
self.publisher.bind('tcp://127.0.0.1:9997')
self.subscriber = self.context.socket(zmq.SUB)
self.thread = threading.Thread(target=self._zmqBasedListener)
def _zmqBasedListener(self):
self.subscriber.connect('tcp://127.0.0.1:9998')
self.subscriber.setsockopt(zmq.SUBSCRIBE, "some_key")
while True:
message = self.subscriber.recv()
print message
sys.exit()
def tenMessageSender(self):
self._decideListener()
for message in range(10):
self.publisher.send("testid : %d: I am a task" %message)
def _decideListener(self):
if not self.thread.is_alive():
print "STARTING THREAD"
self.thread.start()
Client
import zmq
context = zmq.Context()
subscriber = context.socket(zmq.SUB)
subscriber.connect('tcp://127.0.0.1:9997')
publisher = context.socket(zmq.PUB)
publisher.bind('tcp://127.0.0.1:9998')
subscriber.setsockopt(zmq.SUBSCRIBE, "testid")
count = 0
print "Listener"
while True:
message = subscriber.recv()
print message
publisher.send('some_key : Message received %d' %count)
count+=1
Instead of thread you can use greenlet etc.
I have Python 2.7.4 and pyserial-2.5 win32 installed on my PC. Here I am using a microcontroller device as a master (primary) and my pc as a slave (secondary). Here every time microcontroller will transmit data, and my PC has to receive the data through serial port. I want a code in Python to receive continuous data. Here the transmitted data size will vary all the time. Here I wrote a code to transmit data, and the code is
import serial
ser= serial.serial("COM10", 9600)
ser.write("Hello world\n")
x = ser.readline()
print(x)
With this code I can transmit data to the other PC, I crosschecked by opening HyperTerminal on the other PC and I can see the transmitted data (hello world).
I also wrote the code to receive data:
import serial
ser=serial.serial("COM10", 9600)
while 1:
if ser.inwaiting():
val = ser.readline(ser.inwaiting())
print(val)
if I send the data (how are you) from HyperTerminal, I can receive the data in my PC, with the above code.
Until this every thing is fine.
My question now is, when the microcontroller is transmitting variable data at variable time periods, I need to receive that data in my PC with Python. Do I need to use a buffer to store the received data? If yes, how will the code be? Why and how to use a buffer in Python? According to my search in internet, buffer is used to slice the string.
Typically what you do for communicating with a micro is to use single characters for something lightweight or create a communication protocol. Basically you have a start flag, end flag, and some sort of checksum to make sure the data gets across correctly. There are many ways to do this.
The below code is for Python 3. You may have to make changes for bytes data.
# On micro
data = b"[Hello,1234]"
serial.write(data)
On the computer you would run
def read_data(ser, buf=b'', callback=None):
if callback is None:
callback = print
# Read enough data for a message
buf += ser.read(ser.inwaiting()) # If you are using threading +10 or something so the thread has to wait for more data, this makes the thread sleep and allows the main thread to run.
while b"[" not in buf or b"]" not in buf:
buf += ser.read(ser.inwaiting())
# There may be multiple messages received
while b"[" in buf and b']' in buf:
# Find the message
start = buf.find(b'[')
buf = buf[start+1:]
end = buf.find(b']')
msg_parts = buf[:end].split(",") # buf now has b"Hello, 1234"
buf = buf[end+1:]
# Check the checksum to make sure the data is valid
if msg_parts[-1] == b"1234": # There are many different ways to make a good checksum
callback(msg_parts[:-1])
return buf
running = True
ser = serial.serial("COM10", 9600)
buf = b''
while running:
buf = read_data(ser, buf)
Threading is useful if you are using a GUI. Then you can have your thread read data in the background while your GUI displays the data.
import time
import threading
running = threading.Event()
running.set()
def thread_read(ser, callback=None):
buf = b''
while running.is_set():
buf = read_data(ser, buf, callback)
def msg_parsed(msg_parts):
# Do something with the parsed data
print(msg_parsed)
ser = serial.serial("COM10", 9600)
th = threading.Thread(target=thread_read, args=(ser, msg_parsed))
th.start()
# Do other stuff while the thread is running in the background
start = time.clock()
duration = 5 # Run for 5 seconds
while running.is_set():
time.sleep(1) # Do other processing instead of sleep
if time.clock() - start > duration
running.clear()
th.join() # Wait for the thread to finish up and exit
ser.close() # Close the serial port
Note that in the threading example I use a callback which is a function that gets passed as a variable and gets called later. The other way this is done is by putting the data in a Queue and then processing the data in the Queue in a different part of the code.
I try to write a daemon in python. But I have no idea how can I use a thread to start parallel tcp server in this daemon. And even what type of server I should use : asyncore?SocketServer?socket?
this is part of my code:
import os
def demonized():
child_pid = os.fork()
if child_pid == 0:
child_pid = os.fork()
if child_pid == 0: #fork twice for demonize
file = open('###', "r") # open file
event = file.read()
while event:
#TODO check for changes put changes in list variable
event = file.read()
file.close()
else:
sys.exit(0)
else:
sys.exit(0)
if __name__ == "__main__":
demonized()
So in a loop I have a list variable with some data appended every circle, and I want to start a thread with tcp server that wait for connection in the loop and if client connects send it this data(with zeroing variable). So I do not need to handle multiple clients, the client will be only one at time. What is the optimal way to implement this?
Thank you.
In case you want to avoid repeating boilerplate, Python will soon have a standard module that does the fork() pair and standard-I/O manipulations (which you have not added to your program yet?) that make it a daemon. You can download and use this module right now, from:
http://pypi.python.org/pypi/python-daemon
Running a TCP server in a separate thread is often as simple as:
import threading
def my_tcp_server():
sock = socket.socket(...)
sock.bind(...)
sock.listen()
while True:
conn, address = sock.accept()
...
... talk on the connection ...
...
conn.close()
def main():
...
threading.Thread(target=my_tcp_server).start()
...
I strongly recommend against trying to get your file-reader thread and your socket-answering thread talking with a list and lock of your own devising; such schemes are hard to get working and hard to keep working. Instead, use the standard library's Queue.Queue() class which does all of the locking and appending correctly for you.
Do you want to append items to the list in while event:... loop and serving this list simultaneously? If so then you have two writers and you must somehow protect your list.
In the sample SocketServer.TCPServer and threading.Lock was used:
import threading
import SocketServer
import time
class DataHandler(SocketServer.StreamRequestHandler):
def handle(self):
self.server.list_block.acquire()
self.wfile.write(', '.join(self.server.data))
self.wfile.flush()
self.server.data = []
self.server.list_block.release()
if __name__ == '__main__':
data = []
list_block = threading.Lock()
server = SocketServer.TCPServer(('localhost', 0), DataHandler)
server.list_block = list_block
server.data = data
t = threading.Thread(target=server.serve_forever)
t.start()
while True:
list_block.acquire()
data.append(1)
list_block.release()
time.sleep(1)