In Python how do I fill a buffer with lines of data (strings) and consume it with a second process? There are ample of examples here adding and reading lines from a string, but I need to remove the consumed line from the string for the string to work as a buffer.
Example: read sporadic data from a serial port and send it via TCP/IP to a server. Line-by-line within one loop and no buffering = no problem, but in case the destination is unreachable the data should be stored in the buffer and then sent once connection is available.
#!/usr/bin/python
import serial
import socket
from multiprocessing import Process
ip = "someURL"
port = 12345
ser = serial.Serial("/dev/ttyUSB0", 57600, timeout=0)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def serial_reader():
while True:
for line in ser.read():
try:
response = ser.readlines(None)
response = str(response)
message = response[7:]
except:
print datetime.datetime.now(), " No data from serial connection."
##
def data_sender():
s.connect((ip, port))
while True:
for line in queue():
try:
s.send(message)
except:
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ip, port))
continue
except:
s.close()
##
if __name__ == '__main__':
Process(target=serial_reader).start()
Process(target=data_sender).start()
I think the best way to achieve what you want is to use a queue:
from multiprocessing import Queue
specifically use queue.put() to put a string on the queue, queue.get() to retrieve it, and queue.task_done() to indicate that the task is complete.
https://docs.python.org/2/library/queue.html#Queue.Queue
if you need a bigger gun take a look at RabbitMQ and python libraries that implement the AMPQ protocol such as rabbitpy. This is the defacto standard for inter process/inter service communication and has a lot of usefyl stuff already baked in, such as persisting messages in case the processes shut down, load balancing tasks across multiple processes, etc.
Related
I have a code which works perfectly for one connection. I have seen two options for multi-client handling but I don't really understand it.
Here is the server socket code:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listening_sock:
listening_sock.bind(('', port))
listening_sock.listen()
client_soc, client_address = listening_sock.accept()
client_soc.sendall('200#Welcome to my server!'.encode())
print(f'Address {client_soc.getsockname()[0]} connected with port {client_soc.getsockname()[1]}')
while True:
# get message
msg = client_soc.recv(1024).decode()
# receive log print:
print(f'"{msg}" sent from {client_soc.getsockname()[0]}')
if 'Quit' in msg:
client_soc.sendall('200#Thanks for using my server!'.encode())
client_soc.close()
elif '0' < msg.split('#')[0] <= '9': # one of the valid actions
answer = call_action(msg.split('#')[0], db, msg.split('#')[1]) # the answer for given parameter
client_soc.sendall("200#".encode() + answer.encode())
If I have only one connection it works good and last thing I need to add is option for multiple-client handling. What is the shortest and easiest way to do it?
The code only calls accept once. Instead, call accept in a while loop and create a thread for each client connection so they are handled in parallel. Use the following pattern as an example:
import socket
import threading
# Thread to handle each "client_soc" connection
def handler(client_soc):
...
client_soc.close()
with socket.socket() as listening_sock:
listening_sock.bind(('', 8000))
listening_sock.listen()
while True:
client_soc, client_address = listening_sock.accept()
# Send each "client_soc" connection as a parameter to a thread.
threading.Thread(target=handler,args=(client_soc,), daemon=True).start()
There is also a built-in socket server that simplifies this process. Here's a tested example echo server that echoes back newline-terminated data:
from socketserver import ThreadingTCPServer,StreamRequestHandler
class echohandler(StreamRequestHandler):
def handle(self):
print(f'Connected: {self.client_address[0]}:{self.client_address[1]}')
while True:
# get message
msg = self.rfile.readline()
if not msg:
print(f'Disconnected: {self.client_address[0]}:{self.client_address[1]}')
break # exits handler, framework closes socket
print(f'Received: {msg}')
self.wfile.write(msg)
self.wfile.flush()
server = ThreadingTCPServer(('',8000),echohandler)
server.serve_forever()
Your code blocks itself.
For instance: client_soc, client_address = listening_sock.accept()
Accepts client, then while True: runs forever, so you can work with 1 connection only, because socket.accept() is called once. You should learn some of these to solve your problem: asyncio, threading, multiprocessing. These libraries will help your code to accept and work with clients concurrently. Sockets can use every, but often they are paired with asyncio: https://asyncio.readthedocs.io/
Hi i have an exercise to build with sockets select and msvcrt, server and clients of mltiplie chat(the server and the clients need to be built non-blocking) that every client will send message and the server will send the message to all the clients except the one who sent it, the server:
import socket
import select
IP = "192.168.1.154"
port = 123
default_buffer_size = 1024
open_client_sockets = []
messages_to_send = []
def send_waiting_messages(wlist):
for message in messages_to_send:
(client_sock, data) = message
if client_sock in wlist:
for sock in open_client_sockets:
if sock is not client_sock:
sock.send(data)
messages_to_send.remove(message)
def main():
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((IP, port))
sock.listen(5)
print("The server is on and waiting for client...")
while True:
rlist, wlist, xlist = select.select([sock] + open_client_sockets, open_client_sockets, [])
for current_socket in rlist:
if current_socket is sock:
(new_socket, addr) = sock.accept()
open_client_sockets.append(new_socket)
else:
data = current_socket.recv(default_buffer_size)
if data == "":
open_client_sockets.remove(current_socket)
print("Connection with client closed")
else:
messages_to_send.append((current_socket, 'Hello ' + data))
send_waiting_messages(wlist)
if __name__ == '__main__':
main()
Building the server wasnt hard because it was guided(if it was not guided i would never got this code working) by the book but i have problem building the client and the main reason is that i dont understand how select.select works, couldn't find answer that will simplify enough this module for me.
this is what i did with the client:
import socket
import select
import msvcrt
IP = "192.168.1.154"
port = 123
sockets = []
def write():
pass
def main():
sock = socket.socket()
sock.connect((IP, port))
while True:
rlist, wlist, xlist = select.select(sockets, sockets, [])
for current_socket in rlist:
if current_socket is sock:
data = current_socket.recv(1024)
print(data)
else:
sockets.append(current_socket)
write()
if __name__ == '__main__':
main()
This probably shows you that I have low understanding of the module select and the exercise actually. I saw some threads that has similar question but I understand nothing from them so I realy need good explantion.
In conclusion I realy am lost...
select takes as parameters a list of sockets to wait for readablity, a list of sockets to wait for writability, and a list of sockets to wait for errors. It returns lists of ready to read, ready to write, and error sockets. From help:
>>> help(select.select)
Help on built-in function select in module select:
select(...)
select(rlist, wlist, xlist[, timeout]) -> (rlist, wlist, xlist)
Wait until one or more file descriptors are ready for some kind of I/O.
The first three arguments are sequences of file descriptors to be waited for:
rlist -- wait until ready for reading
wlist -- wait until ready for writing
xlist -- wait for an ``exceptional condition''
If only one kind of condition is required, pass [] for the other lists.
A file descriptor is either a socket or file object, or a small integer
gotten from a fileno() method call on one of those.
The optional 4th argument specifies a timeout in seconds; it may be
a floating point number to specify fractions of seconds. If it is absent
or None, the call will never time out.
The return value is a tuple of three lists corresponding to the first three
arguments; each contains the subset of the corresponding file descriptors
that are ready.
*** IMPORTANT NOTICE ***
On Windows, only sockets are supported; on Unix, all file
descriptors can be used.
So to fix your client, you need to add the socket you opened (sock) to the sockets list. Your write function can then be called if your socket is ready to be written.
In write, use msvcrt.kbhit() to test for characters typed. You can't just use input because it will block. Then read the character if one has been typed. Collect up the characters until you hit enter, then build a message and write it to the socket. Something like:
message = []
def write(sock):
if msvcrt.kbhit():
c = msvcrt.getche()
if c == '\r':
data = ''.join(message)
print 'sending:',data
sock.sendall(data)
message.clear()
else:
message.append(c)
I am creating a socket client and trying to obtain some data. In order to do so, I need to connect to a web server via socket and the server actually creates another socket which listens and awaits for the data after which sends back to the client.
The problem I have with the code below is that my socket client does not wait for the incoming data from the server and just accepts empty data.
How can I wait for a non-empty data from the server using Python sockets?
My code:
import sys
import json
import socketIO_client
import time
host = 'https://SOME_URL'
socketIO = socketIO_client.SocketIO(host, params={"email" : "edmund#gmail.com"})
def on_connect(*args):
print "socket.io connected"
def on_disconnect(*args):
print "socketIO diconnected"
socketIO.on('connect', on_connect)
socketIO.on('disconnect', on_disconnect)
def on_response_state(*args):
print args # Prints ()
socketIO.emit('receive_state',on_response_state)
socketIO.wait_for_callbacks(seconds=3)
Here's an example using socket. Using s.accept(), the client will wait till a client accepts the connection before starting the while loop to receive data. This should help with your problem.
def receiver():
PORT = 123
CHUNK_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('0.0.0.0', PORT))
s.listen(1)
conn,address=s.accept() # accept an incoming connection using accept() method which will block until a new client connects
while True:
datachunk = conn.recv(CHUNK_SIZE) # reads data chunk from the socket in batches using method recv() until it returns an empty string
if not datachunk:
break # no more data coming in, so break out of the while loop
data.append(datachunk) # add chunk to your already collected data
conn.close()
print(data)
return
receiver()
put the recv socket in a while thread.
like this:
def rec(self):
while 1:
sleep 0.01
rdata = self.clientsocket.recv(self.buffsize)
print("rec from server: ", rdata.decode('utf8'),'\n','press enter to continue')
....
t2 = threading.Thread(target=y.rec, name="rec")
t2.start()
Since you're using the SocketIO library to include parameters (achieved using requests), and want to emit a message, you can wait indefinitely for a response by not specifying a wait time.
with SocketIO(host, params={"email" : "edmund#gmail.com"}) as socketIO:
def on_response_state(*args):
print args # Prints ()
socketIO.emit('receive_state', on_response_state)
socketIO.wait()
I'm trying to access socket objects from memory address "socket._socketobject object at 0x7f4c39d78b40" and use it for another function at different times. The clients are connected to port 9999 and I want the server to react with each one at a later stage while keeping the connection up.
def sock_con(host,port):
host = host
port = port
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((host, port))
sock.listen(5)
while True:
client, address = sock.accept()
print client
print type(client)
print "Server (%s, %s) connected" % address
mongoconn = connectionx('IP_Clients')
key = {'addresses':'192.168.11.1'}
data = {'client':str(client), 'addresses':address}
mongoconn.update(key, data)
client.settimeout(60)
The next code is at a different module which can be used at anytime:
import os,sys
import socket
currentdir = os.path.dirname(os.path.realpath(__file__))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir)
from mgodb import connectionx
mongoconn = connectionx('IP_Clients')
x= mongoconn.find_one({'addresses':'192.168.11.1'})
client= eval(x['client'])
def send_stuff(client,addresses,arg1):
while True:
try:
#data = client.recv(size)
print data
client.send(arg1)
return data
except:
#raise error('Client disconnected')
client.close()
return False
send_stuff(client,x['addresses'],'test10')
To use sockets later in the same process, just store them at their arrival and find them later. Something like this:
...
clients = {}
while True:
client, addr = server.accept()
clients[addr[0]] = client
So, if you stop the listening loop, or run it in a thread, or you run something else in a thread (doesn't matter), you can get the opened socket object from dictionary clients by the client's IP address.
client = clients.get("192.168.1.1")
But you should count in the port as well for detection, because there may be two different clients contacting you from same IP address.
If you want to send an opened socket to another process, well, it is doable but not worth the trouble.
You would need to send the socket's filedescriptor ( socket.fileno() ) to another process, and that can be done using Python module sendfds. It can be found on pypi.python.org.
Then, in receiving process, you would have to construct the socket wrapper object around it manually or trick somehow the existing _socket.dll/.so and socket.py modules to do it for you.
A lot of work and success dubious. What you should do instead is to use the dictionary to store sockets and create an interface (over socket, PIPE or whatever IPC) to forward messages to and from needed connected sockets.
Finally, you do not have to worry about this mess at all, because Python has asyncore module.
It already does the socket storing into dictionary and other useful stuff. The thing is, you need to know what you want to achieve to be able to adequately tune the asyncore client handler. Set correct buffer sizes etc. etc. But asyncore is elegant and you can easily mix it with existing GUI event loop. asyncore and asynchat are often used when creating push servers or instant-messaging-like systems.
I have a device that continually outputs data and I would like to send that data to a client on the same network as it is produced and I'm not finding a good solution. Here is what I'm trying.
Server:
import SocketServer
from subprocess import Popen,PIPE
class Handler(SocketServer.BaseRequestHandler):
def handle(self):
if not hasattr(self, 'Proc'):
self.Proc = Popen('r.sh', stdout=PIPE)
socket = self.request[1]
socket.sendto(self.Proc.stdout.readline(),self.client_address)
if __name__ == "__main__":
HOST, PORT = "192.168.1.1", 6001
server = SocketServer.UDPServer((HOST, PORT), Handler)
server.serve_forever()
Client:
import socket
data = " ".join(sys.argv[1:])
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(data + "\n", ("192.168.1.1", 6001))
try:
received = sock.recv(1024)
while True:
print "Sent: {}".format(data)
print "Received: {}".format(received)
sock.sendto('more' + "\n", ("192.168.1.1", 6001))
received = sock.recv(1024)
except:
print "No more messages"
arg[1] for the client is a program that outputs lines of data for several minutes that I need to process as it is created. The problem seems to be that every time the client sends another request, a new Handler object is created, so I loose Proc. How can I stream Proc.stdout?
Edit: The device is a Korebot2, so I have limited access to other python libraries due to space.
Using UDP you get a new "connection" each time you send a datagram, which is the reason you notice that a new object instance is created each time you send something. You're probably using the wrong kind of protocol here... UDP is used mostly for sending distinct "datagrams", or when a longer connection is not needed. TCP is also called a "streaming" protocol, and is often used for data that has no fixed end.
Also remember that UDP is not a reliable protocol, if used over a network it is almost guaranteed that you will loose packets.