Develop Reference

using multiprocessing to handle socket

I have a server with main process acepting socket connections and put them in Queue stack and another process monitoring this stack and applying it to pool processes handling connections. All works fine except for one thing:
last connection allways at stuck until another connection appears, it's look like last connection can't be closed, but why?
from multiprocessing import Queue, Process, Pool, Manager
import datetime
import socket
def get_date():
return datetime.datetime.now().strftime('%H:%M:%S')
class Server:
def __init__(self, host, port):
self.server_address = host, port
self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def run(self):
self.server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.server_socket.bind(self.server_address)
self.server_socket.listen(1)
print('listen at: %s:%s' % self.server_address)
q = Manager().Queue()
Process(target=self.handle_request, args=(q,)).start()
while True:
client_socket, adress = self.server_socket.accept()
print('\n[%s] request from: %s:%s' % (get_date(), *adress))
q.put(client_socket)
client_socket.close()
del client_socket # client_socket.close() not working
def help(self, client_socket):
data = client_socket.recv(512)
client_socket.send(data)
client_socket.close()
print(data[:50])
def handle_request(self, q):
with Pool(processes=2) as pool:
while True:
pool.apply_async(self.help, (q.get(),))
Server('localhost', 8000).run()

close doesn't realy close connection unless no other process holding a reference, but shutdown will affect all processes. you could call client_socket.shutdown(socket.SHUT_WR) before client_socket.close().
Update:
the reason close doesn't fully close connection is there is a process started by Manager() is holding a reference. Use Queue instead would make close works as you expected.

How to send and receive from the same socket in Python?

I'm am trying to write a client program in Python that can send and receive from the same socket, but it is always giving me the same error which address is already in use. Here is the function I'm trying to write.
def Login():
username=raw_input()
password=raw_input()
message=raw_input()
array=[username,password,message]
TCP_IP = '127.0.0.1'
TCP_PORT = 5563
BUFFER_SIZE = 1024 # Normally 1024, but we want fast response
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((TCP_IP, TCP_PORT))
array_string=pickle.dumps(array)
sock.send(array_string)
sock.close()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((TCP_IP, TCP_PORT))
sock.listen(1)
conn, info = sock.accept()
while 1:
data = serverSocket.recv(1024)
if not data:break
conn.send(data)
conn.close()

There is a bunch of truly newbie errors here.
You can't ever connect a TCP socket to itself. There must be two different sockets.
If you really want to get the data you sent earlier at a listening socket, this listening socket must be created, bound and configured to listen before the client side connects (or, at least, in parallel to this connect attempt, in a few seconds, so the connect attempt will try - but this very likely won't work on localhost).
You can't wait on connect and on accept in the same thread if both are blocking. The simplest approach is to separate the client side and the server side to 2 different programs and run them manually in parallel. Then, after successful debugging, you will be able to do this in different threads of the same process, or using an event-driven engine.

While you may not be able to connect a socket to itself to send and receive data, you might be able to learn from the following example inspired by your code that attempts to do something similar.
import _thread
import pickle
import socket
import time
def main():
"""Run a server in a thread and start a client to talk to it."""
_thread.start_new_thread(run_server, ('', 5563))
run_client('localhost', 5563)
def run_server(host, port):
"""Handle all incoming connections by spawning worker threads."""
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host, port))
server.listen(5)
while True:
_thread.start_new_thread(handle_connection, server.accept())
def handle_connection(client, address):
"""Answer an incoming question from the connected client."""
print('Incoming connection from', address)
client.settimeout(0.1)
data = recvall(client)
client.shutdown(socket.SHUT_RD)
question = pickle.loads(data)
answer = '''len(username) = {}
len(password) = {}
len(message) = {}'''.format(*map(len, question))
client.sendall(answer.encode())
client.shutdown(socket.SHUT_WR)
client.close()
print('Finished with', address)
def recvall(connection):
"""Receive all data from a socket and return as a bytes object."""
buffer = bytearray()
while True:
try:
data = connection.recv(1 << 12)
except socket.timeout:
pass
else:
if data:
buffer.extend(data)
else:
return bytes(buffer)
def run_client(host, port):
"""Collect information from question and display returned answer."""
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
time.sleep(0.1) # wait for server to start listening for clients
client.connect((host, port))
time.sleep(0.1) # wait for handler thread to display connection
username = input('Username: ')
password = input('Password: ')
message = input('Message: ')
question = pickle.dumps((username, password, message))
client.sendall(question)
client.shutdown(socket.SHUT_WR)
answer = recvall(client)
client.shutdown(socket.SHUT_RD)
client.close()
print(answer.decode())
time.sleep(0.1) # wait for handler to cleanly terminate execution
if __name__ == '__main__':
main()

Python socket allow only one connection, but disconnect on new, not refuse

In python, you can define maximum number of socket connections by parameter of listen() function... for example:
serversocket = socket.socket(
socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind((socket.gethostname(), 80))
serversocket.listen(1) // allow only 1 connection
But the problem is that when second client wants to connect, connection is being refused. And I would like to disconnect the old user and connect the new one. Could anybody help me with that?
Probably an answer:
I am posting it in question as it is probable answer (I didn't have time to check it)
serversocket = socket.socket(
socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind((socket.gethostname(), 80))
serversocket.listen(10) // allow 10 connections, but disconnect previous later
someone_connected = 0
while 1:
(clientsocket, address) = serversocket.accept()
if(someone_connected) someone_connected.close()
someone_connected = clientsocket

I am not sure that I fully understand you question, but I think the following example can meet your requirement. the server can disconnect the old user and serve the new one.
the sever side:
#!/usr/bin/env python
import socket
import multiprocessing
HOST = '127.0.0.1'
PORT = 50007
# you can do your real staff in handler
def handler(conn, addr):
try:
print 'processing...'
while 1:
data = conn.recv(1024)
if not data:
break
print data
conn.sendall(data)
conn.close()
print 'processing done'
except:
pass
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(5)
processes = []
while True:
conn, addr = s.accept()
print conn, addr
[p.terminate() for p in processes] # to disconnect the old connection
# start process newer connection and save it for next kill
p = multiprocessing.Process(target=handler, args=(conn, addr))
processes = [p]
p.start()
newest_conn = conn # this is the newest connection object, if you need it
For test, the client side:
#!/usr/bin/env python
import socket
import time
import multiprocessing
HOST = '127.0.0.1'
PORT = 50007
def client():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
time.sleep(0.1)
try:
for n in range(20):
s.send(str(n))
data = s.recv(1024)
print data
time.sleep(0.5)
s.send('')
s.close()
except:
pass
if __name__ == "__main__":
for i in range(5):
print 'user %i connect' %i
p = multiprocessing.Process(target=client)
p.start() # simulate a new user start connect
time.sleep(3)
Try it :-)

You have a wrong assumption built into your question - the single argument to socket listen() is not the "number of connections", but a backlog - number of pending, but not yet accepted client connections the kernel holds for you for a while.
Your problem then seems to be that you have accepted one connection, and reading/writing to it in a loop, and not calling accept() again. The kernel holds the request for any new client connection for some timeout, then notifies the client that the server is not accepting it.
You want to look into select() functionality, as suggested in the comments.

Inheritance of file descriptors - Python 3.4

The documentation under https://docs.python.org/3.5/library/os.html#fd-inheritance about "Inheritance of file descriptors" says:
"On UNIX, non-inheritable file descriptors are closed in child processes at the execution of a new program, other file descriptors are inherited."
Also the documentation to the sockets says that, "the newly created socket is non-inheritable."
I have just tested it with following code:
import socket, os
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('localhost', 9999))
sock.listen(512)
sock.settimeout(1)
print("Socket inheritable?: {}".format(sock.get_inheritable()))
pid = os.fork()
if not pid: # child process
print(sock)
else:
pass
By calling "sock.get_inheritable()" I get False, that means the socket is not inheritable.
But the child process seems to have inherited the socket descriptor.
Am I missing something?
Why is that so?
Thanks
Update:
Here is the "server.py" waiting on the accept in the child process:
import socket, os, time
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('localhost', 9999))
sock.listen(512)
sock.setblocking(True)
print("Socket inheritable?: {}".format(sock.get_inheritable()))
pid = os.fork()
if not pid: # child process
sock, addr = sock.accept()
data = sock.recv(100)
print(data.decode())
else:
while True:
time.sleep(1)
"client.py" sends "Hello" to the socket:
import socket, time, select, sys
msg = "Hello".encode()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost',9999))
s.setblocking(1)
s.send(msg)
s.close()
After starting the server.py and running the "client.py" I see "Hello" message printed in the terminal of "server.py".

Non-inheritable means that the descriptor is closed, as it says in your quote. It does not mean the reference to the socket object in Python disappears.
The more meaningful test would be to try and read from the socket in both branches of the conditional.

Threading udp datas with Python

I'm trying to implement UDP socket's threading.
I want to be able to wait for clients to send me some data in a thread and wait for first datas in an other.
import threading
import socket
class Broker():
def __init__(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.sock.bind(('127.0.0.1', 4242))
self.clients_list = []
def talkToClient(self, ip):
self.sock.sendto("ok", ip)
def listen_clients(self):
while True:
msg, client = self.sock.recvfrom(1024)
t = threading.Thread(None, self.talkToClient, None, (client,), None)
b = Broker()
b.listen_clients()
and my client
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
sock.sendto("connection", ('127.0.0.1', 4242))
while True:
msg, b = sock.recvfrom(1024)
print msg
Problem is that my client is never receiving "ok"

Your main problem is that you are not starting the thread that you have created.
t.start()
Should do it. Please make sure you are using four spaces for indentation as well.
I didn't see the error first myself, but once I added some logging statements it was pretty obvious. The code ended up looking like this:
import threading
import socket
import logging
class Broker():
def __init__(self):
logging.info('Initializing Broker')
self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.sock.bind(('127.0.0.1', 4242))
self.clients_list = []
def talkToClient(self, ip):
logging.info("Sending 'ok' to %s", ip)
self.sock.sendto("ok", ip)
def listen_clients(self):
while True:
msg, client = self.sock.recvfrom(1024)
logging.info('Received data from client %s: %s', client, msg)
t = threading.Thread(target=self.talkToClient, args=(client,))
t.start()
if __name__ == '__main__':
# Make sure all log messages show up
logging.getLogger().setLevel(logging.DEBUG)
b = Broker()
b.listen_clients()
I'm afraid you will run into other problems however, because of your threaded solution. Most python modules are not thread-safe by default, unfortunately this is true for the socket module as well. I'm pretty sure that eventually your socket's internal state will be corrupted since you are reading in one thread and writing in another, or potentially in many others since you spawn a new process for each client.
If you look at multi-threaded socket code examples in Python, a socket is usually owned and used by only one thread. The key is to not reuse the listening socket for clients, but to use socket.accept to create a new socket for each client once it has connected.