Develop Reference

Client that communicates with multiple servers concurrently

I am trying to learn multithreading programming with Python and specifically I am trying to build a programm that 1 Client sends data to multiple servers and get some messages back. In the first version of my programm I want my client to communicate back and forth with each server that I have spawned with each thread, till I type 'bye'. Now I have 2 issues with my implementation that I don't understand how to deal with. The first one is that I don't want the connection with the server to close after I type 'bye' (I want to add extra functionality after that) and the second one is that the servers doesn't get the messages that I type at the same type but I can communicate with the second server only if the first thread terminates (which as I said, I don't want to terminate). Any suggestions would be appreciated. Cheers!
Client.py
import sys
import threading
from _thread import *
import socket
host_1 = '127.0.0.1'
port_1 = 6000
host_2 = '127.0.0.2'
port_2 = 7000
def connect_to_server(host, port):
client_socket = socket.socket() # instantiate
client_socket.connect((host, port)) # connect to the server
message = input(" -> ") # take input
while message.lower().strip() != 'bye':
client_socket.send(message.encode()) # send message
data = client_socket.recv(1024).decode() # receive response
print('Received from server: ' + data) # show in terminal
message = input(" -> ") # again take input
threads_dict = {}
th_1 = threading.Thread(target=connect_to_server, args=(host_1, port_1))
th_2 = threading.Thread(target=connect_to_server, args=(host_2, port_2))
th_1.start()
th_2.start()
th_1.join()
th_2.join()
Server.py
import socket
import sys
def server_program():
host = sys.argv[1] # '127.0.0.1', '127.0.0.2'
port = int(sys.argv[2]) # 6000, 7000
server_socket = socket.socket() # get instance
server_socket.bind((host, port)) # bind host address and port together
# configure how many client the server can listen simultaneously
server_socket.listen(2)
conn, address = server_socket.accept() # accept new connection
print("Connection from: " + str(address))
while True:
# receive data stream. it won't accept data packet greater than 1024 bytes
data = conn.recv(1024).decode()
if not data:
# if data is not received break
break
print("from connected user: " + str(data))
data = input(' -> ')
conn.send(data.encode()) # send data to the client
if __name__ == '__main__':
server_program()

python multithreading server

I am new to networking programming and python.
I am trying to figure out how to run different jobs at the server side.
For example, I want one function to create connections for incoming clients but in the same time I can still do some administration work from the terminal.
My code is as below but it doesn't work:
Edited: it doesn't work means it will get stuck in the init_conn() function
Like:
starting up on localhost port 8887
Thread: 0 Connected with 127.0.0.1:48080
# waiting
I am looking into SocketServer framework but don't know how that works.
from thread import *
import socket
def init_conn():
thread_count =0
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Bind the socket to the port
server_address = ('localhost', 8887)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)
# Listen for incoming connections
sock.listen(10)
#now keep talking with the client
while 1:
#wait to accept a connection - blocking call
conn, addr = sock.accept()
print 'Thread: '+ str(thread_count) + ' Connected with ' + addr[0] + ':' + str(addr[1])
#start new thread takes 1st argument as a function name to be run, second is the tuple of arguments to the function.
start_new_thread(clientthread ,(conn,))
thread_count +=1
sock.close()
def clientthread(conn):
# receive data from client and send back
def console():
print 'this is console'
option = raw_input('-v view clients')
if option == 'v':
print 'you press v'
def main():
start_new_thread( init_conn(),() )
start_new_thread( console(),() )
if __name__ == "__main__":
main()

Your problem is probably that you start the program, sometimes it prints "this is console" and then it ends.
The first bug is that you call the methods instead of passing the handle to start_new_thread. It must be:
start_new_thread( init_conn, () )
i.e. no () after the function name.
The program doesn't do much because start_new_thread() apparent starts a thread and then waits for it to stop. The documentation is pretty unclear. It's better to use the new threading module; See http://pymotw.com/2/threading/
def main():
t = threading.Thread( target=init_conn )
t.daemon = True
t.start()
console()
so the code will run until console() ends.
I suggest to split the server and the command line tool. Create a client which accepts commands from the command line and sends them to the server. That way, you can start the console from anywhere and you can keep the code for the two separate.

Seeing that you're new to python, have you tried taking a look at the threading module that comes with the standard library?
import threading
... #rest of your code
while conditions==True:
i = threading.Thread(target=init_conn)
c = threading.Thread(target=console)
i.start()
c.start()
Can't say I've done too much with networking programming with python, so I don't really have much to say in that manner, but at least this should get you started with adding multithreading to your project.

Using SocketServer you may implement a client/server system. The documentation gives small examples which may be useful for you. Here is an extended example from there:
server.py :
import SocketServer
import os
import logging
FORMAT = '[%(asctime)-15s] %(message)s'
logging.basicConfig(format=FORMAT, level=logging.DEBUG)
class MyServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
# By setting this we allow the server to re-bind to the address by
# setting SO_REUSEADDR, meaning you don't have to wait for
# timeouts when you kill the server and the sockets don't get
# closed down correctly.
allow_reuse_address = True
request_queue_size = 10
def __init__(self, port):
self.host = os.uname()[1]
self.port = port
SocketServer.TCPServer.__init__(self, (self.host,self.port), MyTCPHandler)
logging.info( "Server has been started on {h}:{p}".format(h=self.host,p=self.port) )
class MyTCPHandler(SocketServer.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
# max length is here 1024 chars
self.data = self.request.recv(1024).strip()
logging.info( "received: {d}".format(d=self.data) )
# here you may execute different functions according to the
# request string
# here: just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
PORT = 8887
if __name__ == "__main__":
# Create the server, binding to localhost on port 8887
#server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
server = MyServer( PORT )
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()
client.py
import socket
import sys
import logging
FORMAT = '[%(asctime)-15s] %(message)s'
logging.basicConfig(format=FORMAT, level=logging.DEBUG)
HOST, PORT = "workstation04", 8887
logging.info( "connect to server {h}:{p}".format(h=HOST,p=PORT ) )
# read command line
data = " ".join(sys.argv[1:])
# Create a socket (SOCK_STREAM means a TCP socket)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
# Connect to server and send data
sock.connect((HOST, PORT))
sock.sendall(data + "\n")
# Receive data from the server and shut down
received = sock.recv(1024)
finally:
sock.close()
logging.info( "Sent: {}".format(data) )
logging.info( "Received: {}".format(received) )
The output looks something like:
server side:
> python server.py
[2015-05-28 11:17:49,263] Server has been started on disasterarea:8887
[2015-05-28 11:17:50,972] received: my message
client side:
[2015-05-28 11:17:50,971] connect to server disasterarea:8887
[2015-05-28 11:17:50,972] Sent: my message
[2015-05-28 11:17:50,972] Received: MY MESSAGE
You can run several clients (from different consoles) in parallel. You may implement a request processor on the server side which processes the incoming requests and executes certain functions.
Alternatively, you may use the python module ParallelPython which executes python code locally on a multicore system or on a cluster and clusters. Check the http examples.
I had to force pip to install this module:
pip install --allow-external pp --allow-unverified pp pp

Python broadcasting message to all clients in a socket

I have made a simple chat server using threads like the following:
#-*- coding:utf-8 -*-
import _thread as thread
import time
import socket
def now():
return time.asctime(time.localtime())
def handleclient(connection, ADDR):
sod = str(ADDR)
msg = sod+"joined the chat"
msg2 = msg.encode("utf-8")
connection.sendall(msg2)
while True:
recieved = connection.recv(1024)
adsf = recieved.decode("utf-8")
print(now(),"(%s):%s" % (ADDR, recieved))
output = "%s:%s"%(ADDR, recieved.decode("utf-8"))
message = output.encode("utf-8")
connection.sendall(message)
if __name__ == "__main__":
addr = ("", 8080)
r =socket.socket()
print("socket object created at", now())
r.bind(addr)
r.listen(5)
while True:
print("Waiting for clients...")
connection, ADDR = r.accept()
print("We have connection from ", ADDR)
thread.start_new_thread(handleclient, (connection, ADDR))
However, it looks like the sendall isnt working and sending the message to only the person who sent it. How can I make it send it to all clients?

There is nothing like what you're trying to do, because as pointed out in the commends, sendall() means "definitely send all my bytes and keep trying until you have," not "send these bytes to lots of clients."
You will want to use either UDP multicast (if you're on a relatively reliable network which supports it, such as a LAN or corporate WAN), or you will simply need to send explicitly to every connected client. The other alternative is peer-to-peer: send to several clients and instruct those clients to send to more clients until all clients are taken care of. Obviously this requires more coding. :)

You may have a look at Zero MQ, which provides high-level facilities over sockets by implementing several patterns ( publish/subscribe , push/pull, etc...).

Python: Passing Multiple Variables Across TCP Client/Server

I'm creating a TCP client and server using Python 3.3 and I'm new to using both Python and sockets. My issue is that I need to be able to store anything passed across the connection as a separate variable for writing to various files, etc.
I'm unsure how to do this as it all seems to be one stream of data that I cannot store separately. Below is my latest piece of working code and all it does is send the data I need across the connection. Should I be trying to send all the data across as a list and de-construct it into separate variables? Can I already access them separately and I just haven't figured it out yet? Is my approach all wrong?
Server code:
import os
from socket import *
HOST = '' #We are the host
PORT = 29876
ADDR = (HOST, PORT)
BUFFSIZE = 4096
serv = socket( AF_INET,SOCK_STREAM)
serv.bind(ADDR,) #Create tuple with one element
serv.listen(5)
print ('listening...')
conn,addr = serv.accept()
print (conn,addr)
print ('...connected')
with open(os.path.expanduser("~/.ssh/id_rsa.pub")) as f:
key = f.read()
conn.sendall(key)
print(key)
while True:
data = conn.recv(BUFFSIZE)
print(data)
conn.close()
Client code:
from socket import *
import os
import socket
HOST = 'xxx.xxx.xxx.xxx'
PORT = 29876 #Must match server.py
ADDR = (HOST,PORT)
BUFFSIZE = 4096
cli = socket.socket( AF_INET, SOCK_STREAM)
cli.connect(ADDR,)
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("gmail.com",80))
ip = ((s.getsockname()[0]))
ip = ip.encode('utf-8')
cli.send(ip)
while True:
data = cli.recv(BUFFSIZE)
print (data)
cli.close()

server:
from socket import *
from os.path import isfile
s = socket()
s.bind(('', 1234))
s.listen(4)
ns, na = s.accept()
loopdata = {}
i = 0
while 1:
try:
data = ns.recv(8192)
except:
break
for line data.split('\n'):
if line == 'version':
print na[0] + ' requested a version'
ns.send('1.0\n')
elif line == 'key':
print na[0] + ' is requesting a key'
if isfile(na[0] + '.key'):
with open(na[0] + '.key') as f:
ns.send(f.read())
else:
ns.send('Missing key file!\n')
loopdata[i] = line
#ns.send('OK\n')
i += 1
ns.close()
s.close()
print loopdata # <- Print all the lines
client:
from socket import *
s = socket()
s.connect(('127.0.0.1', 1234))
s.send('version\n')
print 'Client got:', s.recv(8192)
s.close()
I'm not sure as to what you want to save/store/respond to,
You mentioned something about a key in your code and in your client code you created multiple sockets but only really used one.. and that was for printing whatever the server was sednding?
Begin clean, try to figure out things before mixing it all up.
And state a clear goal that you want to achieve with your code? what do you want to send? and when sending X, what do you want the server to respond?
Tactics:
1. You want to define a protocol,
1.1 Command separator
1.2 Command structure (ex: command:parameter:data\n)
1.3 sates (ex login state etc etc..)
After you know what you want to do, ex file share:
c->s: get:file:/root/storage/file.txt\n
c<-s: file:content\n
c<-s: <file data>\n\n
c<-s: file:close\n
c->s: file:recieved
For instance.

Python chat client-server modification goes horribly wrong

As an exercise, I looked for a simple Python chat client-server system to modify and toy with. The first major failing I found in the system was that it used a single tcp connection for the server and client to communicate. The second was that you could only have two people (one using the client, and the other using the server) communicate. Thirdly, consecutive posts were impossible. One person sent a message, then had to wait for the other person to send a single message before talking again. Very, very limiting.
So I began threading it and experimenting with the sockets. Clients connect to the server once, give their IP addresses, create a listening thread, and then reconnect to the server's message receiver.
All posts are sent to that receiver, which iterates through a list of connected clients and connects to each of them and sends the message (with the sender's name in the beginning; misc feature).
(I know that opening a new connection so often like that is inefficient, but I wanted to keep with tcp connections until I had it working, and THEN go to UDP)
However, weird crap began happening. Suffice it to say that I have nightmares of Error 91.
Could anyone identify how to render this code operable within this structure and feature-set?
(Python version 2.6 yey; ignore the infinite loop that is just a placeholder)
SERVER CODE:
from socket import *
from time import time, ctime
import Queue, threading
IP = ''
PORT = 5000
PORTPlus = 2
PORTRec = 1000
ADS = (IP, PORT)
namelist = []
clientlist = []
class clientRec(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
print "I'm this far:", (IP, (PORT + PORTRec))
self.receiver = socket(AF_INET, SOCK_STREAM)
self.receiver.bind((IP, PORT + PORTRec))
self.sender = socket(AF_INET, SOCK_STREAM)
def run(self):
global clientlist, namelist
self.receiver.listen(10)
connected = True
while connected:
tcpcli, addr = receiver.accept()
message = tcpcli.recv(1024) # Accept clien't IP for home-dialing
for i in range(clientlist.__len__()): # For each connected client
try:
sender.connect(clientlist(i)) # connect
sender.send(namelist[i] + message) # and deliver message with sender's name
sender.close()
except:
del clientlist[i], namelist[i]
print "ADS:", (IP, 5000)
handle = clientRec()
tcpsoc = socket(AF_INET, SOCK_STREAM) # Paperwork
tcpsoc.bind(ADS) # Bind self to port
tcpsoc.listen(5) # Listen on that port0
handle.start() # Start thread
# Main
while 1:
print "Waiting for connection"
tcpcli, addr = tcpsoc.accept() # Accept unknown client
print "Connection received; handling..."
namelist.append(tcpcli.recv(1024)) # Accept client's name
client_IP = tcpcli.recv(1024) # Accept clien't IP for home-dialing
client_port = int(tcpcli.recv(1024))# Accept clien't listening port
port_assign = PORT + PORTRec
tcpcli.send(str(port_assign)) # Tell the client that port
tcpcli.close() # Close client connection
clientlist.append((client_IP, client_port))# Add client to send-list
print "Handled."
tcpsoc.close()
CLIENT CODE:
#!/usr/bin/env python
from socket import *
import threading, cgi, os
IP = ''
PORT = 5000
PORTmy = 100
ADS = (IP, PORT)
class iListen(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.receiver = socket(AF_INET, SOCK_STREAM)# Paperwork
self.receiver.bind(('', PORT + PORTmy)) # Listen on that port
self.receiver.listen(5) # Listen for posts
def run(self):
while listening:
tcpcli, addr = receiver.accept() # Accept unknown client
message = tcpcli.recv(1024)
if message == "/q":
listening = False
tcpcli.close()
# Initial CONNECT
myname = raw_input("Tell me yer name partnah: ")
tcpsoc = socket(AF_INET, SOCK_STREAM)
tcpsoc.connect(ADS) # First Connect
tcpsoc.send(myname) # Declare name
tcpsoc.send(gethostbyname(gethostname()))# Give IP address
tcpsoc.send(str(PORT + PORTmy)) # Give listening port
ADS = (IP, int(tcpsoc.recv(1024))) # Get new connect details
tcpsoc.close() # Close old connection
listen = iListen() # Create listener thread
listen.start() # Start listening
# RECONNECT
print ADS
tcpsoc = socket(AF_INET, SOCK_STREAM)
tcpsoc.connect(ADS) # reconnect to new port
connected = True
# Main Chat-loop
while connected:
mes = raw_input(">>>")
tcpsoc.send(mes)
if mes == "/q":
tcpsoc.close()
connected = False
time.sleep(4)
sys.exit()

I am working on something alot like this, but I am instead going to implement text encryption. I see you are suing lists for the client list... but I would say there is a better way of doing that. I am using a dictionary.
if you are familiar with dictionaries, skip the next paragraph.
Dicionaries can handle basicly 2 variables, and are defined using the {}.
>>> stuff = {'a':'hello','b':'world'}
>>> print stuff['a']
hello
>>> print stuff['a'],stuff['b']
hello world
so using this, you can can make a dictionary like {'username':'ipaddr'} this way you can make it so that both usernames and ips are all in one variable. If you want the end product like me, you will be making it so all the server does is repeat the message, and send it to everyone who is connected. then the server can just cycle through the usernames.
as another note, I think the tcpsoc.listen(5) is how many people can beconnected at once... I think thats what i read somewhere.
I have no idea why you would be having that error, but if you want to look at my halfway constucted code, you are more than welcome too. (ignore the import random, this is not yet used, but will be part of the encryption system)
http://piratepad.net/PwQzdU0bkk