I have a client-server model where the client will constantly checking a log file and as soon as a new line comes in the log file it sends that line to the server.
Somehow I managed to work this thing using the following code.
server.py
import SocketServer
class MyTCPSocketHandler(SocketServer.BaseRequestHandler):
def handle(self):
# self.request is the TCP socket connected to the client
data = self.request.recv(1024).strip()
print data
# process the data..
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
server = SocketServer.TCPServer((HOST, PORT), MyTCPSocketHandler)
server.serve_forever()
client.py
import time
import socket
def follow(thefile):
thefile.seek(0, 2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line
def connect_socket():
HOST, PORT = "localhost", 9999
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
return sock
if __name__ == '__main__':
logfile = open("my_log.log")
loglines = follow(logfile)
for line in loglines:
sock = connect_socket()
# send data
sock.sendall(bytes(line))
the problem is every time I need to call the connect_socket() method to send a new line.
I'm quite new to this topic so somebody please let me know is there any workaround for this to work in a such a way that once the connection is established between client and server I need to send data continuously to the server without making a new connection again and again.
If I'm connecting only one time and using the same socket object to send data it was throwing
socket.error: [Errno 32] Broken pipe
Some StackOverflow links which I have followed are given below,
1, 2, 3
One thing I found is
Broken Pipe occurs when one end of the connection tries sending data while the other end has already closed the connection.
How can I keep the connection open on both ends?
For this use case should I go for an asynchronous method and if so which framework will be the best match tornado or twisted?
After a line is transmitted, you close the connection on the client, but don't close it on the server.
From the docs:
RequestHandler.finish()
Called after the handle() method to perform any clean-up actions required. The default implementation does nothing.
So you should implement finish as well and close the socket (self.request) there.
Another option is not to close the connection on the client:
sock = connect_socket()
for line in loglines:
# send data
sock.sendall(bytes(line))
sock.sendall(b'bye') # EOF
sock.close()
However in this case you should modify the server code and make sure it can serve multiple clients and it understands when to close the socket. With all these difficulties it is nevertheless the preferable way of doing things. Ideally, a client has a single TCP connection per session for they're costly to establish.
Related
I want to create a multiprocessing echo server. I am currently using telnet as my client to send messages to my echo server.Currently I can handle one telnet request and it echos the response. I initially, thought I should intialize the pid whenever I create a socket. Is that correct?
How do I allow several clients to connect to my server using multiprocessing.
#!/usr/bin/env python
import socket
import os
from multiprocessing import Process
def create_socket():
# Create socket
sockfd = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Port for socket and Host
PORT = 8002
HOST = 'localhost'
# bind the socket to host and port
sockfd.bind((HOST, PORT))
# become a server socket
sockfd.listen(5)
start_socket(sockfd)
def start_socket(sockfd):
while True:
# Establish and accept connections woth client
(clientsocket, address) = sockfd.accept()
# Get the process id.
process_id = os.getpid()
print("Process id:", process_id)
print("Got connection from", address)
# Recieve message from the client
message = clientsocket.recv(2024)
print("Server received: " + message.decode('utf-8'))
reply = ("Server output: " + message.decode('utf-8'))
if not message:
print("Client has been disconnected.....")
break
# Display messags.
clientsocket.sendall(str.encode(reply))
# Close the connection with the client
clientsocket.close()
if __name__ == '__main__':
process = Process(target = create_socket)
process.start()
It's probably a good idea to understand which are blocking system calls and which are not. listen for example is not blocking and accept is blocking one. So basically - you created one process through Process(..), that blocks at the accept and when a connection is made - handles that connection.
Your code should have a structure - something like following (pseudo code)
def handle_connection(accepted_socket):
# do whatever you want with the socket
pass
def server():
# Create socket and listen to it.
sock = socket.socket(....)
sock.bind((HOST, PORT))
sock.listen(5)
while True:
new_client = sock.accept() # blocks here.
# unblocked
client_process = Process(target=handle_connection, args=(new_client))
client_process.start()
I must also mention, while this is a good way to just understand how things can be done, it is not a good idea to start a new process for every connection.
The initial part of setting up the server, binding, listening etc (your create_socket) should be in the master process.
Once you accept and get a socket, you should spawn off a separate process to take care of that connection. In other words, your start_socket should be spawned off in a separate process and should loop forever.
I just started programming Python.
My goal is to built a digital Picture Frame with three Screens. Therefore I use 3 Raspis, one for each Monitor.
For the communication of these Raspis I need to program a server and a Client.
For a first test I want to built a server which is able to send and receive messages to/from multiple clients.
So I started with a few socket tutorials an created the following program.
Server Class (TcpServer.py)
class TcpServer:
clients = []
serverIsRunning = 0
port = 0
def __init__(self, port):
self.port = port
self.serverIsRunning = 0
self.serverRunning = 0
def startServer (self):
print("start Server...")
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind(("", self.port))
self.server.listen(1)
self.serverRunning = 1
while self.serverRunning:
read, write, oob = select.select([self.server] + self.clients, [], [])
for sock in read:
if sock is self.server:
client, addr = self.server.accept()
self.clients.append(client)
print ("+++ Client ", addr[0], " verbunden")
else:
nachricht = sock.recv(1024)
ip = sock.getpeername()[0]
if nachricht:
print (ip, nachricht)
else:
print ("+++ Verbindung zu ", ip , " beendet")
sock.close()
self.clients.remove(sock)
for c in self.clients:
c.close()
self.clients.remove(c)
self.server.close()
def send(self, message):
message = message.encode()
self.server.send(message)
Client class (TcpClient.py)
import socket
class TcpClient:
def __init__(self, ip, port):
self.serverAdress = (ip, port)
self.connected = 0
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.connection.connect(self.serverAdress)
print ("connectet to ", self.serverAdress)
def send(self, message):
message = message.encode()
self.connection.send(message)
Server:
import threading
import TcpServer
tcpServer = TcpServer.TcpServer(50000)
threadTcpServer = threading.Thread(target = tcpServer.startServer)
threadTcpServer.start()
while True:
tcpServer.send(input("Nachricht eingeben: "))
Client:
import threading
import TcpClient
tcpClient = TcpClient.TcpClient("192.168.178.49", 50000)
while True:
tcpClient.send(input("Nachricht eingeben: "))
I can send messages from the Client to the server, but when I want to send a Message from the server to the client it generates the following error:
BrokenPipeError: [Errno 32] Broken pipe
I assume it is because the server thread blocks the socket while waiting of a incoming message. But I have no idea how to handle this.
How can I program a server who can send and receive messages? Can you recommend a tutorial? I didn't found a tutorial who describes a solution for my problem.
Edit:
Now I tried to solve the problem with the socketserver library, but I still can't solve may problem.
here is my new code for the server:
import socketserver
import threading
import time
class MyTCPHandler(socketserver.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print("{} wrote:".format(self.client_address[0]))
print(self.data)
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
threadTcpServer = threading.Thread(target = server.serve_forever)
threadTcpServer.start()
print("server started")
time.sleep(10)
print("sending Data")
server.request.sendall("Server is sending...")
it generates the error:
AttributeError: 'TCPServer' object has no attribute 'request'
My goal is to write a server with a thread who receives Data and still be able to send data from a other thread.
Is this even possible with only one socket?
You should use the provided socketserver rather than writing all the handling of sockets and select etc.
There are multiple problems with your code -
1 - The server is trying to write to the listening socket!! The client communication socket is the one that you get from the accept() call and that is the one you have to use for reading and writing.
2 - The client is sending the data and completing immediately, but it should really wait for getting a response. Otherwise, the python / OS will close the client socket as soon as the program completes and it will mostly be before the server gets a chance to respond.
I believe with the Handler code you are able to receive the data sent by the client on the server and are also able to send some data back from the Handler to the client? You must have understood that the server cannot send any data back unless there is a client connected to it?
Now, to send data to the client (or clients) from "another" thread, you will need a way to make the handler objects or the client sockets (available inside the Handler object as self.request) available to the "another" thread.
One way is to override the def __init__(self, request, client_address, server): method and save this object's reference in a global list. Remember to do the below as the last line of the overridden init -
# BaseRequestHandler __init__ must be the last statement as all request processing happens in this method
socketserver.BaseRequestHandler.__init__(self, request, client_address, server)
Once you have all the client handlers in the global list, you can easily write to all the clients from any thread as per your needs. You must read about synchronization (Locks) and understand that using same object / socket from multiple threads can create some logical / data issues with your application.
Another thing that you have to worry about and code for is cleaning up this global list whenever a client closes the connection.
I have read several different SO posts on using python as a TCP client and am not able to connect to a server sending data via TCP at 1hz which is hosted locally. The connection parameters I am using are:
import socket
client = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
ip=socket.gethostbyname("127.0.0.1")
port=1234
address=(ip,port)
client.connect(address)
while True:
print("test1")
data = client.recv(1024)
print("test2")
print(data)
I believe that it is failing on the second line of the while statement but do not know why because it hangs and I am not given an error. Below are links to the SO articles, I have read and I have attached a screenshot from a TCP client tool that I am able to connect to the data server with. I'm expecting the data to stream in my print statement, is this not how it works? Whats the best way to make a persistent connection to a TCP connection with python?
Researched:
(Very) basic Python client socket example,Python continuous TCP connection,Python stream data over TCP
Working with sockets: In order to communicate over a socket, you have to open a connection to an existing socket (a "client"), or create an open socket that waits for a connection (a "server"). In your code, you haven't done either, so recv() is waiting for data that will never arrive.
The simple case is connecting as a client to a server which is waiting/listening for connections. In your case, assuming that there is a server on your machine listening on port 1234, you simply need to add a connect() call.
import socket
client = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
ip=socket.gethostbyname("127.0.0.1")
port=1234
address=(ip,port)
client.connect(address) ## <--Add this line.
while True:
print("test1")
data = client.recv(1024)
print("test2")
print(data)
I am new to python and I am using socketserver to try to create a server that broadcasts all the received message from one client to all the connected clients, but facing a problem. I get the following error in the end:
Exception happened during processing of request from ('127.0.0.1', 14872)
Traceback (most recent call last):
File "C:\Users\umair\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 628, in process_request_thread
self.finish_request(request, client_address)
File "C:\Users\umair\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 357, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Users\umair\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 684, in init
self.handle()
File "C:\Users\umair\Desktop\socketserverthread_server.py", line 52, in handle
clients.send(data)
OSError: [WinError 10038] An operation was attempted on something that is not a socket
My server code is as follows:
import socketserver
import threading
host = '127.0.0.1'
port = 6666
all_clients = []
class ThreadingHandler (socketserver.BaseRequestHandler):
def handle(self):
if (self.request) not in all_clients:
all_clients.append(self.request)
data = self.request.recv(1024)
print('%s writes: ' % str(self.client_address), end = " ")
print(data.decode())
for clients in all_clients:
clients.send(data)
class ThreadingServer (socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
myserv = ThreadingServer((host, port), ThreadingHandler)
t = threading.Thread(target = myserv.serve_forever)
t.setDaemon(True)
t.start()
print('The server is online')
The client code is:
import socket
host = '127.0.0.1'
port = 6666
while True:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((host, port))
data = input('You: ')
sock.send(data.encode())
received = sock.recv(1024)
print('Received: ', received.decode())
I am running two of these client codes to test it. The first message from a client doesn't get broadcast, and is just received by the same client. On sending the second message, I receive the above mentioned error. Is it the problem due to the fact that I am creating a new socket in every cycle? I have to create the socket in the loop because if I don't, then I cannot send and receive continuously. I think the socket object is destroyed after a request and a response. I have no idea what's going on. So any help is appreciated. Thank you.
On sending the second message, I receive the above mentioned error. Is
it the problem due to the fact that I am creating a new socket in
every cycle? … I think the socket
object is destroyed after a request and a response. I have no idea
what's going on.
You do have an idea what's going on; indeed the client's socket object sock from a previous loop cycle is destroyed at the time when the new socket is assigned to the variable, which causes the old socket to be closed. But independently thereof the server already closes its request socket due to the use of socketserver, just after one request has been handled, i. e. after your handle(self) returns. The above mentioned error arises from the omission to take this closed socket object in all_clients into account, which could be done e. g. so:
for clients in all_clients[:]:
if clients._closed: all_clients.remove(clients)
else: clients.send(data)
The first message from a client doesn't get broadcast, and is just
received by the same client.
The message does get broadcast (sent to each client), it's just that the other client doesn't bother about receiving it, because it is waiting in input('You: ').
I have to create the socket in the loop because if I don't, then I
cannot send and receive continuously.
That's only true in a sense because you use socketserver, which closes the connection after each request - not quite helpful in case of a chat server.
See the question "Handle multiple requests with select" for an example server without the hassle brought by socketserver et al.
I'm teaching myself Python networking, and I recalled that back when I was teaching myself threading, I came across this page, so I copied the scripts, updated them for Python 3.1.1 and ran them. They worked perfectly.
Then I made a few modifications. My goal is to do something simple:
The client pickles an integer and sends it to the server.
The server receives the pickled integer, unpickles it, doubles it, then pickles it and sends it back to the client.
The client receives the pickled (and doubled) integer, unpickles it, and outputs it.
Here's the server:
import pickle
import socket
import threading
class ClientThread(threading.Thread):
def __init__(self, channel, details):
self.channel = channel
self.details = details
threading.Thread.__init__ ( self )
def run(self):
print('Received connection:', self.details[0])
request = self.channel.recv(1024)
response = pickle.dumps(pickle.loads(request) * 2)
self.channel.send(response)
self.channel.close()
print('Closed connection:', self.details [ 0 ])
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('', 2727))
server.listen(5)
while True:
channel, details = server.accept()
ClientThread(channel, details).start()
And here is the client:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
for x in range(10):
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
The server runs fine, and when I run the client it successfully connects and starts sending integers and receiving them back doubled as expected. However, very quickly it exceptions out:
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python30\lib\threading.py", line 507, in _bootstrap_inner
self.run()
File "C:\Users\Imagist\Desktop\server\client.py", line 13, in run
print('Received:',repr(pickle.loads(client.recv(1024))))
socket.error: [Errno 10053] An established connection was aborted by the softwar
e in your host machine
The server continues to run and receives connections just fine; only the client crashes. What's causing this?
EDIT: I got the client working with the following code:
import pickle
import socket
import threading
class ConnectionThread(threading.Thread):
def run(self):
for x in range(10):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('localhost', 2727))
client.send(pickle.dumps(x))
print('Sent:',str(x))
print('Received:',repr(pickle.loads(client.recv(1024))))
client.close()
for x in range(5):
ConnectionThread().start()
However, I still don't understand what's going on. Isn't this just opening and closing the socket a bunch of times? Shouldn't there be time limitations to that (you shouldn't be able to open a socket so soon after closing it)?
Your client is now correct - you want to open the socket send the data, receive the reply and then close the socket.
The error original error was caused by the server closing the socket after it sent the first response which caused the client to receive a connection closed message when it tried to send the second message on the same connection.
However, I still don't understand
what's going on. Isn't this just
opening and closing the socket a bunch
of times?
Yes. This is acceptable, if not the highest performance way of doing things.
Shouldn't there be time
limitations to that (you shouldn't be
able to open a socket so soon after
closing it)?
You can open a client socket as quickly as you like as every time you open a socket you will get a new local port number, meaning that the connections won't interfere. In the server code above, it will start a new thread for each incoming connection.
There are 4 parts to every IP connection (source_address, source_port, destination_address, destination_port) and this quad (as it is known) must change for ever connection. Everything except source_port is fixed for a client socket so that is what the OS changes for you.
Opening server sockets is more troublesome - if you want to open a new server socket quickly, your
server.bind(('', 2727))
Above then you need to read up on SO_REUSEADDR.