I have read several different SO posts on using python as a TCP client and am not able to connect to a server sending data via TCP at 1hz which is hosted locally. The connection parameters I am using are:
import socket
client = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
ip=socket.gethostbyname("127.0.0.1")
port=1234
address=(ip,port)
client.connect(address)
while True:
print("test1")
data = client.recv(1024)
print("test2")
print(data)
I believe that it is failing on the second line of the while statement but do not know why because it hangs and I am not given an error. Below are links to the SO articles, I have read and I have attached a screenshot from a TCP client tool that I am able to connect to the data server with. I'm expecting the data to stream in my print statement, is this not how it works? Whats the best way to make a persistent connection to a TCP connection with python?
Researched:
(Very) basic Python client socket example,Python continuous TCP connection,Python stream data over TCP
Working with sockets: In order to communicate over a socket, you have to open a connection to an existing socket (a "client"), or create an open socket that waits for a connection (a "server"). In your code, you haven't done either, so recv() is waiting for data that will never arrive.
The simple case is connecting as a client to a server which is waiting/listening for connections. In your case, assuming that there is a server on your machine listening on port 1234, you simply need to add a connect() call.
import socket
client = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
ip=socket.gethostbyname("127.0.0.1")
port=1234
address=(ip,port)
client.connect(address) ## <--Add this line.
while True:
print("test1")
data = client.recv(1024)
print("test2")
print(data)
Related
I am currently trying to learn networking with python. I am really new to this topic so I replicated some examples from somewhere like here
I want to achieve a continous data transfer with TCP. This means I want to send data as long as some condition is met. So I slightly modified the example to this code below:
My Setup is Win10 with Python 3.8
My client.py copied and modified form above:
# Echo client program
import socket
HOST = '192.168.102.127' # The remote host
PORT = 21
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
i=0 #For counting how often the string was sent
while True: #for testing this is forever
s.sendall(b'Hello, world')
data = s.recv(1024)#
print(i)
i=i+1
print('Received', repr(data))
My server.py:
# Echo server program
import socket
HOST = '' # Symbolic name meaning all available interfaces
PORT = 21
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
with conn:
print('Connected by', addr)
while True:
data = conn.recv(1024)
if not data: break
conn.sendall(data)
The error I am getting is
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
after i=5460 (in multiple tries) on the Client side and
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host on the Server side
The longer my text message is, less messages got sent before the error.
This leads me to believe I sent the data to some sort of buffer which is (over-)written to until the error is thrown.
When looking for possible solutions I only found different implementations, which did not cover my problem or used other software.
As stated in some answers for similar questions, I disabled my firewall and stopped my antivirus, but with no noticable difference.
When looking up the error, there is also the possibilty of protocol errors but I do not expect that to be a problem.
When reading into the socket/TCP documentation, I found somewhere that TCP is not really designed for this kind of problem, but rather for
client connects to server
|
V
client sends request to server
|
V
server sends request answer
|
V
server closes connection.
Is this really true?
But I cannot believe that for every data that is sent a new socket must be connected, like in this question. This solution is also really slow.
But if this is the case, what could I use alternatively?
To illustrate the bigger picture:
I have a some other code which is giving me status data (text) at 500Hz. In Python, I am processing this data and sending the processed data to an Arduino with Ethernet shield. This data is "realtime" data, so I need the data sent to the arduino as fast as possible. Here the client is Python and the Server is the Arduino with the Ethernet module. The connection and everthing is working fine, only the continous sending of data is my problem.
I am writing a UDP server application that serves as a back end to Teltonika FMB630 car mounted devices.
I already took care of the protocol specifics and decoding, the problem I am facing relates to the UDP socket used.
My UDP server has to send an acknowledgement to the client device upon receiving a message (that is the protocol), however, if I send those ACKs, the server socket stops receiving data after a while.
The server's UDP socket object is passed to an concurrent.futures.ThreadPoolExecutor that fires a function (send_ack) that sends the ACK, however this is not the issue because I tried calling send_ack in the main thread, after receiving data and the same issue occurs.
I suspect the problem is the remote device somehow breaks the connection or the ISP or MNO doesn't route the reply packet (this is a GPRS device) and then the socket.send() method that is used to send the acknowledge, somehow freezes other socket operations, specifically recvfrom_into called in the main thread loop.
I wrote two scripts to illustrate the situation:
udp_test_echo.py :
#!/usr/env/bin python
import socket
import concurrent.futures
def send_ack(sock, addr, ack):
print("Sending ACK to {}".format(addr))
sock.connect(addr)
print("connected to {}".format(addr))
sock.send(ack)
print("ACK sent to {}".format(addr))
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(("127.0.0.1", 1337))
data = bytearray([0] * 10)
executor = concurrent.futures.ThreadPoolExecutor(max_workers=4)
while True:
print("listening")
nbytes, address = s.recvfrom_into(data)
print("Socket Data received {} bytes Address {}".format(nbytes, address))
print("Data received: ", data, " Echoing back to client")
executor.submit(send_ack, s, address, data[:nbytes])
udp_test_client.py:
#!/usr/env/bin python
import socket
import time
import random
def get_random_bytes():
return bytearray([random.randint(0,255) for b in range(10)])
ip = "127.0.0.1"
port = 1337
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect((ip, port))
while True:
stuff_to_send = get_random_bytes()
print("Sending stuff", stuff_to_send)
s.sendall(stuff_to_send)
print("reply: ", s.recvfrom(10))
time.sleep(0.1)
Running udp_test_echo.py in one terminal and udp_test_client.py in another, we see normal operation but if you Ctrl+C the test client and re run it, you will see that the server doesn't respond until it is restarted.
Is there a way to timeout a specific sending operation from a specific call to socket.send() method without affecting other calls ? (I want my socket.recvfrom_into call to block on the main thread)
If I settimeout on the entire socket object, I am going to have to deal with many exceptions while waiting for data in the main thread and I don't like to have to rely on exceptions for proper program operation.
The culprit was the socket.connect() call in send_ack, when being called on the server's socket object it causes the socket to no longer be bound and listen on the port specified in the start of the program.
Instead the send_ack function was changed to be:
def send_ack(sock, addr, ack):
print("Sending ACK to {}".format(addr))
sock.sendto(ack, addr)
print("ACK sent to {}".format(addr))
socket.sendto(data, address) uses the existing connection instead of starting a new one.
I have a client-server model where the client will constantly checking a log file and as soon as a new line comes in the log file it sends that line to the server.
Somehow I managed to work this thing using the following code.
server.py
import SocketServer
class MyTCPSocketHandler(SocketServer.BaseRequestHandler):
def handle(self):
# self.request is the TCP socket connected to the client
data = self.request.recv(1024).strip()
print data
# process the data..
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
server = SocketServer.TCPServer((HOST, PORT), MyTCPSocketHandler)
server.serve_forever()
client.py
import time
import socket
def follow(thefile):
thefile.seek(0, 2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line
def connect_socket():
HOST, PORT = "localhost", 9999
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST, PORT))
return sock
if __name__ == '__main__':
logfile = open("my_log.log")
loglines = follow(logfile)
for line in loglines:
sock = connect_socket()
# send data
sock.sendall(bytes(line))
the problem is every time I need to call the connect_socket() method to send a new line.
I'm quite new to this topic so somebody please let me know is there any workaround for this to work in a such a way that once the connection is established between client and server I need to send data continuously to the server without making a new connection again and again.
If I'm connecting only one time and using the same socket object to send data it was throwing
socket.error: [Errno 32] Broken pipe
Some StackOverflow links which I have followed are given below,
1, 2, 3
One thing I found is
Broken Pipe occurs when one end of the connection tries sending data while the other end has already closed the connection.
How can I keep the connection open on both ends?
For this use case should I go for an asynchronous method and if so which framework will be the best match tornado or twisted?
After a line is transmitted, you close the connection on the client, but don't close it on the server.
From the docs:
RequestHandler.finish()
Called after the handle() method to perform any clean-up actions required. The default implementation does nothing.
So you should implement finish as well and close the socket (self.request) there.
Another option is not to close the connection on the client:
sock = connect_socket()
for line in loglines:
# send data
sock.sendall(bytes(line))
sock.sendall(b'bye') # EOF
sock.close()
However in this case you should modify the server code and make sure it can serve multiple clients and it understands when to close the socket. With all these difficulties it is nevertheless the preferable way of doing things. Ideally, a client has a single TCP connection per session for they're costly to establish.
I am trying simulate mysql connection process through python program.
In python script, I am opening a tcp socket to mysql server and writing pre-captured on socket.
On login packet I get an error "#28000Access denied for user 'root'#'10.x.x.xxx' (using password: YES)"
import socket
import sys
import time
Host = '10.x.x.xxx'
Port = '3306'
t_con = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clust_vip = (Host, int(Port))
try:
t_con.connect(clust_vip)
print ('Socket connection established')
print "TCP connection established:", t_con.recv(4096)
byte1 = open("req_r1").read()
t_con.send(byte1)
print "Response for packet1:", t_con.recv(4096)
bytes2 = open("req_r2").read()
t_con.send(byte2)
print "Response for packet2:", t_con.recv(4096)
finally:
t_con.close()
"req_r1" and "req_r2" file used above contains raw packets (mysql protocol raw packets and not entire frame/tcp layer)
I am replaying the capture through socket
I have skipped the TCP connection packet (as I am establishing the socket connection through python)
I am trying to write raw packet (mysql protocol packet) and not entire frame on the socket.
Can anyone guide me how I can overcome this issue. I think error is due to salt used to establish the mysql connection.
I have tried Passwordless connection as well however it didn't worked.
Update: What I understand is on establishing tcp socket connection, db server replies with salt and I need to reuse this salt to generate encrypted password and use it in next connect packet.
If anyone has idea if I am on right track and how i can extract/reuse it it would be great help.
You can't log in by replaying a previous session. As has been pointed out in comments, that would be terribly insecure. It's a challenge/response mechanism, and your response varies with the challenge received.
See https://dev.mysql.com/doc/internals/en/client-server-protocol.html for a breakdown of the protocol.
I have a small server side program which accepts the connections from the clients and writes into the socket. I want to detect from the server side as soon as the client gets disconnected. I see the client has sent a TCP Fin when it disconnected and the server is detecting it only when it has some data to send towards the client and I see Broken pipe exception. Instead I want the server to be able to detect that the client has sent a Finish and immediately close the session. Can someone please help me by giving some inputs.
>>> Error sending data: [Errno 32] Broken pipe
[DEBUG] (Keepalive-Thread) Exiting
Server program to accept connections:
def enable_pce(ip):
#Step 1: Create a TCP/IP socket to listen on. This listens for IPv4 AFI
srv = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
#Step 2: Prevent from "address in use" upon server restart
srv.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
#Step 3: Bind the socket to you server ip address
srv_address = (ip,55555)
print 'starting server on %s port %s' % srv_address
srv.bind(srv_address)
#Step 4: Listen for connections
srv.listen(1)
#Step 5: Wait for incoming connections
connection, peer_address = peer.accept()
print 'connection from', connection.getpeername()
return connection
the server is detecting it only when it has some data to send towards the client and I see Broken pipe exception.
Close of connection by the peer can be detected either by getting an error when writing to the socket (broken pipe) or by reading from the socket and getting nothing (no error, no data) back.
In most scenarios it only matters to detect disconnection if one wants to communicate with the peer (that is read or write). If you want to do this without actually reading or writing data you might check readability of the socket with select.select and if the socket is readable peek into the read buffer with recv and MSG_PEEK.