Hi I have multiple systems communicating via message using TCP connections.
My send function looks like the following
def _send(self, message, dest):
self.sendLock.acquire()
message = pickle.dumps(message)
#sending length
message_length = len(message)
self.outChan[dest].send('<MESSAGELENGTH>%s</MESSAGELENGTH>'
% str(message_length))
for message_i in range(0, message_length, 1024):
self.outChan[dest].send(message[:1024])
message = message[1024:]
self.sendLock.release()
And the receive thread looks like this:
def readlines(self, sock):
while True:
msg = ''
opTag = '<MESSAGELENGTH>'
clTag = '</MESSAGELENGTH>'
while not all(tag in msg for tag in (opTag, clTag)):
msg = sock.recv(1024)
msglen = int(msg.split(clTag)[0].split(opTag)[1])
msg = msg.split(clTag)[1]
while len(msg) < msglen:
msg += sock.recv(msglen-len(msg))
self.rec.put(pickle.loads(msg))
After the message is read from self.rec a confirmation message is sent to the sender.
I have implemented my own buffer to control the traffic in the network. At any moment I would have sent atmost MAX_BUFFER_SIZE messages with no confirmation received.
Here is the problem: When program starts, it sends MAX_BUFFER_SIZE messages without waiting for the confirmation. But only a few of these MAX_BUFFER_SIZE messages are received.
In one of the simulations with MAX_BUFFER_SIZE = 5, total 100 messages were sent and m2,m3 and m4 were not received. All other messages were received (in the order they were sent).
I doubt the error is in the initial sending burst, but I am unable to figure out the exact problem.
There are a few errors in the receive thread:
While inspecting the received message for the opening and closing tags, you are not appending to the already received part, but overwriting it.
After detecting the message length, you are losing the subsequent messages, that have their closing tag already received, but not analyzed yet.
You are possibly putting several messages together into self.rec.
Here is a corrected form, with comments explaining the changes:
def readlines(self, sock):
msg = '' # initialize outside since otherwise remiander of previous message would be lost
opTag = '<MESSAGELENGTH>' # no need to repeat this in each iteration
clTag = '</MESSAGELENGTH>' # no need to repeat this in each iteration
while True:
while not all(tag in msg for tag in (opTag, clTag)):
msg += sock.recv(1024) # += rather than =
msglen = int(msg.split(clTag)[0].split(opTag)[1])
msg = msg.split(clTag, 1)[1] # split just once, starting from the left
while len(msg) < msglen:
msg += sock.recv(msglen-len(msg))
self.rec.put(pickle.loads(msg[:maglen])) # handle just one message
msg = msg[msglen:] # prepare for handling future messages
Related
I have a server that sends some messages to a client. The print(trades) statement shows that file reader reads the entire csv correctly:
def send_past_trades(self):
with open('OTC_trade_records.csv',newline='') as f:
connectionSocket, addr = self.client
trades = f.read()
#print(trades)
connectionSocket.send(trades.encode())
My client receiver is like this:
msg = b""
while(True):
print("Batch receiving")
tmp = client_socket.recv(4096)
msg += tmp
if len(tmp) < 4096:
print(len(tmp))
break
msg = msg.decode()
print(msg)
The message is always partial. I can see that the statement "Batch receiving" is printed once and when the break statement is initiated, the length of the last message is 1228.
Another point is, this code works fine in my local system. The problem occurs when I put the server program to a remote server machine. Is there a possibility that server intervenes with the message?
Note: I tried different ways to solve the problem such as sending only package size of 1024b messages in a loop. Still partial messages received.
The problem is here:
if len(tmp) < 4096:
print(len(tmp))
break
The point is that bufsize in recv(bufsize) is a maximum size to receive. The recv will return fewer bytes if there are fewer available.
I suggest to define a simple communication protocol that describes the structure of a message with a header and payload. The header must contain the payload size. This allows you to parse data from the incoming TCP stream and get the exact size of the received data. Then you can receive requested amount of data.
A client will look like this:
import struct
# Receive a header
header = connection.recv(8)
(length,) = struct.unpack('>Q', header) # Parse payload length
# Receive the payload
payload = b''
while len(payload) < length:
to_read = length - len(payload)
payload += connection.recv(4096 if to_read > 4096 else to_read)
Server:
import struct
with open('OTC_trade_records.csv',newline='') as f:
connectionSocket, addr = self.client
trades = f.read()
length = struct.pack('>Q', len(trades))
connectionSocket.sendall(length)
connectionSocket.sendall(trades)
I am trying to implement a reliable UDP messaging scheme between a single client and a server.
In my current code I can send incrementing numbers by 1 to the server if he only uses the receive command. if the server tries replying with the received data using the send command, it seems to work for 1-3 messages back and forth, and then I enter a deadlock. I do not understand where the source of the deadlock comes from. Below is my implementation of send and receive.
Both the client and server start with their self.seqnumber set to 0 and both sockets are set to timeout after 1 second. Both the client and server share these methods as they belong to a class both the client and server import.
def sendCommand(self, command):
self.s.sendto((self.seqnumber).to_bytes(8, "little") + command, self.address)
try:
data, self.address = self.s.recvfrom(1200)
if int.from_bytes(data[:8], 'little') == self.seqnumber:
self.seqnumber += 1
return 0
else:
return self.sendCommand(command)
except:
return self.sendCommand(command)
def getCommand(self):
while(1):
try:
data, self.address = self.s.recvfrom(1200)
if int.from_bytes(data[:8], 'little') == self.seqnumber:
self.s.sendto(data, self.address)
self.seqnumber += 1
break
elif int.from_bytes(data[:8], 'little') < self.seqnumber:
self.s.sendto(data, self.address)
else:
continue
except:
continue
return data[8:]
The code running on the server (commInf is the class in which the get and send command are defined):
while (1):
command = self.commInf.getCommand()
print(command.decode())
self.commInf.sendCommand(command)
and the code running on the client:
for i in range(100):
self.commInf.sendCommand(f"{i}".encode())
command = self.commInf.getCommand()
print(command.decode())
I expect the output to allow me to reliably send messages and return them using the sendCommand(with the received data) since the returned data from getCommand does not include its sequence number and is just the raw data.
I'm fairly new to Python and I'm writing a script that sends a Skype message when someone sends you a message (And it changes the message dependent on whether a process exists or not.) and I'm struggling to get python to recognize a received message. I've seen another post on this but I'm struggling to put this into the if statement.
The code is included below in-case you need it.
import Skype4Py
import sys
import os
while 0 == 0: #infinite loop
if """user sends message to me""":
if #process is open (Still haven't figured this out, either!)
print("Process Found")
client = Skype4Py.Skype()
client.Attach()
user = ChatMessage.Sender #may need changing
message = 'SERVER ONLINE'
client.SendMessage(user, message)
print("Message sent to", ChatMessage.Sender)
else:
print("Process Non-Existant")
client = Skype4Py.Skype()
client.Attach()
user = ChatMessage.Sender #same as above
message = 'SERVER OFFLINE'
client.SendMessage(user, message)
print("Message sent to", ChatMessage.Sender)
else:
print("Awaiting Message")
else:
print("Physics is broken.")
That's how far I got.
Thanks - James Whybrow
I send data to socket on one side every second, but I can read that data on another side in any moment. Here's the writer:
from settings import Config
filename = Config.NAVIGATION_SOCKET_FILE
client = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
client.settimeout(None)
while True:
try:
client.connect(filename)
break
except Exception:
continue
messages = ["$GPRMC,125504.049,A,5542.2389,N,03741.6063,E,0.06,25.82,200906,,,*17",
"$GPRMC,155604.049,A,5542.2389,N,03741.6063,E,0.06,25.82,200906,,,*19",]
while True:
msg = random.choice(messages)
client.send(msg)
print msg
time.sleep(1)
And here's reader:
navigation_socket = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
if os.path.exists(app.config['NAVIGATION_SOCKET_FILE']):
os.remove(app.config['NAVIGATION_SOCKET_FILE'])
navigation_socket.bind(app.config['NAVIGATION_SOCKET_FILE'])
class NavigationInfo(restful.Resource):
def get(self):
msg = navigation_socket.recv(1024)
regex = re.compile(r"^\$GPRMC,(?P<time>\d{6}\.\d{3}),(?P<status>A|V),"
r"(?P<latitude>\d{4}\.\d{4}),(?P<lat_n_s>N|S),"
r"(?P<longitude>\d{5}\.\d{4}),(?P<long_e_w>E|W),"
r"(?P<hor_speed>\d+.\d+),(?P<track_angle>\d+.\d+),"
r"(?P<date>\d{6}),(?P<magnetic_declination>\d+\.\d+)?,"
r"(?P<magnetic_decl_direction>\d)?,"
r"(?P<mode>A|D|E|N)?\*(?P<checksum>\d\d)")
result = regex.match(msg)
navigation_info = result.groupdict()
return navigation_info
So the first problem is that writer just stops writing data to socket when buffer is full (at least that's what I see) and when I request data on the other side, it's too old.
Can just I store one value in buffer and then rewrite it? Or maybe I'm getting it all wrong?
I think that you are using the solution in reverse.
Instead of pushing messaging, while not pulling messages ?
Your server may look like:
Wait for a connection
Give a random message
go to step 1
And your client may just connect to the server when he needs a message.
In your case, the connection is "opened all the time", in my solution the socket is opened only when needed. and closed right after the message is delivered.
I have two programs, sendfile.py and recvfile.py that are supposed to interact to send a file across the network. They communicate over TCP sockets. The communication is supposed to go something like this:
sender =====filename=====> receiver
sender <===== 'ok' ======= receiver
or
sender <===== 'no' ======= receiver
if ok:
sender ====== file ======> receiver
I've got
The sender and receiver code is here:
Sender:
import sys
from jmm_sockets import *
if len(sys.argv) != 4:
print "Usage:", sys.argv[0], "<host> <port> <filename>"
sys.exit(1)
s = getClientSocket(sys.argv[1], int(sys.argv[2]))
try:
f = open(sys.argv[3])
except IOError, msg:
print "couldn't open file"
sys.exit(1)
# send filename
s.send(sys.argv[3])
# receive 'ok'
buffer = None
response = str()
while 1:
buffer = s.recv(1)
if buffer == '':
break
else:
response = response + buffer
if response == 'ok':
print 'receiver acknowledged receipt of filename'
# send file
s.send(f.read())
elif response == 'no':
print "receiver doesn't want the file"
# cleanup
f.close()
s.close()
Receiver:
from jmm_sockets import *
s = getServerSocket(None, 16001)
conn, addr = s.accept()
buffer = None
filename = str()
# receive filename
while 1:
buffer = conn.recv(1)
if buffer == '':
break
else:
filename = filename + buffer
print "sender wants to send", filename, "is that ok?"
user_choice = raw_input("ok/no: ")
if user_choice == 'ok':
# send ok
conn.send('ok')
#receive file
data = str()
while 1:
buffer = conn.recv(1)
if buffer=='':
break
else:
data = data + buffer
print data
else:
conn.send('no')
conn.close()
I'm sure I'm missing something here in the sorts of a deadlock, but don't know what it is.
With blocking sockets, which are the default and I assume are what you're using (can't be sure since you're using a mysterious module jmm_sockets), the recv method is blocking -- it will not return an empty string when it has "nothing more to return for the moment", as you seem to assume.
You could work around this, for example, by sending an explicit terminator character (that must never occur within a filename), e.g. '\xff', after the actual string you want to send, and waiting for it at the other end as the indication that all the string has now been received.
TCP is a streaming protocol. It has no concept of message boundaries. For a blocking socket, recv(n) will return a zero-length string only when the sender has closed the socket or explicitly called shutdown(SHUT_WR). Otherwise it can return a string from one to n bytes in length, and will block until it has at least one byte to return.
It is up to you to design a protocol to determine when you have a complete message. A few ways are:
Use a fixed-length message.
Send a fixed-length message indicating the total message length, followed by the variable portion of the message.
Send the message, followed by a unique termination message that will never occur in the message.
Another issue you may face is that send() is not guaranteed to send all the data. The return value indicates how many bytes were actually sent, and it is the sender's responsibility to keep calling send with the remaining message bytes until they are all sent. You may rather use the sendall() method.