Right now we are working on a multiplayer game with a client in python and a server in C++.
We send the size of the header, a header with an object Id and the object. When parsing, we read the size of the header, parse the header and then start parsing the object it indicates. Everything works fine until I try to send a response from C++ to the python client, then _DecodeVarInt32 reads only a byte and determines that the size of the header is 0.
We suspect its because of the int encoding, but we haven't found any material regarding this issue and protocol buffers
Socket send on C++
int sz = header.size()+4+obj.size();
char* pkg = new char[sz];
google::protobuf::io::ArrayOutputStream aos(pkg,sz);
CodedOutputStream *cos= new CodedOutputStream(&aos);
cos->WriteVarint32(header.size());
cos->WriteString(header.data());
cos->WriteString(obj.data());
int rc = sendto(sd,(void*)pkg,sz,0,&sock.sa,sock.sa_len);
Socket receive in C++
google::protobuf::uint32 sz;
google::protobuf::io::ArrayInputStream ais(buffer,MAX_MESSAGE_SIZE);
CodedInputStream cis(&ais);
cis.ReadVarint32(&sz);
auto lim = cis.PushLimit(sz);
sample.from_bin(cis);
cis.PopLimit(lim);
obj.from_bin(cis);
Socket send on python
header = Header(headerId)
header.to_bin()
object.to_bin()
headerSize = _VarintBytes(header._size)
print(len(headerSize))
print(header._size)
print(object._size)
btarray = headerSize + header._data + object._data
print(len(btarray))
self.sock.sendto(btarray,(self.host,self.port))
And socket receive in python
def recvHeader(self):
buf = self.sock.recv(2000,socket.MSG_PEEK)
print(buf)
rc = Header(serialMsg.MessageID.LOGINPETITION)
n = 0
hdSize, n =_DecodeVarint32(buf,n)
rc.from_bin(buf[n:n+hdSize])
n+=hdSize
return (rc,n)
def loadObject(self,object,startPos):
print('start load')
buf = self.sock.recv(2000)
object.from_bin(buf[startPos:])
print('loaded')
Related
I am trying to send a file from python client to a c# server and present it on screen by saving it first and then showing it on my MainWindow.
I came across a couple of problems I can't figure out why happen (I'm new to C#)
I followed this guide : http://snippetbank.blogspot.com/2014/04/csharp-client-server-file-transfer-example-1.html
The problems were :
1. The file was not being saved to my folder.
2. when I used message box to try and detect if it passes all the info it looks like it gets stuck in the middle .
I've been stuck on this for quite some time now but can't figure out what I'm missing
Python code :
def send_file(conn, name):
try:
full_path = "Output/"+name
file_to_send = open(full_path, "rb")
size = os.path.getsize(full_path)
file_name = name + "\n"
size_to_send = str(size) + "\n"
conn.send(size_to_send.encode())
conn.send(file_name.encode())
while size > 0:
data = file_to_send.read(1024)
conn.send(data)
size -= len(data)
file_to_send.close()
return conn
except IOError:
print("FILE IS ALREADY OPEN")
C# CODE :
public static string ReceiveFile(StreamReader reader, TcpClient tcpClient)
{
string folder = #"C:\Users\Roy\Desktop\GUI243\GUI243\";
// The first message from the client is the file size
string cmdFileSize = reader.ReadLine();
MessageBox.Show(cmdFileSize);
// The first message from the client is the filename
string cmdFileName = reader.ReadLine();
MessageBox.Show(cmdFileName);
string full_path = folder + cmdFileName;
int length = Convert.ToInt32(cmdFileSize);
byte[] buffer = new byte[length];
int received = 0;
int read = 0;
int size = 1024;
int remaining = 0;
// Read bytes from the client using the length sent from the client
while (received < length)
{
remaining = length - received;
if (remaining < size)
{
size = remaining;
}
read = tcpClient.GetStream().Read(buffer, received, size);
if (read == 0)
{
break;
}
received += read;
}
// Save the file using the filename sent by the client
using (FileStream fStream = new FileStream(Path.GetFileName(cmdFileName), FileMode.Create))
{
fStream.Write(buffer, 0, buffer.Length);
fStream.Flush();
fStream.Close();
}
return full_path;
}
In your C# code looks like a while cycle is missing, unless you call the function ReceiveFile() iterating somewhere else.
The class StreamReader needs to constantly check the tcp client socket to see if new data has been received because this is how a TCP stream works.
You don't know when the client will connect and send the data so you can't just call the ReceiveFile() function once.
In the example there is a perpetual while (true) cycle that makes that reader.Readline() work:
while (true)
{
// Accept a TcpClient
TcpClient tcpClient = tcpListener.AcceptTcpClient();
Console.WriteLine("Connected to client");
StreamReader reader = new StreamReader(tcpClient.GetStream());
// The first message from the client is the file size
string cmdFileSize = reader.ReadLine();
...
}
Do you have an equivalent in your code?
Also, is not pratical to use MessageBox for debugging purposes.
Try:
Debug.WriteLine ("Expected size is: " + cmdFileSize);
Note you need System.Diagnostics to use that.
Posting this here out of desperation. Any help is appreciated. Thank you.
Backstory:
I am helping my friend with a device that he got from China. The device supposedly sends a audio file to my server using UDP.
assuming you want some Python code to do this automatically, here's how I'd validate and decode the packet:
import struct
def decode_packet(packet):
framehead, version, command, datalen = struct.unpack_from('!HBBH', packet)
valid = (
framehead == 0x55aa and
version == 0x00 and
command == 0x1e and
len(packet) <= datalen + 11
)
if not valid:
# ignore other protocols using this address/port
print(
' header invalid',
f'{framehead:04x} {version:02x} {command:02x} {datalen:04x}'
)
return
if len(packet) < datalen + 11:
print(' warning: packet was truncated')
offset, = struct.unpack_from('!I', packet, 6)
if datalen == 4:
print(f' end of data: file size={offset}')
return
data = packet[10:10+datalen]
print(f' got data: offset={offset} len={len(data)} hex(data)={data.hex()}')
if len(packet) == datalen + 11:
print(f' hex(checksum)={packet[datalen + 10:].hex()}')
it obviously prints out a lot of stuff, but this is good to seeing if the device is actually following the documented protocol. it doesn't seem to be, as the +4 on the data length doesn't seem to be being applied. you can test this with:
decode_packet(bytes.fromhex('55aa001e038400000000a9b6ad98d2923...'))
assuming you can get this to function correctly, you can put this into some code that listens for packets on the correct port:
import socket
def server(portnum):
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
sock.bind(('', portnum))
while True:
packet, addr = sock.recvfrom(10240)
print(f'received {len(packet)} bytes from {addr[0]}')
decode_packet(packet)
again, doesn't do much. you'd want to write the data to a file rather than printing it out, but you can pull the offset out and you get a signal for when the data has finished transferring
For a networking project, I'm using UDP Multicast to build an overlay network with my own implementation of IP.
I use the following to parse and build my Header first, then append the payload:
def __init__(buffer_size_bytes):
self.__buffer = bytearray(buffer_size_bytes)
def read_sock(self, listening_socket):
n_bytes, addr = listening_socket.recvfrom_into(self.__buffer, Packet.HEADER_SIZE)
packet = Packet.parse_header(self.__buffer)
if packet.payload_length is not 0:
packet.payload = parse_payload(packet.payload_length, listening_socket)
self.__router.add_to_route_queue(packet, listening_socket.locator)
def parse_payload(to_read, socket):
payload = bytearray(to_read)
view = memoryview(payload)
while to_read:
n_bytes, addr = socket.recvfrom_into(view, to_read)
view = view[n_bytes:]
to_read -= n_bytes
return payload
The header seems to be parsed correctly, but the payload gets corrupted every time. I can't figure out what I'm doing wrong when parsing the payload, and I can confirm I'm sending a bytearray from the other side.
For example, when I send a packet with the payload "Hello World" encoded in utf-8, I receive the following:
b'`\x00\x00\x00\x00\x0b\x00\x1f\x00\x00\x00'
The Packet.parse_header method:
def parse_header(cls, packet_bytes):
values = struct.unpack(cls.ILNPv6_HEADER_FORMAT, packet_bytes[:cls.HEADER_SIZE])
flow_label = values[0] & 1048575
traffic_class = (values[0] >> 20 & 255)
version = values[0] >> 28
payload_length = values[1]
next_header = values[2]
hop_limit = values[3]
src = (values[4], values[5])
dest = (values[6], values[7])
return Packet(src, dest, next_header, hop_limit, version, traffic_class, flow_label, payload_length)
For reference, the entire sent packet looks like this:
b'`\x00\x00\x00\x00\x0b\x00\x1f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01Hello World'
On receiving the first packet, the socket.recvfrom_into blocks when reading for the payload, and doesn't return until I send another message. It then seems to discard the payload of the previous message and use the second packet received as the payload...
Found my explanation here.
So the key thing was that I'm using UDP. And UDP sockets discard anything that doesn't fit in the buffer you give it.
TCP sockets however behave more like the bytestream I was expecting.
Fun!
I am sending .jpg images from Qt client to python server. Qt client sends as follows:
void Sender::sendToServer(QByteArray imageData)
{
QDataStream stream(serverSocket);
int t = imageData.size();
stream << t;
stream << imageData;
serverSocket->waitForBytesWritten(-1);
qDebug() << "Frame size:" << imageData.size();
}
And the server is:
unpacker = struct.Struct('!i')
conn, addr = s.accept()
bool = True
data = b''
while True:
while len(data) < 4:
try:
data += conn.recv(100)
except:
break
img_size = unpacker.unpack(data[:4])[0]
print('size: ', img_size)
print('Data: ',data)
print('Image: ',data[:8]) # deleting preceding 8 bytes
# image processing
The output is:
//Client:
Frame size: 49993
//Server:
size: 49993
Data: b'\x00\x00\xc3\r\x00\x00\xc3\r\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01...
Image: b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00...
In the output, .jpg file starts from '\xff\xd8'. As C++ int is 4 byte, the server expects "{4 byte}\xff\xd8...". However, preceding bytes are always of different size. Sometimes it's 8 bytes, but sometimes even different (such as 6 or 9 bytes). I could not figure out the reason behind it. What is the problem here?
Platform: win7, Qt 5.9.1, Python 3.4.
Edit
Following #Frank's answer, I changed the code as follows and everything works fine:
Qt client send method:
void Sender::sendToServer(QByteArray imageData)
{
QDataStream stream(serverSocket);
QString size = QString::number(data.size());
QString size_8_byte = size.rightJustified(8, ' '); // add spaces to make it exactly 8 bytes
imageData.prepend(size_8_byte.toUtf8());
stream.writeRawData((char*)(imageData.data()), imageData.size());
serverSocket->waitForBytesWritten(-1);
}
Python server:
unpacker = struct.Struct('8s')
conn, addr = s.accept()
data = b''
bool = True
while True:
while len(data) < 8:
try:
data += conn.recv(100)
except:
break
img_size = unpacker.unpack(data[:8])[0]
img_size = int(img_size.decode("utf-8").strip()) #remove spaces and convert to int
data = data[8:] #correct data
#image processing
QDataStream is not a simple binary dumper, it formats the data in some internally defined way. This helps with stuff like endianness correction and the like,
However, there is a very simple fix available to you: QDataStream provides the writeRawData() function, which bypasses all formatting operations.
Simply change your output operations to using it and you should be good to go.
I'm using the following script:
import socket
import struct
username = "username_value"
verification_key = "verification_key"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # boilerplate
s.connect(("example.com", 1234)) # adjust accordingly
# now for the packet
# note that the String type is specified as having a length of 64, we'll pad that
packet = ""
packet += struct.pack("B", 1) # packet type
packet += struct.pack("B", 7) # protocol version
packet += "%-64s" % username # magic!
packet += "%-64s" % verification_key
packet += struct.pack("B", 0) # that unused byte, assuming a NULL byte here
# send what we've crafted
s.send(packet)
and getting a response of:
packet += struct.pack("B", 1) # packet type
TypeError: Can't convert 'bytes' object to str implicitly
I am almost brand-new to Python, and just started, but I understand the language. I read up and found something about Python 3 changing the way you use packets. I feel kind of hopeless. Help? Thank you
In python 3 you have to implicitly define your string packet as a bytes
packet = b""
instead of packet = ""