I'm using the following script:
import socket
import struct
username = "username_value"
verification_key = "verification_key"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # boilerplate
s.connect(("example.com", 1234)) # adjust accordingly
# now for the packet
# note that the String type is specified as having a length of 64, we'll pad that
packet = ""
packet += struct.pack("B", 1) # packet type
packet += struct.pack("B", 7) # protocol version
packet += "%-64s" % username # magic!
packet += "%-64s" % verification_key
packet += struct.pack("B", 0) # that unused byte, assuming a NULL byte here
# send what we've crafted
s.send(packet)
and getting a response of:
packet += struct.pack("B", 1) # packet type
TypeError: Can't convert 'bytes' object to str implicitly
I am almost brand-new to Python, and just started, but I understand the language. I read up and found something about Python 3 changing the way you use packets. I feel kind of hopeless. Help? Thank you
In python 3 you have to implicitly define your string packet as a bytes
packet = b""
instead of packet = ""
Related
I'm using the socket module for a UDP server。The incoming packets always have a different size(0-65535), so client send package length first, then send the package；server receives data according to the package length and 1024 per server receives。but server didn't work like I thought，it can receives package only once。
# -*- coding: utf-8 -*-
import json
import socket
ip_port = ('127.0.0.1', 8080)
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind(ip_port)
print('server listen......')
def recv_header(s_socket):
try:
msg, addr = s_socket.recvfrom(32)
return json.loads(msg), addr
except Exception as e:
return str(e), None
while True:
header_msg, client_addr = recv_header(s_socket=server)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
receiverBufsize = 1024
if client_addr is None:
print (header_msg)
continue
data_body = ""
package_len = int(header_msg['length'])
print ("package_len: {}".format(package_len))
print ("client_addr: {}".format(str(client_addr)))
while package_len > 0:
if package_len > receiverBufsize:
body_part = server.recvfrom(receiverBufsize)
else:
body_part = server.recvfrom(package_len)
data_body += body_part[0]
package_len -= receiverBufsize
print ("data_body: {}".format(data_body))
# -*- coding: utf-8 -*-
import json
import socket
import random
ip_port = ('127.0.0.1', 8080)
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def get_data(length):
base_str = 'ABCDEFGHIGKLMNOPQRSTUVWXYZabcdefghigklmnopqrstuvwxyz0123456789'
return ''.join([random.choice(base_str) for _ in range(length)])
while 1:
msg = raw_input('>>>：')
if msg == 'q':
break
if not msg:
continue
len = int(msg)
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
data = get_data(len)
header = json.dumps({"length": len})
aa = client.sendto(header.encode('utf-8'), ip_port)
print aa
aa = client.sendto(data.encode('utf-8'), ip_port)
print aa
If the packet length is less than 1024(my receiverBufsize), it works normally
enter image description here
enter image description here
If the packet length is more than 1024(my receiverBufsize), socket.recvfrom will block
enter image description here
enter image description here
UDP is a message oriented protocol, not a data stream like TCP. This means in UDP a single send is matched by a single recv.
This means you cannot use multiple recv with a small buffer (size 1024 in your case) to get data which were sent in a single send with a larger buffer - you need to use recv with a large enough buffer instead.
This means also that you don't actually need to send the length upfront, since you don't need to add message semantics to a protocol which already has message semantic. In contrary, sending the length in a separate message as you do could actually be harmful since UDP is an unreliable protocol and you cannot rely on packets getting delivered in order, at all or only once.
I'm trying to send a json containing text fields and a buffer in a bytearray, from a micro-controller to a Windows server
msg = {"some_stuff": "some_stuff", "buf": bytearray(b'\xfe\xc2\xf1\xfe\xd5\xc0 ...')}
Note that the buffer is quite long (so that I can't put it here as reference) len(buf) -> 35973
I'm sending the length of the message before to the server so that it knows how long is the message to be received
def send_json(conn, msg):
msg = json.dumps(msg).encode('utf-8')
msg_length = len(msg)
header = str(msg_length).encode('utf-8')
header += b' ' * (64 - len(header))
conn.send(header)
conn.send(msg)
The receiving function is then
def receive_json(conn) -> dict:
msg_length = int(
conn.recv(64).decode('utf-8').replace(' ', '')
)
msg_b = conn.recv(msg_length)
msg_s = msg_b.decode('utf-8')
try:
msg_d = json.loads(msg_s)
except:
msg_d = eval(msg_s)
return msg_d
The problem is that the received message is truncated.
msg_b = b'{"buf": bytearray(b\'\\xfe\\xc2\\xf1 ... \\x06u\\xd0\\xff\\xb'
It's worth mentioning that while in debug, if I stop for a while with a breakpoint on line msg_b = conn.recv(msg_length), before running it, the received message is complete.
So it seems that in the receiving function the conn.recv(msg_length) instruction does not wait to receive a message of the specified length (msg_length)
Why is it the case? What can I do to receive a complete message?
I could introduce time.sleep between receiving the length of the message and the message, but how to know how much to wait depending on the message length?
Thank you
My solution was to check for how much of the message is missing and iterate till the message is complete
def receive_json(conn) -> dict:
msg_length = int(
conn.recv(64).decode('utf-8').replace(' ', '')
)
buf = bytearray(b'')
while len(buf) < msg_length:
missing_length = msg_length - len(buf)
packet = conn.recv(missing_length)
buf.extend(packet)
msg_s = buf.decode('utf-8')
try:
msg_d = json.loads(msg_s)
except:
msg_d = eval(msg_s)
return msg_d
TCP is a streaming protocol that guarantees delivery of bytes in the order sent, but not with the same send breaks. You need to define a protocol (which you have, as a 64-byte header of message size, then the message data), and then buffer reads until you have a complete message.
Python sockets have a .makefile method that handles the buffering for you, where you can .read(n) a specific number of bytes or .readline() to read a newline-terminated line. With this you can implement the following client and server:
server.py
import socket
import json
import time
s = socket.socket()
s.bind(('',5000))
s.listen()
while True:
c,a = s.accept()
print(f'{a} connected')
# wrap socket in a file-like buffer
with c, c.makefile('rb') as r: # read binary so .read(n) gets n bytes
while True:
header = r.readline() # read header up to a newline
if not header: break # if empty string, client closed connection
size = int(header)
data = json.loads(r.read(size)) # read exactly "size" bytes and decode JSON
print(f'{a}: {data}')
print(f'{a} disconnected')
client.py
import socket
import json
def send_json(conn, msg):
# smaller data size if non-ASCII used.
data = json.dumps(msg, ensure_ascii=False).encode()
msg_length = len(data) # length in encoded bytes
# send newline-terminated header, then data
conn.sendall(f'{msg_length}\n'.encode())
conn.sendall(data)
s = socket.socket()
s.connect(('localhost',5000))
with s:
send_json(s, {'name':'马克'}) # check to support non-ASCII properly
send_json(s, [1,2,3])
Start server.py, then run client.py a couple of times:
Output:
('127.0.0.1', 26013) connected
('127.0.0.1', 26013): {'name': '马克'}
('127.0.0.1', 26013): [1, 2, 3]
('127.0.0.1', 26013) disconnected
('127.0.0.1', 26015) connected
('127.0.0.1', 26015): {'name': '马克'}
('127.0.0.1', 26015): [1, 2, 3]
('127.0.0.1', 26015) disconnected
Currently trying to make handshake process on python using raw sockets but for some reason I can't send any packet with TCP protocol receiving OSError: [WinError 10022] An invalid argument was supplied. Here is my code:
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_TCP)
s.sendto(packet, ('8.8.8.8', 80))
As packet I have tried to use scapy's TCP packet, TCP bytes from wireshark after successful sendings with other libraries and also hand-made byte strings:
def chksum(packet: bytes) -> int:
if len(packet) % 2 != 0:
packet += b'\0'
res = sum(array.array("H", packet))
res = (res >> 16) + (res & 0xffff)
res += res >> 16
return (~res) & 0xffff
class TCPPacket:
def __init__(self,
src_host: str,
src_port: int,
dst_host: str,
dst_port: int,
flags: int = 0):
self.src_host = src_host
self.src_port = src_port
self.dst_host = dst_host
self.dst_port = dst_port
self.flags = flags
def build(self) -> bytes:
packet = struct.pack(
'!HHIIBBHHH',
self.src_port, # Source Port
self.dst_port, # Destination Port
0, # Sequence Number
0, # Acknoledgement Number
5 << 4, # Data Offset
self.flags, # Flags
8192, # Window
0, # Checksum (initial value)
0 # Urgent pointer
)
pseudo_hdr = struct.pack(
'!4s4sHH',
socket.inet_aton(self.src_host), # Source Address
socket.inet_aton(self.dst_host), # Destination Address
socket.IPPROTO_TCP, # PTCL
len(packet) # TCP Length
)
checksum = chksum(pseudo_hdr + packet)
packet = packet[:16] + struct.pack('H', checksum) + packet[18:]
return packet
So literally no idea why my socket doesn't like any packet
I found out what is wrong. Windows doesn't allow to send TCP packets with raw sockets so this code will never work. Probably it is possible to write the same with scapy or using other libraries but that's not what I need so the only way to make it work is to run on linux. Still not sure if the packet creation is correct but TCP protocol with raw sockets sure works fine on linux.
Posting this here out of desperation. Any help is appreciated. Thank you.
Backstory:
I am helping my friend with a device that he got from China. The device supposedly sends a audio file to my server using UDP.
assuming you want some Python code to do this automatically, here's how I'd validate and decode the packet:
import struct
def decode_packet(packet):
framehead, version, command, datalen = struct.unpack_from('!HBBH', packet)
valid = (
framehead == 0x55aa and
version == 0x00 and
command == 0x1e and
len(packet) <= datalen + 11
)
if not valid:
# ignore other protocols using this address/port
print(
' header invalid',
f'{framehead:04x} {version:02x} {command:02x} {datalen:04x}'
)
return
if len(packet) < datalen + 11:
print(' warning: packet was truncated')
offset, = struct.unpack_from('!I', packet, 6)
if datalen == 4:
print(f' end of data: file size={offset}')
return
data = packet[10:10+datalen]
print(f' got data: offset={offset} len={len(data)} hex(data)={data.hex()}')
if len(packet) == datalen + 11:
print(f' hex(checksum)={packet[datalen + 10:].hex()}')
it obviously prints out a lot of stuff, but this is good to seeing if the device is actually following the documented protocol. it doesn't seem to be, as the +4 on the data length doesn't seem to be being applied. you can test this with:
decode_packet(bytes.fromhex('55aa001e038400000000a9b6ad98d2923...'))
assuming you can get this to function correctly, you can put this into some code that listens for packets on the correct port:
import socket
def server(portnum):
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
sock.bind(('', portnum))
while True:
packet, addr = sock.recvfrom(10240)
print(f'received {len(packet)} bytes from {addr[0]}')
decode_packet(packet)
again, doesn't do much. you'd want to write the data to a file rather than printing it out, but you can pull the offset out and you get a signal for when the data has finished transferring
For a networking project, I'm using UDP Multicast to build an overlay network with my own implementation of IP.
I use the following to parse and build my Header first, then append the payload:
def __init__(buffer_size_bytes):
self.__buffer = bytearray(buffer_size_bytes)
def read_sock(self, listening_socket):
n_bytes, addr = listening_socket.recvfrom_into(self.__buffer, Packet.HEADER_SIZE)
packet = Packet.parse_header(self.__buffer)
if packet.payload_length is not 0:
packet.payload = parse_payload(packet.payload_length, listening_socket)
self.__router.add_to_route_queue(packet, listening_socket.locator)
def parse_payload(to_read, socket):
payload = bytearray(to_read)
view = memoryview(payload)
while to_read:
n_bytes, addr = socket.recvfrom_into(view, to_read)
view = view[n_bytes:]
to_read -= n_bytes
return payload
The header seems to be parsed correctly, but the payload gets corrupted every time. I can't figure out what I'm doing wrong when parsing the payload, and I can confirm I'm sending a bytearray from the other side.
For example, when I send a packet with the payload "Hello World" encoded in utf-8, I receive the following:
b'`\x00\x00\x00\x00\x0b\x00\x1f\x00\x00\x00'
The Packet.parse_header method:
def parse_header(cls, packet_bytes):
values = struct.unpack(cls.ILNPv6_HEADER_FORMAT, packet_bytes[:cls.HEADER_SIZE])
flow_label = values[0] & 1048575
traffic_class = (values[0] >> 20 & 255)
version = values[0] >> 28
payload_length = values[1]
next_header = values[2]
hop_limit = values[3]
src = (values[4], values[5])
dest = (values[6], values[7])
return Packet(src, dest, next_header, hop_limit, version, traffic_class, flow_label, payload_length)
For reference, the entire sent packet looks like this:
b'`\x00\x00\x00\x00\x0b\x00\x1f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01Hello World'
On receiving the first packet, the socket.recvfrom_into blocks when reading for the payload, and doesn't return until I send another message. It then seems to discard the payload of the previous message and use the second packet received as the payload...
Found my explanation here.
So the key thing was that I'm using UDP. And UDP sockets discard anything that doesn't fit in the buffer you give it.
TCP sockets however behave more like the bytestream I was expecting.
Fun!