Develop Reference

Generating a UDP message in python with a header and payload in python3

I am new to Networking and trying to implement a network calculator using python3 where the client's responsibility is to send operands and operators and the server will calculate the result and send it back to the client. Communication is through UDP messages and I am working on client side. Each message is comprised of a header and a payload and they are described as shown in the below figures.
UDP header:
I am familiar with sending string messages using sockets but having a hard-time with how to make a message with both header and payload and how to assign the bits for various attributes or how to generate message/client id's in the header and If there is any way to automatically generate the Id's. Any help or suggestions will be highly appreciated.
Thanks in advance

I will only do a portion of your homework.
I hope it will help you to find energy to work on missing parts.
import struct
import socket
CPROTO_ECODE_REQUEST, CPROTO_ECODE_SUCCESS, CPROTO_ECODE_FAIL = (0,1,2)
ver = 1 # version of protocol
mid = 0 # initial value
cid = 99 # client Id (arbitrary)
sock = socket.socket( ...) # to be customized
def sendRecv( num1, op, num2):
global mid
ocs = ("+", "-", "*", "/").index( op)
byte0 = ver + (ocs << 3) + (CPROTO_ECODE_REQUEST << 6)
hdr = struct.pack( "!BBH", byte0, mid, cid)
parts1 = (b'0000' + num1.encode() + b'0000').split(b'.')
parts2 = (b'0000' + num2.encode() + b'0000').split(b'.')
msg = hdr + parts1[0][-4:] + parts1[1][:4] + parts2[0][-4:] + parts2[1][:4]
socket.send( msg) # send request
bufr = socket.recv( 512) # get answer
# to do:
# complete socket_send and socket.recv
# unpack bufr into: verr,ecr,opr,value_i, value_f
# verify that verr, ecr, opr, are appropriate
# combine value_i and value_f into answer
mid += 1
return answer
result = sendRecv( '2.47', '+', '46.234')
There are many elements that haven't be specified by your teacher:
what should be the byte-ordering on the network (bigEndian or littleEndian)? The above example suppose it's bigEndian but you can easily modify the 'pack' statement to use littleEndian.
What should the program do if the received packet header is invalid?
What should the program do if there's no answer from server?
Payload: how should we interpret "4 most significant digits of fraction"? Does that mean that the value is in ASCII? That's not specified.
Payload: assuming the fraction is in ASCII, should it be right-justified or left-justified in the packet?
Payload: same question for integer portion.
Payload: if the values are in binary, are they signed or unsigned. It will have an affect on the unpacking statement.
In the program above, I assumed that:
values are positive and in ASCII (without sign)
integer portion is right-justified
fractional portion is left justified
Have fun!

Best way to check checksum of bytes in python

I receive some packets from serial port. Packet example:
last 2 bytes
/\
[ data length ] [ data ] [packet crc]
I get crc, for example, b'w\x06'. Value is 125 (sum(b'w\x06') = 125). I calculate crc of packet: sum(data). And I get 1655. I know that 1655 is the same as b'w\06', but I do not know, how to translate it simply to 125 and compare with right checksum. What I need to convert 1655 to the same value as received packet bytes (125 or b'w\x06')? binascii/struct/something else?
Thanks

You can use int.from_bytes:
int.from_bytes(b'w\x06', 'little')
or with struct:
struct.unpack("<H", b'w\x06')[0]

Using Scapy to send Fragment Packets with random Offsets

I would like to send fragmented packets size of 8 bytes and a random starting offset. Also want to leave out the last fragmented packet.
So far I got everything except the fragment of
from scapy.all import *
from random import randint
dip="MY.IP.ADD.RESS"
payload="A"*250+"B"*500
packet=IP(dst=dip,id=12345,off=123)/UDP(sport=1500,dport=1501)/payload
frags=fragment(packet,fragsize=8)
print(packet.show())
for f in frags:
send(f)
What does the above code do?
It sends IP Fragment Packets size of 8 byte to a destination IP address.
I would like to send IP Fragment Packets with a random Frag Offset.
I can't find anything about fragment() and the only field, I was able to edit was in IP packet instead of each fragmented IP packet.
Does someone have an idea to accomplish this?
Infos: Python2.7, latest version of scapy (pip)

If you want to generate "broken" fragment offset fields, you have to do that yourself. The scapy fragment() function is simple enough:
def fragment(pkt, fragsize=1480):
"""Fragment a big IP datagram"""
fragsize = (fragsize + 7) // 8 * 8
lst = []
for p in pkt:
s = raw(p[IP].payload)
nb = (len(s) + fragsize - 1) // fragsize
for i in range(nb):
q = p.copy()
del(q[IP].payload)
del(q[IP].chksum)
del(q[IP].len)
if i != nb - 1:
q[IP].flags |= 1
q[IP].frag += i * fragsize // 8 # <---- CHANGE THIS
r = conf.raw_layer(load=s[i * fragsize:(i + 1) * fragsize])
r.overload_fields = p[IP].payload.overload_fields.copy()
q.add_payload(r)
lst.append(q)
return lst
Source: https://github.com/secdev/scapy/blob/652b77bf12499451b47609b89abc663aa0f69c55/scapy/layers/inet.py#L891
If you change the marked code line above, you can set the fragment offset to whatever you want.

parsing a text file for integers on a given line with python

I've got this test suite that outputs test results as a text file.
Here is a sample:
File Opened: Tuesday, June 26, 2016, 10:17:13 AM
File Opened: Tuesday, June 26, 2016, 10:17:29 AM
Radio Test BER LOOP BACK successful
Radio Test PAUSE successful
Radio Test BER LOOP BACK successful
File Opened: Tuesday, June 28, 2016, 10:18:11 AM
Bits received 10152
Bits in error 117
Access code bit errors 0
Packets received 49
Packets expected 2707
Packets w/ header error 0
Packets w/ CRC error 0
Packets w/ uncorr errors 0
Sync timeouts 3
==================================
Bits received 10368
Bits in error 85
Access code bit errors 0
Packets received 52
Packets expected 2758
Packets w/ header error 0
Packets w/ CRC error 0
Packets w/ uncorr errors 0
Sync timeouts 1
==================================
Bits received 10152
Bits in error 93
Access code bit errors 0
Packets received 49
Packets expected 2707
Packets w/ header error 0
Packets w/ CRC error 0
Packets w/ uncorr errors 0
Sync timeouts 3
I'm trying to extract the number after Bits received and Bits in error, and divide them to get a percentage.
Then, I'd like to plot those as a scatter plot with matplotlib.pyplot.
I'm having a hard time getting those numbers out of this file, however...I'm messing up something with how I'm parsing this.
Either way, I'm just feeling my way through this, and I'm sure I'm not doing this the most elegant way possible. This seems like such a simple task for Python and I'm surely making it much harder than it needs to be.
How would you handle this?
Thanks

Create two arrays, once for the received data and one for the error data, then just loop through the file and parse:
receivedData = []
errorData = []
with open("data.txt") as f:
for line in f:
if line.startswith("Bits received"):
receivedData.append(int(line.split()[-1]))
elif line.startswith("Bits in error"):
errorData.append(int(line.split()[-1]))
else:
#do normal stuff with other lines
pass

Another easy way would be to use the regex library re. (https://docs.python.org/2/library/re.html)
import re
pattern1 = re.compile(r'Bits received\s+(\d+)') # \d means any digit character
pattern2 = re.compile(r'Bits in error\s+(\d+)')
with open('path/file.txt', 'r') as f:
text = f.read()
received = int(pattern1.match(text).group(1))
in_error = int(pattern2.match(text).group(1))
value_of_interest = in_error/received
This approach assumes that every input file has those two lines. If that assumption can't be made, break up the matches to check for their presence:
match1 = pattern1.match(text) # re.MatchObject if the pattern is found
if match1: # None if it's not found
received = int(match1.group(1)) # re.MatchObject.group(1) is the first parenthesized group
match2 = pattern2.match(text)
if match2:
in_error = int(match2.group(1))

Ack number to acknowledge data in scapy

I'm playing around with scapy, trying to do a nmap-like, I succeeded doing a three way handshake and I automatically receive data if I connect to a smtp server but I can't acknowledge it.
I have connected to it via netcat to see why it wouldn't work and nc actually send an ACK packet with the TCP ack field set to the length of the data segment received + 1.
So I am trying to do the same with scapy but I don't know where to find the length of a TCP segment. I tried a lsc(TCP) but I can not see the field.
Here is a wireshark capture of the netcat connection :
And the detail of the packet :
As you can see, the ack field of the packet just below the data is 37 + 1 = 38.
Does someone knows where or how it can be found ?
EDIT:
There is actually no field that gives the segment length but it can be calculated with two methods:
tcp_seg_len = len(rp.getlayer(Raw).load)
# or
ip_total_len = rp.getlayer(IP).len
ip_header_len = rp.getlayer(IP).ihl * 32 / 8
tcp_header_len = rp.getlayer(TCP).dataofs * 32 / 8
tcp_seg_len = ip_total_len - ip_header_len - tcp_header_len
ans_ack,unans_ack = sr(IP(dst=ip)/TCP(sport=pkt[1].dport, \
dport=pkt[1].sport, \
seq=rp[1].ack, \
ack=tcp_seg_len + 1, \
flags="A"), \
verbose=0, timeout=1)
However, when I print the values I get the good ack number but when I watch the wireshark capture, I have a huge ack number and the packet is marked as "ACKed unseen segment". When I look at the bytes in hexa, I got the right value though. Does anyone knows why that happens and how to solve it?

Wireshark uses relative numbers in its display so the right scapy line is :
ans_ack,unans_ack = sr(IP(dst=ip)/TCP(sport=pkt[1].dport, \
dport=pkt[1].sport, \
seq=rp[1].ack, \
ack=rp[1].seq + tcp_seg_len, \
flags="A"), \
verbose=0, timeout=1)
And the following code (see EDIT in the question) does work :
ip_total_len = rp.getlayer(IP).len
ip_header_len = rp.getlayer(IP).ihl * 32 / 8
tcp_header_len = rp.getlayer(TCP).dataofs * 32 / 8
tcp_seg_len = ip_total_len - ip_header_len - tcp_header_len
ans_ack,unans_ack = sr(IP(dst=ip)/TCP(sport=pkt[1].dport, \
dport=pkt[1].sport, \
seq=rp[1].ack, \
ack=rp[1].seq + tcp_seg_len, \
flags="A"), \
verbose=0, timeout=1)

To get the TCP Segment Length (This has been verified for IPv6 traffic):
tcp_header_len = rp.getlayer(TCP).dataofs * 32 / 8
tcp_seg_len = rp.plen - tcp_header_len