Develop Reference

Generating a UDP message in python with a header and payload in python3

I am new to Networking and trying to implement a network calculator using python3 where the client's responsibility is to send operands and operators and the server will calculate the result and send it back to the client. Communication is through UDP messages and I am working on client side. Each message is comprised of a header and a payload and they are described as shown in the below figures.
UDP header:
I am familiar with sending string messages using sockets but having a hard-time with how to make a message with both header and payload and how to assign the bits for various attributes or how to generate message/client id's in the header and If there is any way to automatically generate the Id's. Any help or suggestions will be highly appreciated.
Thanks in advance

I will only do a portion of your homework.
I hope it will help you to find energy to work on missing parts.
import struct
import socket
CPROTO_ECODE_REQUEST, CPROTO_ECODE_SUCCESS, CPROTO_ECODE_FAIL = (0,1,2)
ver = 1 # version of protocol
mid = 0 # initial value
cid = 99 # client Id (arbitrary)
sock = socket.socket( ...) # to be customized
def sendRecv( num1, op, num2):
global mid
ocs = ("+", "-", "*", "/").index( op)
byte0 = ver + (ocs << 3) + (CPROTO_ECODE_REQUEST << 6)
hdr = struct.pack( "!BBH", byte0, mid, cid)
parts1 = (b'0000' + num1.encode() + b'0000').split(b'.')
parts2 = (b'0000' + num2.encode() + b'0000').split(b'.')
msg = hdr + parts1[0][-4:] + parts1[1][:4] + parts2[0][-4:] + parts2[1][:4]
socket.send( msg) # send request
bufr = socket.recv( 512) # get answer
# to do:
# complete socket_send and socket.recv
# unpack bufr into: verr,ecr,opr,value_i, value_f
# verify that verr, ecr, opr, are appropriate
# combine value_i and value_f into answer
mid += 1
return answer
result = sendRecv( '2.47', '+', '46.234')
There are many elements that haven't be specified by your teacher:
what should be the byte-ordering on the network (bigEndian or littleEndian)? The above example suppose it's bigEndian but you can easily modify the 'pack' statement to use littleEndian.
What should the program do if the received packet header is invalid?
What should the program do if there's no answer from server?
Payload: how should we interpret "4 most significant digits of fraction"? Does that mean that the value is in ASCII? That's not specified.
Payload: assuming the fraction is in ASCII, should it be right-justified or left-justified in the packet?
Payload: same question for integer portion.
Payload: if the values are in binary, are they signed or unsigned. It will have an affect on the unpacking statement.
In the program above, I assumed that:
values are positive and in ASCII (without sign)
integer portion is right-justified
fractional portion is left justified
Have fun!

Using Scapy to send Fragment Packets with random Offsets

I would like to send fragmented packets size of 8 bytes and a random starting offset. Also want to leave out the last fragmented packet.
So far I got everything except the fragment of
from scapy.all import *
from random import randint
dip="MY.IP.ADD.RESS"
payload="A"*250+"B"*500
packet=IP(dst=dip,id=12345,off=123)/UDP(sport=1500,dport=1501)/payload
frags=fragment(packet,fragsize=8)
print(packet.show())
for f in frags:
send(f)
What does the above code do?
It sends IP Fragment Packets size of 8 byte to a destination IP address.
I would like to send IP Fragment Packets with a random Frag Offset.
I can't find anything about fragment() and the only field, I was able to edit was in IP packet instead of each fragmented IP packet.
Does someone have an idea to accomplish this?
Infos: Python2.7, latest version of scapy (pip)

If you want to generate "broken" fragment offset fields, you have to do that yourself. The scapy fragment() function is simple enough:
def fragment(pkt, fragsize=1480):
"""Fragment a big IP datagram"""
fragsize = (fragsize + 7) // 8 * 8
lst = []
for p in pkt:
s = raw(p[IP].payload)
nb = (len(s) + fragsize - 1) // fragsize
for i in range(nb):
q = p.copy()
del(q[IP].payload)
del(q[IP].chksum)
del(q[IP].len)
if i != nb - 1:
q[IP].flags |= 1
q[IP].frag += i * fragsize // 8 # <---- CHANGE THIS
r = conf.raw_layer(load=s[i * fragsize:(i + 1) * fragsize])
r.overload_fields = p[IP].payload.overload_fields.copy()
q.add_payload(r)
lst.append(q)
return lst
Source: https://github.com/secdev/scapy/blob/652b77bf12499451b47609b89abc663aa0f69c55/scapy/layers/inet.py#L891
If you change the marked code line above, you can set the fragment offset to whatever you want.

Signal strength for scapy beacon packet, negative Integer to two's complement hex

I am trying to convert a negative integer to a two's complement hex that can be put into the hex code of a scapy packet (bellow) in the signal_strength variable.
The negative number comes from this line of code:
sigstren = int(-1*(20*math.log10(distance) + 20*math.log10(frequency) - 37.55))
An example number is -54, whose hex two's complement is FFCA, but I need it in the format of \xca.
RadioTap(len=18, present='Flags+Rate+Channel+dBm_AntSignal+Antenna', notdecoded='\x00\x6c' + get_frequency(CHANNEL) + '\xc0\x00' + signal_strength + '\x01\x00\x00')
I have tried this:
signal_string = struct.pack("<h", signal)
but it returns \xca\xff which is too big for the packet.
I have also tried this:
def int_to_hex_string(value, bits):
return "{0:0{1}X}".format(value & ((1<<bits) - 1), bits//4)
signalstrength = int_to_hex_string(sigstren, 8)
but that only returns CA, which I cannot add \x to because of ValueError: invalid \x escape.
I was getting that error when trying to add \x to get the right format for the packet:
signal_strength = '\x' + signalstrength

I know exactly what you are doing, you are trying to populate the radio signal manually into notdecoded, because it was not accessible as a field in previous scapy versions.
Note that the latest developpement version of scapy now allows you to access those fields directly. Doing it yourself is more complicated (you also need to take the padding of each field into account...)
Install the dev version: zip of latest dev github version then extract it and python setup.py install (with Sudo on Linux)
Use the newest RadioTap abilities:
Here is the list of fields you can set with the latest dev version of scapy
['TSFT',
'Flags',
'Rate',
'Channel',
'FHSS',
'dBm_AntSignal',
'dBm_AntNoise',
'Lock_Quality',
'TX_Attenuation',
'dB_TX_Attenuation',
'dBm_TX_Power',
'Antenna',
'dB_AntSignal',
'dB_AntNoise',
'RXFlags',
'ChannelPlus',
'MCS',
'A_MPDU',
'VHT',
'timestamp',
'RadiotapNS',
'VendorNS',
'Ext']
Here is what you should do
RadioTap(len=18, present='Flags+Rate+Channel+dBm_AntSignal+Antenna',
dBm_AntSignal=-54,
Antenna=0x1)
It works:
###[ RadioTap dummy ]###
version= 0
pad= 0
len= 18
present= Flags+Rate+Channel+dBm_AntSignal+Antenna
Flags=
Rate= 0
Channel= 0
ChannelFlags=
dBm_AntSignal= -54dBm
Antenna= 1
notdecoded= ''

Is this any help?
>>> hex(0x100 - 0x54)
'0xac'
You just have to flip the half-bytes around and change 0 to \

How to get the size of struct in python if the data inside struct has variable length binary string?

This code will read data from block device and pack data using struct.pack() and again unpack data using struct.unpack(). This is part of my main socket program. But I am facing an issue in calculating struct size. So I am putting a small code here to demonstrate my problem.
import sys,os
import struct as st
dev_read_data = os.open("/dev/sdc1",os.O_RDONLY)
buffer_size = 230400
offset = 0
while True:
data = os.pread(dev_read_data,buffer_size,offset)
packed_data = st.pack("%dsi" %(len(data)),data,offset) # This statement packs data and assign it to packed_data. Till here code works fine.
print ('Packed_Data {]'.format(packed_data))
unpacked_data = st.unpack("%dsi" %(len(data)),packed_data) # This unpacks data successfully.
offset += buffer_size
if buffer_size == 10036977152:
break
Now I want to calculate the size of struct using function:
struct.calcsize(format)
But in this case only one parameter can be passed. So how to get struct size in case of variable length binary string?
I will be very thankful if experts can find some time to answer my query.

like this :
import os
import binascii
import zlib
path = "/dev/sdc1"
stat = os.stat(path)
block_SIZE = stat.st_blksize
block_COUNT = os.statvfs(path).f_blocks
image = file(path,"rb")
indicator = 0
while True :
try :
if indicator > 2 : break #if indicator > block_Count : break
image.seek(indicator*block_SIZE)
data = image.read(block_SIZE)
HEX_CRC32 = binascii.unhexlify(str(hex(zlib.crc32(data) & 0xffffffff))[2:])
header = binascii.unhexlify(("%010X" % indicator)+("%04x"%block_SIZE))
""" NOW SEND TO SOCKET
My_socket.write(header+data+HEX_CRC32)
check Frame number : first 5 byte
Data Length : 2 Byte (block_SIZE)
Data : Data content
CRC32 : 2 Byte (32-bit value)
Client side : Check CRC32 , get `data[7:-2]` for current block
"""
indicator += 1
except Exception,e :
"""
very important (if failed,re send packet )
"""
print e
if indicator > block_Count : break
Not only brake ! Tell to client all packet has ben succesfully send, please close to socket on client side.

How to gzip a bytearray in Python?

I have binary data inside a bytearray that I would like to gzip first and then post via requests. I found out how to gzip a file but couldn't find it out for a bytearray. So, how can I gzip a bytearray via Python?

Have a look at the zlib-module of Python.
Python 3: zlib-module
A short example:
import zlib
compressed_data = zlib.compress(my_bytearray)
You can decompress the data again by:
decompressed_byte_data = zlib.decompress(compressed_data)
Python 2: zlib-module
A short example:
import zlib
compressed_data = zlib.compress(my_string)
You can decompress the data again by:
decompressed_string = zlib.decompress(compressed_data)
As you can see, Python 3 uses bytearrays while Python 2 uses strings.

In case the bytearray is not too large to be stored in memory more than once and known as b, you can just:
b_gz = str(b).encode('zlib')
If you need to do deocding first, have a look at the decode() method of the bytearray.

The zlib module of Python Standard Library should meet your requirements :
>>> import zlib
>>> a = b'abcdefghijklmn' * 10
>>> ca = zlib.compress(a)
>>> len(a)
140
>>> len(ca)
25
>>> b = zlib.decompress(ca)
>>> b == a
True
>>> b
b'abcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmnabcdefghijklmn'
This is the output under Python3.4, but it works same under Python 2.7 -

import zlib
import binascii
def compress_packet(packet):
return zlib.compress(buffer(packet),1)
def decompress_packet(compressed_packet):
return zlib.decompress(compressed_packet)
def demo_zlib() :
packet1 = bytearray()
packet1.append(0x41)
packet1.append(0x42)
packet1.append(0x43)
packet1.append(0x44)
print "before compression: packet:{0}".format(binascii.hexlify(packet1))
cpacket1 = compress_packet(packet1)
print "after compression: packet:{0}".format(binascii.hexlify(cpacket1))
print "before decompression: packet:{0}".format(binascii.hexlify(cpacket1))
dpacket1 = decompress_packet(buffer(cpacket1))
print "after decompression: packet:{0}".format(binascii.hexlify(dpacket1))
def main() :
demo_zlib()
if __name__ == '__main__' :
main()
This should do. The zlib requires access to bytearray content, use buffer() for that.