Develop Reference

Python UDP socket receive - server.recvfrom package size longer than bufsize

I'm using the socket module for a UDP server。The incoming packets always have a different size(0-65535), so client send package length first, then send the package；server receives data according to the package length and 1024 per server receives。but server didn't work like I thought，it can receives package only once。
# -*- coding: utf-8 -*-
import json
import socket
ip_port = ('127.0.0.1', 8080)
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind(ip_port)
print('server listen......')
def recv_header(s_socket):
try:
msg, addr = s_socket.recvfrom(32)
return json.loads(msg), addr
except Exception as e:
return str(e), None
while True:
header_msg, client_addr = recv_header(s_socket=server)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
receiverBufsize = 1024
if client_addr is None:
print (header_msg)
continue
data_body = ""
package_len = int(header_msg['length'])
print ("package_len: {}".format(package_len))
print ("client_addr: {}".format(str(client_addr)))
while package_len > 0:
if package_len > receiverBufsize:
body_part = server.recvfrom(receiverBufsize)
else:
body_part = server.recvfrom(package_len)
data_body += body_part[0]
package_len -= receiverBufsize
print ("data_body: {}".format(data_body))
# -*- coding: utf-8 -*-
import json
import socket
import random
ip_port = ('127.0.0.1', 8080)
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def get_data(length):
base_str = 'ABCDEFGHIGKLMNOPQRSTUVWXYZabcdefghigklmnopqrstuvwxyz0123456789'
return ''.join([random.choice(base_str) for _ in range(length)])
while 1:
msg = raw_input('>>>：')
if msg == 'q':
break
if not msg:
continue
len = int(msg)
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
data = get_data(len)
header = json.dumps({"length": len})
aa = client.sendto(header.encode('utf-8'), ip_port)
print aa
aa = client.sendto(data.encode('utf-8'), ip_port)
print aa
If the packet length is less than 1024(my receiverBufsize), it works normally
enter image description here
enter image description here
If the packet length is more than 1024(my receiverBufsize), socket.recvfrom will block
enter image description here
enter image description here

UDP is a message oriented protocol, not a data stream like TCP. This means in UDP a single send is matched by a single recv.
This means you cannot use multiple recv with a small buffer (size 1024 in your case) to get data which were sent in a single send with a larger buffer - you need to use recv with a large enough buffer instead.
This means also that you don't actually need to send the length upfront, since you don't need to add message semantics to a protocol which already has message semantic. In contrary, sending the length in a separate message as you do could actually be harmful since UDP is an unreliable protocol and you cannot rely on packets getting delivered in order, at all or only once.

Python network scanner (host discovery tool) is telling me that my source and destination are the same when sending UDP datagrams

I'm following Black Hat Python (2ed.), in which I'm writing a network scanning tool. The tool is in theory supposed to send UDP packets out to a given subnet, and if a host is up on that subnet, the response packet is decoded, found to contain the message in the original datagram, and used to indicate the host is up. This seems to generally be working well to capture packets; I can go to a website, or ping another host, and the tool reliably provides the correct source and destination addresses for those cases.
Here is the meat of the code (I have not included the class creation, or the passing of the host argument for brevity, but the host is 192.168.10.85).
class IP:
"""layer 3 (IP) packet header decoder"""
def __init__(self, buff=None):
header = struct.unpack('<BBHHHBBH4s4s', buff)
self.ver = header[0] >> 4
self.ihl = header[0] & 0xF
self.tos = header[1]
self.len = header[2]
self.id = header[3]
self.offset = header[4]
self.ttl = header[5]
self.protocol_num = header[6]
self.sum = header[7]
self.src = header[8]
self.dst = header[9]
# make IP addrs human readable
self.src_address = ipaddress.ip_address(self.src)
self.dst_address = ipaddress.ip_address(self.dst)
# the protocol_num is actually a code for the protocol name
self.protocol_name = {1: 'ICMP', 6: 'TCP', 17: 'UDP'}
# try to provide the human version of the protocol, otherwise just give the code
try:
self.protocol = self.protocol_name[self.protocol_num]
except KeyError as error:
self.protocol = self.protocol_num
print(f'Protocol is unrecognized, try googling "IP protocol {self.protocol_num}"')
class ICMP:
"""layer 4 (ICMP) packet header decoder"""
def __init__(self, buff):
header = struct.unpack('<BBHHH', buff)
self.type = header[0]
self.code = header[1]
self.checksum = header[2]
self.ident = header[3]
self.seq_num = header[4]
def udp_sender():
# blasts udp packets into the network to solicit responses
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sender:
for ip in ipaddress.ip_network(SUBNET).hosts():
# time.sleep(1)
print(f'sending a test message to {ip}')
# send our test message out to port 65212 on the destination
sender.sendto(bytes(MESSAGE, 'utf8'), (str(ip), 65212))
class Scanner:
def __init__(self, host):
self.host = host
# create raw socket, bind to public interface
# if windows:
if os.name == 'nt':
socket_protocol = socket.IPPROTO_IP
# if linux/mac:
else:
socket_protocol = socket.IPPROTO_ICMP
self.socket = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket_protocol)
self.socket.bind((host, 0))
# socket options, include header
self.socket.setsockopt(socket_protocol, socket.IP_HDRINCL, 1)
# enable promiscuous mode for windows
if os.name == 'nt':
self.socket.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON)
def sniff(self):
# set of all hosts that are up (respond to our ICMP message)
hosts_up = {f'{str(self.host)} *'}
try:
while True:
# read a packet, and parse the IP header
raw_buffer = self.socket.recvfrom(65535)[0]
# create IP header from the first 20 bytes
ip_header = IP(raw_buffer[0:20])
# if the protocol is ICMP, do some additional things
# print(f'src={ip_header.src_address}, dst={ip_header.dst_address}, prot_name={ip_header.protocol}')
if ip_header.protocol == 'ICMP':
# calculate where the ICMP packet starts
offset = ip_header.ihl * 4
buf = raw_buffer[offset:offset + 8]
# create ICMP structure
icmp_header = ICMP(buf)
print(f'type: {icmp_header.type}, code: {icmp_header.code}')
print(f'src={ip_header.src_address}, dst={ip_header.dst_address}, prot_name={ip_header.protocol}')
if icmp_header.type == 3 and icmp_header.code == 3:
print(f'type: {icmp_header.type}, code: {icmp_header.code}')
print(f'src={ip_header.src_address}, dst={ip_header.dst_address}, prot_name={ip_header.protocol}')
if ipaddress.ip_address(ip_header.src_address) in ipaddress.IPv4Network(SUBNET):
# make sure the packet has our test message
if raw_buffer[len(raw_buffer) - len(MESSAGE):] == bytes(MESSAGE, 'utf8'):
tgt = str(ip_header.src_address)
if tgt != self.host and tgt not in hosts_up:
hosts_up.add(str(ip_header.src_address))
print(f'Host Up: {tgt}')
However, when receiving the ICMP responses as a result of my datagram, the tool reports that the source and destination addresses are the same (my host, 192.168.10.85). Furthermore, while I should be receiving responses with Type 3 and Code 3 (destination unreachable, and port unreachable), but I am receiving (in my program) Type 3 and Code 1.
Here is an example of the output when I issue a ping command while the scanner is running, which seems correct:
src=192.168.10.85, dst=192.168.10.200, prot_name=ICMP type: 0, code: 0 src=192.168.10.200, dst=192.168.10.85, prot_name=ICMP type: 8, code: 0
Here is an example of the output to what I am assuming is the UDP packet response, which seems incorrect):
src=192.168.10.85, dst=192.168.10.85, prot_name=ICMP type: 3, code: 1
If I open wireshark while I'm running my code, I can correctly see the ICMP Type 3/Code 3 responses, so I know they are going through, here is a screen grab of one host on the target subnet as an example:
Why is my scanner not seeing these responses that are in wireshark?
I've tried running wireshark alongside my program, to see if the packets are being correctly decoded, and that the message in the UDP packet is properly in place. All signs indicate that the packets are going out to the hosts I'm trying to detect, and the correct responses are coming back, but my scanner refuses to find them.

The raw sockets can't create an ARP request packet with source (MAC and IP )taken as some other machine .. Any suggestions?

I'm using the below script for injecting an ARP packet request. When I keep the source (MAC and IP) as my machine, I can happily see the packets in the wire and receive ARP replies however on changing the source to a different machine in the LAN, the ARP requests don't get back the ARP replies.
I am dicey if the RAW sockets can only frame up an ARP request for the base machine or am I going wrong somewhere ?
Below is the code ...
#!/usr/bin/python
import sys
import socket
import binascii
import struct
from itertools import chain
try:
iFace = raw_input("Enter the interface using which the Injection needs to be done ...\n")
rawSocket = socket.socket(socket.PF_PACKET, socket.SOCK_RAW,socket.htons(0x0800))
rawSocket.bind((iFace, socket.htons(0x0800)))
print "Raw Socket got created .... with the Ethernet Protocol Id : 0x0806 at interface %s"%str(iFace)
except:
print "Something unexpected happened during the Program execution."
else:
def checkMac(mac):
if len(mac.split(":")) != 6:
print "The MAC is in correct. It should be in Hexadecimal Format with each byte separated with colon...\n"
sys.exit(0)
else:
macList = mac.split(":")
macLen = len(macList)
return tuple ([int(macList[index],16) for index in range(macLen)])
def checkIp(ip):
ipList = ip.split(".")
ipLen = len(ipList)
return int( "".join( [ "{:02X}".format(int(ele)) for ele in ipList ] ), 16 )
dMac = raw_input("Enter the Destination MAC .. hexadecimal charaters separated with ':' \n")
# dMac = "0X:XX:XX:XX:XX:4X"
dMacTup = checkMac(dMac)
# sMac = raw_input("Enter the Source MAC .. hexadecimal charaters separated with ':' \n")
sMac = "XX:XX:XX:XX:XX:XX"
sMacTup = checkMac(sMac)
type = 0x0806
# Creating an Ethernet Packet .... using dMac, sMac, type
etherPack = struct.pack ("!6B6BH",*tuple(chain(dMacTup,sMacTup,[type])))
# Creating an ARP Packet .... now
hardwareType = 0x0001
protocolType = 0x0800
hln = 0x06
pln = 0x04
op = 0x0001
# srcIp = raw_input("Enter the Source IP ':' \n")
srcIp = "10.0.2.216"
intSrcIp = checkIp(srcIp)
destIp = raw_input("Enter the Destination IP .. \n")
# destIp = "10.0.2.1"
intDestIp = checkIp(destIp)
arpPack = struct.pack("!HHBBH6BI6BI", *tuple(chain( [hardwareType,protocolType,hln,pln,op], sMacTup,[intSrcIp], dMacTup,[intDestIp] )))
# Framing the final Packet
finalPack = etherPack + arpPack
for i in range(50):
rawSocket.send(finalPack + "Hacker in the wires ...")
print "Sending Packet %d"%i
finally:
print "Closing the created Raw Socket ..."
rawSocket.close()

Socket Programming; File corrupts when transferring over multiple devices

The problem I'm having is to get a file from the server to client across devices. Everything works fine on localhost.
Lets say I want to "get ./testing.pdf" which sends the pdf from the server to the client. It sends but it is always missing bytes. Is there any problems with how I am sending the data. If so how can I fix it? I left out the code for my other functionalities since they are not used for this function.
sending a txt file with "hello" in it works perfectly
server.py
import socket, os, subprocess # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
#host = ''
port = 5000 # Reserve a port for your service.
bufsize = 4096
s.bind((host, port)) # Bind to the port
s.listen(5) # Now wait for client connection.
while True:
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
while True:
userInput = c.recv(1024)
.... CODE ABOUT OTHER FUNCTIONALITY
elif userInput.split(" ")[0] == "get":
print "inputed get"
somefile = userInput.split(" ")[1]
size = os.stat(somefile).st_size
print size
c.send(str(size))
bytes = open(somefile).read()
c.send(bytes)
print c.recv(1024)
c.close()
client.py
import socket, os # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
#host = '192.168.0.18'
port = 5000 # Reserve a port for your service.
bufsize = 1
s.connect((host, port))
print s.recv(1024)
print "Welcome to the server :)"
while 1 < 2:
userInput = raw_input()
.... CODE ABOUT OTHER FUNCTIONALITY
elif userInput.split(" ")[0] == "get":
print "inputed get"
s.send(userInput)
fName = os.path.basename(userInput.split(" ")[1])
myfile = open(fName, 'w')
size = s.recv(1024)
size = int(size)
data = ""
while True:
data += s.recv(bufsize)
size -= bufsize
if size < 0: break
print 'writing file .... %d' % size
myfile = open('Testing.pdf', 'w')
myfile.write(data)
myfile.close()
s.send('success')
s.close

I can see two problems right away. I don't know if these are the problems you are having, but they are problems. Both of them relate to the fact that TCP is a byte stream, not a packet stream. That is, recv calls do not necessarily match one-for-one with the send calls.
size = s.recv(1024) It is possible that this recv could return only some of the size digits. It is also possible that this recv could return all of the size digits plus some of the data. I'll leave it for you to fix this case.
data += s.recv(bufsize) / size -= bufsize There is no guarantee that that the recv call returns bufsize bytes. It may return a buffer much smaller than bufsize. The fix for this case is simple: datum = s.recv(bufsize) / size -= len(datum) / data += datum.

Sending low level raw tcp packets python

I have been working on a program lately for raw packets. We recently had a lecture about raw packets so I have been trying to learn and do exactly what my professor told me. I have a problem with my program it comes up with an error saying destination address required, its raw so I don't want to do socket.connect(destaddr) even though that will fix the error. Here is my code:
Here is the class and function:
#not real mac address to protect privacy also removed preamble
class packet(object):
b = ""
def __init__(self, payload):
self.payload = payload
def ether(self):
#preamble = "55555555555555D5"
macdest = "123456789101" #my mac address - needed to remove colons
macsource = "123456789101" #router mac address without colons
ethertype = "0800" #removed 0x because it is not needed
fcs = "" #frame check sequence none so far
frame = macdest+macsource+ethertype
return frame
def ip(self): #in hexadecimal
version = "4" #ipv4 hex
ihl = "5" #header length hex
dscp = "00" #default
ecn = "00" #default
length = "36" #ether-24 + ip-20 + tcp-30 = 54 to hexa = 35
idip="0000" #random id
flags = "40" #dont fragment flag is 2 to hex is 4
offset = "00" #space taker
ttl = "40"#hex(64) = 40
protocol = "06" #for tcp
checksum = "0000"
ipaddrfrom = "c0a8010a"
ipaddrto = "c0a80101"
datagram = version+ihl+dscp+ecn+length+idip+flags+offset+ttl+protocol+checksum+ipaddrfrom+ipaddrto
return datagram
def tcp(self):
portsrc = "15c0" #5568
portdest = "0050" #80
syn = "00000000"
ack = "00000000"
nonce = "80"
fin = "10"
windowscale = "813b"
checksum = "0000"
segment = portsrc+portdest+syn+ack+nonce+fin+windowscale + checksum
return segment
def getpacket(self):
frame = self.ether()
datagram = self.ip()
segment = self.tcp()
payload = self.payload
packet = frame+datagram+segment+payload
a = 0
b = ""
for char in packet:
a = a+1
b = b + char
if a == 4:
b = b + " "
a=0
self.fmtpacket = b
return packet
def raw():
s = socket(AF_INET, SOCK_RAW, IPPROTO_IP)
s.bind(('192.168.1.10', 0))
pckt = packet("")
netpacket = pckt.getpacket()
print "Sending: " + pckt.fmtpacket
print ""
s.sendall(netpacket)
data = s.recv(4096)
print data

If your professor is okay with it, you may find Scapy a lot easier to work with in creating raw packets in python.
From their website:
Scapy is a powerful interactive packet manipulation program. It is able to forge or decode packets of a wide number of protocols, send them on the wire, capture them, match requests and replies, and much more. It can easily handle most classical tasks like scanning, tracerouting, probing, unit tests, attacks or network discovery (it can replace hping, 85% of nmap, arpspoof, arp-sk, arping, tcpdump, tethereal, p0f, etc.)

Is there a reason for binding to '0.0.0.0'? When you create a raw socket, you'll need to bind it to an interface.
One thing I notice is that you'll need the '\x' prefix for hex.
Right now, you're stringing together chars.
For example, in ip(), version + ihl = '45'. That's a string, not a hex value. When you're sending this along, as a raw packet, that's two bytes instead of the one that you want. You want to send '\x45', not '45'.

packet to be sent should contain the actual bytes and not the string.