Develop Reference

Python - Pcapy to Scapy on SPAN port, odd behaviour

I built a network sniffer in Scapy but it can't handle the rate of packets I am sniffing (it adds 15-20 minutes of latency which is just unacceptable). I have used Pcapy before in the past at this speed with success, but this time to save me having to re-write all my parsing code that uses Scapy, I want to convert a packet received by Pcapy into a Scapy IP object. The problem is when I try to do this, the IP's and protocol numbers I get are scrambled/unusable, like Scapy is reading the wrong section of the packet.
Some example code below:
#!/usr/bin/python
from pcapy import findalldevs, open_live
from impacket import ImpactDecoder, ImpactPacket
from scapy.all import *
def sniff():
interface = "eth3"
print "Listening on: %s" % interface
# Open a live capture
reader = open_live(interface, 65535, 1, 100)
# Set a filter to be notified only for TCP packets
reader.setfilter('ip proto \\tcp')
# Run the packet capture loop
reader.loop(0, callback)
def callback(hdr, data):
pkt = IP(data)
if IP in pkt:
print pkt[IP].dst
# Parse the Ethernet packet
#decoder = ImpactDecoder.EthDecoder()
#ether = decoder.decode(data)
# Parse the IP packet inside the Ethernet packet
#iphdr = ether.child()
# Parse the TCP packet inside the IP packet
#tcphdr = iphdr.child()
# Only process SYN packets
#if tcphdr.get_SYN() and not tcphdr.get_ACK():
# # Get the source and destination IP addresses
# src_ip = iphdr.get_ip_src()
# dst_ip = iphdr.get_ip_dst()
# # Print the results
# print "Connection attempt %s -> %s" % (src_ip, dst_ip)
def main():
sniff()
if __name__ == "__main__":
main()
And an example of the output:
30.184.113.84
0.120.231.205
30.184.113.91
5.64.113.97
0.120.231.206
21.248.113.98
0.120.231.207
0.120.231.208
0.120.231.209
0.120.231.210
0.120.231.211
0.48.243.73
As you can see these IP's dont make sense, where do you think I am going wrong. Eth3 is connected to a NetGear mirror port.
Thanks for your time.

Never mind, just me being an idiot, I blame bank-holiday Mondays. I was trying to detect the packet from the wrong layer. Convert raw to Ether and Scapy does the rest of the work for me.
def callback(hdr, data):
pkt = Ether(data)
if IP in pkt:
print pkt[IP].dst
else:
print list(pkt)
Cheers

Socket Programming in Python

I am working on a proof of concept using python that emulates a server/client communication using sockets to send UDP packets. I can easily do a simple client to server and back to client comms, but I am trying to introduce a "middle-man" into that communication. Conceptually the problem can be descirbed as, if "Joe" is the main client, he will send a message to "Steve" who is the middle man who will do something with that message before sending it to "Carol" who acts as the server that will process the new message and send a response back to the middle-man, "Steve". Eventually the middle-man will then send that message on elsewhere, but at the moment I am not worrying about that.
My current code looks like:
"Joe" (original client) looks like
# create dgram udp socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
except socket.error:
print ('Failed to create socket')
sys.exit()
host = 'localhost'
port = 8888
print("start comms")
while 1:
arr = ['Dog', 'cat', 'treE', 'Paul']
num = random.randrange(0,4)
#Send the string
s.sendto(arr[num].encode(), (host, port))
"Steve" (middle man) looks like
host = ''
hostRT = 'localhost'
portVM = 8888
portRT = 8752
# socket to receive from "Joe"
s1 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s1.bind((host, portVM))
# socket to send to "Carol"
s2 = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
print("start comms")
while 1:
# receive from "Joe"
data = s1.recvfrom(1024)
num = data[0].decode()
addrVM = data[1]
# print data from Joe
print(num)
# add some excitement to Joe's message
num += '!!!'
# show received message address + port number
print ("message[" + addrVM[0] + ":" + str(addrVM[1]) + ']')
# Send to "Carol"
s2.sendto(num.encode(), (hostRT, portRT))
# receive from "Carol"
d = s2.recvfrom(1024)
reply = d[0].decode()
addrRT = d[1]
# show received message address + port number
print ("message[" + addrRT[0] + ":" + str(addrRT[1]) + ']')
# show Carol's response
print ('Server reply : ' + reply)
s1.close()
s2.close()
"Carol" (server) looks like
host = ''
port = 8752
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
print ("socket created")
s.bind((host, port))
print ("Socket bind complete")
while 1:
d = s.recvfrom(1024)
data = d[0].decode()
addr = d[1]
print(data)
reply = "Upper case client data = " + data.upper()
print(reply)
s.sendto(reply.encode(), addr)
print ("message[" + addr[0] + ":" + str(addr[1]) + '] - ' + data.strip())
s.close()
Currently I can receive a message from Joe but then it hangs on the sending to the server Carol. I'm completely new to socket programming so any help would be greatly appreciated.
Edit for clarification
Using Python 3.4
Joe is sending packets non stop as to emulate the real life application that this proof of concept is for. Joe will be sending packets at a rate of roughly 1 packet / 4ms, but I am only concerned with the most recent packet. However, since the average turn around time for the round trip from Steve to Carol is around 10ms, I had originally thought to cache Joe's most recent packet in a local memory location and overwrite that location until Steve is ready to send a packet to Carol once she has responded with the last packet. However, for this simple proof of concept I haven't tried to implement that. Any suggestions on that would also be helpful.

There are multiple faults that contribute to the overall failure, some of which are not apparent (i.e. it sort of works until it crashes down somewhere else).
First of all, at the moment sends packets as fast as he cans. That alone can lead to significant packet loss everywhere else (that might be a good thing, since you now have to make sure your code survives packet loss). Unless you truly want to stress the network, something like time.sleep(0.1) would be appropriate in the send loop.
More importantly, steve's socket setup is all messed up. He needs two sockets at the most, not three. The way it is currently set up, carol answers steve to the IP address and port she got the packet from (which is quite sensible), but steve reads on a distinct socket that never gets data sent to.
To make matters worse, the port steve's s3 listens on is actually the same one that carol uses! Since you are not using multicast. You can simply remove every reference to s3 from the code and use s2 exclusively.
Another problem is that you don't deal with packet loss. For example, if a packet gets lost between steve and carol, the code
# Send to "Carol"
s2.sendto(num.encode(), (hostRT, portRT))
# receive from "Carol"
d = s2.recvfrom(1024) # s3 in the original
will hang forever, since Carol does not send any new packets after the one that got lost. As mentioned before, packet loss is way more likely since joe is blasting out packets as fast as he can.
To detect packet loss, there are a few options:
Use multiple threads or processes for sending and receinv. This is going to make your program way more complex.
Switch to asynchronous / non-blocking IO, in Python with the high-level asyncore or the more low-level select.
Set and correctly handle socket timeouts. This is probably the easiest option for now, but is quite limited.
Switch to TCP if you actually need reliable communication.
Apart from the aforementioned network problems, there are also some potential problems or inaccuracies:
If you are using Python 2.x, the behavior of decode and encode on strings depends on the system configuration. Like many other potential problems, this has been fixed in Python 3.x by mandating UTF-8 (in 2.x you have to explicitly request that). In your case, that's fine as long as you only send ASCII characters.
while(1) : looks really strange in Python - why the whitespace after the argument, and why parentheses . Why not while 1: or while True:?
You can use tuple unpacking to great effect. Instead of
data = s1.recvfrom(1024)
num = data[0].decode()
addrVM = data[1]
how about:
data, addrVM = s1.recvfrom(1024)
num = data.decode('utf-8')

Receiving and returning a packet after the function has ended

I have two functions. One sends a UDP packet to a port number and returns the port number if it gets a response. The second cycles through addresses calling the first function repeatedly incrementing the port number. It stops when a port number is returned. I'm basically spamming an IP to find which port number is being used.
All works well with very long time outs but I'm trying to speed up the rate at which I send my test packets. For example I might get a packet back from port 27018 even though the spam port function is sending to 27022. It then incorrectly reports that 27022 is the port to use. This happens even when I return the returned packet info from the first function since you can't tell the arguments which were used originally.
def check_port(s, serverip, serverport):
payload = "ffffffff54536f7572636520456e67696e6520517565727900".decode('hex')
s.sendto(payload, (serverip, serverport))
while 1:
try:
s.settimeout(0.1) # time to wait for response
d = s.recvfrom(1400)
data = d[0]
addr = d[1]
if len(data) > 1:
break
except socket.error:
return False
return addr[1]
def spam_ports(serverip):
s = serverComms.port_config()
start_highport = 27015
end_highport = 28015
start_lowport = 2299
end_lowport = 4000
port = start_highport
while check_port(s,serverip, port) == False:
port += 1
if port == end_highport:
port = start_lowport
if port == end_lowport:
return 'no query ports found in range'
else:
return check_port(s,serverip, port)
I really appreciate any help with this.

I think I know what happens.
It takes some time for the server to reply. If the delay is shorter than that, your application becomes confused. Let me explain:
You send packages to port 1000, 1001, 1002, ...
Say port 1010 produces a reply. But lets assume the server needs a full second to reply. Your application has progressed well bejond 1010 since the timeout is less then a second. By the time the reply arrives your application is already at, say, 1020. Now it looks like the received package is the result of sending something to 1020.
Possible approch
What you need is a way to know to which port a received reply belongs. Here it gets tricky:
Each package has a source and a destination port. With the packages you send the destination port is incremented. The source port is probably arbitrarly assigned. When the server answers it will send a package with an arbitrary source port and the destination port equal the source port of your package.
What you could do is check with the documentation and see how you can control the source port of the packages you're sending. With that you make sure that each sent package has a different source port. This identifies each package uniquely. Now you can use the destination port of the reply to know were it belongs to.
Then you can use, for example, two threads. One sending out packages and one receiving the answers. You keep a dict that maps my_source_port to server_port. The sender fills the dict and the receiver reads it. You can let the sender send as fast as it can, now timeout required. Once the sender is done you give the receiver thread a second or so to catch up.
port_map = {}
active_ports = []
def send(min, max):
for dest_port in range(min,max):
src_port = get_free_port()
port_map[src_port] = dest_port
send_package(src_port, dest_port, 'somedata')
def recv():
while True:
src_port, dest_port, data = receive_package()
port = port_map[dest_port]
active_ports.append(port)
This is not a working example, just the basic idea. The methods don't exist in that form, thread synchronization is missing and recv() would run forever, but it shows the basic idea.
You probably have to create a socket for each package you send.

Python raw socket not receiving ICMP packets

I'm trying to use raw sockets in Python to send UDP packets to a host and then get the ICMP response back for the packet -- basically reimplementing traceroute.
I've managed to correctly construct my IP and UDP headers and send the packet. I can see it in Wireshark. I also see the ICMP response in Wireshark telling me that the TTL exceeded.
I have the following code:
me = gethostbyname(gethostname())
my_socket = socket(AF_INET, SOCK_RAW)
my_socket.setsockopt(IPPROTO_IP, IP_HDRINCL, 1)
my_socket.bind((me, 0))
hostname = 'www.google.com'
hostip = gethostbyname(hostname)
packet = create_packet(hostname)
send_socket.sendto(packet, (hostip , 0))
Then after the packet is sent I call another function to listen for incoming packets which includes this snippet:
while True:
ready = select.select([my_socket], [], [], time_left)
if ready[0] == []:
print "timeout"
time_now = time.time()
rec_packet, addr = my_socket.recvfrom(5120)
unpacked_ip = unpack('!BBHHHBBH4s4s', rec_packet[0:20]) #0-20 is IP header
prot = unpacked_ip[6] #gives the protocol id
if prot == 1:
#this is ICMP , let's do things
I'm able to successfully unpack the IP header and check the protocol, but it is always either 6 or 17 (TCP or UDP). I never get the IP packet containing the ICMP payload even though it appears in Wireshark.
I've tried comparing the ICMP packet in Wireshark to other packets in Wireshark that my program does see and the IP headers are pretty much identical. I don't know what is wrong.
Thanks for the help

Judging from this answer, it looks like you need to pass the IPPROTO_ICMP option in when you create your socket.
You can do this like:
my_socket = socket.socket(socket.AF_INET,socket.SOCK_RAW,socket.IPPROTO_ICMP)

python icmp raw socket implementation

i am relatively new to python, so please be considerate...
i'm implementing a server and a client via raw_sockets.
i have the necessary privileges.
now, the server i defined so:
host = socket.gethostbyname(socket.gethostname())
address = (host, 22224)
sockSer = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
sockSer.bind(address)
sockSer.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON)
packet, addr = sockSer .recvfrom(4096) # wait for packet from client
Q1) why can't i simply type: hosts = 'localhost'.
if i do so, it doesn't allow me to write the line: sockSer.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON). and then the server doesn't receive my client's messages.
only when doing gethostbyname(socket.gethostname()) i get 192.168.1.101
and then it works.
in a different class:
the client socket:
host = socket.gethostbyname(socket.gethostname())
address = (host, 22224)
sockCli = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
Q2) do i also need to type: sockCli.ioctl(socket.SIO_RCVALL, socket.RCVALL_ON)
or maybe sockCli.connect(address)? seems that it works without the connect command.
for the client socket?
now, the problems arise when i do the following:
1) send a packet from client to server:
header=...
payload='a'
sockCli.sendto(header + payload, address)
2) receive packet in server and send something back to client:
while(true):
data, addr = sockSer.recvfrom(4096)
header2=...
payload2='b'
sockSer.sendto(header2 + payload2, addr)
now, my important question is:
Q3) the server sent only 1 packet to client, with payload 'b'.
what happens is, my client actually receives 2 packets in the while loop:
first packet is what the client itself sent to server, and the other packet is what the client got from the server.
hence my output is 'ab' instead of simply 'b'
why is this happening???
NOTE: i didn't type the entire code, but i think my syntax,parsing,header composition etc.. are correct.
is there an obvious problem in my code?
if necessary i'll upload the entire code.
thanks

I got this too.
my solution is add a judge in the receive code,such as if I send Ping package so I only want ECHO　Reply( type 0 code 0), I write
if type != 0:
continue
and you also can write as
if addr == my_ip:
continue
It seems not has any smooth solution

Q1: I was able to bind to localhost and call IOCTL with both parameters just fine. Assuming your client is also running on the same system, ensure the client is sending to "localhost", otherwise your server will never receive the packets. If your client is on another system, obviously your server will never receive the packets.
Q2: You do not need IOCTL for sending the packet. Just send it via sendto().
Q3: The reason you're seeing two replies is, the kernel is also processing the echo request, in addition to your own user-space code.
Although you can use ICMP for arbitrary message passing, as someone else pointed out this isn't its intended design. You may find that your data portion is truncated out in message replies. For example, when sending echo requests, your reply likely will contain everything you sent; however, a reply that is type 3 code 3 may not include your data, but only the first 8 bytes of the ICMP header.