How can I implement a manual MTU discovery with Python?
From: https://networkengineering.stackexchange.com/a/28988/23983
Send a ping to a target, in my example, I'll use Google's DNS server (8.8.8.8). Set your DF bit in your ping to on, to prevent your ping from being fragmented. Set your packet size to some large number, or the standard MTU of 1500. Note that some ping implementations set the size of just the payload, which means you have to account for the 8 byte ICMP header, and 20 byte IP header.
I'd like to add that you could to this with raw sockets as well.
scapy is a nice abstraction layer that does much of the magic for you, but in all fairness if you're going to go low you could go all the way for the learning experience. (Note that raw sockets require elevated permissions in most modern OS'es, and as you go deeper implementations may vary from Windows and Linux.)
import socket
s = socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.ntohs(0x0003))
This will give you a raw packet based socket that literally delivers all the frames for you. It varies a bit between Windows and Linux but I'll stick to Linux in this answer. Also note that all outgoing packets might not get picked up by this socket, if you need that functionality (sniffing stuff) consider searching for promiscuous mode related hits.
All you need to do now is treat each packet as the segments they come or go in, for instance - unpacking a ethernet and IP frame would look something like this:
frame, meta = s.recvfrom(65565)
print(frame, meta)
ethernet = frame[0:14]
ethernet_segments = struct.unpack("!6s6s2s", ethernet)
mac_source, mac_dest = (binascii.hexlify(mac) for mac in ethernet_segments[:2])
ip = frame[14:34]
ip_segments = struct.unpack("!12s4s4s", ip)
ip_source, ip_dest = (socket.inet_ntoa(section) for section in ip_segments[1:3])
print('MAC Source:', b':'.join(mac_source[i:i+2] for i in range(0, len(mac_source), 2)))
print('MAC Dest:', b':'.join(mac_dest[i:i+2] for i in range(0, len(mac_dest), 2)))
print('IP Source:', ip_source)
print('IP Dest:', ip_dest)
Payloading would be "easy", considering you're building the packets yourself.
All be it not the most conventional of ways or the initially fastest way, but you'd be able to implement whatever you wanted.
Sending is just as easy, use struct and have a look at the many ICMP examples out there, including those with checksum calculations:
https://www.g-loaded.eu/2009/10/30/python-ping/
https://gist.github.com/pklaus/856268
Regarding MTU, this is logic you'd have to implement yourself because there's no pre-built library that does this that I'm aware of.
But this is my contribution to sending raw IP traffic with Python.
Sending anything raw in python, you are looking at using at using scapy.
For info on how to send ping messages:
http://www.secdev.org/projects/scapy/doc/usage.html#icmp-ping
To set the DF Flag:
IP(flags='DF')
For how to adjust the particular size so that you can simulate fragmentation:
Adding payload in packet (scapy)
Putting it all together:
data = "x" * 1473
ans,unans = sr(IP(dst="<target_ip>", flags='DF' )/ICMP() / Raw(load=data))
If you aren't actually that interested in creating these things raw, this question is a dup of: How to find mtu value of network through code(in python)?
Related
Ok, I realize this situation is somewhat unusual, but I need to establish a TCP connection (the 3-way handshake) using only raw sockets (in C, in linux) -- i.e. I need to construct the IP headers and TCP headers myself. I'm writing a server (so I have to first respond to the incoming SYN packet), and for whatever reason I can't seem to get it right. Yes, I realize that a SOCK_STREAM will handle this for me, but for reasons I don't want to go into that isn't an option.
The tutorials I've found online on using raw sockets all describe how to build a SYN flooder, but this is somewhat easier than actually establishing a TCP connection, since you don't have to construct a response based on the original packet. I've gotten the SYN flooder examples working, and I can read the incoming SYN packet just fine from the raw socket, but I'm still having trouble creating a valid SYN/ACK response to an incoming SYN from the client.
So, does anyone know a good tutorial on using raw sockets that goes beyond creating a SYN flooder, or does anyone have some code that could do this (using SOCK_RAW, and not SOCK_STREAM)? I would be very grateful.
MarkR is absolutely right -- the problem is that the kernel is sending reset packets in response to the initial packet because it thinks the port is closed. The kernel is beating me to the response and the connection dies. I was using tcpdump to monitor the connection already -- I should have been more observant and noticed that there were TWO replies one of which was a reset that was screwing things up, as well as the response my program created. D'OH!
The solution that seems to work best is to use an iptables rule, as suggested by MarkR, to block the outbound packets. However, there's an easier way to do it than using the mark option, as suggested. I just match whether the reset TCP flag is set. During the course of a normal connection this is unlikely to be needed, and it doesn't really matter to my application if I block all outbound reset packets from the port being used. This effectively blocks the kernel's unwanted response, but not my own packets. If the port my program is listening on is 9999 then the iptables rule looks like this:
iptables -t filter -I OUTPUT -p tcp --sport 9999 --tcp-flags RST RST -j DROP
You want to implement part of a TCP stack in userspace... this is ok, some other apps do this.
One problem you will come across is that the kernel will be sending out (generally negative, unhelpful) replies to incoming packets. This is going to screw up any communication you attempt to initiate.
One way to avoid this is to use an IP address and interface that the kernel does not have its own IP stack using- which is fine but you will need to deal with link-layer stuff (specifically, arp) yourself. That would require a socket lower than IPPROTO_IP, SOCK_RAW - you need a packet socket (I think).
It may also be possible to block the kernel's responses using an iptables rule- but I rather suspect that the rules will apply to your own packets as well somehow, unless you can manage to get them treated differently (perhaps applying a netfilter "mark" to your own packets?)
Read the man pages
socket(7)
ip(7)
packet(7)
Which explain about various options and ioctls which apply to types of sockets.
Of course you'll need a tool like Wireshark to inspect what's going on. You will need several machines to test this, I recommend using vmware (or similar) to reduce the amount of hardware required.
Sorry I can't recommend a specific tutorial.
Good luck.
I realise that this is an old thread, but here's a tutorial that goes beyond the normal SYN flooders: http://www.enderunix.org/docs/en/rawipspoof/
Hope it might be of help to someone.
I can't help you out on any tutorials.
But I can give you some advice on the tools that you could use to assist in debugging.
First off, as bmdhacks has suggested, get yourself a copy of wireshark (or tcpdump - but wireshark is easier to use). Capture a good handshake. Make sure that you save this.
Capture one of your handshakes that fails. Wireshark has quite good packet parsing and error checking, so if there's a straightforward error it will probably tell you.
Next, get yourself a copy of tcpreplay. This should also include a tool called "tcprewrite".
tcprewrite will allow you to split your previously saved capture files into two - one for each side of the handshake.
You can then use tcpreplay to play back one side of the handshake so you have a consistent set of packets to play with.
Then you use wireshark (again) to check your responses.
I don't have a tutorial, but I recently used Wireshark to good effect to debug some raw sockets programming I was doing. If you capture the packets you're sending, wireshark will do a good job of showing you if they're malformed or not. It's useful for comparing to a normal connection too.
There are structures for IP and TCP headers declared in netinet/ip.h & netinet/tcp.h respectively. You may want to look at the other headers in this directory for extra macros & stuff that may be of use.
You send a packet with the SYN flag set and a random sequence number (x). You should receive a SYN+ACK from the other side. This packet will have an acknowledgement number (y) that indicates the next sequence number the other side is expecting to receive as well as another sequence number (z). You send back an ACK packet that has sequence number x+1 and ack number z+1 to complete the connection.
You also need to make sure you calculate appropriate TCP/IP checksums & fill out the remainder of the header for the packets you send. Also, don't forget about things like host & network byte order.
TCP is defined in RFC 793, available here: http://www.faqs.org/rfcs/rfc793.html
Depending on what you're trying to do it may be easier to get existing software to handle the TCP handshaking for you.
One open source IP stack is lwIP (http://savannah.nongnu.org/projects/lwip/) which provides a full tcp/ip stack. It is very possible to get it running in user mode using either SOCK_RAW or pcap.
if you are using raw sockets, if you send using different source mac address to the actual one, linux will ignore the response packet and not send an rst.
So I have some questions about sockets and sniffer programming...
I've just started programming and have a project where I would like to use information that is sent across my network.
I tried watching several videos on youtube that talk about this process a little, and tried to find better material to research it further, but I haven't been able to find a source that makes sense to me.
The code I included came from a video on youtube and seemed to make sense as they explained it, but I guess he might have been using Linux or something else because Windows did not support AF_PACKET. After some research I found that people use AF_INET, but I got the error:
OSError: [WinError 10043] The requested protocol has not been configured into the system, or no implementation for it exists
Is there a place or a way someone might be able to explain sockets a little bit for me? I don't plan to use windows for the final version of this project, and I also plan to modify it for bluetooth in the future, so I would like to learn the reasoning behind things if I can find a way to do that.
`
import socket
import struct
import textwrap
def main():
conn = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.ntohs(3))
while True:
raw_data, addr = conn.recvfrom(65535)
dest_mac, src_mac, eth_proto, data = ethernet_frame(raw_data)
print('\nEthernet Frame:')
print('Destination: {}, Source: {}, Protocol: {}'.format(dest_mac, src_mac, eth_proto, data[:14]))
#unpack ethernet frame
def ethernet_frame(data):
dest_mac, src_mac, proto = struct.unpack('! 6s 6s H', data[:14])
return get_mac_addr(dest_mac), get_mac_addr(src_mac), socket.htons(proto), data[14:]
#Get Mac Address
def get_mac_addr(bytes_addr):
bytes_str = map('{:02x}'.format, bytes_addr)
return ':'.join(bytes_str).upper()
main()
`
The OSes on which you use a socket for packet sniffing are:
Linux
Irix
Windows is not on that list (neither are anything with "BSD" in the name, OS X, Solaris, HP-UX, AIX, etc.). Linux and Irix both happen to use sockets to do sniffing, but that's just their choice (and they didn't choose the same type of socket, they just happened to choose sockets).
If you want to write a sniffer, you're probably best advised to use a wrapper around libpcap/WinPcap, and let them deal with the painful details of the way packet sniffing is done on a particular operating system. Wrappers for Python include pylibpcap and pcapy; I don't know whether either of them work on Windows with WinPcap.
(Note that you are not guaranteed to get Ethernet headers on sniffed packets; you should call pcap_datalink(), or whatever the wrapper's equivalent is, and check its return value - if it's not DLT_EN10MB, or the wrapper's equivalent, you won't be getting Ethernet headers.)
AF_INET raw sockets, on any platform, aren't going to give you Ethernet headers. I don't know what you'll get with a protocol argument of 3 - 3 is the Internet protocol number for GGP, as per RFC 823 Appendix A, and that protocol is ancient and not used as far as I know; you'll probably end up with a socket on which you can send GGP packets and from which you can receive GGP packets, for what that's worth (which is not much). (Also, the arguments to the socket() call in C are in host byte order, and Python probably works the same, so you probably don't want that socket.ntohs() in there, not that it'll make a difference.)
I have two apps sending tcp packages, both written in python 2. When client sends tcp packets to server too fast, the packets get concatenated. Is there a way to make python recover only last sent package from socket? I will be sending files with it, so I cannot just use some character as packet terminator, because I don't know the content of the file.
TCP uses packets for transmission, but it is not exposed to the application. Instead, the TCP layer may decide how to break the data into packets, even fragments, and how to deliver them. Often, this happens because of the unterlying network topology.
From an application point of view, you should consider a TCP connection as a stream of octets, i.e. your data unit is the byte, not a packet.
If you want to transmit "packets", use a datagram-oriented protocol such as UDP (but beware, there are size limits for such packets, and with UDP you need to take care of retransmissions yourself), or wrap them manually. For example, you could always send the packet length first, then the payload, over TCP. On the other side, read the size first, then you know how many bytes need to follow (beware, you may need to read more than once to get everything, because of fragmentation). Here, TCP will take care of in-order delivery and retransmission, so this is easier.
TCP is a streaming protocol, which doesn't expose individual packets. While reading from stream and getting packets might work in some configurations, it will break with even minor changes to operating system or networking hardware involved.
To resolve the issue, use a higher-level protocol to mark file boundaries. For example, you can prefix the file with its length in octets (bytes). Or, you can switch to a protocol that already handles this kind of stuff, like http.
First you need to know if the packet is combined before it is sent or after. Use wireshark to check it the sender is sending one packet or two. If it is sending one, then your fix is to call flush() after each write. I do not know the answer if the receiver is combining packets after receiving them.
You could change what you are sending. You could send bytes sent, followed by the bytes. Then the other side would know how many bytes to read.
Normally, TCP_NODELAY prevents that. But there are very few situations where you need to switch that on. One of the few valid ones are telnet style applications.
What you need is a protocol on top of the tcp connection. Think of the TCP connection as a pipe. You put things in one end of the pipe and get them out of the other. You cannot just send a file through this without both ends being coordinated. You have recognised you don't know how big it is and where it ends. This is your problem. Protocols take care of this. You don't have a protocol and so what you're writing is never going to be robust.
You say you don't know the length. Get the length of the file and transmit that in a header, followed by the number of bytes.
For example, if the header is a 64bits which is the length, then when you receive your header at the server end, you read the 64bit number as the length and then keep reading until the end of the file which should be the length.
Of course, this is extremely simplistic but that's the basics of it.
In fact, you don't have to design your own protocol. You could go to the internet and use an existing protocol. Such as HTTP.
I am currently working on a project where I need to send packets to a particular networking hardware, receive responses back, and generate packets based on the response in real time.
I came across Scapy, and to my understanding it is capable of doing the first two parts: sending and receiving. Is it possible through Python to retrieve the necessary fields in a response and respond back?
Thanks!
If I understand what you mean, you have (at least) two options with Scapy:
The clean one is to create an AnsweringMachine that matches your needs (you have several examples in Scapy's code).
The dirty (but maybe quicker if that's what you need) one is to give as prn parameter to sniff() a function that will receive the packet, craft an answer (or anything you like) and send() it on the network.
As an example, this code will send RST-ACK packets to any TCP packet seen with SYN flag on:
def rst(p):
ans = IP(src=p[IP].dst, dst=p[IP].src)/TCP(
flags='RA',
sport=p[TCP].dport,
dport=p[TCP].sport,
seq = 0,
ack = p[TCP].seq + 1,
)
send(ans, verbose=False)
return "%s\n => %s" % (p[IP].summary(), ans.summary())
sniff(iface="eth0", filter="tcp and tcp[tcpflags] & tcp-syn == tcp-syn",
prn=rst)
I need to monitor how long it takes for a certain website to respond when addressed. I would like to sniff the traffic on port 80 but only when there is traffic being exchanged with the targeted site. I have searched SO and it seems like pcapy or scapy is the right tool for the job, but they seem deeper than I need. I have studying the following script:
Network traffic monitor with pcapy in python
and I think I need to change the
def __handle_packet(self, header, data):
# method is called for each packet by dispatch call (pcapy)
self._dispatch_bytes_sum += header.getlen() #header.getlen() #len(data)
logger.debug("header: (len:{0}, caplen:{1}, ts:{2}), d:{3}".format(header.getlen(), header.getcaplen(), header.getts(), len(data)))
#self.dumper.dump(header, data)
to somehow only unpack/handle packets that are destined for the target site. Note that this is for a Windows XP machine on a LAN and it is critical that the browser initiate the traffic.
Any pointers appreciated?
The problem with scapy is it doesn't handle reassembling TCP streams. Your HTTP that you're looking for is likely to be embedded in a TCP stream. To quote the docs:
Scapy is based on a stimulus/response model. This model does not work well for a TCP stack. On the other hand, quite often, the TCP stream is used as a tube to exchange messages that are stimulus/response-based.
Like you said scapy is more ideal for lower-layer things. You could, for instance, probably track IP packets on DHCP requests. Like many network tools, the complexities and stream-based nature of TCP means once you cross that layer it gets harder to reassemble everything and deal with all the retransmission and what not edge cases and coherently pull data out.
Could you use something like curl or urllib and see how long it takes for the response to come back?