I'm coding to tcp client/server using python twisted
in order to replace for Java or C#.
I have to parse length prefixed string messages based on ANS(alpha numeric string) in connected permanent session.
like this :
message format : [alpha numeric string:4byte][message data]
example-1 : 0004ABCD ==> ABCD
example-2 : 0002AB0005HELLO ==> AB, HELLO
it can't be solved by IntNProtocol, NetStringProtocol.
And if a client send a 2kb message from application layer, the kernel split message data by MSS(maximum segment size) and send packets are splitted.
in TCP send/receive environment, it often raise like this :
example : 1000HELLO {not yet arrived 995 byte data}
so it has to wait for receiving spare data using array, queue...
in the twisted, I don't know how to parse multiple large-message.
Anybody help me to give some information or URL?
class ClientProtocol(protocol.Protocol):
def dataReceived(self, data):
# how can I code to parse multiple large message?
# is there solution to read specific size for data ?
It looks like you can implement this protocol using StatefulProtocol as a base. Your protocol basically has two states. In the first state, you're waiting for 4 bytes which you will interpret as a zero-padded base 10 integer. In the second state, you're waiting for a number of bytes equal to the integer read in the first state.
from twisted.protocols.stateful import StatefulProtocol
class ANSProtocol(StatefulProtocol):
def getInitialState(self):
return (self._state_length, 4)
def _state_length(self, length_bytes):
length = int(length_bytes)
return self._state_content, length
def _state_content(self, content):
self.application_logic(content)
return self.getInitialState()
def application_logic(self, content):
# Application logic operating on `content`
# ...
I am using srp1() of Scapy to replay pcap file to a device as follows:
for p in rdpcap(pcapfile):
...
rcv = srp1(p, 'eth0')
print rcv[IP].len
print rcv[TCP].seq
...
When the device sends 1 packet I can get its IP.len and TCP.seq, but when it sends 2 packets, I can get only the information of the first packet while I need the information of the second one.
Where did I go wrong?
Both Scapy's user manual and Scapy's API documentation state that srp1() is a variant of srp() that returns just the first packet that constitutes an answer for the sent packet/s.
Therefore, try using srp() instead of srp1(), as follows:
for p in rdpcap(pcapfile):
...
answers, unanswered = srp(p, 'eth0')
last_request, last_answer = answers[-1]
print last_answer[IP].len
print last_answer[TCP].seq
...
I found this code to detect the length of encrypted data in the frame :
header = self.request.recv(5)
if header == '':
#print 'client disconnected'
running = False
break
(content_type, version, length) = struct.unpack('>BHH', header)
data = self.request.recv(length)
Souce :
https://github.com/EiNSTeiN-/poodle/blob/master/samples/poodle-sample-1.py
https://gist.github.com/takeshixx/10107280
https://gist.github.com/ixs/10116537
This code, listen the connection between a client and a server. When the client talk to the server, self.request.recv(5) can get you the length of the header in the frame. Then we use that length to take the data.
If we print the exchange between the client and the server :
Client --> [proxy] -----> Server
length : 24 #why 24 ?
Client --> [proxy] -----> Server
length: 80 #length of the data
Client <-- [proxy] <----- Server
We can see that the client will send two packet to the server.
If i change
data = self.request.recv(length)
to
data = self.request.recv(4096)
Only one exchange is made.
Client --> [proxy] -----> Server
length: 109 #length of the data + the header
Client <-- [proxy] <----- Server
My question is why we only need to take a size of 5 to get the lenght, content_type informations ? Is there an understandable doc about this ?
Why there is two request: one with 24 and another with the lenght of our data ?
why we only need to take a size of 5 to get the lenght, content_type
informations ?
Because obviously that's the way the protocol was designed.
Binary streams only guarantee that when some bytes are put into one end of the stream, they arrive in the same order on the other end of the stream. For message transmission through binary streams the obvious problem is: where are the message boundaries? The classical solution to this problem is to add a prefix to messages, a so-called header. This header has a fixed size, known to both communication partners. That way, the recipient can safely read header, message, header, message (I guess you grasp the concept, it is an alternating fashion). As you see, the header does not contain message data -- it is just communication "overhead". This overhead should be kept small. The most efficient (space-wise) way to store such information is in binary form, using some kind of code that must, again, be known to both sides of the communication. Indeed, 5 bytes of information is quite a lot.
The '>BHH' format string indicates that this 5 byte header is built up like this:
unsigned char (1 Byte)
unsigned short (2 Bytes)
unsigned short (2 Bytes)
Plenty of room for storing information such as length and content type, don't you think? This header can encode 256 different content types, 65536 different versions, and a message length between 0 and 65535 bytes.
Why there is two request: one with 24 and another with the lenght of
our data ?
If your network forensics / traffic analysis does not correspond to what you have inferred from code, one of both types of analyses is wrong/incomplete. In this case, I guess that your traffic analysis is correct, but that you have not understood all relevant code for this kind of communication. Note that I did not look at the source code you linked to.
After asking this, I just wanted to make a simple test. I captured a traffic using tcpdump. Filtered out a TCP ACK packet in Wireshark and exported the filtered packet to sample.pcap.
Now this is pretty much my code for TCP checksum recalculation:
from scapy.all import *
ack_pkt = sniff(offline="sample.pcap", count=1)[0]
print "Original:\t", ack_pkt[TCP].chksum
del ack_pkt[TCP].chksum
print "Deleted:\t", ack_pkt[TCP].chksum
ack_pkt[TCP]=ack_pkt[TCP].__class__(str(ack_pkt[TCP]))
print "Recalculated:\t", ack_pkt[TCP].chksum
The output I'm getting is:
WARNING: No route found for IPv6 destination :: (no default route?)
Original: 30805
Deleted: None
Recalculated: 55452
Is the checksum recalculation process is correct or is there something else to recalculate the checksum? Since scapy is being used widely for a long time, I guess, there is something wrong in my checksum recalculation.
Updated with packet information: (Ethernet header is not shown for a better view.)
To view the packet in hex string:
from binascii import hexlify as hex2
ack_pkt = sniff(offline="sample.pcap", count=1)[0]
print ack_pkt.chksum, ack_pkt[TCP].chksum
print hex2(str(ack_pkt[IP]))
del ack_pkt.chksum
del ack_pkt[TCP].chksum
print ack_pkt.chksum, ack_pkt[TCP].chksum
print hex2(str(ack_pkt[IP]))
ack_pkt=ack_pkt.__class__(str(ack_pkt))
print ack_pkt.chksum, ack_pkt[TCP].chksum
print hex2(str(ack_pkt[IP]))
ack_pkt[TCP].chksum=0
print hex2(str(ack_pkt[IP]))
And the output I get is:
26317 30805
450000345bc840004006*66cd*0e8b864067297c3a0016a2b9f11ddc3fe61e9a8d801000f7*7855*00000101080a47e8a8af0b323857
None None
450000345bc840004006*66cd*0e8b864067297c3a0016a2b9f11ddc3fe61e9a8d801000f7*d89c*00000101080a47e8a8af0b323857
26317 55452
450000345bc840004006*66cd*0e8b864067297c3a0016a2b9f11ddc3fe61e9a8d801000f7*d89c*00000101080a47e8a8af0b323857
450000345bc840004006*66cd*0e8b864067297c3a0016a2b9f11ddc3fe61e9a8d801000f7*0000*00000101080a47e8a8af0b323857
(* is only for marking the checksum bytes.)
Isn't it strange? After deleting the checksum, when I put a ack_pkt.show(), I see both checksum fields are None. But while converting to hex-string, is it been recalculated?
ack_pkt[TCP].chksum=0 with this, the recalculated checksum comes 0 only.
Note:
I've tried with ack_pkt[TCP].show2() and I'm getting the same value as I'm getting above.
1) Did you try to copy a complete packet, not just its TCP part:
ack_pkt=ack_pkt.__class__(str(ack_pkt))
2) Did you try the following approach, explicitly converting the packet to a string and recreating it using the string?
3) If none of the above works, please post the TCP ACK packet you are working with
I tried a manual checksum recalculation both with python and C (with libpcap) by following one's compliment of one's compliment sum and I'm getting the same value as scapy gets for the mentioned packet. So I guess the checksum calculation in linux-kernel is modified somehow.
(It's better to say one guy is wrong rather than saying all other 3 were wrong. :P)
I am trying to implement an ICMP based Traceroute in Python. I found a very helpful guide ( https://blogs.oracle.com/ksplice/entry/learning_by_doing_writing_your ) that has allowed me to create a UDP based Traceroute so just needs modification. However I have looked around and am having trouble changing the send socket and making it work. Is anybody able to assist me?
#!/usr/bin/python
import socket
def main(dest_name):
dest_addr = socket.gethostbyname(dest_name)
port = 33434
max_hops = 30
icmp = socket.getprotobyname('icmp')
udp = socket.getprotobyname('udp')
ttl = 1
while True:
recv_socket = socket.socket(socket.AF_INET, socket.SOCK_RAW, icmp)
send_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, udp)
send_socket.setsockopt(socket.SOL_IP, socket.IP_TTL, ttl)
recv_socket.bind(("", port))
send_socket.sendto("", (dest_name, port))
curr_addr = None
curr_name = None
try:
_, curr_addr = recv_socket.recvfrom(512)
curr_addr = curr_addr[0]
try:
curr_name = socket.gethostbyaddr(curr_addr)[0]
except socket.error:
curr_name = curr_addr
except socket.error:
pass
finally:
send_socket.close()
recv_socket.close()
if curr_addr is not None:
curr_host = "%s (%s)" % (curr_name, curr_addr)
else:
curr_host = "*"
print "%d\t%s" % (ttl, curr_host)
ttl += 1
if curr_addr == dest_addr or ttl > max_hops:
break
if __name__ == "__main__":
main('google.com')
Not sure why you chose scapy (nice module though it is), as this is certainly possible using only python. To send an ICMP packet, you simply send out your recv_socket. In order to send out this socket, you'll need to first create an ICMP packet.
However, it seems what you want is to send out a UDP packet over an ICMP socket. This won't work as you might think.
First, let me say apparently there exists a patch to the Linux kernel that'll allow SOCK_DGRAM with IPPROTO_ICMP: ICMP sockets (linux). I've not tested this.
Generally, though, this combination of socket flags won't work. This is because an ICMP socket expects an ICMP header. If you were to send an empty string over recv_socket similarly to send_socket, the kernel will drop the packet. Further, if you were to layer a UDP header over an ICMP header, the receiving system will only react to the received ICMP header, and treat the UDP header as nothing more than just data appended to ICMP. In fact, in its ICMP reply to you, the remote system will include your UDP header simply as data you sent to it in the first place.
The reason you're able to send an empty string over send_socket is because the kernel has created the UDP header for you, and what you send over that UDP socket is simply appended as data to the UDP header. It's not like that with an ICMP socket. As I wrote, you'll need to create the icmp header to send on this socket.
The actuality of what happens in a UDP "ping" is this: A UDP packet is sent over a UDP socket to a remote system, using a (hopefully) unopened port as the destination port. This elicits an ICMP response back from the remote system (type 3, code 3). At this point, you will need an ICMP-registered socket to handle the ICMP reply back, which requires root (or administrator on Windows) privileges.
To create an icmp echo header is quite easy:
import struct
icmp = struct.pack(">BBHHH", 8, 0, 0, 0, 0)
icmp = struct.pack(">BBHHH", 8, 0, checksum(icmp), 0, 0)
A couple things to note with this header. First, arguments four and five are the "Identifier" and "Sequence Number" fields. Typically the identifier is set to a process ID while the sequence number starts at 1 so you can track the sequence of sends to replies. In this example, I've set the numbers to zero just for illustration (although, it is perfectly normal to keep it this way if you don't care about them).
Second, see the checksum(icmp) function I allude to? I'm assuming you have access to code that does this. It is known as the 1's complement checksum, and is ratified under RFC 1071. The checksum is vital because, without it being correctly computed, the receiving system might not forward the packet to any of the ICMP socket handlers.
Third, if you're wondering why I initially created the header with a zero checksum, it's because the kernel bases a checksum result on that field being zero, before the actual checksum is created and added back to the header.
Because the struct module deals with what's called "packed" binary data, I advise that you familiarize yourself with bit shifting and bitwise operators. You're going to need this as you advance further into raw sockets, particulary if and when you need to extract certain bits from bit fields that may or may not overlap. For example, the following is the IP header:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 |Version| IHL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4 | Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
8 | Time to Live | Protocol | Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
12 | Source Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 | Destination Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
20 | Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Assuming you wanted to insert IP address 192.168.1.10 in the source field, first you must note its length, 4 bytes. To correctly insert this into struct (just this field, not rest of header, as an example), you must:
struct.pack(">I", 192 << 24| 168 << 16| 1 << 8| 10)
Doing this adds the proper integer for this field, 3232235786L.
(Some readers may point out that the same result could be had with struct.pack(">BBBB", 192, 168, 1, 10). Although this would work for this use case, it is incorrect in general. In this case, it works because an IP address is byte-oriented; however, fields that are not byte-oriented, such as the checksum field, will fail because the resulting value of a checksum is greater than 255, that is to say, it is expected to be a 16-bit value. So don't do this in general i.e. use the exact bits expected by a protocol field.)
As another example for learning bit shifting and operations, take the VER field of the IP header. This is a nibble-sized field, i.e. 4 bits. In order to extract this, you would do the following as an example (assuming IHL is zero):
# Assume ip[] already has a full IP header.
ver = struct.unpack("!B", ip[0])[0]
# This produces the integer 4, which is required for sending IPV4
ver >> 4
In a nutshell, I've right-shifted a one-byte value down 4 bits. In this downshift, the new value compared to the old value now looks like this:
# OLD VALUE IN BINARY; value is 64
# 01000000
# NEW VALUE IN BINARY; value is 4
# 00000100
It's important to note that, without this shift, checking the raw binary value would result in 64, which isn't what is expected. For the inverse, in order to "pack" the value 4 into the upper 4 bits of one byte, do this:
# New value will be 64, or binary 01000000
x = 4 << 4
Hope this points you in the right direction.
Old question, but adding another point for clarity as I recently had to do just this.
You can write a native Python version of ICMP traceroute using sockets. The two important issues to solve are:
creating the ICMP header with the correct checksum (as Eugene states above)
setting the TTL of the socket using sockopts
For the first issue, there's a great example in the pyping module that works perfectly; it's what I used when I wrote my traceroute utility.
For the second, it's as easy as:
current_socket = socket.socket(socket.AF_INET, socket.SOCK_RAW,
socket.getprotobyname("icmp"))
current_socket.setsockopt(socket.IPPROTO_IP, socket.IP_TTL, ttl)
where 'ttl' is the integer value of the TTL you're setting
Then it's just a matter of doing a send/recv and watching the type/code in the return packets, in a control structure that increments the TTL.
I modeled my header pack/unpack after what was already written in the pyping module; just look for header type = 11 with code 0 for the intermediate hops, and header type = 0 for when you've reached your destination.
Scapy works fine, but it's slower, and is an additional external dependency. I was able to write a robust traceroute in an afternoon with AS-lookup (via RADb) and geolocation, using nothing but sockets.
Ended up writing my own using scapy as this wasn't possible.