I am attempting to work with large packet captures from wireshark that have been output in pdml format. These captures are then loaded into python using the lxml library to traverse over them. The issue I am having is that I can pull out information regarding a single HTTP response packet and then I need a way to associate this with its HTTP request packet.
The current solution I was thinking of implementing is to search for an HTTP request packet that is part of the same TCP stream as the response, however this seems like an inefficient solution to the problem, having to continually separate out TCP streams and then search through them for the request packet.
Is there a simple way to associate response packets with requests that I am missing?
Best solution I have come up with thus far is to use xpath under the assumption that each TCP connection only contains one request/response pair.
#Get the stream index from the packet
streamIndex = packet.xpath('proto/field[#name="tcp.stream"]')[0].attrib['show']
#Use that stream index to get the matching response packet
return packet.xpath('/pdml/packet[proto/field[#name="tcp.stream" and #show="' + streamIndex + '"] and proto/field[#name="http.request.full_uri"]]')[0]
Related
How would I use scapy 2.4.3+ to spoof a http response? I tried using the packet below, however on the target machine (which has been arp spoofed, where the machine of the spoofer does not have port fowarding, resulting in the http request having to be answered by the spoofed packet, that is what I am trying to do), the provided HTML from the packet does not get rendered, once its sent using scapy.send(packet). So how would I adapt the packet below, to send a HTTP packet that would render on the target machine?
packet = scapy.IP(src=server_ip, dst=target_ip)/scapy.TCP() / HTTP() / HTTPResponse(Server=server_ip) / "<html><p>Hi</p></html>"
I am attempting to transmit a GET request to a server to update particular json values. I have been making use of these similar posts to do so:
Python-Scapy or the like-How can I create an HTTP GET request at the packet level
Filter HTTP Get requests packets using scapy
How can I alter the JSON object and override the values highlight in the image that are being sent?
I'm using Python requests to play with a REST API. The response format is JSON and let's assume the server always send correct data. Given the fact that HTTP uses TCP for transmission, do I still have to check the existence of a required key if no exception is thrown by requests?
For TCP transmissions, you don't need to verify the response if you assume that the server always sends correct data:
TCP provides reliable, ordered, and error-checked delivery of a stream of octets between applications running on hosts communicating by an IP network.
Source: Wikipedia
Of course, it's always a good idea to add some error handling and verification to your code just in case the server doesn't send what you'd expect.
I am working on a simple program written in Python which sniffs coming network packets. Then, let user use added modules like DoS detection or Ping prevention. With the help of sniffer, I can get incoming connections' IP address, MAC address, protocol flag and packet content. Now, what I want to do is adding a new module that detects whether sender using proxy or not and do some thing according to it. I was searched on the methods that can be used with Python but can not find useful one. How many ways are there to detect proxy for Python?
My sniffer code part is something like that:
.....
sock = socket.socket(socket.PF_PACKET, socket.SOCK_RAW, 8)
while True:
packet = sock.recvfrom(2048)
ipheader = packet[0][14:34]
ip_hdr = struct.unpack("!8sB3s4s4s", ipheader)
sourceIP = socket.inet_ntoa(ip_hdr[3])
tcpheader = packet[0][34:54]
tcp_hdr = struct.unpack("!HH9ss6s", tcpheader)
protoFlag = binascii.hexlify(tcp_hdr[3])
......
Firstly, you mean incoming packets.
secondly,
From the server TCP's point of view it is connected to the proxy, not the downstream client.
so your server can't identify that there is a proxy involved from the packet.
however, if you are in the application level like http proxy, there might be a X-forwarded-for header available in which there will be the original client IP. I said it might be because proxy server will decide whether or not send this header to you. If you are expecting incoming http connections to your server, you can take a look at python's urllib2 although I'm not sure if you can access the X-forwarded-for using this library.
From the docs:
urllib2.urlopen(url[, data][, timeout])
...
This function returns a file-like object with two additional methods:
geturl() — return the URL of the resource retrieved, commonly used to determine if a redirect was followed
info() — return the meta-information of the page, such as headers, in the form of an mimetools.Message instance (see Quick Reference to HTTP Headers)
so using info() will retrieve the headers. hope you find what you're looking for in there.
There aren't many ways to do this, as proxies / VPNs look like real traffic. To add to what Mid said, you can look for headers and/or user agents to help you determine if the user is using a proxy or a VPN.
The only free solution I know is getIPIntel that uses block lists, machine learning, and statistics to determine if the IP is a proxy / VPN or not.
There are other paid solutions like maxmind and blocked.
What you'll need to do is send API queries to these services and parse the results.
I'm trying to code my own Python 3 http library to learn more about sockets and the Http protocol. My question is, if a do a recv(bytesToRead) using my socket, how can I get only the header and then with the Content-Length information, continue recieving the page content? Isn't that the purpose of the Content-Length header?
Thanks in advance
In the past to accomplish this, I will read a portion of the socket data into memory, and then read from that buffer until a "\r\n\r\n" sequence is encountered (you could use a state machine to do this or simply use the string.find() function. Once you reach that sequence you know all of the headers have been read and you can do some parsing of the headers and then read the entire content length. You may need to be prepared to read a response that does not include a content-length header since not all responses contain it.
If you run out of buffer before seeing that sequence, simply read more data from the socket into your buffer and continue processing.
I can post a C# example if you would like to look at it.