Develop Reference

Problem with reading destination IP from pcap file

I am trying to read list of destination IP's from a pcap file, the problem is when I run the while loop a get this error
Traceback (most recent call last):
File "/root/PycharmProjects/pcap/pcap.py", line 10, in <module>
print(pcap[4]['IP'].show())
File "/root/venv/pcap/lib/python3.7/site-packages/scapy/packet.py", line 1171, in __getitem__
raise IndexError("Layer [%s] not found" % lname)
IndexError: Layer ['IP'] not found
When I checked Wireshark, I found that error appears because of requests made by vmware, since I wrote this code on a Kali virtual machine. Here is my code
from scapy.all import *
from nmap import *
from collections import OrderedDict
scanner = nmap.PortScanner()
pcap = rdpcap('/root/Downloads/nakerah.pcap')
ip_list = []
x = 0
while x < 4:
host_ip = pcap[x]['IP'].dst
ip_list.append(host_ip)
final_list = list(OrderedDict.fromkeys(ip_list))
x += 1
print(final_list)

The error tells you exactly what you need to know.
IndexError: Layer ['IP'] not found
One of the packets in your packet captures that does not contain an IP layer. You need to check if the IP layer exists before accessing it. For example, an ARP packet will not have an IP layer and will break your code.
Using this pcap from wireshark's sample captures, we can get the dest IPs by checking if the IP layer exists.
# print_ips.py
from scapy.all import rdpcap
ip_list = []
pkts = rdpcap('allen_test.pcap')
# Limit analysis to 20 packets for brevity
twenty_pkts = pkts[:20]
for packet in twenty_packets:
# This check is what you are missing
if 'IP' in packet:
dest_ip = packet['IP'].dst
ip_list.append(dest_ip)
print("Out of", len(twenty_packets), "packets,", len(ip_list), "were IP packets.")
print("Dest IPs", ip_list)
Running this in the shell, we get
$ python print_ips.py
WARNING: DNS decompression loop detected
Out of 20 packets, 7 were IP packets.
Dest IPs ['172.19.255.255', '172.19.255.255', '172.19.255.255', '172.19.255.255', '224.0.0.9', '172.19.0.240', '172.19.0.240']

How to send multiple music files using sockets? [duplicate]

This question already has an answer here:
sending multiple files in python
(1 answer)
Closed 3 years ago.
I am using python 3.6 and ubuntu 18.04.
I am able to send single music file using socket-python (in binary mode) and i want to send multiple music files from server to client.
Problem is, at the receiver end (that is client), all the music files (approx 120 files sent from server) gets collected in one single file making it a 9 hour long single music file.
I have tried using time.sleep method (does not work), tried sending bogus element (error was shown) and tried sending some random character to end the file writing at the client side and initiate new file write (but random character requires encoding and decoding, so again error as binary data was unable to decode).
SERVER CODE
import socket
import os
import send_file
import time
s = socket.socket()
host = ""
port = 9997
s.bind((host, port))
s.listen(5)
print("Binding Done\n")
socket_object, address = s.accept()
print("Connection Established\n")
print("Sending file...")
file_class = send_file.send_files() #ignore
file_names = file_class.files #ignore - contains list of path of music file
socket_object.sendall( str(len(file_names)).encode() )
for i in file_names:
f = open(i, 'rb')
buf = f.read(1024)
while buf:
socket_object.sendall(buf)
buf = f.read(1024)
f.close()
print("Files Send")
socket_object.close()
s.close()
CLIENT CODE
import socket
import os
import time
def recv_file(i):
f = open("/home/ravi/PycharmProjects/File_Transfer/B/"+"M"+str(i)+".mp3", 'wb')
buf = s.recv(1024)
while buf:
f.write(buf)
buf = s.recv(1024)
f.close()
s = socket.socket()
host = "127.0.0.1"
port = 9997
s.connect((host, port))
print("Receiving data...")
l = s.recv(1024).decode() #ignore - length of total number of files i.e., 120 approx
for i in range(int(l)):
recv_file(i+1)
print("Files Collected")
s.close()
Any suggestions would be appreciated. Thank You.

TCP is a stream-oriented protocol. It is designed for sending streams of data. At TCP level there are no files. It doesn't split streams in any file-oriented chunks.
Look at your code:
for i in file_names:
f = open(i, 'rb')
buf = f.read(1024)
while buf:
socket_object.sendall(buf)
buf = f.read(1024)
f.close()
You just glue all files in a single stream, and your client has no idea when one file ends and the next file starts.
The task of sending multiple files over TCP could be solved in many ways:
Develop your own protocol. E.g.: first send the number of files, then send an array of 8-byte-encoded file lengths, and then the stream of file contents. The receiving end reads number of files, then parses file lengths. Knowing the lengths the receiver correctly splits the stream into files.
Use existing multi-file packaging formats: tar, cpio, zip, etc. Pack files before sending, then send the resulting package. On the receiving end unpack the package after receiving.
Recommended way Use existing protocols for sending files over TCP: TFTP or FTP.
I'd recommend using TFTP. It is very simple and reasonably efficient. There are several implementations in Python, such as tftpy
On the remote machine where you want to upload your files to, start TFTP server:
import tftpy
server = tftpy.TftpServer('<destination-folder>')
server.listen('0.0.0.0', 54321)
On the machine with files start the client:
import tftpy
client = tftpy.TftpClient('your.server.address', 54321)
for name in ("fileA", "fileB", "fileC"):
# the first argument is the name on the remote machine
# the second argument is the name on the local machine
client.upload(name, name)

Download several files from a local server to a client

The following codes let me download from server to client three files called tmp.bsp, tmp.seq and tmp.dms. However, just the first file tmp.dms is completely downloaded. The other one tmp.seq is filled up with the informations of tmp.bsp and tmp.bsp stay 0KB.
client:
import socket
import socket
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
sData = "Temp"
sData2 = "Temp"
sData3 = "Temp"
while True:
sData = skClient.recv(1024)
fDownloadFile = open("tmp.dms","wb")
sData2 = skClient.recv(1024)
fDownloadFile2 = open("tmp.seq","wb")
sData3 = skClient.recv(1024)
fDownloadFile3 = open("tmp.bsp","wb")
while sData:
fDownloadFile.write(sData)
sData = skClient.recv(1024)
fDownloadFile.close()
fDownloadFile2.write(sData2)
sData2 = skClient.recv(1024)
fDownloadFile2.close()
fDownloadFile3.write(sData3)
sData3 = skClient.recv(1024)
fDownloadFile3.close()
print "Download over"
break
skClient.close()
n is a counter and the prints are for debugging.
sFileName is to download one file, and used to work but since I want three files I just commented it.
server:
import socket
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
while True:
Content,Address = skServer.accept()
print Address
files = "C:\Users\Name_user\Desktop\Networking\Send_Receive/"
fUploadFile = open(files+str('tmp.dms'),"rb")
sRead = fUploadFile.read(1024)
fUploadFile2 = open(files+str('tmp.seq'),"rb")
sRead2 = fUploadFile2.read(1024)
fUploadFile3 = open(files+str('tmp.bsp'),"rb")
sRead3 = fUploadFile3.read(1024)
while sRead:
Content.send(sRead)
sRead = fUploadFile.read(1024)
Content.send(sRead2)
sRead2 = fUploadFile2.read(1024)
# Content.send(sRead3)
# sRead3 = fUploadFile3.read(1024)
Content.close()
print "Sending is over"
break
skServer.close()
files I'm using:
server2.py is my server
Execution

The main issue with your code is that you're sending / receiving an arbitrary number of data. If your buffer (1024) is smaller than the file size then the client's file will contain less information, and if it's larger the file may contain more information (data from the next file).
You could solve this issue by sending a value that signifies the end of a file. The problem with this method is that this value can't be contained in any file, and the client must be scanning the received data for this value.
Another possible solution is to calculate the file size and send that infomation in front of the file data. This way the cilent will know how many data to expect for each file.
Using struct.pack we can create a minimal four bytes header with the file size.
def send_file(soc, path):
with open(path, 'rb') as f:
data = f.read()
size = struct.pack('!I', len(data))
soc.send(size + data)
Tthen the client can get the file size by reading four bytes and unpacking to int.
def recv_file(soc, path):
size_header = soc.recv(4)
size = struct.unpack('!I', size_header)[0]
data = soc.recv(size)
with open(path, 'wb') as f:
f.write(data)
Note that sending/receiving files with one call may raise a socket error if the file size is larger than the socket buffer. In that case you'll have to read the data in smaller chunks in a loop, or increase the buffer size with socket.setsockopt.
Here is a modified version of the above functions that can handle large files:
import struct
import os.path
def send_file(soc, path):
file_size = os.path.getsize(path)
size_header = struct.pack('!Q', file_size)
soc.send(size_header)
with open(path, 'rb') as f:
while True:
data = f.read(1024)
if not data:
break
soc.send(data)
def recv_file(soc, path):
size_header = soc.recv(8)
file_size = struct.unpack('!Q', size_header)[0]
chunks = [1024 for i in range(file_size / 1024)]
with open(path, 'wb') as f:
for chunk in chunks:
f.write(soc.recv(chunk))
f.write(soc.recv(file_size % 1024))
I haven't tested this code thoroughly, but it should work for files of any size.
An example using the send_file function in your server:
host = ''
skServer = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
skServer.bind((host,2525))
skServer.listen(10)
print "Server currently active"
Content,Address = skServer.accept()
print Address
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
send_file(Content, file)
Content.close()
print "Sending is over"
skServer.close()
Using recv_file in the client:
skClient = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
skClient.connect(("127.0.0.1",2525))
files = ['tmp.bsp', 'tmp.seq', 'tmp.dms']
for file in files:
recv_file(skClient, file)
print "Download over"
skClient.close()

Yes you are right, I did run your program and found exactly same issue. I dont have enough time to work more on this issue but I found few key points which might lead you to the right work around.
https://docs.python.org/2/howto/sockets.html
The above official doc says:
When a recv returns 0 bytes, it means the other side has closed (or is in the process of closing) the connection. You will not receive any more data on this connection. Ever. You may be able to send data successfully
This is what it is happening when the third file returns 0 bytes.
But why 2nd and 3rd file is merged, I guess its because sockets are just buffered files and we might need to try making sure buffer is clear before sending another.
Read this,
Now there are two sets of verbs to use for communication. You can use send and recv, or you can transform your client socket into a file-like beast and use read and write. The latter is the way Java presents its sockets. I’m not going to talk about it here, except to warn you that you need to use flush on sockets. These are buffered “files”, and a common mistake is to write something, and then read for a reply. Without a flush in there, you may wait forever for the reply, because the request may still be in your output buffer.
But if you plan to reuse your socket for further transfers, you need to realize that there is no EOT on a socket. I repeat: if a socket send or recv returns after handling 0 bytes, the connection has been broken. If the connection has not been broken, you may wait on a recv forever, because the socket will not tell you that there’s nothing more to read (for now). Now if you think about that a bit, you’ll come to realize a fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited (shrug), or indicate how long they are (much better), or end by shutting down the connection. The choice is entirely yours, (but some ways are righter than others).
Hope this helps.

I'm not totally fluent in Python, but I think your while statement should be something like:
while: sData or sData2 or sData3
I may have the syntax wrong, but currently it looks like you will stop when "sData" is done and stop downloading sData2 and aData3 at that time even if they haven't finished.
Hmm--either that or the "While" isn't looping at all and it's just being used as an "if"? hard to tell without knowing the API.

Python - Pcapy to Scapy on SPAN port, odd behaviour

I built a network sniffer in Scapy but it can't handle the rate of packets I am sniffing (it adds 15-20 minutes of latency which is just unacceptable). I have used Pcapy before in the past at this speed with success, but this time to save me having to re-write all my parsing code that uses Scapy, I want to convert a packet received by Pcapy into a Scapy IP object. The problem is when I try to do this, the IP's and protocol numbers I get are scrambled/unusable, like Scapy is reading the wrong section of the packet.
Some example code below:
#!/usr/bin/python
from pcapy import findalldevs, open_live
from impacket import ImpactDecoder, ImpactPacket
from scapy.all import *
def sniff():
interface = "eth3"
print "Listening on: %s" % interface
# Open a live capture
reader = open_live(interface, 65535, 1, 100)
# Set a filter to be notified only for TCP packets
reader.setfilter('ip proto \\tcp')
# Run the packet capture loop
reader.loop(0, callback)
def callback(hdr, data):
pkt = IP(data)
if IP in pkt:
print pkt[IP].dst
# Parse the Ethernet packet
#decoder = ImpactDecoder.EthDecoder()
#ether = decoder.decode(data)
# Parse the IP packet inside the Ethernet packet
#iphdr = ether.child()
# Parse the TCP packet inside the IP packet
#tcphdr = iphdr.child()
# Only process SYN packets
#if tcphdr.get_SYN() and not tcphdr.get_ACK():
# # Get the source and destination IP addresses
# src_ip = iphdr.get_ip_src()
# dst_ip = iphdr.get_ip_dst()
# # Print the results
# print "Connection attempt %s -> %s" % (src_ip, dst_ip)
def main():
sniff()
if __name__ == "__main__":
main()
And an example of the output:
30.184.113.84
0.120.231.205
30.184.113.91
5.64.113.97
0.120.231.206
21.248.113.98
0.120.231.207
0.120.231.208
0.120.231.209
0.120.231.210
0.120.231.211
0.48.243.73
As you can see these IP's dont make sense, where do you think I am going wrong. Eth3 is connected to a NetGear mirror port.
Thanks for your time.

Never mind, just me being an idiot, I blame bank-holiday Mondays. I was trying to detect the packet from the wrong layer. Convert raw to Ether and Scapy does the rest of the work for me.
def callback(hdr, data):
pkt = Ether(data)
if IP in pkt:
print pkt[IP].dst
else:
print list(pkt)
Cheers

Python Client/Server send big size files

I am not sure if this topic have been answered or not, if was I am sorry:
I have a simple python script the sends all files in one folder:
Client:
import os,sys, socket, time
def Send(sok,data,end="292929"):
sok.sendall(data + end);
def SendFolder(sok,folder):
if(os.path.isdir(folder)):
files = os.listdir(folder);
os.chdir(folder);
for file_ in files:
Send(sok,file_);#Send the file name to the server
f = open(file_, "rb");
while(True):
d = f.read();#read file
if(d == ""):
break;
Send(sok, d, "");#Send file data
f.close();
time.sleep(0.8);#Problem here!!!!!!!!!!!
Send(sok,"");#Send termination to the server
time.sleep(1);#Wait the server to write the file
os.chdir("..");
Send(sok,"endfile");#let the server know that we finish sending files
else:
Send("endfile")#If not folder send termination
try:
sok1 = socket.socket();
sok1.connect(("192.168.1.121",4444))#local ip
time.sleep(1);
while(True):
Send(sok1,"Enter folder name to download: ");
r = sok1.recv(1024);
SendFolder(sok1,r);
time.sleep(0.5);
except BaseException, e:
print "Error: " + str(e);
os._exit(1);
Server:
import sys,os,socket,time
# receive data
def Receive(sock, end="292929"):
data = "";
while(True):
if(data.endswith(end)):
break;
else:
data = sock.recv(1024);
return data[:-len(end)];#return data less termination
def FolderReceive(sok):
while(True):
r = Receive(sok);# recv filename or folder termination("endfile")
if(r == "endfolder"):
print "Folder receive complete.";
break;
else:
filename = r;#file name
filedata = Receive(sok);# receive file data
f = open(filename,"wb");
f.write(filedata);
f.close();#finish to write the file
print "Received: " + filename;
try:
sok1 = socket.socket();
sok1.bind(("0.0.0.0",4444));
sok1.listen(5);
cl , addr = sok1.accept();#accepts connection
while(True):
r = Receive(cl);
sys.stdout.write("\n" + r);
next = raw_input();
cl.sendall(next);#send folder name to the client
FolderReceive(cl);
except BaseException, e:
print "Error: " + str(e);
os._exit(1);
I know this not best server ever...but is what I know. This just work for a folder with small files because if I send big files(like 5mb...) it crashes because the time the client waits for the server is not enough.
So my question is how can I send the files to the server without client need to wait??or know exactly how many time the client needs to wait for the server to receive the file?? Some code that does the same but handling any file size, any help?

TCP sockets are byte streams, not message streams. If you want to send a series of separate messages (like your separate files), you need to define a protocol, and then write a protocol handler. There is no way around that; just guessing at the timing or trying to take advantage of packet boundaries cannot possibly work.
The blog post linked above shows one way to do it. But you can do it with string delimiters if you want. But you have to deal with two problems:
The delimiter can appear anywhere in a read packet, not just at the end.
The delimiter can be split on packet boundaries—you may get "2929" at the end of one read, and the other "29" at the start of the next.
The usually way you do that is to accumulate a buffer, and search for the delimiter anywhere in the buffer. Something like this:
def message(sock, delimiter):
buf = ''
while True:
data = sock.read(4096)
if not data:
# If the socket closes with no delimiter, this will
# treat the last "partial file" as a complete file.
# If that's not what you want, just return, or raise.
yield buf
return
buf += data
messages = buf.split(delimiter)
for message in messages[:-1]:
yield message
buf = message[-1]
Meanwhile, you have another problem with your delimiter: There's nothing stopping it from appearing in the files you're trying to transmit. For example, what if you tried to send your script, or this web page?
That's one of the reasons that other protocols are often better than delimiters, but this isn't hard to deal with: just escape any delimiters found in the files. Since you're sending the whole file at once, you can just use replace on right before the sendall, and the reverse replace right before the split.