I've got problem trying open .pcap file. In scapy.utils there is RawPcapReader
try:
self.f = gzip.open(filename,"rb")
magic = self.f.read(4)
except IOError:
self.f = open(filename,"rb")
magic = self.f.read(4)
if magic == "\xa1\xb2\xc3\xd4": #big endian
self.endian = ">"
elif magic == "\xd4\xc3\xb2\xa1": #little endian
self.endian = "<"
else:
raise Scapy_Exception("Not a pcap capture file (bad magic)")
hdr = self.f.read(20)
if len(hdr)<20:
raise Scapy_Exception("Invalid pcap file (too short)")
My magic has value "\n\r\r\n" but RawPcapReader is expecting magic == "\xa1\xb2\xc3\xd4" or magic == "\xd4\xc3\xb2\xa1".
Could you tell me what can be the problem? With .pcap file? I'm using python version 2.7
The magic value of "\n\r\r\n" (\x0A\x0D\x0D\x0A) indicates that your file is actually in .pcapng format, rather than libpcap
The solution is simple
In Wireshark 'Save As': Wireshark/tcpdump - pcap
Or use tshark:
$tshark -r old.pcapng -w new.pcap -F libpcap
As an alternative to saving the file in pcap format, scapy now has PcapNgReader so you could do:
mypcap = PcapNgReader(filename)
Related
I'm French, sorry if my english isn't perfect !
Before starting, if you want to try my code, you can download a pcap sample file here : https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=ipv4frags.pcap
I succeed to open pcap file, read packets and write them to another file with this code :
# Python 3.6
# Scapy 2.4.3
from scapy.utils import PcapReader, PcapWriter
import time
i_pcap_filepath = "inputfile.pcap" # pcap to read
o_filepath = "outputfile.pcap" # pcap to write
i_open_file = PcapReader(i_pcap_filepath) # opened file to read
o_open_file = PcapWriter(o_filepath, append=True) # opened file to write
while 1:
# I will have EOF exception but anyway
time.sleep(1) # in order to see packet
packet = i_open_file.read_packet() # read a packet in file
o_open_file.write(packet) # write it
So now I want to write in a FIFO and see the result in a live Wireshark window.
To do that, I just create a FIFO :
$ mkfifo /my/project/location/fifo.fifo
and launch Wireshark application on it : $ wireshark -k -i /my/project/location/fifo.fifo
I change my filepath in my Python script : o_filepath = "fifo.fifo" # fifo to write
But I have a crash ... Here is the traceback :
Traceback (most recent call last):
File "fifo.py", line 25, in <module>
o_open_file = PcapWriter(o_pcap_filepath, append=True)
File "/home/localuser/.local/lib/python3.6/site-packages/scapy/utils.py", line 1264, in __init__
self.f = [open, gzip.open][gz](filename, append and "ab" or "wb", gz and 9 or bufsz) # noqa: E501
OSError: [Errno 29] Illegal seek
Wireshark also give me an error ("End of file on pipe magic during open") : wireshark error
I don't understand why, and what to do. Is it not possible to write in FIFO using scapy.utils library ? How to do then ?
Thank you for your support,
Nicos44k
Night was useful because I fix my issue this morning !
I didn't undestand the traceback yesterday but it give me in reality a big hint : we have a seek problem.
Wait ... There is no seek in FIFO file !!!
So we cannot set "append" parameter to true.
I changed with : o_open_file = PcapWriter(o_filepath)
And error is gone.
However, packets were not showing in live...
To solve this problem, I needed to force FIFO flush with : o_open_file.flush()
Remember that you can download a pcap sample file here : https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=ipv4frags.pcap
So here is the full code :
# Python 3.6
# Scapy 2.4.3
from scapy.utils import PcapReader, PcapWriter
import time
i_pcap_filepath = "inputfile.pcap" # pcap to read
o_filepath = "fifo.fifo" # pcap to write
i_open_file = PcapReader(i_pcap_filepath) # opened file to read
o_open_file = PcapWriter(o_filepath) # opened file to write
while 1:
# I will have EOF exception but anyway
time.sleep(1) # in order to see packet
packet = i_open_file.read_packet() # read a packet in file
o_open_file.write(packet) # write it
o_open_file.flush() # force buffered data to be written to the file
Have a good day !
Nicos44k
Error:
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
MemoryError
def download(self):
print("start thread:%s at %s" % (self.getName(), time.time()))
headers = {"Range": "bytes=%s-%s" % (self.startpos, self.endpos)}
res = requests.get(self.url, headers=headers, stream=True)
# res.text 是将get获取的byte类型数据自动编码，是str类型， res.content是原始的byte类型数据
# 所以下面是直接write(res.content)
with open(self.filename, "wb") as fp:
fp.seek(self.startpos)
fp.write(res.content)
print("stop thread:%s at %s" % (self.getName(), time.time()))
# f.close()
def run(self):
self.download()
I had the same problem on Windows 10. If your platform is NOT Windows NT or newer, this answer will not help.
The problem is: on Windows the socket input is always buffered. This can NOT be avoided. No way. And this causes MemoryError because of Windows (not by Python code) and by Python C internals - socket input is buffered to a C variable, these have limited size so we have to "help" that buffer - the solution is: creating a bytearray object, calling its join method (second parameter - sockobj.recv(1024)) in a while loop. To accomplish this, navigate to <Python Installdir>\Lib\requests.py (can also be Lib\site-packages\requests\__init__.py - notice TWO underscores!) and fix all socket usage with the fix specified above. Also perform such fix for all files in the requests package and also urllib3 package.
Hope this helps.
No. Python 64bit will NOT fix your problem: Windows Socket Buffer will remain THE SAME! Only looped recv with explicit Python bytearray buffer will help!
You're using stream=True, which explicitly has the use of buffering the data so you can asynchronously empty the buffer.
You're not doing that. So simply don't use stream=True but directly save to a file.
I have been pulling my hair out trying to get a proxy working. I need to decrypt the packets from a server and client ((this may be out of order..)), then decompress everything but the packet header.
The first 2 packets ((10101 and 20104)) are not compressed, and decrypt, destruct, and decompile properly.
Alas, but to no avail; FAIL!; zlib.error: Error -5 while decompressing data: incomplete or truncated stream
Same error while I am attempting to decompress the encrypted version of the packet.
When I include the packet header, I get a randomly chosen -3 error.
I have also tried changing -zlib.MAX_WBITS to zlib.MAX_WBITS, as well as a few others, but still get the same error.
Here's the code;
import socket, sys, os, struct, zlib
from Crypto.Cipher import ARC4 as rc4
cwd = os.getcwd()
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ss = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('192.168.2.12',9339))
s.listen(1)
client, addr = s.accept()
key = "fhsd6f86f67rt8fw78fw789we78r9789wer6renonce"
cts = rc4.new(key)
stc = rc4.new(key)
skip = 'a'*len(key)
cts.encrypt(skip)
stc.encrypt(skip)
ss.connect(('game.boombeachgame.com',9339))
ss.settimeout(0.25)
s.settimeout(0.25)
def io():
while True:
try:
pack = client.recv(65536)
decpack = cts.decrypt(pack[7:])
msgid, paylen = dechead(pack)
if msgid != 10101:
decopack = zlib.decompress(decpack, -zlib.MAX_WBITS)
print "ID:",msgid
print "Payload Length",paylen
print "Payload:\n",decpack
ss.send(pack)
dump(msgid, decpack)
except socket.timeout:
pass
try:
pack = ss.recv(65536)
msgid, paylen = dechead(pack)
decpack = stc.decrypt(pack[7:])
if msgid != 20104:
decopack = zlib.decompress(decpack, -zlib.MAX_WBITS)
print "ID:",msgid
print "Payload Length",paylen
print "Payload:\n",decpack
client.send(pack)
dump(msgid, decpack)
except socket.timeout:
pass
def dump(msgid, decpack):
global cwd
pdf = open(cwd+"/"+str(msgid)+".bin",'wb')
pdf.write(decpack)
pdf.close()
def dechead(pack):
msgid = struct.unpack('>H', pack[0:2])[0]
print int(struct.unpack('>H', pack[5:7])[0])
payload_bytes = struct.unpack('BBB', pack[2:5])
payload_len = ((payload_bytes[0] & 255) << 16) | ((payload_bytes[1] & 255) << 8) | (payload_bytes[2] & 255)
return msgid, payload_len
io()
I realize it's messy, disorganized and very bad, but it all works as intended minus the decompression.
Yes, I am sure the packets are zlib compressed.
What is going wrong here and why?
Full Traceback:
Traceback (most recent call last):
File "bbproxy.py", line 68, in <module>
io()
File "bbproxy.py", line 33, in io
decopack = zlib.decompress(decpack, zlib.MAX_WBITS)
zlib.error: Error -5 while decompressing data: incomplete or truncated stream
I ran into the same problem while trying to decompress a file using zlib with Python 2.7. The issue had to do with the size of the stream (or file input) exceeding the size that could be stored in memory. (My PC has 16 GB of memory, so it was not exceeding the physical memory size, but the buffer default size is 16384.)
The easiest fix was to change the code from:
import zlib
f_in = open('my_data.zz', 'rb')
comp_data = f_in.read()
data = zlib.decompress(comp_data)
To:
import zlib
f_in = open('my_data.zz', 'rb')
comp_data = f_in.read()
zobj = zlib.decompressobj() # obj for decompressing data streams that won’t fit into memory at once.
data = zobj.decompress(comp_data)
It handles the stream by buffering it and feeding in into the decompressor in manageable chunks.
I hope this helps to save you time trying to figure out the problem. I had help from my friend Jordan! I was trying all kinds of different window sizes (wbits).
Edit: Even with the below working on partial gz files for some files when I decompressed I got empty byte array and everything I tried would always return empty though the function was successful. Eventually I resorted to running gunzip process which always works:
def gunzip_string(the_string):
proc = subprocess.Popen('gunzip',stdout=subprocess.PIPE,
stdin=subprocess.PIPE, stderr=subprocess.DEVNULL)
proc.stdin.write(the_body)
proc.stdin.close()
body = proc.stdout.read()
proc.wait()
return body
Note that the above can return a non-zero error code indicating that the input string is incomplete but it still performs the decompression and hence the stderr being swallowed. You may wish to check errors to allow for this case.
/edit
I think the zlib decompression library is throwing an exception because you are not passing in a complete file just a 65536 chunk ss.recv(65536). If you change from this:
decopack = zlib.decompress(decpack, -zlib.MAX_WBITS)
to
decompressor = zlib.decompressobj(-zlib.MAX_WBITS)
decopack = decompressor(decpack)
it should work as that way can handle streaming.
A the docs say
zlib.decompressobj - Returns a decompression object, to be used for decompressing data streams that won’t fit into memory at once.
or even if it does fit into memory you might just want to do the beginning of the file
Try this:
decopack = zlib.decompressobj().decompress(decpack, zlib.MAX_WBITS)
I'm trying to write a json dumps string using linux bash shell echo in a text file. My problem is it removes all double quotes.
example code.
d = {"key": "value"}
"echo %s > /home/user/%s" % (simplejson.dumps(d), 'textfile'))
Output in textfile
{key: value}
It removes all double quotes so I can't load it to json because it is not a valid json anymore.
Thanks
You need to escape quotes for Bash usage:
("echo %s > /home/user/%s" % (simplejson.dumps(d), 'textfile')).replace('"', '\\"')
Since you said you're using paramiko, writing to the file directly is perfect. Edited code to reflect paramiko:
You can write to the file directly after logging onto the server, no need to pass in the bash command (which is hackish as is).
You will need two try-catch's: one to catch any error in opening the file, the other to catch any write in the file. If you'd prefer an exception to be thrown up in either of those cases, remove the try-catch.
import paramiko
*do your ssh stuff to establish an SSH session to server*
sftp = ssh.open_sftp()
try:
file = sftp.file('/home/user/textfile', 'a+')
try:
file.write(simplejson.dumps(d))
except IOError:
...*do some error handling for the write here*
except IOError:
...*do some error handling for being unable to open the file here*
else:
file.close()
sftp.close()
Could I unblock a file in windows(7), which is automatically blocked by windows (downloaded from Internet) from a python script? A WindowsError is raised when such a file is encountered. I thought of catching this exception, and running a powershell script that goes something like:
Parameter Set: ByPath
Unblock-File [-Path] <String[]> [-Confirm] [-WhatIf] [ <CommonParameters>]
Parameter Set: ByLiteralPath
Unblock-File -LiteralPath <String[]> [-Confirm] [-WhatIf] [ <CommonParameters>]
I don't know powershell scripting. But if I had one I could call it from python. Could you folks help?
Yes, all you have to do is call the following command line from Python:
powershell.exe -Command Unblock-File -Path "c:\path\to\blocked file.ps1"
From this page about the Unblock-File command: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/unblock-file?view=powershell-7.2
Internally, the Unblock-File cmdlet removes the Zone.Identifier alternate data stream, which has a value of 3 to indicate that it was downloaded from the internet.
To remove an alternate data stream ads_name from a file path\to\file.ext, simply delete path\to\file.ext:ads_name:
try:
os.remove(your_file_path + ':Zone.Identifier')
except FileNotFoundError:
# The ADS did not exist, it was already unblocked or
# was never blocked in the first place
pass
# No need to open up a PowerShell subprocess!
(And similarly, to check if a file is blocked you can use os.path.isfile(your_file_path + ':Zone.Identifier'))
In a PowerShell script, you can use Unblock-File for this, or simply Remove-Item -Path $your_file_path':Zone.Identifier'.
Remove-Item also has a specific flag for alternate data streams: Remove-Item -Stream Zone.Identifier (which you can pipe in multiple files to, or a single -Path)
Late to the party . . . .
I have found that the Block status is simply an extra 'file' (stream) attached in NTFS and it can actually be accessed and somewhat manipulated by ordinary means. These are called Alternative Data Streams.
The ADS for file blocking (internet zone designation) is called ':Zone.Identifier' and contains, I think, some useful information:
[ZoneTransfer]
ZoneId=3
ReferrerUrl=https://www.google.com/
HostUrl=https://imgs.somewhere.com/product/1969297/some-pic.jpg
All the other info I have found says to just delete this extra stream.... But, personally, I want to keep this info.... So I tried changing the ZoneId to 0, but it still shows as Blocked in Windows File Properties.
I settled on moving it to another stream name so I can still find it later.
The below script originated from a more generic script called pyADS. I only care about deleting / changing the Zone.Identifier attached stream -- which can all be done with simple Python commands. So this is a stripped-down version. It has several really nice background references listed. I am currently running the latest Windows 10 and Python 3.8+; I make no guarantees this works on older versions.
import os
'''
References:
Accessing alternative data-streams of files on an NTFS volume https://www.codeproject.com/Articles/2670/Accessing-alternative-data-streams-of-files-on-an
Original ADS class (pyADS) https://github.com/RobinDavid/pyADS
SysInternal streams applet https://learn.microsoft.com/en-us/sysinternals/downloads/streams
Windows: killing the Zone.Identifier NTFS alternate data stream https://wiert.me/2011/11/25/windows-killing-the-zone-identifier-ntfs-alternate-data-stream-from-a-file-to-prevent-security-warning-popup/
Zone.Information https://learn.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/6e3f7352-d11c-4d76-8c39-2516a9df36e8
About URL Security Zones https://learn.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/ms537183(v=vs.85)?redirectedfrom=MSDN
GREAT info: How Windows Determines That the File.... http://woshub.com/how-windows-determines-that-the-file-has-been-downloaded-from-the-internet/
Dixin's Blog: Understanding File Blocking and Unblocking https://weblogs.asp.net/dixin/understanding-the-internet-file-blocking-and-unblocking
'''
class ADS2():
def __init__(self, filename):
self.filename = filename
def full_filename(self, stream):
return "%s:%s" % (self.filename, stream)
def add_stream_from_file(self, filename):
if os.path.exists(filename):
with open(filename, "rb") as f: content = f.read()
return self.add_stream_from_string(filename, content)
else:
print("Could not find file: {0}".format(filename))
return False
def add_stream_from_string(self, stream_name, bytes):
fullname = self.full_filename(os.path.basename(stream_name))
if os.path.exists(fullname):
print("Stream name already exists")
return False
else:
fd = open(fullname, "wb")
fd.write(bytes)
fd.close()
return True
def delete_stream(self, stream):
try:
os.remove(self.full_filename(stream))
return True
except:
return False
def get_stream_content(self, stream):
fd = open(self.full_filename(stream), "rb")
content = fd.read()
fd.close()
return content
def UnBlockFile(file, retainInfo=True):
ads = ADS2(file)
if zi := ads.get_stream_content("Zone.Identifier"):
ads.delete_stream("Zone.Identifier")
if retainInfo: ads.add_stream_from_string("Download.Info", zi)
### Usage:
from unblock_files import UnBlockFile
UnBlockFile(r"D:\Downloads\some-pic.jpg")
Before:
D:\downloads>dir /r
Volume in drive D is foo
Directory of D:\downloads
11/09/2021 10:05 AM 8 some-pic.jpg
126 some-pic.jpg:Zone.Identifier:$DATA
1 File(s) 8 bytes
D:\downloads>more <some-pic.jpg:Zone.Identifier:$DATA
[ZoneTransfer]
ZoneId=3
ReferrerUrl=https://www.google.com/
HostUrl=https://imgs.somewhere.com/product/1969297/some-pic.jpg
After:
D:\downloads>dir /r
Volume in drive D is foo
Directory of D:\downloads
11/09/2021 10:08 AM 8 some-pic.jpg
126 some-pic.jpg:Download.Info:$DATA
1 File(s) 8 bytes