I am trying to pipe the output of xz to a custom python script:
xz -cd file.pcap.xz | myscripy.py
However, I get an error when the script attempts to run this line:
#!/usr/bin/env python2.7
from __future__ import print_function
import pcap
import io
STDIN_ALIAS = '/proc/self/fd/0'
pcap.pcap(io.open(STDIN_ALIAS, 'r'))
and received an error
pcap.pcap(io.open(STDIN_ALIAS, 'r'))
File "pcap.pyx", line 196, in pcap.pcap.__init__
TypeError: expected string or Unicode object, _io.TextIOWrapper found
I am on Ubuntu 18.04 and running under python 2.7.
You can't use Python to pass in packets from a file to pcap.pcap(). The pypcap library you are using is a thin wrapper around the pcap_open_offline() and pcap_create() C functions, and offers no facilities for passing in a Python file object. This wrapper only accepts a filename or a network interface name, nothing else.
The pcap_open_offline() function does accept - as an alias for stdin, so just pass that in directly:
import pcap
sniffer = pcap.pcap('-')
The error message already tell you what happened. You need a string to the pcap() function, not a file object. To fix this, try
pcap.pcap(io.open(STDIN_ALIAS, 'r').read())
But I am not sure this will work as your file might be binary instead of text. In such case, you may want to open with 'rb' instead of 'r' flag, and do some conversion afterwards (especially if you use Python 3 instead of Python 2.7).
I see another issue: your code is not portable as it depends on this:
STDIN_ALIAS = '/proc/self/fd/0'
A pythonic way to read the stdin is the follows (see Reading binary data from stdin)
import sys
string = sys.stdin.read()
Have you tried upgrading PyPcap to work on Python 3? This could help, since Unicode handling is a lot cleaner and less prone to surprises on Python 3. The appropriate package is available, at least on Debian (and probably derived distros as well). Look for: python3-pypcap.
Related
To preface, this is not my code and am just trying to resolve it. I am not accustomed to Python (just have been introduced with it recently with this project)
We run python scripts via UiPath. We upgraded the version of all python dependencies to Py3.10 which meant I had to convert the scripts using 2to3 as well, I've pinpointed the issue but I don't have any idea why this part of the script would fail.
Debugging technique used - slip ctypes.windll.user32.MessageBoxW in between lines of the script.
Here's the part of the code that was no longer able to show the message box (which I assumed is where this problematic). Line formatted as bold.
dObj = open(param_response_result_file)
ogdata = json.load(dObj)
ogdata['results']['fields'] = field_value_dictionary
with open(param_response_result_file, 'w+') as outfile:
**json.dump(ogdata, outfile)**
outfile.write("\n")
What can be the root causes for this line to fail? I've checked param_response_result_file and it contains the correct values, what more can I check to trace why this json.dump is failing?
Thank you
EDIT I found the exception being thrown:
Object of type bytes is not JSON serializable
How do I resolve this? I am aware that there has been variable changes between Py 2 to 3 upgrade with json instantiations, but how do I get it to pass the correct variable?
I'm trying to read the duration of video files using mediainfo. This shell command works
mediainfo --Inform="Video;%Duration/String3%" file
and produces an output like
00:00:33.600
But when I try to run it in python with this line
subprocess.check_output(['mediainfo', '--Inform="Video;%Duration/String3%"', file])
the whole --Inform thing is ignored and I get the full mediainfo output instead.
Is there a way to see the command constructed by subprocess to see what's wrong?
Or can anybody just tell what's wrong?
Try:
subprocess.check_output(['mediainfo', '--Inform=Video;%Duration/String3%', file])
The " in your python string are likely passed on to mediainfo, which can't parse them and will ignore the option.
These kind of problems are often caused by shell commands requiring/swallowing various special characters. Quotes such as " are often removed by bash due to shell magic. In contrast, python does not require them for magic, and will thus replicate them the way you used them. Why would you use them if you wouldn't need them? (Well, d'uh, because bash makes you believe you need them).
For example, in bash I can do
$ dd of="foobar"
and it will write to a file named foobar, swallowing the quotes.
In python, if I do
subprocess.check_output(["dd", 'of="barfoo"', 'if=foobar'])
it will write to a file named "barfoo", keeping the quotes.
I'm newbie here and I wouldn't want to ask such a easy question as my first post but I don't know anything about Python even I'm a PHP/C programmer.
I have a python script in Figway tools which is called RegisterDevice.py to register my own sensor hardware to FIWARE Lab. But some code lines of that script doesn't work as I expected because of Python3.4. This may not be my problem but I don't have too much time to wait an official solution that's why I thought that I could resolve it as a person who is familiar to the programming.
I've searched on the web for solution but I couldn't find any exact solution for it yet. As far as I read bytes and unicode strings are two different types in Python3.x but I couldn't realize where I have to encode or maybe decode string to other type on the code. Maybe I have to do something else...
Here is the part of script which gave me error like above.
# Load the configuration file
with open(CONFIG_FILE,'r+') as f:
sample_config = f.read()
#config = ConfigParser.RawConfigParser(allow_no_value=True)
config = configparser.RawConfigParser(allow_no_value=True)
config.readfp(io.BytesIO(sample_config))
Error:
Traceback (most recent call last):
File "RegisterDevice.py", line 47, in <module>
config.readfp(io.BytesIO(sample_config))
TypeError: 'str' does not support the buffer interface
Firstly readfp() is deprecated in Python3 and you should use read_file().
The best way is probably using the read() function directly when you want to work with a file. You should set encoding as the second parameter if you expect non-ASCII characters inside the file.
The alternative is to read_string() and give it a string directly.
I have been doing work very similar to this, and I believe this script runs, but you will have to verify if it gives you the desired results:
import configparser
with open('.coveragerc','r+') as f:
#config = ConfigParser.RawConfigParser(allow_no_value=True)
config = configparser.RawConfigParser(allow_no_value=True)
config.readfp(f)
In Perl, the interpreter kind of stops when it encounters a line with
__END__
in it. This is often used to embed arbitrary data at the end of a perl script. In this way the perl script can fetch and store data that it stores 'in itself', which allows for quite nice opportunities.
In my case I have a pickled object that I want to store somewhere. While I can use a file.pickle file just fine, I was looking for a more compact approach (to distribute the script more easily).
Is there a mechanism that allows for embedding arbitrary data inside a python script somehow?
With pickle you can also work directly on strings.
s = pickle.dumps(obj)
pickle.loads(s)
If you combine that with """ (triple-quoted strings) you can easily store any pickled data in your file.
If the data is not particularly large (many K) I would just .encode('base64') it and include that in a triple-quoted string, with .decode('base64') to get back the binary data, and a pickle.loads() call around it.
In Python, you can use """ (triple-quoted strings) to embed long runs of text data in your program.
In your case, however, don't waste time on this.
If you have an object you've pickled, you'd be much, much happier dumping that object as Python source and simply including the source.
The repr function, applied to most objects, will emit a Python source-code version of the object. If you implement __repr__ for all of your custom classes, you can trivially dump your structure as Python source.
If, on the other hand, your pickled structure started out as Python code, just leave it as Python code.
I made this code. You run something like python comp.py foofile.tar.gz, and it creates decomp.py, with foofile.tar.gz's contents embedded in it. I don't think this is really portable with windows because of the Popen though.
import base64
import sys
import subprocess
inf = open(sys.argv[1],"r+b").read()
outs = base64.b64encode(inf)
decomppy = '''#!/usr/bin/python
import base64
def decomp(data):
fname = "%s"
outf = open(fname,"w+b")
outf.write(base64.b64decode(data))
outf.close()
# You can put the rest of your code here.
#Like this, to unzip an archive
#import subprocess
#subprocess.Popen("tar xzf " + fname, shell=True)
#subprocess.Popen("rm " + fname, shell=True)
''' %(sys.argv[1])
taildata = '''uudata = """%s"""
decomp(uudata)
''' %(outs)
outpy = open("decomp.py","w+b")
outpy.write(decomppy)
outpy.write(taildata)
outpy.close()
subprocess.Popen("chmod +x decomp.py",shell=True)
Well, this could be a simple question, to be frank I'm a little confused with encodings an all those things.
Let's suppose I have the file 01234.txt which is iso-8859-1.
When I do:
iconv --from-code=iso-8859-1 --to-code=utf-8 01234.txt > 01234_utf8.txt
It gives me the desired result, but when I do the same thing with python and using subprocess:
import subprocess
p0 = subprocess.Popen([<here the same command>], shell=True)
p0.wait()
I get almost the same result, but the new file is missing e.g. part of the line before the last one and the last one.
Here the last three lines of both files:
iconv result:
795719000|MARIA TERESA MARROU VILLALOBOS|107
259871385|CHRISTIAM ALBERTO SUAREZ VILLALOBOS|107
311015100|JORGE MEZA CERVANTES|09499386
python result:
795719000|MARIA TERESA MARROU VILLALOBOS|107
259871385|CHRISTIAM
EDIT: In the python file I've tried using coding: utf-8 and coding: iso-8859-1 (not both at the same time).
EDIT: I've used codecs in bpython it works great. When using it from a file I get the not desired result.
EDIT: I'm using linux (Ubuntu 9.10) and python 2.6.2.
Any suggestions?
You wrote: "In the python file I've used coding: utf-8 and coding: iso-8859-1."
Only the first of those will be used. Secondly, that specifies the encoding of the Python source file in which it appears, so that the Python compiler can do its job. Consequently it is absolutely nothing to do with the encodings of your input file and output file. A script to transcode data from encoding X to encoding Y can be written using only ASCII characters.
Now to your problem:
You wrote: "p0 = subprocess.Popen([<here the same command>], shell=True)"
Please (always) when asking a question, show the EXACT code that was run, not what you hoped/thought was run. Use copy/paste, don't retype it. Don't try to put it in a comment; edit your question.
Update: Here is a GUESS, based on the symptoms: you are losing the last few bytes of a file -- looks like failure to flush a buffer before fading away. Is the size of the truncated output file an integral power of 2?
Perhaps you should not rely on the command line processor doing > 01234_utf8.txt reliably. If you omit that part of the command, does the full payload appear on stdout? If, so you may be able to work around the problem by opening the output file yourself, passing its handle as the stdout arg, and later doing handle.flush() and handle.close().