Beginner here, want to read in data with the file ending p.
My code looks like this :
import pickle
training_file = "/home/sk/CarND-Traffic-Sign-Classifier-Project/train.p"
testing_file = "/home/sk/CarND-Traffic-Sign-Classifier-Project/test.p"
with open(training_file, mode='rb') as f:
train = pickle.load(f)
with open(testing_file, mode='rb') as f:
test = pickle.load(f)
I get the following error:
ValueError: unsupported pickle protocol: 3
Can someone point out how i can fix it, either changing protocol or reading in the data some other way ?
Had the same issue when i created a pickle file using python3 and then tried loading it in python2. Try running your program with python3 or try creating a pickle file using python2.
Pickle uses different protocols to convert your data to a binary stream.
In python 2 there are 3 different protocols (0, 1, 2) and the default is 0.
In python 3 there are 5 different protocols (0, 1, 2, 3, 4) and the default is 3.
You must specify in python 3 a protocol lower than 3 in order to be able to load the data in python 2. You can specify the protocol parameter when invoking pickle.dump.
It seems as those files was created with a protocol >=3 (probably 3). So the only option you get is to load it into python 3 and then dump it with a lower protocol.
Evidently pickle protocol 3 was used in whatever python 3 code pickled the object. You can't unpickle with protocol 3 in python 2. You could however write a short python 3 program that loads it and then dumps it with protocol = 2. Then you can load them in python 2.
https://docs.python.org/2/library/pickle.html#usage
https://github.com/zopefoundation/zodbpickle
Under Python2, this package forks both Python 2.7's pickle and cPickle modules, adding support for the protocol 3 opcodes.
Related
I am trying to pipe the output of xz to a custom python script:
xz -cd file.pcap.xz | myscripy.py
However, I get an error when the script attempts to run this line:
#!/usr/bin/env python2.7
from __future__ import print_function
import pcap
import io
STDIN_ALIAS = '/proc/self/fd/0'
pcap.pcap(io.open(STDIN_ALIAS, 'r'))
and received an error
pcap.pcap(io.open(STDIN_ALIAS, 'r'))
File "pcap.pyx", line 196, in pcap.pcap.__init__
TypeError: expected string or Unicode object, _io.TextIOWrapper found
I am on Ubuntu 18.04 and running under python 2.7.
You can't use Python to pass in packets from a file to pcap.pcap(). The pypcap library you are using is a thin wrapper around the pcap_open_offline() and pcap_create() C functions, and offers no facilities for passing in a Python file object. This wrapper only accepts a filename or a network interface name, nothing else.
The pcap_open_offline() function does accept - as an alias for stdin, so just pass that in directly:
import pcap
sniffer = pcap.pcap('-')
The error message already tell you what happened. You need a string to the pcap() function, not a file object. To fix this, try
pcap.pcap(io.open(STDIN_ALIAS, 'r').read())
But I am not sure this will work as your file might be binary instead of text. In such case, you may want to open with 'rb' instead of 'r' flag, and do some conversion afterwards (especially if you use Python 3 instead of Python 2.7).
I see another issue: your code is not portable as it depends on this:
STDIN_ALIAS = '/proc/self/fd/0'
A pythonic way to read the stdin is the follows (see Reading binary data from stdin)
import sys
string = sys.stdin.read()
Have you tried upgrading PyPcap to work on Python 3? This could help, since Unicode handling is a lot cleaner and less prone to surprises on Python 3. The appropriate package is available, at least on Debian (and probably derived distros as well). Look for: python3-pypcap.
I am trying to use Mxnet-js library to visualize my Mxnet trained model in browser. I am following Mxnet-js git readme file.
They provided a python script. ./tool/model2json, to convert model to json file.
When i am running this script with my model i am getting error:
TypeError: write() argument must be str, not bytes
Getting this error make sense because, how can i write byte to a file that is opened in string mode. At line
model = base64.b64encode(bytes(open(sys.argv[3], 'rb').read()))
they are reading it in bytes but in line
with open(sys.argv1, 'w') as fo:
they are opening file in string mode and in line
fo.write(model)
they are writing bytes to string.
Am i missing something here ? Why they are trying to write bytes to string?
#!/usr/bin/env python
"""Simple util to convert mxnet model to json format."""
import sys
import json
import base64
if len(sys.argv) < 4:
print('Usage: <output.json> <symbol.json> <model.param>
[mean_image.nd] [synset]')
exit(0)
symbol_json = open(sys.argv[2]).read()
model = base64.b64encode(bytes(open(sys.argv[3], 'rb').read()))
mean_image = None
synset = None
if len(sys.argv) > 4:
mean_image = base64.b64encode(bytes(open(sys.argv[4],
'rb').read()))
if len(sys.argv) > 5:
synset = [l.strip() for l in open(sys.argv[5]).readlines()]
with open(sys.argv[1], 'w') as fo:
fo.write('{\n\"symbol\":\n')
fo.write(symbol_json)
if synset:
fo.write(',\n\"synset\": ')
fo.write(json.dumps(synset))
fo.write(',\n\"parambase64\": \"')
fo.write(model)
fo.write('\"\n')
if mean_image is not None:
fo.write(',\n\"meanimgbase64\": \"')
fo.write(mean_image)
fo.write('\"\n')
fo.write('}\n')
TL;DR
Are you using Python3? If so - use Python2 and things should work!
More details:
The code opens the model's binary weight file, reads the binary data, constructs a Bytes sequence (Python builtin type), and converts it into String.
Now, while Python 2 implicitly converts Bytes to String, Python 3 does not do it. So I suspect you are using Python 3, and then your conversion is incorrect.
To check your version run python --version
If you are indeed using Python 3, you can try and update line 12 of model2json.py to have explicit conversion:
model = str(base64.b64encode(bytes(open(sys.argv[3], 'rb').read())))
Note that for Python 3 you will also need to launch the local web server using a different command than the one noted on the readme.md: $ python3 -m http.server
My recommendation is that you use Python 2 since this entire repo is written for it, and using Python3 you might encounter other issues.
When I am trying to load something I dumped using cPickle, I get the error message:
ValueError: insecure string pickle
Both the dumping and loading work are done on the same computer, thus same OS: Ubuntu 8.04.
How could I solve this problem?
"are much more likely than a never-observed bug in Python itself in a functionality that's used billions of times a day all over the world": it always amazes me how cross people get in these forums.
One easy way to get this problem is by forgetting to close the stream that you're using for dumping the data structure. I just did
>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: insecure string pickle
Which is why I came here in the first place, because I couldn't see what I'd done wrong.
And then I actually thought about it, rather than just coming here, and realized that I should have done:
>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> out.close() # close it to make sure it's all been written
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Easy to forget. Didn't need people being told that they are idiots.
I've get this error in Python 2.7 because of open mode 'rb':
with open(path_to_file, 'rb') as pickle_file:
obj = pickle.load(pickle_file)
So, for Python 2 'mode' should be 'r'
Also, I've wondered that Python 3 doesn't support pickle format of Python 2, and in case when you'll try to load pickle file created in Python 2 you'll get:
pickle.unpicklingerror: the string opcode argument must be quoted
Check this thread. Peter Otten says:
A corrupted pickle. The error is
raised if a string in the dump does
not both start and end with " or '.
and shows a simple way to reproduce such "corruption". Steve Holden, in the follow-up post, suggests another way to cause the problem would be to mismatch 'rb' and 'wb' (but in Python 2 and on Linux that particular mistake should pass unnoticed).
What are you doing with data between dump() and load()? It's quite common error to store pickled data in file opened in text mode (on Windows) or in database storage in the way that doesn't work properly for binary data (VARCHAR, TEXT columns in some databases, some key-value storages). Try to compare pickled data that you pass to storage and immediately retrieved from it.
If anyone has this error using youtube-dl, this issue has the fix: https://github.com/rg3/youtube-dl/issues/7172#issuecomment-242961695
richiecannizzo commented on Aug 28
brew install libav
Should fix it instantly on mac or
sudo apt-get install libav
#on linux
This error may also occur with python 2 (and early versions of python 3) if your pickle is large (Python Issue #11564):
Python 2.7.11 |Anaconda custom (64-bit)| (default, Dec 6 2015, 18:08:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cPickle as pickle
>>> string = "X"*(2**31)
>>> pp = pickle.dumps(string)
>>> len(pp)
2147483656
>>> ss = pickle.loads(pp)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: insecure string pickle
This limitation was addressed with the introduction of pickle protocol 4 in python 3.4 (PEP 3154). Unfortunately, this feature has not been back-ported to python 2, and probably won't ever be. If this is your problem and you need to use python 2 pickle, the best you can do is reduce the size of your pickle, e.g., instead of pickling a list, pickle the elements individually into a list of pickles.
Same problem with a file that was made with python on windows, and reloaded with python on linux.
Solution : dos2unix on the file before reading in linux : works as a charm !
I got the Python ValueError: insecure string pickle message in a different way.
For me it happened after a base64 encoding a binary file and passing through urllib2 sockets.
Initially I was wrapping up a file like this
with open(path_to_binary_file) as data_file:
contents = data_file.read()
filename = os.path.split(path)[1]
url = 'http://0.0.0.0:8080/upload'
message = {"filename" : filename, "contents": contents}
pickled_message = cPickle.dumps(message)
base64_message = base64.b64encode(pickled_message)
the_hash = hashlib.md5(base64_message).hexdigest()
server_response = urllib2.urlopen(url, base64_message)
But on the server the hash kept coming out differently for some binary files
decoded_message = base64.b64decode(incoming_base64_message)
the_hash = hashlib.md5(decoded_message).hexdigest()
And unpickling gave insecure string pickle message
cPickle.loads(decoded_message)
BUT SUCCESS
What worked for me was to use urlsafe_b64encode()
base64_message = base64.urlsafe_b64encode(cPickle.dumps(message))
And decode with
base64_decoded_message = base64.urlsafe_b64decode(base64_message)
References
http://docs.python.org/2/library/base64.html
https://www.rfc-editor.org/rfc/rfc3548.html#section-3
This is what happened to me, might be a small section of population, but I want to put this out here nevertheless, for them:
Interpreter (Python3) would have given you an error saying it required the input file stream to be in bytes, and not as a string, and you may have changed the open mode argument from 'r' to 'rb', and now it is telling you the string is corrupt, and thats why you have come here.
The simplest option for such cases is to install Python2 (You can install 2.7) and then run your program with Python 2.7 environment, so it unpickles your file without issue. Basically I wasted a lot of time scanning my string seeing if it was indeed corrupt when all I had to do was change the mode of opening the file from rb to r, and then use Python2 to unpickle the file. So I'm just putting this information out there.
I ran into this earlier, found this thread, and assumed that I was immune to the file closing issue mentioned in a couple of these answers since I was using a with statement:
with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
pickle.dump(foo, temp_file)
# Push file to another machine
_send_file(temp_file.name)
However, since I was pushing the temp file from inside the with, the file still wasn't closed, so the file I was pushing was truncated. This resulted in the same insecure string pickle error in the script that read the file on the remote machine.
Two potential fixes to this: Keep the file open and force a flush:
with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
pickle.dump(foo, temp_file)
temp_file.flush()
# Push file to another machine
_send_file(temp_file.name)
Or make sure the file is closed before doing anything with it:
file_name = ''
with tempfile.NamedTemporaryFile(mode='wb', delete=False) as temp_file:
file_name = temp_file.name
pickle.dump(foo, temp_file)
# Push file to another machine
_send_file(file_name)
I use pickle to dump a file on python 3, and I use pickle to load the file on python 2, the ValueError appears.
So, python 2 pickle can not load the file dumped by python 3 pickle?
If I want it? How to do?
You should write the pickled data with a lower protocol number in Python 3. Python 3 introduced a new protocol with the number 3 (and uses it as default), so switch back to a value of 2 which can be read by Python 2.
Check the protocolparameter in pickle.dump. Your resulting code will look like this.
pickle.dump(your_object, your_file, protocol=2)
There is no protocolparameter in pickle.load because pickle can determine the protocol from the file.
Pickle uses different protocols to convert your data to a binary stream.
In python 2 there are 3 different protocols (0, 1, 2) and the default is 0.
In python 3 there are 5 different protocols (0, 1, 2, 3, 4) and the default is 3.
You must specify in python 3 a protocol lower than 3 in order to be able to load the data in python 2. You can specify the protocol parameter when invoking pickle.dump.
When I am trying to load something I dumped using cPickle, I get the error message:
ValueError: insecure string pickle
Both the dumping and loading work are done on the same computer, thus same OS: Ubuntu 8.04.
How could I solve this problem?
"are much more likely than a never-observed bug in Python itself in a functionality that's used billions of times a day all over the world": it always amazes me how cross people get in these forums.
One easy way to get this problem is by forgetting to close the stream that you're using for dumping the data structure. I just did
>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: insecure string pickle
Which is why I came here in the first place, because I couldn't see what I'd done wrong.
And then I actually thought about it, rather than just coming here, and realized that I should have done:
>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> out.close() # close it to make sure it's all been written
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Easy to forget. Didn't need people being told that they are idiots.
I've get this error in Python 2.7 because of open mode 'rb':
with open(path_to_file, 'rb') as pickle_file:
obj = pickle.load(pickle_file)
So, for Python 2 'mode' should be 'r'
Also, I've wondered that Python 3 doesn't support pickle format of Python 2, and in case when you'll try to load pickle file created in Python 2 you'll get:
pickle.unpicklingerror: the string opcode argument must be quoted
Check this thread. Peter Otten says:
A corrupted pickle. The error is
raised if a string in the dump does
not both start and end with " or '.
and shows a simple way to reproduce such "corruption". Steve Holden, in the follow-up post, suggests another way to cause the problem would be to mismatch 'rb' and 'wb' (but in Python 2 and on Linux that particular mistake should pass unnoticed).
What are you doing with data between dump() and load()? It's quite common error to store pickled data in file opened in text mode (on Windows) or in database storage in the way that doesn't work properly for binary data (VARCHAR, TEXT columns in some databases, some key-value storages). Try to compare pickled data that you pass to storage and immediately retrieved from it.
If anyone has this error using youtube-dl, this issue has the fix: https://github.com/rg3/youtube-dl/issues/7172#issuecomment-242961695
richiecannizzo commented on Aug 28
brew install libav
Should fix it instantly on mac or
sudo apt-get install libav
#on linux
This error may also occur with python 2 (and early versions of python 3) if your pickle is large (Python Issue #11564):
Python 2.7.11 |Anaconda custom (64-bit)| (default, Dec 6 2015, 18:08:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cPickle as pickle
>>> string = "X"*(2**31)
>>> pp = pickle.dumps(string)
>>> len(pp)
2147483656
>>> ss = pickle.loads(pp)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: insecure string pickle
This limitation was addressed with the introduction of pickle protocol 4 in python 3.4 (PEP 3154). Unfortunately, this feature has not been back-ported to python 2, and probably won't ever be. If this is your problem and you need to use python 2 pickle, the best you can do is reduce the size of your pickle, e.g., instead of pickling a list, pickle the elements individually into a list of pickles.
Same problem with a file that was made with python on windows, and reloaded with python on linux.
Solution : dos2unix on the file before reading in linux : works as a charm !
I got the Python ValueError: insecure string pickle message in a different way.
For me it happened after a base64 encoding a binary file and passing through urllib2 sockets.
Initially I was wrapping up a file like this
with open(path_to_binary_file) as data_file:
contents = data_file.read()
filename = os.path.split(path)[1]
url = 'http://0.0.0.0:8080/upload'
message = {"filename" : filename, "contents": contents}
pickled_message = cPickle.dumps(message)
base64_message = base64.b64encode(pickled_message)
the_hash = hashlib.md5(base64_message).hexdigest()
server_response = urllib2.urlopen(url, base64_message)
But on the server the hash kept coming out differently for some binary files
decoded_message = base64.b64decode(incoming_base64_message)
the_hash = hashlib.md5(decoded_message).hexdigest()
And unpickling gave insecure string pickle message
cPickle.loads(decoded_message)
BUT SUCCESS
What worked for me was to use urlsafe_b64encode()
base64_message = base64.urlsafe_b64encode(cPickle.dumps(message))
And decode with
base64_decoded_message = base64.urlsafe_b64decode(base64_message)
References
http://docs.python.org/2/library/base64.html
https://www.rfc-editor.org/rfc/rfc3548.html#section-3
This is what happened to me, might be a small section of population, but I want to put this out here nevertheless, for them:
Interpreter (Python3) would have given you an error saying it required the input file stream to be in bytes, and not as a string, and you may have changed the open mode argument from 'r' to 'rb', and now it is telling you the string is corrupt, and thats why you have come here.
The simplest option for such cases is to install Python2 (You can install 2.7) and then run your program with Python 2.7 environment, so it unpickles your file without issue. Basically I wasted a lot of time scanning my string seeing if it was indeed corrupt when all I had to do was change the mode of opening the file from rb to r, and then use Python2 to unpickle the file. So I'm just putting this information out there.
I ran into this earlier, found this thread, and assumed that I was immune to the file closing issue mentioned in a couple of these answers since I was using a with statement:
with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
pickle.dump(foo, temp_file)
# Push file to another machine
_send_file(temp_file.name)
However, since I was pushing the temp file from inside the with, the file still wasn't closed, so the file I was pushing was truncated. This resulted in the same insecure string pickle error in the script that read the file on the remote machine.
Two potential fixes to this: Keep the file open and force a flush:
with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
pickle.dump(foo, temp_file)
temp_file.flush()
# Push file to another machine
_send_file(temp_file.name)
Or make sure the file is closed before doing anything with it:
file_name = ''
with tempfile.NamedTemporaryFile(mode='wb', delete=False) as temp_file:
file_name = temp_file.name
pickle.dump(foo, temp_file)
# Push file to another machine
_send_file(file_name)