I want to read a .pyc file. However, I cannot find any documentation on the format.
The only one I found does not work for Python 3 (although it does for Python 2):
>>> f = open('__pycache__/foo.cpython-34.pyc', 'rb')
>>> f.read(4)
b'\xee\x0c\r\n'
>>> f.read(4)
b'\xf8\x17\x08W'
>>> marshal.load(f)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: bad marshal data (unknown type code)
marshal only consumes one byte: \x00, which indeed is not a valid first character for marshall (as a comparison, the first byte of Python 2 bytecode for the same empty module is c)
So, how can I decode what comes after the header?
Try this. It worked a while back. They added another int32 in v3.
def load_file(self, source):
if isinstance(source, str):
import os.path
if not os.path.exists(source):
raise IOError("Cannot load_file('"
+ source
+ "'): does not exist")
with open(source, "rb") as fh:
header_bytes = fh.read(12)
# ignore header
self.code = marshal.load(fh)
return self.code
Have you looked at the dissembler?
https://docs.python.org/3/library/dis.html
Related
I tried unzipping a file through Python using zipfile.extractAll but it gave BAD zip file, hence I tried this:
zipfile cant handle some type of zip data?
As mentioned in this answer, i used the code:
def fixBadZipfile(zipFile):
f = open(zipFile, 'r+b')
data = f.read()
pos = data.find('\x50\x4b\x05\x06') # End of central directory signature
if (pos > 0):
self._log("Truncating file at location " + str(pos + 22) + ".")
f.seek(pos + 22) # size of 'ZIP end of central directory record'
f.truncate()
f.close()
else:
# raise error, file is truncated enter code here
but it gave the error
Message File Name Line Position Traceback
C:\Users\aditya1.r\Desktop\Python_pyscripter\module1.py 50
main C:\Users\aditya1.r\Desktop\Python_pyscripter\module1.py 17
fixBadZipfile C:\Users\aditya1.r\Desktop\Python_pyscripter\module1.py 37
TypeError: 'str' does not support the buffer interface
I'm using Python 3.4
How can i unzip this file?
import subprocess
subprocess.Popen('unzip ' + file_name, shell = True).wait()
Hope this help you :)
You are reading the file as a bytes object but trying to find passing a string object so just simply change this line -
pos = data.find('\x50\x4b\x05\x06')
to
pos = data.find(b'\x50\x4b\x05\x06')
Note that I have casted it to a byte object by simply prepending a b.
You don't need to do this is Python 2.X but in python 3.X you need to explicitly serialize a string object to a byte object.
I'm using python3.3 and I'm having a cryptic error when trying to pickle a simple dictionary.
Here is the code:
import os
import pickle
from pickle import *
os.chdir('c:/Python26/progfiles/')
def storvars(vdict):
f = open('varstor.txt','w')
pickle.dump(vdict,f,)
f.close()
return
mydict = {'name':'john','gender':'male','age':'45'}
storvars(mydict)
and I get:
Traceback (most recent call last):
File "C:/Python26/test18.py", line 31, in <module>
storvars(mydict)
File "C:/Python26/test18.py", line 14, in storvars
pickle.dump(vdict,f,)
TypeError: must be str, not bytes
The output file needs to be opened in binary mode:
f = open('varstor.txt','w')
needs to be:
f = open('varstor.txt','wb')
Just had same issue. In Python 3, Binary modes 'wb', 'rb' must be specified whereas in Python 2x, they are not needed. When you follow tutorials that are based on Python 2x, that's why you are here.
import pickle
class MyUser(object):
def __init__(self,name):
self.name = name
user = MyUser('Peter')
print("Before serialization: ")
print(user.name)
print("------------")
serialized = pickle.dumps(user)
filename = 'serialized.native'
with open(filename,'wb') as file_object:
file_object.write(serialized)
with open(filename,'rb') as file_object:
raw_data = file_object.read()
deserialized = pickle.loads(raw_data)
print("Loading from serialized file: ")
user2 = deserialized
print(user2.name)
print("------------")
pickle uses a binary protocol, hence only accepts binary files. As the document said in the first sentence, "The pickle module implements binary protocols for serializing and de-serializing".
I am getting an interesting error while trying to use Unpickler.load(), here is the source code:
open(target, 'a').close()
scores = {};
with open(target, "rb") as file:
unpickler = pickle.Unpickler(file);
scores = unpickler.load();
if not isinstance(scores, dict):
scores = {};
Here is the traceback:
Traceback (most recent call last):
File "G:\python\pendu\user_test.py", line 3, in <module>:
save_user_points("Magix", 30);
File "G:\python\pendu\user.py", line 22, in save_user_points:
scores = unpickler.load();
EOFError: Ran out of input
The file I am trying to read is empty.
How can I avoid getting this error, and get an empty variable instead?
Most of the answers here have dealt with how to mange EOFError exceptions, which is really handy if you're unsure about whether the pickled object is empty or not.
However, if you're surprised that the pickle file is empty, it could be because you opened the filename through 'wb' or some other mode that could have over-written the file.
for example:
filename = 'cd.pkl'
with open(filename, 'wb') as f:
classification_dict = pickle.load(f)
This will over-write the pickled file. You might have done this by mistake before using:
...
open(filename, 'rb') as f:
And then got the EOFError because the previous block of code over-wrote the cd.pkl file.
When working in Jupyter, or in the console (Spyder) I usually write a wrapper over the reading/writing code, and call the wrapper subsequently. This avoids common read-write mistakes, and saves a bit of time if you're going to be reading the same file multiple times through your travails
I would check that the file is not empty first:
import os
scores = {} # scores is an empty dict already
if os.path.getsize(target) > 0:
with open(target, "rb") as f:
unpickler = pickle.Unpickler(f)
# if file is not empty scores will be equal
# to the value unpickled
scores = unpickler.load()
Also open(target, 'a').close() is doing nothing in your code and you don't need to use ;.
It is very likely that the pickled file is empty.
It is surprisingly easy to overwrite a pickle file if you're copying and pasting code.
For example the following writes a pickle file:
pickle.dump(df,open('df.p','wb'))
And if you copied this code to reopen it, but forgot to change 'wb' to 'rb' then you would overwrite the file:
df=pickle.load(open('df.p','wb'))
The correct syntax is
df=pickle.load(open('df.p','rb'))
As you see, that's actually a natural error ..
A typical construct for reading from an Unpickler object would be like this ..
try:
data = unpickler.load()
except EOFError:
data = list() # or whatever you want
EOFError is simply raised, because it was reading an empty file, it just meant End of File ..
You can catch that exception and return whatever you want from there.
open(target, 'a').close()
scores = {};
try:
with open(target, "rb") as file:
unpickler = pickle.Unpickler(file);
scores = unpickler.load();
if not isinstance(scores, dict):
scores = {};
except EOFError:
return {}
if path.exists(Score_file):
try :
with open(Score_file , "rb") as prev_Scr:
return Unpickler(prev_Scr).load()
except EOFError :
return dict()
Had the same issue. It turns out when I was writing to my pickle file I had not used the file.close(). Inserted that line in and the error was no more.
I have encountered this error many times and it always occurs because after writing into the file, I didn't close it. If we don't close the file the content stays in the buffer and the file stays empty.
To save the content into the file, either file should be closed or file_object should go out of scope.
That's why at the time of loading it's giving the ran out of input error because the file is empty. So you have two options :
file_object.close()
file_object.flush(): if you don't wanna close your file in between the program, you can use the flush() function as it will forcefully move the content from the buffer to the file.
This error comes when your pickle file is empty (0 Bytes). You need to check the size of your pickle file first. This was the scenario in my case. Hope this helps!
Note that the mode of opening files is 'a' or some other have alphabet 'a' will also make error because of the overwritting.
pointer = open('makeaafile.txt', 'ab+')
tes = pickle.load(pointer, encoding='utf-8')
temp_model = os.path.join(models_dir, train_type + '_' + part + '_' + str(pc))
# print(type(temp_model)) # <class 'str'>
filehandler = open(temp_model, "rb")
# print(type(filehandler)) # <class '_io.BufferedReader'>
try:
pdm_temp = pickle.load(filehandler)
except UnicodeDecodeError:
pdm_temp = pickle.load(filehandler, fix_imports=True, encoding="latin1")
from os.path import getsize as size
from pickle import *
if size(target)>0:
with open(target,'rb') as f:
scores={i:j for i,j in enumerate(load(f))}
else: scores={}
#line 1.
we importing Function 'getsize' from Library 'OS' sublibrary 'path' and we rename it with command 'as' for shorter style of writing. Important is hier that we loading only one single Func that we need and not whole Library!
line 2.
Same Idea, but when we dont know wich modul we will use in code at the begining, we can import all library using a command '*'.
line 3.
Conditional Statement... if size of your file >0 ( means obj is not an empty). 'target' is variable that schould be a bit earlier predefined.
just an Example : target=(r'd:\dir1\dir.2..\YourDataFile.bin')
Line 4.
'With open(target) as file:' an open construction for any file, u dont need then to use file.close(). it helps to avoid some typical Errors such as "Run out of input" or Permissions rights.
'rb' mod means 'rea binary' that u can only read(load) the data from your binary file but u cant modify/rewrite it.
Line5.
List comprehension method in applying to a Dictionary..
line 6. Case your datafile is empty, it will not raise an any Error msg, but return just an empty dictionary.
I have trouble loading excel files into a dataframe using ExcelFile(). I have imported pandas,xlrd and openpyxl. I am using spyder for interactive data analysis.
I'm new to pandas and python, so I would appriciate an answer that is understandable for a beginner. Could someone help me?
>>> import xlrd
>>> import openpyxl
>>> from pandas import *
>>> xls = ExcelFile('C:\RWFC\test.xls')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\pandas\io\parsers.py", line 1294, in __init__
self.book = xlrd.open_workbook(path_or_buf)
File "C:\Python27\lib\site-packages\xlrd\__init__.py", line 400, in open_workbook
f = open(filename, "rb")
IOError: [Errno 22] invalid mode ('rb') or filename: 'C:\\RWFC\test.xls'
The problem is in this line:
>>> xls = ExcelFile('C:\RWFC\test.xls')
The backward slash has a special meaning. For example, the character "\t" in a normal string is the tab character:
>>> "\t"
'\t'
>>> len("\t")
1
That's why in your error message:
IOError: [Errno 22] invalid mode ('rb') or filename: 'C:\\RWFC\test.xls'
You see a double slash in front of the R -- \R doesn't have any special meaning, and so it knew you meant one "real" slash:
>>> s = "\\"
>>> s
'\\'
>>> print s
\
>>> len(s)
1
but \t does have a special meaning. To solve this problem you can either use a "raw string", and add "r" before the string literal:
>>> "C:\RWFC\test.xls"
'C:\\RWFC\test.xls'
>>> r"C:\RWFC\test.xls"
'C:\\RWFC\\test.xls'
or, you can simply use forward slashes instead -- which Windows supports -- and avoid all the trouble:
>>> "C:/RWFC/test.xls"
'C:/RWFC/test.xls'
Either way should work.
I was having a similar problem. I resolved the issue this way:
path = r"Drive:\path\to\your\file.extension"
workbook = xlrd.open_workbook(path) ##assuming you have imported xlrd already
Hope this helps. :)
I'm using python3.3 and I'm having a cryptic error when trying to pickle a simple dictionary.
Here is the code:
import os
import pickle
from pickle import *
os.chdir('c:/Python26/progfiles/')
def storvars(vdict):
f = open('varstor.txt','w')
pickle.dump(vdict,f,)
f.close()
return
mydict = {'name':'john','gender':'male','age':'45'}
storvars(mydict)
and I get:
Traceback (most recent call last):
File "C:/Python26/test18.py", line 31, in <module>
storvars(mydict)
File "C:/Python26/test18.py", line 14, in storvars
pickle.dump(vdict,f,)
TypeError: must be str, not bytes
The output file needs to be opened in binary mode:
f = open('varstor.txt','w')
needs to be:
f = open('varstor.txt','wb')
Just had same issue. In Python 3, Binary modes 'wb', 'rb' must be specified whereas in Python 2x, they are not needed. When you follow tutorials that are based on Python 2x, that's why you are here.
import pickle
class MyUser(object):
def __init__(self,name):
self.name = name
user = MyUser('Peter')
print("Before serialization: ")
print(user.name)
print("------------")
serialized = pickle.dumps(user)
filename = 'serialized.native'
with open(filename,'wb') as file_object:
file_object.write(serialized)
with open(filename,'rb') as file_object:
raw_data = file_object.read()
deserialized = pickle.loads(raw_data)
print("Loading from serialized file: ")
user2 = deserialized
print(user2.name)
print("------------")
pickle uses a binary protocol, hence only accepts binary files. As the document said in the first sentence, "The pickle module implements binary protocols for serializing and de-serializing".