Python hashlib MD5 digest of any UNC file always yields same hash - python

The below code shows that three files which are on a UNC share hosted on another machine have the same hash. It also shows that local files have different hashes. Why would this be? I feel that there is some UNC consideration that I don't know about.
Python 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> fn_a = '\\\\some.host.com\\Shares\\folder1\\file_a'
>>> fn_b = '\\\\some.host.com\\Shares\\folder1\\file_b'
>>> fn_c = '\\\\some.host.com\\Shares\\folder2\\file_c'
>>> fn_d = 'E:\\file_d'
>>> fn_e = 'E:\\file_e'
>>> fn_f = 'E:\\folder3\\file_f'
>>> f_a = open(fn_a, 'r')
>>> f_b = open(fn_b, 'r')
>>> f_c = open(fn_c, 'r')
>>> f_d = open(fn_d, 'r')
>>> f_e = open(fn_e, 'r')
>>> f_f = open(fn_f, 'r')
>>> hashlib.md5(f_a.read()).hexdigest()
'54637fdcade4b7fd7cabd45d51ab8311'
>>> hashlib.md5(f_b.read()).hexdigest()
'54637fdcade4b7fd7cabd45d51ab8311'
>>> hashlib.md5(f_c.read()).hexdigest()
'54637fdcade4b7fd7cabd45d51ab8311'
>>> hashlib.md5(f_d.read()).hexdigest()
'd2bf541b1a9d2fc1a985f65590476856'
>>> hashlib.md5(f_e.read()).hexdigest()
'e84be3c598a098f1af9f2a9d6f806ed5'
>>> hashlib.md5(f_f.read()).hexdigest()
'e11f04ed3534cc4784df3875defa0236'
EDIT: To further investigate the problem, I also tested using a file from another host. It appears that changing the host will change the result.
>>> fn_h = '\\\\host\\share\\file'
>>> f_h = open(fn_h, 'r')
>>> hashlib.md5(f_h.read()).hexdigest()
'f23ee2dbbb0040bf2586cfab29a03634'
...but then I tried a different file on the new host, and got a new result!
>>> fn_i = '\\\\host\\share\\different_file'
>>> f_i = open(fn_i, 'r')
>>> hashlib.md5(f_i.read()).hexdigest()
'a8ad771db7af8c96f635bcda8fdce961'
So, now I'm really confused. Could it have something to do with the fact that the original host is a \\host.com format and the new host is a \\host format?

I did some additional research based on the comments and answers everyone provided. I decided I needed to study permutations of these two features of the code:
A raw string literal is used for the path name, i.e. whether or not:
A. The file path string is raw with single backslashes in the path, vs.
B. The file path string is not raw with double backslashes in the path
(FYI to those who don't know, a raw string is one which is proceeded by an "r" like this: r'This is a raw string')
The open function mode is r or rb.
(FYI again to those who don't know, the b in rb mode indicates to read the file as binary.)
The results demonstrated:
The string literal / backslashes make no difference in whether or not the hashes of different files are different
My error was not opening the file in binary mode. When using rb mode in open, I got different results.
Yay! And thanks for the help.

Use f1.seek(0) if you intend to use it again, otherwise it would be a file completely read and calling read() again would just return a empty string.

I don't reproduce your problem. I'm using Python 3.4 on Windows 7 here with the following test script which accesses files on a network hard disk:
import sys, hashlib
def main():
fn0 = r'\\NAS\Public\Software\Backup\Test\Vagrantfile'
fn1 = r'\\NAS\Public\Software\Backup\Test\z.xml'
with open(fn0, 'rb') as f:
h0 = hashlib.md5(f.read())
print(h0.hexdigest())
with open(fn1, 'rb') as f:
h1 = hashlib.md5(f.read())
print(h1.hexdigest())
if __name__ == '__main__':
sys.exit(main())
Running this results in two different hash values (as expected):
c:\src\python>python hashtest.py
8af202dffb88739c2dbe188c12291e3d
2ff3db61ff37ca5ceac6a59fd7c1018b
If reading the file contents returns different data for the remote files then passing that data into md5 has to result in different hash values. You might want to print out the first 80 bytes of each file as a check that you are getting what you expect.

Related

passing directory with spaces in it from raw_input to os.listdir(path) in Python

I've looked at a few posts regarding this topic but haven't seen anything specifically fitting my usage. This one looks to be close to the same issue. They talk about escaping the escape characters.
My issue is that I want this script to run on both MAC and PC systems - and to be able to process files in a sub-folder that has spaces in the folder name.
Right now I use this code gleaned partially from several different SO posts:
directory = raw_input("Enter the location of the files: ")
path = r"%s" % directory
for file in os.listdir(path):
I don't quite understand what the second line is doing - so perhaps that's the obvious line that needs tweaking. It works fine with regular folder names but not with spaces in the name.
I have tried using "\ " instead of just " ", - that didn't work - but in any case I'm looking for a code solution. I don't want the user to have to specify an escape character
On Windoze using sub folder name "LAS Data" in response to raw_input prompt (without quotation marks), I get an message that:
the system can't find the path specified "LAS Data\*.*"
Your problem is most likely not with the code you present, but with the input you gave. The message
the system can't find the path specified "LAS Data\*.*"
suggest that you (or the user) entered the directory together with a wildcard for files. A directory with the name "LAS Data\*.*" indeed does not exist (unless you did something special to your file system :-).
Try entering just "LAS Data" instead.
The raw string should not be needed either
> python
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec 5 2015, 20:32:19) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> directory = raw_input("Enter the location of the files: ")
Enter the location of the files: Directory with Spaces
>>> for file in os.listdir(directory):
... print(file)
...
hello-1.txt
hello-2.txt
>>>
I don't quite understand what the [path = r"%s" % directory] line is doing
It creates a copy path of directory:
>>> directory = raw_input()
LAS Data
>>> path = r"%s" % directory
>>> path == directory
True
>>> type(path), type(directory)
(<type 'str'>, <type 'str'>)
Really wondering where you got that from. Seems pretty pointless.

What does `if file.find('freq-') != -1` mean?

I'm a chemistry student and want to write a script to extract some data (like coupling constants and interproton distance) from gaussian output files.
I found a script which extracts chemical shifts from gaussian output files. However, I don't understand what does if file.find('freq-') !=-1 mean in the script.
Here's part of the script (since the script also does other things as well so I've just sown the bit relevant to my question):
def read_gaussian_freq_outfiles(list_of_files):
list_of_freq_outfiles = []
for file in list_of_files:
if file.find('freq-') !=-1:
list_of_freq_outfiles.append([file,int(get_conf_number(file)),open(file,"r").readlines()])
return list_of_freq_outfiles
def read_gaussian_outputfiles():
list_of_files = []
for file in glob.glob('*.out'):
list_of_files.append(file)
return list_of_files
I think in the def read_gaussian_outputfiles() bit, we create a list of file and simply add all file with extension '.out' to the list.
The read_gaussian_freq_outfiles(list_of_files) bit has probably list files which has "freq-" in the file name. But what does the file.find('freq-')!=-1 mean?
Does it mean if whatever we find in the file name doesn't equal to -1, or something else?
Some other additional information: the format of the gaussian output filename is: xxxx-opt_freq-conf-yyyy.out where xxxx is the name of your molecule and yyyy is a number.
When s.find(foo) fails to find foo in s, it returns -1. Therefore, when s.find(foo) does not return -1, we know it didn't fail.
read_gaussian_freq_outfiles looks for the term "freq-" in each of the names of files in list_of_files. If it succeeds in finding this phrase in the name of a file, it appends a list containing this file, a "conf number" (not sure what this is), and the contents of the file, to a list called list_of_freq_outfiles.
I created three files, goodbye.txt, hello.txt, and helloworld.txt to demonstrate usage.
In this example, I'll print all files that end with .txt, create a list of files, then print all files that have the phrase "goodbye" in the filename. This should only print goodbye.txt.
09:53 $ ls
goodbye.txt hello.txt helloworld.txt
(venv) ✔ ~/Desktop/ex
09:53 $ python
Python 2.7.11 (default, Dec 5 2015, 14:44:47)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import glob
>>> for file in glob.glob('*.txt'):
... print(file)
...
goodbye.txt
hello.txt
helloworld.txt
>>> list_of_files = [ file for file in glob.glob('*.txt') ]
>>> print(list_of_files)
['goodbye.txt', 'hello.txt', 'helloworld.txt']
>>> for file in list_of_files:
... if file.find('goodbye') != -1:
... print(file)
...
goodbye.txt
Indeed, goodbye.txt is the only file printed.
As the other answers also show: if .find() retrieves -1, it cannot find what you're looking for. This has to do with the fact that .find will return the first index at which it can find your query. So in the following sentence
The cat is on the mat
and sentence.find('cat'), it will return 4 (since 'cat' starts at index 4 (it starts at 0!)).
However, sentence.find('dog') will return the only thing it can return if it cannot find it: -1. If it returned 0 as the 'cannot find', you might think your query starts at index 0. With -1, you know it could not find it.
String find method in python looks at the occurrence of a sub-string in a given string (ref http://www.tutorialspoint.com/python/string_find.htm)
Here it is looking for all the filenames with 'freq-' sub-string in them.

Unable to load .pkl file [duplicate]

When I am trying to load something I dumped using cPickle, I get the error message:
ValueError: insecure string pickle
Both the dumping and loading work are done on the same computer, thus same OS: Ubuntu 8.04.
How could I solve this problem?
"are much more likely than a never-observed bug in Python itself in a functionality that's used billions of times a day all over the world": it always amazes me how cross people get in these forums.
One easy way to get this problem is by forgetting to close the stream that you're using for dumping the data structure. I just did
>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: insecure string pickle
Which is why I came here in the first place, because I couldn't see what I'd done wrong.
And then I actually thought about it, rather than just coming here, and realized that I should have done:
>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> out.close() # close it to make sure it's all been written
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Easy to forget. Didn't need people being told that they are idiots.
I've get this error in Python 2.7 because of open mode 'rb':
with open(path_to_file, 'rb') as pickle_file:
obj = pickle.load(pickle_file)
So, for Python 2 'mode' should be 'r'
Also, I've wondered that Python 3 doesn't support pickle format of Python 2, and in case when you'll try to load pickle file created in Python 2 you'll get:
pickle.unpicklingerror: the string opcode argument must be quoted
Check this thread. Peter Otten says:
A corrupted pickle. The error is
raised if a string in the dump does
not both start and end with " or '.
and shows a simple way to reproduce such "corruption". Steve Holden, in the follow-up post, suggests another way to cause the problem would be to mismatch 'rb' and 'wb' (but in Python 2 and on Linux that particular mistake should pass unnoticed).
What are you doing with data between dump() and load()? It's quite common error to store pickled data in file opened in text mode (on Windows) or in database storage in the way that doesn't work properly for binary data (VARCHAR, TEXT columns in some databases, some key-value storages). Try to compare pickled data that you pass to storage and immediately retrieved from it.
If anyone has this error using youtube-dl, this issue has the fix: https://github.com/rg3/youtube-dl/issues/7172#issuecomment-242961695
richiecannizzo commented on Aug 28
brew install libav
Should fix it instantly on mac or
sudo apt-get install libav
#on linux
This error may also occur with python 2 (and early versions of python 3) if your pickle is large (Python Issue #11564):
Python 2.7.11 |Anaconda custom (64-bit)| (default, Dec 6 2015, 18:08:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cPickle as pickle
>>> string = "X"*(2**31)
>>> pp = pickle.dumps(string)
>>> len(pp)
2147483656
>>> ss = pickle.loads(pp)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: insecure string pickle
This limitation was addressed with the introduction of pickle protocol 4 in python 3.4 (PEP 3154). Unfortunately, this feature has not been back-ported to python 2, and probably won't ever be. If this is your problem and you need to use python 2 pickle, the best you can do is reduce the size of your pickle, e.g., instead of pickling a list, pickle the elements individually into a list of pickles.
Same problem with a file that was made with python on windows, and reloaded with python on linux.
Solution : dos2unix on the file before reading in linux : works as a charm !
I got the Python ValueError: insecure string pickle message in a different way.
For me it happened after a base64 encoding a binary file and passing through urllib2 sockets.
Initially I was wrapping up a file like this
with open(path_to_binary_file) as data_file:
contents = data_file.read()
filename = os.path.split(path)[1]
url = 'http://0.0.0.0:8080/upload'
message = {"filename" : filename, "contents": contents}
pickled_message = cPickle.dumps(message)
base64_message = base64.b64encode(pickled_message)
the_hash = hashlib.md5(base64_message).hexdigest()
server_response = urllib2.urlopen(url, base64_message)
But on the server the hash kept coming out differently for some binary files
decoded_message = base64.b64decode(incoming_base64_message)
the_hash = hashlib.md5(decoded_message).hexdigest()
And unpickling gave insecure string pickle message
cPickle.loads(decoded_message)
BUT SUCCESS
What worked for me was to use urlsafe_b64encode()
base64_message = base64.urlsafe_b64encode(cPickle.dumps(message))
And decode with
base64_decoded_message = base64.urlsafe_b64decode(base64_message)
References
http://docs.python.org/2/library/base64.html
https://www.rfc-editor.org/rfc/rfc3548.html#section-3
This is what happened to me, might be a small section of population, but I want to put this out here nevertheless, for them:
Interpreter (Python3) would have given you an error saying it required the input file stream to be in bytes, and not as a string, and you may have changed the open mode argument from 'r' to 'rb', and now it is telling you the string is corrupt, and thats why you have come here.
The simplest option for such cases is to install Python2 (You can install 2.7) and then run your program with Python 2.7 environment, so it unpickles your file without issue. Basically I wasted a lot of time scanning my string seeing if it was indeed corrupt when all I had to do was change the mode of opening the file from rb to r, and then use Python2 to unpickle the file. So I'm just putting this information out there.
I ran into this earlier, found this thread, and assumed that I was immune to the file closing issue mentioned in a couple of these answers since I was using a with statement:
with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
pickle.dump(foo, temp_file)
# Push file to another machine
_send_file(temp_file.name)
However, since I was pushing the temp file from inside the with, the file still wasn't closed, so the file I was pushing was truncated. This resulted in the same insecure string pickle error in the script that read the file on the remote machine.
Two potential fixes to this: Keep the file open and force a flush:
with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
pickle.dump(foo, temp_file)
temp_file.flush()
# Push file to another machine
_send_file(temp_file.name)
Or make sure the file is closed before doing anything with it:
file_name = ''
with tempfile.NamedTemporaryFile(mode='wb', delete=False) as temp_file:
file_name = temp_file.name
pickle.dump(foo, temp_file)
# Push file to another machine
_send_file(file_name)

Python: Save a list and recover it for AI learning application

I have an application that requires saving a list as it grows. This is a step towards doing AI learning stuff. Anyway here is my code:
vocab = ["cheese", "spam", "eggs"]
word = raw_input()
vocab.append(word)
However when the code is run the finished vocab will return to just being cheese, spam and eggs. How can I make whatever I append to the list stay there permenantly even when windows CMD is closed and I return to the code editing stage. Is this clear enough??
Thanks
You're looking into the more general problem of object persistence. There are bazillions of ways to do this. If you're just starting out, the easiest way to save/restore data structures in Python is using the pickle module. As you get more advanced, there are different methods and they all have their trade-offs...which you'll learn when you need to.
You could use json and files, something like this:
import json
#write the data to a file
outfile = open("dumpFile", 'w')
json.dump(vocab, outfile)
#read the data back in
with open("dumpFile") as infile:
newVocab = json.load(infile)
This has the advantage of being a plain text file, so you can view the data stored in it easily.
Look into pickle
With it, you can serialize a Python data structure and then reload it like so:
>>> import pickle
>>> vocab =["cheese", "spam", "eggs"]
>>> outf=open('vocab.pkl','wb')
>>> pickle.dump(vocab,outf)
>>> outf.close()
>>> quit()
Python interpreter is now exited, restart Python and reload the data structure:
abd-pb:~ andrew$ python
Python 2.7.1 (r271:86882M, Nov 30 2010, 10:35:34)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pklf=open('vocab.pkl','rb')
>>> l=pickle.load(pklf)
>>> l
['cheese', 'spam', 'eggs']
You could use pickle, but as a technique it's limited to python programs. The normal way to deal with persistent data is to write it to a file. Just read your word list from a regular text file (one word per line), and write out an updated word list later. Later you can learn how to keep this kind of information in a database (more powerful but less flexible than a file).
You can happily program for years without actually needing pickle, but you can't do without file i/o and databases.
PS. Keep it simple: You don't need to mess with json, pickle or any other structured format unless and until you NEED structure.

ValueError: insecure string pickle

When I am trying to load something I dumped using cPickle, I get the error message:
ValueError: insecure string pickle
Both the dumping and loading work are done on the same computer, thus same OS: Ubuntu 8.04.
How could I solve this problem?
"are much more likely than a never-observed bug in Python itself in a functionality that's used billions of times a day all over the world": it always amazes me how cross people get in these forums.
One easy way to get this problem is by forgetting to close the stream that you're using for dumping the data structure. I just did
>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: insecure string pickle
Which is why I came here in the first place, because I couldn't see what I'd done wrong.
And then I actually thought about it, rather than just coming here, and realized that I should have done:
>>> out = open('xxx.dmp', 'w')
>>> cPickle.dump(d, out)
>>> out.close() # close it to make sure it's all been written
>>> k = cPickle.load(open('xxx.dmp', 'r'))
Easy to forget. Didn't need people being told that they are idiots.
I've get this error in Python 2.7 because of open mode 'rb':
with open(path_to_file, 'rb') as pickle_file:
obj = pickle.load(pickle_file)
So, for Python 2 'mode' should be 'r'
Also, I've wondered that Python 3 doesn't support pickle format of Python 2, and in case when you'll try to load pickle file created in Python 2 you'll get:
pickle.unpicklingerror: the string opcode argument must be quoted
Check this thread. Peter Otten says:
A corrupted pickle. The error is
raised if a string in the dump does
not both start and end with " or '.
and shows a simple way to reproduce such "corruption". Steve Holden, in the follow-up post, suggests another way to cause the problem would be to mismatch 'rb' and 'wb' (but in Python 2 and on Linux that particular mistake should pass unnoticed).
What are you doing with data between dump() and load()? It's quite common error to store pickled data in file opened in text mode (on Windows) or in database storage in the way that doesn't work properly for binary data (VARCHAR, TEXT columns in some databases, some key-value storages). Try to compare pickled data that you pass to storage and immediately retrieved from it.
If anyone has this error using youtube-dl, this issue has the fix: https://github.com/rg3/youtube-dl/issues/7172#issuecomment-242961695
richiecannizzo commented on Aug 28
brew install libav
Should fix it instantly on mac or
sudo apt-get install libav
#on linux
This error may also occur with python 2 (and early versions of python 3) if your pickle is large (Python Issue #11564):
Python 2.7.11 |Anaconda custom (64-bit)| (default, Dec 6 2015, 18:08:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import cPickle as pickle
>>> string = "X"*(2**31)
>>> pp = pickle.dumps(string)
>>> len(pp)
2147483656
>>> ss = pickle.loads(pp)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: insecure string pickle
This limitation was addressed with the introduction of pickle protocol 4 in python 3.4 (PEP 3154). Unfortunately, this feature has not been back-ported to python 2, and probably won't ever be. If this is your problem and you need to use python 2 pickle, the best you can do is reduce the size of your pickle, e.g., instead of pickling a list, pickle the elements individually into a list of pickles.
Same problem with a file that was made with python on windows, and reloaded with python on linux.
Solution : dos2unix on the file before reading in linux : works as a charm !
I got the Python ValueError: insecure string pickle message in a different way.
For me it happened after a base64 encoding a binary file and passing through urllib2 sockets.
Initially I was wrapping up a file like this
with open(path_to_binary_file) as data_file:
contents = data_file.read()
filename = os.path.split(path)[1]
url = 'http://0.0.0.0:8080/upload'
message = {"filename" : filename, "contents": contents}
pickled_message = cPickle.dumps(message)
base64_message = base64.b64encode(pickled_message)
the_hash = hashlib.md5(base64_message).hexdigest()
server_response = urllib2.urlopen(url, base64_message)
But on the server the hash kept coming out differently for some binary files
decoded_message = base64.b64decode(incoming_base64_message)
the_hash = hashlib.md5(decoded_message).hexdigest()
And unpickling gave insecure string pickle message
cPickle.loads(decoded_message)
BUT SUCCESS
What worked for me was to use urlsafe_b64encode()
base64_message = base64.urlsafe_b64encode(cPickle.dumps(message))
And decode with
base64_decoded_message = base64.urlsafe_b64decode(base64_message)
References
http://docs.python.org/2/library/base64.html
https://www.rfc-editor.org/rfc/rfc3548.html#section-3
This is what happened to me, might be a small section of population, but I want to put this out here nevertheless, for them:
Interpreter (Python3) would have given you an error saying it required the input file stream to be in bytes, and not as a string, and you may have changed the open mode argument from 'r' to 'rb', and now it is telling you the string is corrupt, and thats why you have come here.
The simplest option for such cases is to install Python2 (You can install 2.7) and then run your program with Python 2.7 environment, so it unpickles your file without issue. Basically I wasted a lot of time scanning my string seeing if it was indeed corrupt when all I had to do was change the mode of opening the file from rb to r, and then use Python2 to unpickle the file. So I'm just putting this information out there.
I ran into this earlier, found this thread, and assumed that I was immune to the file closing issue mentioned in a couple of these answers since I was using a with statement:
with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
pickle.dump(foo, temp_file)
# Push file to another machine
_send_file(temp_file.name)
However, since I was pushing the temp file from inside the with, the file still wasn't closed, so the file I was pushing was truncated. This resulted in the same insecure string pickle error in the script that read the file on the remote machine.
Two potential fixes to this: Keep the file open and force a flush:
with tempfile.NamedTemporaryFile(mode='wb') as temp_file:
pickle.dump(foo, temp_file)
temp_file.flush()
# Push file to another machine
_send_file(temp_file.name)
Or make sure the file is closed before doing anything with it:
file_name = ''
with tempfile.NamedTemporaryFile(mode='wb', delete=False) as temp_file:
file_name = temp_file.name
pickle.dump(foo, temp_file)
# Push file to another machine
_send_file(file_name)

Categories

Resources