I have a very strange problem where I can't open a file from one of my larger scripts.
This problem is intermittent and I can't seem to understand the error.
I am getting this error:
IOError: [Errno 22] invalid mode ('w+') or filename: 'D:\\R4a\\2014_05\\R01359\\R01359_data_dwnhi.dat'
from this command in my script:
with open(os.path.normpath(os.path.join(sonpath,base+'_data_dwnhi.dat')), 'w+') as ff:
fp = np.memmap(ff, dtype='int16', mode='w+', shape=np.shape(tmp2))
when I use:
if os.path.exists(os.path.normpath(os.path.join(sonpath,base+'_data_dwnhi.dat')))== True:
print os.path.normpath(os.path.join(sonpath,base+'_data_dwnhi.dat')) + ' is a file'
it evaluates as True. Also when I run the open command by itself in ipython it sucessfully opens the file with:
open('D:\\R4a\\2014_05\\R01359\\R01359_data_dwnhi.dat', 'w+')
I am at a loss for words and think errno 22 error is misleading. As I stated earlier, the problem is intermittent and works on some files. Does anyone have an idea what I am doing wrong?
I am working on a windows 8 machine 64 bit and running python version 2.7.8 from an Anaconda 2.1.0 (64-bit) distribution.
Your opening it like a normal file, hence the error: IOError: [Errno 22] invalid mode. Do this instead:
with open(path, 'wb') as f:
pass
after some trial-errors, instruct np.memmap() to open a file with other than .__doc__ reported mode strings.
#------------------------------------------------------++++++++ TRY:
# ||||||||
# vvvvvvvv
|>>> MM = np.memmap( "temporary_DATA.memmap", mode = 'wNONSENSE+' )
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27.anaconda\lib\site-packages\numpy\core\memmap.py", line 208, in __new__
(valid_filemodes + list(mode_equivalents.keys())))
ValueError: mode must be one of ['r', 'c', 'r+', 'w+', 'readwrite', 'write', 'readonly', 'copyonwrite']
# ^ ^ ^ ^ ^ ^ ^ ^
# | | | | | | | |
#---------------------------------+----+----+-----+-----+------------+--------+-----------+
finally this error Traceback has shown alternative strings, not listed in np.memmap.__doc__, which do work fine
with open( os.path.normpath( os.path.join( sonpath,
base + '_data_dwnhi.dat'
)
),
'w+'
) as ff:
fp = np.memmap( ff, mode = 'readwrite', # options listed in np Traceback
shape = np.shape( tmp2 ),
dtype = 'int16'
)
For hawkish pythoneers, the post intentionally uses non-PEP-8 source code formatting as it is authors experience that during a learning phase, code read-ability improves the focus on task solution and helps getting used to the underlying concepts rather than to spend efforts on formally adhering typography. Hope the principle of providing help is respected and the non-PEP-8 styling format is forgiven in the name of the ease of reading.
Related
I recently made a program using an external document with pickle. But when it tries to load the file with pickle, I got this error (the file is already existing but it also fails when the file in't existing):
python3.6 check-graph_amazon.py
a
b
g
URL to follow www.amazon.com
Product to follow Pool_table
h
i
[' www.amazon.com', ' Pool_table', []]
p
Traceback (most recent call last):
File "check-graph_amazon.py", line 17, in <module>
tab_simple = pickle.load(doc_simple)
io.UnsupportedOperation: read
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "check-graph_amazon.py", line 42, in <module>
pickle.dump(tab_simple, 'simple_data.dat')
TypeError: file must have a 'write' attribute
Here is the code :
import pickle5 as pickle
#import os
try:
print("a")
with open('simple_data.dat', 'rb') as doc_simple:
print("b")
tab_simple = pickle.load(doc_simple)
print("c")
print(tab_simple)
print("d")
URL = tab_simple[0]
produit_nom = tab_simple[1]
tous_jours = tab_simple[2]
print("f")
except :
print("g")
URL = str(input("URL to follow"))
produit_nom = str(input("Product to follow"))
with open('simple_data.dat', 'wb+') as doc_simple:
print("h")
#os.system('chmod +x simple_data.dat')
tab_simple = []
tab_simple.append(URL)
tab_simple.append(produit_nom)
tab_simple.append([])
print(tab_simple)
print("c'est le 2")
print("p")
pickle.dump(tab_simple, 'simple_data.dat')
print("q")
The prints are here to show which lines are executed. The os.system is here to allow writing on the file but the error is persisting.
I don't understand why it's said that the document doesn't have a write attribute because I opened it in writing mode. And I neither understand the first error where it can't load the file.
If it can help you the goal of this script is to initialise the program, with a try. It tries to open the document in reading mode in the try part and then set variables. If the document doesn't exist (because the program is lauched for the first time) it goes in the except part and create the document, before writing informations on it.
I hope you will have any clue, including changing the architecture of the code if you have a better way to make an initialisation for the 1st time the program is launched.
Thanks you in advance and sorry if the code isn't well formated, I'm a beginner with this website.
Quote from the docs for pickle.dump:
pickle.dumps(obj, protocol=None, *, fix_imports=True)
Write a pickled representation of obj to the open file object file. ...
...
The file argument must have a write() method that accepts a single bytes argument. It can thus be an on-disk file opened for binary writing, an io.BytesIO instance, or any other custom object that meets this interface.
So, you should pass to this function a file object, not a file name, like this:
with open("simple_data.dat", "wb"): as File:
pickle.dump(tab_simple, File)
Yeah, in your case the file has already been opened, so you should write to doc_simple.
When I 'print' some hex string, some interesting error information in python, I wonder why this error is caused.
Win10(I tried it on ubuntu, No ERROR),python 2.7
enc_hex = '''f982f01c'''
enc_ascii = enc_hex.decode('hex')
print(enc_ascii)
Traceback (most recent call last):
File ".\xxxx.py", line 7, in <module>
print(enc_ascii)
IOError: [Errno 2] No such file or directory
Well,in fact I want to know why "print" a special set of hex will cause file operation, other hex string will not error
Try using codecs.decode:
import codecs
enc_hex = '''f982f01c'''
enc_ascii = codecs.decode(enc_hex, 'hex')
print(enc_ascii)
Output:
b'\xf9\x82\xf0\x1c'
it seems like directory problems . in windows you have to use forward slash (/) while accessing the directory.similar was happened in my case then i use forward slash in windows then it works.
I ma using Paramiko to do a sftp to transfer file from Linux environment to Windows.
I tried different solutions on Stack Overflow but still having the same problem.
My script
localpath = os.path.join(os.sep, 'Users', 'me', 'Desktop', 'ELK', 'PM_XML')
serverpath = r"***/****/***"
def sftp():
ip=ipAddr.get()
while True:
current_time = time.time()
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ip,username="root",password="root")
sftp = ssh.open_sftp()
for element in sftp.listdir(serverpath):
if element.endswith(".xml"):
creation_time = sftp.stat(serverpath+element).st_mtime
if (current_time+3400 - creation_time) / (3600) <= 1:
sftp.get(serverpath+element,os.path.join(os.sep,localpath,element))
sftp.close()
ssh.close()
I am getting this error:
Traceback (most recent call last):
File "<stdin>", line 5, in <module>
File "C:\Users\me\AppData\Local\Programs\Python\Python37-32\lib\site-packages\paramiko\sftp_client.py", line 801, in g
et
with open(localpath, "wb") as fl:
OSError: [Errno 22] Invalid argument: '\\Users\\me\\Desktop\\ELK\\PM_XML\\A2018-10-18T11:03:00+02:00-2018-10-18T11:04:00
+02:00_user-67-0-test-vm2.lk.fr.xml'
I think the problem is due to the file name
A2018-10-18T11:03:00+02:00-2018-10-18T11:04:00 +02:00_user-67-0-test-vm2.lk.fr.xml'
because when I try to do that with a simple filename my script works fine.
Any suggestions to deal with this filename, because I would like to keep the same name used on the server.
Solved by Martin Prikryl
a suggestion replace colon ":" with "_"
element.replace(":","_")
On Windows, a filename cannot contain a colon (:), among other special characters.
Microsoft documentation on Naming Conventions:
Use any character in the current code page for a name, including Unicode characters and characters in the extended character set (128–255), except for the following:
...
: (colon)
...
There's nothing you can do about it, except for removing/replacing the colon from the filename.
doc_holder_str = ''
sample_H_value = open("C:\testGupixwin\BX-N-H.HED", "r")
standard_conc_value = open("C:\testGupixwin\gupixwin_H_stdConc.txt", "r")
sample_H_value_str = sample_H_value.readline()
while sample_H_value_str is not '' :
stripper_sample_H = float(sample_H_value_str[5:].lstrip(' '))
I'm trying to write a piece of code (as shown above) which reads some values, do some calculations on it and returns the said values. I am using the LiClipse IDE, for python.
I have tested my code and it works, but when I tried to run it with real data, ( for which I created a new folder to put in all the files I will be working with) I received an OS error suggesting I inserted an invalid argument.
The error report says ;
Traceback (most recent call last):
File "C:\Python34\workspace\Gupixwin_Project.py", line 11, in <module>
sample_H_value = open("C:\testGupixwin\BX-N-H.HED", "r")
OSError: [Errno 22] Invalid argument: 'C:\testGupixwin\\BX-N-H.HED'
on clicking on the C:\testGupixwin\\BX-N-H.HED it bring up a message box suggesting, and I quote,
The definition was found at C:\testGupixwin\BX-N-H.HED, (which
cannot be opened because it is a compiled extension)
I must point out that I feel the error is that the system sees ...\\BX-N.... Instead of ..\BX-N... Which I expect.
Some one suggested I do this
[Open Window -> Preferences, goto PyDev -> Editor -> Code Style ->
File Types, look for "Valid source files (comma-separated)" and append
", log".]
I HAVE DONE THIS BUT I STILL GET THE OSERROR REPORT.
Thanks for any assistance
I think the problem is the escaping with \
alternate the string in: open("C:\testGupixwin\BX-N-H.HED", "r")
with:
open( r"C:\testGupixwin\BX-N-H.HED", "r" ) #rawstring
# or
open( "C:\\testGupixwin\\BX-N-H.HED", "r" ) #escaping the '\' with '\'
(also do this in the following line)
I'm doing the Udacity TensorFlow course, first exercise: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/1_notmnist.ipynb
OSX 10.11 (El Capitan)
Python 2.7
virtualenv installation of TF
I am getting an error:
"Exception: Failed to verifynotMNIST_large.tar.gz. Can you get to it with a browser?"
It finds the “small” file, but not the “large.” Appreciate help. Thanks.
Here is the whole block of code:
>>> url = 'http://yaroslavvb.com/upload/notMNIST/'
>>>
>>> def maybe_download(filename, expected_bytes):
... """Download a file if not present, and make sure it's the right size."""
... if not os.path.exists(filename):
... filename, _ = urlretrieve(url + filename, filename)
... statinfo = os.stat(filename)
... if statinfo.st_size == expected_bytes:
... print('Found and verified', filename)
... else:
... raise Exception(
... 'Failed to verify' + filename + '. Can you get to it with a browser?')
... return filename
...
Here is what gets returned:
>>> train_filename = maybe_download('notMNIST_large.tar.gz', 247336696)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 10, in maybe_download
Exception: Failed to verifynotMNIST_large.tar.gz. Can you get to it with a browser?
>>> test_filename = maybe_download('notMNIST_small.tar.gz', 8458043)
Found and verified notMNIST_small.tar.gz
Also face the same situation.
That is a simple thing to handle it and continue.
There's a size mismatch.
Just re-run the code with force=True, and
it works right now!
If you try to download it manually it will also be Ok.
As described in this thread: https://github.com/tensorflow/tensorflow/issues/1475
Hope it helps.
I found the solution (workaround?) of the issue in Udacity forum.
https://discussions.udacity.com/t/not-able-to-load-the-dataset-for-assingment-1/160124
In the page, two solutions are proposed.
a. Download notMNIST_large.tar.gz locally via browser, and copy it using docker command.
or
b. Add "User-Agent" header in fetching script.
I think both approaches works fine.