Develop Reference

shelve module even not creating shelf

I am new to stackoverflow and experimenting with Python, currently just trying tutorial examples. Experienced a wonderful learning curve but got completely stuck with the following (working under windows 10):
import shelve
s = shelve.open("test")
Traceback (most recent call last):
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\dbm\dumb.py", line 82, in _create
f = _io.open(self._datfile, 'r', encoding="Latin-1")
FileNotFoundError: [Errno 2] No such file or directory: 'test.dat'
It would be great to get some help to resolve this.
During handling of the above exception, another exception occurred:

In Python 3, by default, shelve.open tries to open an existing shelf for reading. You have to pass an explicit flag to create a new shelf if if doesn't already exist.
s = shelve.open("test", "c")
This is in contrast to Python 2, where the default flag was "c" instead of "r".

How to read an error message
In general, error messages will do their best to tell you what's wrong. In the case of python, you'll typically start at the bottom; here
No such file or directory: 'test.dat'
tells you exactly why the error's being thrown: test.dat doesn't exist.
Next you would read upward through the stack trace until we got to something that we either understood or had written recently, and we'd try to make sense of the error message from there.
How to troubleshoot an error
Is the stated problem intelligible?
Yes, we asked the software to do something with a (.dat?) file called "test", so we at least know what the hell the error message is talking about.
Do we agree with the underlying premise of the error?
Specifically, does it make sense that it should matter if test.dat exists or not? Chepner covers this.
Do we agree with the specific problem as stated?
For example, it wouldn't be weird at all to get such an error message when there was in fact such a file. Then we would have a more specific question: "Why can't the software find the file?" That's progress.
(Usually the answer would be either "Because it's looking in the wrong place" or "Because it doesn't have permission to access that file".)
Read the documentation for the tools and functions in question.
How can we validate either our own understanding of the situation, or the situation described in the error message?
Depending on the context this may involve some trial and error of re-writing our code to
print out (log) its state during execution
do something similar but different from what it was doing, which we're more certain should work.
do something similar but different from what it was doing, which we're more certain should not work.
Ask for help.

Seems shelve sometimes uses the dumbdbm to serialize.
Use dbm to use dbm instead:
import dbm
with dbm.open($filename, 'n') as db:
# read/write

Configure Django's test debugging to print shorter paths.

I'm not sure how to approach this, whether it's a Django, Python, or even Terminal solution that I can tweak.
The thing is I'm learning Django following a book (reference here, really like it), and whenever I run the tests, I get really long output in the terminal for debugging matters. Obviously there's many traceback functions that get called after another, but what started bugging me is that the file paths are very long and they all have the same project folder... which is long by itself, and then it adds all the virtualenv stuff like this:
Traceback (most recent call last):
File "home/user/code/projects/type_of_projects_like_hobby/my_project_application/this_django_version/virtualenv/lib/python3.6/site-packages/django/db/models/base.py", line 808, in save
force_update=force_update, update_fields=update_fields)
Since the paths take two or more lines, I can't focus on what functions I should be looking at clearly.
I have looked at the verbosity option when calling manage.py test but it doesn't help with the paths. If anyone has an idea on how to ~fix~ go about this issue, it'd be cool.
Thanks guys.

There's really not a way to change the behavior (this is how Python displays tracebacks). But you can pipe the output into something that will reformat it. For example, here's a tool you can pipe traceback output into that will do various types of formatting.

Is there an error log file for linux that shows file open failures?

I just blew a few days tracking down a bug in a python script (unknown to me, web2py used a different root directory from what i was expecting, leading to a file read to fail silently. Thus when run from command line, code was fine, but when run from web, code failed).
Having finally tracked down the culprit I can just fix the silent fail (which is in a library, openCV in this case) but a smarter detective would have seen the fail in some sort of system log , if such exists. Then , no matter where the silent fail is, I still see it and don't have to painstakingly track down the fail.
So - is there some sort of global error logfile for linux that logs such things as file-read errors?
And yes I know there is python specific error logging but the question still holds. e.g. if i have a complex project with some python, some C, some whatever, and someone somewhere is silently failing, a system-wide error log would be of immense help.

This solution may not be performant enough for your needs, but regardless I wanted to report on some research I did that may lead you in the right direction.
First of all, there is logging in linux systems under /var/log. Of interest are the syslog and messages files, which log all kinds of system events. But file read "errors" are not logged, as explained below.
In the case of opening a file that doesn't exist, we are ultimately looking for an open system call that fails (python's open calls this). But there is no notion of an exception at this low level - if open fails it just returns a negative number. In C, you can open files that don't exist all day long and still have your program return a 0 error code.
This means you have to do some work yourself to track this problem. I took your question to be, "How can I track these errors at a level below python's exceptions?" For this you can use a combination of strace and grep. You attach strace per process and it logs all the system calls that take place.
So imagine we have a C program that looks like this:
#include <stdio.h>
int main()
{
fopen("nothere.txt","r");
}
By running strace ./test 2>&1 | grep ENOENT, we get:
open("nothere.txt", O_RDONLY) = -1 ENOENT (No such file or directory)
You could of course run strace on a python process to achieve the same results.
Things to be wary of:
You have to attach this per process. If you don't, we're back to silent errors.
Python generates a lot of system calls. Your log files might get big.
There are a lot of IO errors out there. ENOENT is only one of them.
You will need more complex string parsing to filter out system calls you don't care about.

Perhaps you could post the code that is reading the file? open() failures should always generate an IOError exception:
with open('no-such-file') as f:
print f.read()
Traceback (most recent call last):
File "app.py", line 1, in <module>
with open('no-such-file') as f:
IOError: [Errno 2] No such file or directory: 'no-such-file'
The cause of your frustration is most likely bad exception handling, as in the following code:
try:
with open('no-such-file') as f:
print f.read()
except Exception, e:
print 'bad exception handling here'

Opening TOPCAT through command line Via Python Script

I'm trying to incorporate the program TOPCAT (which has really amazing plotting capabilities) into a python script I have written. The problem is that when I make a call to the program it tells me:
OSError: [Errno 2] No such file or directory
Here's some background to the problem:
1) The way I usually open up topcat through the command line is through the alias I have created:
alias topcat='java -jar /home/username/topcat/topcat-full.jar'
2) If I'd like to open TOPCAT with a file in mind (let's use a csv file since that's what I'd like it to work with), I would type this into the command line:
topcat -f csv /home/username/path_to_csv_file/file.csv
And that also works just fine. The problem comes about when I try to call these commands while in my python script. I've tried both subprocess.call and os.system, and they don't seem to know of the existence of the topcat alias for some reason. Even doing a simple call like:
import subprocess
subprocess.call(['topcat'])
doesn't work... However, I can get topcat to open if I run this:
import subprocess
subprocess.call(['java','-jar','/home/username/topcat/topcat-full.jar'])
The problem with this is that it simply opens the program, and doesn't allow for me to tell it which file to take in and what type it happens to be.
Could somebody tell me what I'm doing incorrectly here? I've also looked into the shell=True option and it doesn't seem to be doing any better.

Okay - so I'm really excited that figured it out. What worked before was:
import subprocess
subprocess.call(['java','-jar','/home/username/topcat/topcat-full.jar'])
It turns out it can take more command line arguments. This is what eventually got it open up with the correct comma separated file through the command line:
import subprocess
subprocess.call(['java','-jar','/home/username/topcat/topcat-full.jar','-f','csv','/full/path/to/data.csv'])
Hopefully this is enough information to help other people who come across this specific task.

If anyone else comes across this, pystilts might be of interest.
https://github.com/njcuk9999/pystilts
edit: It is a native Python wrapper of Topcat/STILTS.

Flaky file deletion under Windows 7?

I have a Python test suite that creates and deletes many temporary files. Under Windows 7, the shutil.rmtree operations sometimes fail (<1% of the time). The failure is apparently random, not always on the same files, not always in the same way, but it's always on rmtree operations. It seems to be some kind of timing issue. It is also reminiscent of Windows 7's increased vigilance about permissions and administrator rights, but there are no permission issues here (since the code had just created the files), and there are no administrator rights in the mix.
It also looks like a timing issue between two threads or processes, but there is no concurrency here either.
Two examples of (partial) stack traces:
File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean
shutil.rmtree(cleandir)
File "c:\python23\lib\shutil.py", line 142, in rmtree
raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg)
WindowsError: [Errno 5] Access is denied removing xml_1
File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean
shutil.rmtree(cleandir)
File "c:\python23\lib\shutil.py", line 142, in rmtree
raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg)
WindowsError: [Errno 3] The system cannot find the path specified removing out
On Windows XP, it never failed. On Windows 7, it fails like this, across a few different Python versions (2.3-2.6, not sure about 3.1).
Anyone seen anything like this and have a solution? The code itself is on bitbucket for the truly industrious.

It's a long shot, but are you running anything that scans directories in the background? I'm thinking antivirus/backup (maybe Windows 7 has something like that built in? I don't know). I have experienced occasional glitches when deleting/moving files from the TSVNCache.exe process that TortoiseSVN starts -- seems it watches directories for changes, then presumably opens them for scanning the files.

We had similar problems with shutil.rmtree on Windows, particularly looking like your first stack trace. We solved it by using an exception handler with rmtree. See this answer for details.

Just a thought, but if the test behavior (creating and deleting lots of temp files) isn't typical of what the app actually does, maybe you could move those test file operations to (c)StringIO, and keep a suite of function tests that exercises your app's actual file creation/deletion behavior.
That way, you can make sure that your app behaves correctly, without introducing extra complexity not related to the app.

My guess is that you should check up on the code that creates the file, and make SURE they are closed explicitly before moving on to delete it. If nothing is obvious there in the code, download a copy of Process Monitor and watch what's happening on the file system there. This tool will give you the exact error code coming from Windows, and should shed some light on the situation.

That "The system cannot find the path specified:" error will appear intermittently if the path is too long for Windows (260 chars). Automated tasks often create folder hierarchies using relative references that produce fully qualified paths longer than 260 characters. Any script that attempts to delete those folders using fully qualified paths will fail.
I built a quick workaround that used relative path references, but don't have generic code solution to offer, just a warning that the excellent answers provided might not help you.

I met same problem with shutil.rmtree command and this issue maybe cause by special file name.(Ex:Other country language:леме / Ö)
Please use the following format to delete the directory which you want:
import shutil
shutil.rmtree(os.path.join("<folder_name>").decode('ascii'))
Enjoy it !