Develop Reference

python sqlite3 connection path with spaces

I am trying to connect with a full path but I get this problem
>>> path = "/home/astro/Fun LAB/DBlist"
>>> db = sql.connect(path)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
sqlite3.OperationalError: unable to open database file
>>>
i 've also tried this
>>> path = "/home/astro/Fun\ LAB/DBlist"
>>> db = sql.connect(path)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
sqlite3.OperationalError: unable to open database file
>>>
I know it's because of the whitespace cause I've tried this and it works
>>> path = "/home/astro/DBlist"
>>> db = sql.connect(path)
>>>
so is there is an easy way to escape the whitespaces in the path or am I doing something wrong?

You might try the uri style approach - it worked for me in getting past connection issues due to spaces in file paths with sqlite3:
Escaping the spaces with %20
prefixing the path with file:///
Examples:
file:///C:/Documents%20and%20Settings/fred/Desktop/data.db
file:///C:/Documents%20and%20Settings/fred/Desktop/

I would suggest naming files or folders without spaces. (I don't think it's possible either) Instead, use dash or underscores.
The next problem is to see where you're running your Python interpreter. If it's nested inside a directory and you're trying to create a path from the outside, then your link would raise an error.
>>> path = "/home/astro/Fun LAB/DBlist"
or
>>> path = "/home/astro/Fun\ LAB/DBlist"
or
>>> path = "/home/astro/DBlist"
wouldn't work if you run python inside these directories or from another.
To correct this, try:
>>> path = "../home/path/to/file"

Having spaces is indeed a problem.
I would suggest bring a symlink and work with that:
!ln -sf '$path_to_db_file' .
db = sql.connect('db_file')

notMNIST not downloadable in TensorFlow Udacity course

I'm doing the Udacity TensorFlow course, first exercise: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/udacity/1_notmnist.ipynb
OSX 10.11 (El Capitan)
Python 2.7
virtualenv installation of TF
I am getting an error:
"Exception: Failed to verifynotMNIST_large.tar.gz. Can you get to it with a browser?"
It finds the “small” file, but not the “large.” Appreciate help. Thanks.
Here is the whole block of code:
>>> url = 'http://yaroslavvb.com/upload/notMNIST/'
>>>
>>> def maybe_download(filename, expected_bytes):
... """Download a file if not present, and make sure it's the right size."""
... if not os.path.exists(filename):
... filename, _ = urlretrieve(url + filename, filename)
... statinfo = os.stat(filename)
... if statinfo.st_size == expected_bytes:
... print('Found and verified', filename)
... else:
... raise Exception(
... 'Failed to verify' + filename + '. Can you get to it with a browser?')
... return filename
...
Here is what gets returned:
>>> train_filename = maybe_download('notMNIST_large.tar.gz', 247336696)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 10, in maybe_download
Exception: Failed to verifynotMNIST_large.tar.gz. Can you get to it with a browser?
>>> test_filename = maybe_download('notMNIST_small.tar.gz', 8458043)
Found and verified notMNIST_small.tar.gz

Also face the same situation.
That is a simple thing to handle it and continue.
There's a size mismatch.
Just re-run the code with force=True, and
it works right now!
If you try to download it manually it will also be Ok.
As described in this thread: https://github.com/tensorflow/tensorflow/issues/1475
Hope it helps.

I found the solution (workaround?) of the issue in Udacity forum.
https://discussions.udacity.com/t/not-able-to-load-the-dataset-for-assingment-1/160124
In the page, two solutions are proposed.
a. Download notMNIST_large.tar.gz locally via browser, and copy it using docker command.
or
b. Add "User-Agent" header in fetching script.
I think both approaches works fine.

How to use python-magic 5.19-1

I need to determine MIME-types from files without suffix in python3 and I thought of python-magic as a fitting solution therefor.
Unfortunately it does not work as described here:
https://github.com/ahupp/python-magic/blob/master/README.md
What happens is this:
>>> import magic
>>> magic.from_file("testdata/test.pdf")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'from_file'
So I had a look at the object, which provides me with the class Magic for which I found documentation here:
http://filemagic.readthedocs.org/en/latest/guide.html
I was surprised, that this did not work either:
>>> with magic.Magic() as m:
... pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __init__() missing 1 required positional argument: 'ms'
>>> m = magic.Magic()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __init__() missing 1 required positional argument: 'ms'
>>>
I could not find any information about how to use the class Magic anywhere, so I went on doing trial and error, until I figured out, that it accepts instances of LP_magic_set only for ms.
Some of them are returned by the module's methods
magic.magic_set() and magic_t().
So I tried to instanciate Magic with either of them.
When I then call the file() method from the instance, it will always return an empty result and the errlvl() method tells me error no. 22.
So how do I use magic anyway?

I think that you are confusing different implementations of "python-magic"
You appear to have installed python-magic-5.19.1, however, you reference firstly the documentation for python-magic-0.4.6, and secondly filemagic-1.6. I think that you are better off using python-magic-0.4.6 as it is readily available at PYPI and easily installed via pip into virtualenv environments.
Documentation for python-magic-5.19.1 is hard to come by, but I managed to get it to work like this:
>>> import magic
>>> m=magic.open(magic.MAGIC_NONE)
>>> m.load()
0
>>> m.file('/etc/passwd')
'ASCII text'
>>> m.file('/usr/share/cups/data/default.pdf')
'PDF document, version 1.5'
You can also get different magic descriptions, e.g. MIME type:
>>> m=magic.open(magic.MAGIC_MIME)
>>> m.load()
0
>>> m.file('/etc/passwd')
'text/plain; charset=us-ascii'
>>> m.file('/usr/share/cups/data/default.pdf')
'application/pdf; charset=binary'
or for more recent versions of python-magic-5.30
>>> import magic
>>> magic.detect_from_filename('/etc/passwd')
FileMagic(mime_type='text/plain', encoding='us-ascii', name='ASCII text')
>>> magic.detect_from_filename('/etc/passwd').mime_type
'text/plain'

Print an error message without printing a traceback and close the program when a condition is not met

I've seen similar questions to this one but none of them really address the trackback.
If I have a class like so
class Stop_if_no_then():
def __init__(self, value one, operator, value_two, then, line_or_label, line_number):
self._firstvalue = value_one
self._secondvalue = value_two
self._operator = operator
self._gohere = line_or_label
self._then = then
self._line_number = line_number
def execute(self, OtherClass):
"code comparing the first two values and making changes etc"
What I want my execute method to be able to do is if self._then is not equal to the string "THEN" (in allcaps) then I want it to raise a custom error message and terminate the whole program while also not showing a traceback.
If the error is encountered the only thing that should print out would look something like (I'm using 3 as an example, formatting is not a problem) this.
`Syntax Error (Line 3): No -THEN- present in the statement.`
I'm not very picky about it actually being an exception class object, so there's no issue in that aspect. Since I will be using this in a while loop, simple if, elif just repeats the message over and over (because obviously I am not closing the loop). I have seen sys.exit() but that also prints out a giant block of red text, unless I am not using it correctly. I don't want to catch the exception in my loop because there are other classes in the same module in which I need to implement something like this.

You can turn off the traceback by limiting its depth.
Python 2.x
import sys
sys.tracebacklimit = 0
Python 3.x
In Python 3.5.2 and 3.6.1, setting tracebacklimit to 0 does not seem to have the intended effect. This is a known bug. Note that -1 doesn't work either. Setting it to None does however seem to work, at least for now.
In Python 3.6.2 and above you should set tracebacklimit to 0 or -1, as setting it to None does not disable the traceback output.
Python 3.6.1 and bellow results:
>>> import sys
>>> sys.tracebacklimit = 0
>>> raise Exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception
>>> sys.tracebacklimit = -1
>>> raise Exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception
>>> sys.tracebacklimit = None
>>> raise Exception
Exception
Python 3.6.2 and above results:
>>> import sys
>>> sys.tracebacklimit = 0
>>> raise Exception
Exception
>>> sys.tracebacklimit = -1
>>> raise Exception
Exception
>>> sys.tracebacklimit = None
>>> raise Exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception
Nevertheless, for better or worse, if multiple exceptions are raised, they can all still be printed. For example:
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>

You can use SystemExit exception:
except Exception as err:
raise SystemExit(err)
https://docs.python.org/3/library/exceptions.html

You can use a try: and then except Exception as inst:
What that will do is give you your error message in a variable named inst and you can print out the arguments on the error with inst.args. Try printing it out and seeing what happens, and is any item in inst.args is the one you are looking for.
EDIT Here is an example I tried with pythons IDLE:
>>> try:
open("epik.sjj")
except Exception as inst:
d = inst
>>> d
FileNotFoundError(2, 'No such file or directory')
>>> d.args
(2, 'No such file or directory')
>>> d.args[1]
'No such file or directory'
>>>
EDIT 2: as for closing the program you can always raise and error or you can use sys.exit()

The cleanest way that I know is to use sys.excepthook.
You implement a three argument function that accepts type, value, and traceback and does whatever you like (say, only prints the value) and assign that function to sys.excepthook.
Here is an example:
import sys
def excepthook(type, value, traceback):
print(value)
sys.excepthook = excepthook
raise ValueError('hello')
This is available in both python 2 and python 3.

If you want to get rid of any traceback for customs exceptions and have line number,
you can do this trick
Python 3
import sys
import inspect
class NoTraceBackWithLineNumber(Exception):
def __init__(self, msg):
try:
ln = sys.exc_info()[-1].tb_lineno
except AttributeError:
ln = inspect.currentframe().f_back.f_lineno
self.args = "{0.__name__} (line {1}): {2}".format(type(self), ln, msg),
sys.exit(self)
class MyNewError(NoTraceBackWithLineNumber):
pass
raise MyNewError("Now TraceBack Is Gone")
Will give this output, and make the raise keyword useless
MyNewError (line 16): Now TraceBack Is Gone

"Exception chaining can be disabled by using from None " - Python docs
>>> try:
... open('database.sqlite')
... except IOError:
... raise RuntimeError from None
Traceback (most recent call last):
File "<stdin>", line 4, in <module>

In general, if you want to catch any exception except SystemExit, and exit with the exception's message without the traceback, define your main function as below:
>>> import sys
>>> def main():
... try:
... # Run your program from here.
... raise RandomException # For testing
... except (Exception, KeyboardInterrupt) as exc:
... sys.exit(exc)
...
>>> main()
name 'RandomException' is not defined
$ echo $?
1
Note that in the case of multiple exceptions being raised, only one message is printed.
This answer is meant to improve upon the one by The-IT.

How do I disable and then re-enable a warning?

I'm writing some unit tests for a Python library and would like certain warnings to be raised as exceptions, which I can easily do with the simplefilter function. However, for one test I'd like to disable the warning, run the test, then re-enable the warning.
I'm using Python 2.6, so I'm supposed to be able to do that with the catch_warnings context manager, but it doesn't seem to work for me. Even failing that, I should also be able to call resetwarnings and then re-set my filter.
Here's a simple example which illustrates the problem:
>>> import warnings
>>> warnings.simplefilter("error", UserWarning)
>>>
>>> def f():
... warnings.warn("Boo!", UserWarning)
...
>>>
>>> f() # raises UserWarning as an exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
UserWarning: Boo!
>>>
>>> f() # still raises the exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 2, in f
UserWarning: Boo!
>>>
>>> with warnings.catch_warnings():
... warnings.simplefilter("ignore")
... f() # no warning is raised or printed
...
>>>
>>> f() # this should raise the warning as an exception, but doesn't
>>>
>>> warnings.resetwarnings()
>>> warnings.simplefilter("error", UserWarning)
>>>
>>> f() # even after resetting, I'm still getting nothing
>>>
Can someone explain how I can accomplish this?
EDIT: Apparently this is a known bug: http://bugs.python.org/issue4180

Reading through the docs and few times and poking around the source and shell I think I've figured it out. The docs could probably improve to make clearer what the behavior is.
The warnings module keeps a registry at __warningsregistry__ to keep track of which warnings have been shown. If a warning (message) is not listed in the registry before the 'error' filter is set, any calls to warn() will not result in the message being added to the registry. Also, the warning registry does not appear to be created until the first call to warn:
>>> import warnings
>>> __warningregistry__
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
NameError: name '__warningregistry__' is not defined
>>> warnings.simplefilter('error')
>>> __warningregistry__
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
NameError: name '__warningregistry__' is not defined
>>> warnings.warn('asdf')
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
UserWarning: asdf
>>> __warningregistry__
{}
Now if we ignore warnings, they will get added to the warnings registry:
>>> warnings.simplefilter("ignore")
>>> warnings.warn('asdf')
>>> __warningregistry__
{('asdf', <type 'exceptions.UserWarning'>, 1): True}
>>> warnings.simplefilter("error")
>>> warnings.warn('asdf')
>>> warnings.warn('qwerty')
------------------------------------------------------------
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
UserWarning: qwerty
So the error filter will only apply to warnings that aren't already in the warnings registry. To make your code work you'll need to clear the appropriate entries out of the warnings registry when you're done with the context manager (or in general any time after you've used the ignore filter and want a prev. used message to be picked up the error filter). Seems a bit unintuitive...

Brian Luft is correct about __warningregistry__ being the cause of the problem. But I wanted to clarify one thing: the way the warnings module appears to work is that it sets module.__warningregistry__ for each module where warn() is called. Complicating things even more, the stacklevel option to warnings causes the attribute to be set for the module the warning was issued "in the name of", not necessarily the one where warn() was called... and that's dependent on the call stack at the time the warning was issued.
This means you may have a lot of different modules where the __warningregistry__ attribute is present, and depending on your application, they may all need clearing before you'll see the warnings again. I've been relying on the following snippet of code to accomplish this... it clears the warnings registry for all modules whose name matches the regexp (which defaults to everything):
def reset_warning_registry(pattern=".*"):
"clear warning registry for all match modules"
import re
import sys
key = "__warningregistry__"
for mod in sys.modules.values():
if hasattr(mod, key) and re.match(pattern, mod.__name__):
getattr(mod, key).clear()
Update: CPython issue 21724 addresses issue that resetwarnings() doesn't clear warning state. I attached an expanded "context manager" version to this issue, it can be downloaded from reset_warning_registry.py.

Brian is spot on about the __warningregistry__. So you need to extend catch_warnings to save/restore the global __warningregistry__ too
Something like this may work
class catch_warnings_plus(warnings.catch_warnings):
def __enter__(self):
super(catch_warnings_plus,self).__enter__()
self._warningregistry=dict(globals.get('__warningregistry__',{}))
def __exit__(self, *exc_info):
super(catch_warnings_plus,self).__exit__(*exc_info)
__warningregistry__.clear()
__warningregistry__.update(self._warningregistry)

Following on from Eli Collins' helpful clarification, here is a modified version of the catch_warnings context manager that clears the warnings registry in a given sequence of modules when entering the context manager, and restores the registry on exit:
from warnings import catch_warnings
class catch_warn_reset(catch_warnings):
""" Version of ``catch_warnings`` class that resets warning registry
"""
def __init__(self, *args, **kwargs):
self.modules = kwargs.pop('modules', [])
self._warnreg_copies = {}
super(catch_warn_reset, self).__init__(*args, **kwargs)
def __enter__(self):
for mod in self.modules:
if hasattr(mod, '__warningregistry__'):
mod_reg = mod.__warningregistry__
self._warnreg_copies[mod] = mod_reg.copy()
mod_reg.clear()
return super(catch_warn_reset, self).__enter__()
def __exit__(self, *exc_info):
super(catch_warn_reset, self).__exit__(*exc_info)
for mod in self.modules:
if hasattr(mod, '__warningregistry__'):
mod.__warningregistry__.clear()
if mod in self._warnreg_copies:
mod.__warningregistry__.update(self._warnreg_copies[mod])
Use with something like:
import my_module_raising_warnings
with catch_warn_reset(modules=[my_module_raising_warnings]):
# Whatever you'd normally do inside ``catch_warnings``

I've run into the same issues, and while all of the other answers are valid I choose a different route. I don't want to test the warnings module, nor know about it's inner workings. So I just mocked it instead:
import warnings
import unittest
from unittest.mock import patch
from unittest.mock import call
class WarningTest(unittest.TestCase):
#patch('warnings.warn')
def test_warnings(self, fake_warn):
warn_once()
warn_twice()
fake_warn.assert_has_calls(
[call("You've been warned."),
call("This is your second warning.")])
def warn_once():
warnings.warn("You've been warned.")
def warn_twice():
warnings.warn("This is your second warning.")
if __name__ == '__main__':
__main__=unittest.main()
This code is Python 3, for 2.6 you need the use an external mocking library as unittest.mock was only added in 2.7.