I don't understand why python (at least 2.7) is not validating the exception handling mechanism for errors.
Example:
try:
some code selecting data from pymongo
except pymongo.errors.OperationFailure:
exception
In this case, if the exception is not called for the first time, python will not validate if I actually did import the pymongo lib.
Any idea why?
If I'm reading your question right, you want to know why except pymongo.errors.OperationFailure doesn't cause an error when the module is loaded if you haven't already imported the pymongo module.
Like most things in Python, the arguments to except clauses are evaluated at runtime. In fact, they can be expressions! Python does not validate them at "compile" time any more than it validates any other names at that time.
The reason is that Python is a dynamic language. Imports can be done conditionally, or performed based on names that are not known at "compile" time, and modules and other namespaces can be replaced, modified, or removed by code. As a result, Python literally cannot know whether pymongo.errors.OperationFailure is a valid name at that point in your code's execution without running your code.
According to PyMongo documentation, exception pymongo.errors.OperationFailure will be "raised when a database operation fails". AS such, your exceptblock gets evaluated only when such an error is raised.
I'm assuming that by "validation of pymongo's existence" you are referring to somethine like:
try:
import pymongo
except:
print("PyMongo not found!")
sys.exit(-1)
This method is often used to provide fallbacks (and backwards compatibity) not to "validate" imports. For instance in the case of json encoder/decoder, we can try whether we have simplejson library available and use jsonlibrary as a fallback as follows:
try:
import simplejson as json
except ImportError:
import json
Assuming that in the beginning of your script, you already have import pymongo, I don't see a reason why you should be checking or "validating" that pymongo has been imported: import pymongo will already raise an ImportError if pymongo library is not found.
First of all, pymongo.errors.OperationFailure may be defined anywhere, not only as a part of pymongo module, but also as a property of property of pymongo object defined in the same file.
Thus when handling exceptions Python should not check if specific module has been imported.
But if you do something like that:
import pymongo
you will see that import error is actually raised if module is not found:
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
import pymongo
ImportError: No module named pymongo
If my answer is not enough for you and you want to know more about why inclusion of pymongo.errors.OperationFailure does not throw any error when you run your script for the first time, even though you do not have any import pymongo statement in your code, then please see kindall's answer on Python being a dynamic language.
Related
I want to be able to get the path to the file which is importing my python library. How can I do that?
For example:
The user creates a file at C:\Users\Bob\bobsproject\main.py. From within the library, I want to be able to get the path to the file, and read it as a txt. How can I do that?
If you want to get the name of the driver script that is (possibly indirectly) loading your library, you can use the fact that python runs a script under the name __main__. You can get it from sys.modules just like any other module and access its __file__ attribute if it exists:
import sys
try:
print(sys.modules['__main__'].__file__)
except KeyError:
print('libray not loaded from script')
except AttributeError:
print('script not loaded from file')
The KeyError is unlikely to ever occur (not even if you run the script with python -m), but it's useful to be safe. The AttributeError is much more likely, and can easily be demonstrated with something like python -c.
If you want something more complex, like the file containing the code that actually called your library function, you will likely have to use the inspect module or similar. This will be even less robust as a matter of course, but may still suit your needs:
import inspect
module = inspect.getmodule(inspect.stack()[1][0])
try:
print(module.__file__)
except AttributeError:
print(f'module "{module.__name__}" not loaded from file')
Notice that inspect.getmodule explicitly uses the word "guess" in its official documentation, while inspect.stack can be a fidgety beast sometimes.
Code for second part referenced from here: https://stackoverflow.com/a/1095621/2988730.
Remember that there are two options here. If you place this code directly in your library module, it will be executed exactly once, when the module is first imported. If you place it in a function that the user can call directly, you will see the printouts every time. If you place the second snippet it in a utility function that you then call from your public module functions, don't forget to increment the frame index to reflect that:
module = inspect.getmodule(inspect.stack()[2][0])
Some automated tests in a larger system need to be able to import a module, and then restore sys.modules to its original condition.
But this code fragment:
import sys
sys.modules = dict(sys.modules)
import pickle
causes this KeyError in Python 3.6-3.8:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "[...]/python3.6/pickle.py", line 1562, in <module>
from _pickle import (
KeyError: '_compat_pickle'
It seems as if only pickle and modules that depend on it like multiprocessing are affected. I've investigated _compat_pickle - it's a module for pickling compatibility with Python 2 - but nothing jumps out that would cause this.
Is there a safe way to restore sys.modules back to an earlier state? And what is the mechanism behind this unexpected KeyError?
The problem is that sys.modules is a lie (I think). It is not actually the true source of the modules dict. That is stored on a C level in the current interpreter, and sys.modules is just a copy to that. _pickle is special, since it imports a module from C source, which I assume leads to this error (mismatch between what tstate->interp->modules says is imported and what sys.modules thinks is imported).
This might be considered a bug in python. I am not sure if a bug report already exists. Here is the bug report: https://bugs.python.org/issue12633 .
You could just save which keys are in modules before and after the code, and delete all other entries afterwards.
I have imported a user created file into my main code but an error in imported file is being displayed before. How can I suppress that error and display it only when the function is called
Importing file and its function :
import userValidation
NameString = input("Enter your name : ")
I have called user_validation function later in the code :
user_validation(names)
user_validation() has some error which I know and is being displayed just after the code start running.
I want to suppress the error till the time user_validation is called.How can i do that.
Use exception handling appropriately.
try:
#code with exception
except:
#handle it here
In the except part you may use pass to just move on if no action is required or use raise to handle it in the calling function.
Your terminology is a bit confusing, and in particular you aren't telling us the nature of the error (syntax, function definition, etc).
When you do:
import userValidation
The code in that module is run - up to the if __name__ =='__main__': block. Normally that code will contain other imports and a set of function and class definitions. That if block is supposed to contain variable definitions and function calls that will be used in a stand alone script call.
With proper separation, the only things that will be run during the import are the function and class definitions. The functions don't have to run correctly, they just have to have a valid syntax.
You should be able to wrap the import in a try/except block
try:
import userValidation
except ImportError: # or other error type
userValidation = None
Typically this structure is used to import modules that might not be present on the system. I'm not sure what errors you get if there is a syntax or other error in the imported file. But keep in mind that if there is an error, that module (or any of its functions) will not be available for latter use.
If this answer isn't enough, you need to describe the errors, maybe even give us a working (or not) example.
I have a Python 2.x program with this line of code:
from types import SliceType
When running the file with python3, the following error is printed:
ImportError: cannot import name 'SliceType'
How can I fix this so both Python 2.x and Python 3.x can execute the file?
This is not a duplicate of any random other ImportError question like this one. It is intended to be found via Google/SO search when you got the same error message. Before writing this question/answer, I wasn't able to find any solution for the issue described here.
You can use a try-based construct to get full 2.x/3.x compatibility:
try:
from types import SliceType
except ImportError:
SliceType = slice
See this reference for a table on the Python3 names for objects in the Python2 types module.
Code
def test_get_network_info(self):
with open(dirname(abspath(__file__)) + '/files/fake_network_info.txt', 'r') as mock_network_info:
with patch('subprocess.check_output', Mock(return_value=mock_network_info.read())):
self.assertEqual('192.168.1.100', get_network_info()[0])
self.assertEqual('255.255.255.0', get_network_info()[1])
self.assertEqual('192.168.1.0', get_network_info()[2])
Error
======================================================================
ERROR: test_get_network_info (tests.test_tools.ToolsTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/tim/Documents/overseer/app/tests/test_tools.py", line 21, in test_get_network_info
with patch('subprocess.check_output', Mock(return_value=mock_network_info.read())):
File "/usr/local/lib/python2.7/dist-packages/mock.py", line 1268, in __enter__
original, local = self.get_original()
File "/usr/local/lib/python2.7/dist-packages/mock.py", line 1242, in get_original
"%s does not have the attribute %r" % (target, name)
AttributeError: <module 'subprocess' from '/usr/local/lib/python2.7/dist-packages/twill/other_packages/subprocess.pyc'> does not have the attribute 'check_output'
What I understand
My understanding of the problem is that mock is trying to mock twill's subprocess module instead of the python one.
Questions
Am I doing something wrong ?
How can I specify that I want to patch the python subprocess module and not the twill's one ? (that may have been imported earlier in the test suite)**
Is there another way to patch the subprocess module ?
What I tried
I tried with patch('tools.subprocess.check_output', ...
Doesn't work.
I tired to use a decorator ...
Doesn't work either
I tired to patch directly the subprocess module subprocess.check_output = Mock( ...
Works but it's not good since it doesn't undo the patching.
Some more informations
If I run just this test and no other tests, it works because twill's subprocess module never got imported. But as soon as I run a test using twill, the above test will fail.
Here is the twill's version of subprocess wich looks like it has been copy pasted from an old version of python. It doesn't have any check_output function and that's why the test fails.
Twill's package comes from the Flask-Testing plugin which I use extensively. I submitted an issue on github here.
I hope someone from the lovely python community can help. :)
See my comment up there, due to bad practices in twill, the proper way would be to either fix twill, which may take some work, or move away to something else, but since you now heavily depend on Flask-Testing, it's not a cheap move either.
So this leaves us with a dirty trick: make sure to import subprocess anywhere before twill is imported. Internally, this will add a reference to the right subprocess module in sys.modules. Once a module is loaded, all subsequents import won't look anymore in sys.path but just use the reference already cached in sys.modules.
Unfortunately this is maybe not the end of the problem. Apparently twill uses a patched version of subprocess for some reason ; and those patches won't be available for him, since the plain built-in subprocess will be loaded instead. It's very likely it'll crash or behave in an unexpected way. If it's the case, well ... back to the suggestions above.