Currently one of my packages requires a JSON parser/encoder, and is designed to use simplejson if available falling back to the json module (in the standard library) if necessary (as benchmarks show simplejson is faster).
However, recently it's been hit or miss as to whether simplejson will install when using zc.buildout - something with the move to github, I believe. Which got me wondering; is it possible to define optional packages in my setup.py file which, if unavailable, won't stop the installation of my package?
optional packages at installation time.
I am assuming you are talking about your setup.py script.
You could change it to have:
# mypackage/setup.py
extras = {
'with_simplejson': ['simplejson>=3.5.3']
}
setup(
...
extras_require=extras,
...)
then you can do either of:
pip install mypackage,
pip install mypackage[with_simplejson]
with the latter installing simplejson>=3.5.3.
Instead of trying to install everything and fallback to a known good version,
you would want to install the subset of packages you know work.
optional packages at execution time.
Once you have two different sets of packages that could be installed, you need
to make sure you can use them if they are available. E.g. for your json import:
try:
# helpful comment saying this should be faster.
import simplejson as json
except ImportError:
import json
Another more complex example:
try:
# xml is dangerous
from defusedxml.cElementTree import parse
except ImportError:
try:
# cElementTree is not available in older python
from xml.cElementTree import parse
except ImportError:
from xml.ElementTree import parse
But you can also find this pattern in some packages:
try:
optional_package = None
import optional.package as optional_package
except ImportError:
pass
...
if optional_package:
# do addtional behavior
AFAIK there is no way to define an optional package and there would be no use to do so. What do you expect when you define an optional package? That it is installed when it is not yet available? (that would somehow make it mandatory)
No, IMHO the correct way to address this is in your imports where you want to use the package. E.g:
try:
from somespecialpackage import someapi as myapi
except ImportError:
from basepackage import theapi as myapi
This of course requires that the two APIs are compatible, but this is the case with simplejson and the standard library json package.
Related
If my library has a contrib extra that has dependencies in it (say requests) that I want users to have to install to have access to a CLI API, but I install the contrib extra during my tests in CI how do I use pytest's MonkeyPatch to remove the dependencies during tests to ensure my detection is correct?
For example, if the contrib extra will additionally install requests and so I want users to have to do
$ python -m pip install mylib[contrib]
to then be able to at the command line have a CLI API that would look like
$ mylib contrib myfunction
where myfunction uses the requests dependency
# mylib/src/mylib/cli/contrib.py
import click
try:
import requests
except ModuleNotFoundError:
pass # should probably warn though, but this is just an example
# ...
#click.group(name="contrib")
def cli():
"""
Contrib experimental operations.
"""
#cli.command()
#click.argument("example", default="-")
def myfunction(example):
requests.get(example)
# ...
How do I mock or monkeypatch out requests in my pytest tests so that I can make sure that a user would properly get a warning along with the ModuleNotFoundError if they just do
$ python -m pip install mylib
$ mylib contrib myfunction
? After reading some other questions on the pytest tag I still don't think I understand how to do this, so I'm asking here.
What I ended up doing that worked, and which I had confirmed was a reasonable method thanks to Anthony Sottile, was to mock that the extra dependency (here requests) does not exist by setting it to None in sys.modules and then reloading the module(s) that would require use of requests.
I test that there is an actual complaint that requests doesn't exist to be imported by using caplog.
Here is the test I'm currently using (with the names changed to match my toy example problem in the question above)
import mylib
import sys
import logging
import pytest
from unittest import mock
from importlib import reload
from importlib import import_module
# ...
def test_missing_contrib_extra(caplog):
with mock.patch.dict(sys.modules):
sys.modules["requests"] = None
if "mylib.contrib.utils" in sys.modules:
reload(sys.modules["mylib.contrib.utils"])
else:
import_module("mylib.cli")
with caplog.at_level(logging.ERROR):
# The 2nd and 3rd lines check for an error message that mylib throws
for line in [
"import of requests halted; None in sys.modules",
"Installation of the contrib extra is required to use mylib.contrib.utils.download",
"Please install with: python -m pip install mylib[contrib]",
]:
assert line in caplog.text
caplog.clear()
I should note that this is actually advocated in #Abhyudai's answer to "Test for import of optional dependencies in init.py with pytest: Python 3.5 /3.6 differs in behaviour" which #hoefling linked to above (posted after I had solved this problem but before I got around to posting this).
If people are interested in seeing this in an actual library, c.f. the following two PRs:
pyhf PR #1046
pyhf PR #1119
A note:
Anthony Sottile has warned that
reload() can be kinda iffy -- I'd be careful with it (things which have old references to the old module will live on, sometimes it can introduce new copies of singletons (doubletons? tripletons?)) -- I've tracked down many-a-test-pollution problems to reload()
so I'll revise this answer if I implement a safer alternative.
I'm having a challenging time getting the Python azure-cosmos library to correctly load for the purposes of locally testing a function in VS Code.
The specific error I'm getting (with the file path shortened) is: Exception: ImportError: cannot import name 'exceptions' from 'azure.cosmos' ([shortened]/.venv/lib/python3.8/site-packages/azure/cosmos/__init__.py)
Things I've checked/tried so far:
Check that requirements.txt specifies azure-cosmos
Manually go into python for each of the interpreters available within VS code and ensure I can manually import azure.cosmos
As instructed here, attempt to reinstall the azure-cosmos library using pip3 and ensuring the --pre flag is used.
[Updated] Verified I can successfully import azure.cosmos.cosmos_client as cosmos_client without any errors
Any ideas? Thanks! Below is the relevant section of my code.
import datetime
import logging
import tempfile
import requests
import os
import zipfile
import pandas as pd
import azure.functions as func
from azure.cosmos import exceptions, CosmosClient, PartitionKey
def main(mytimer: func.TimerRequest, calendars: func.Out[func.Document]) -> None:
logging.info("Timer function has initiated.")
This is what you face now:
This is the offcial doc:
https://github.com/Azure-Samples/azure-cosmos-db-python-getting-started
This doc tells you how to solve this problem.
So the solution is to install pre version.(George Chen's solution is right.)
Didn't install the pre version is the root reason, but please notice that, you need to first delete the package. Otherwise, the pre version will not be installed.(Only run install pre will not solve this problem, you need to delete all of the related packages first. And then install the pre package.)
Whether azure.cosmos is needed depends on whether function binding meets your needs, if the binding could do what you want suppose you don't need to use azure.cosmos.
About this import error, I could reproduce this exception, and I check the github solution it have to add a --pre flag.
So my solution is go to task.json under .vscde, add the flag to the command like below.
If you want to get more details about cosmos binding you could refer to this doc:Azure Cosmos DB trigger and bindings
In python3.6, when executing the following command:
from sklearn.model_selection import GridSearchCV
Reported Error:
from _bz2 import BZ2Compressor, BZ2Decompressor
ModuleNotFoundError: No module named '_bz2'
However, issue is that I don't have root access to machine to use the solutions posted on the same issue on stackoverflow.
On trying to locate libbz2.so, I received following:
/usr/lib64/libbz2.so.1
/usr/lib64/libbz2.so.1.0.6
To be noted: in my bash_profile the LIBRARY_PATH did not have '/usr/lib64' explicitly specified when I compiled python3.6 from source.
bz2 is an optional dependency of python, but sklearn assumes your python installation has this module.
There are at least two possible ways to fix this:
update your version of joblib to make its dependence on bz2 optional. (Thanks to sascha for pointing this out.)
or, install libbz2 and then re-build python3.6.
I'm working on a plugin architecture and need to convert a package name like "foo.bar" to the absolute path where the code resides. imp.find_module seems to do what I want, except when the code in question is installed via an egg-link (installed via 'pip install develop').
If there are two modules foo.bar and foo.bar2 which are installed via egg-links (and which live at completely separate file system locations like /home/bob/foo/bar and /home/alice/foo/bar2), find_modules doesn't work because I look up the package "foo" and get the location to foo/bar, but not foo/bar2.
Anyone have suggestions for an alternative function? find_modules doesn't accept hierarchical names, so I can't just pass "foo.bar2" into it.
The easiest way would be to just import the module and inspect its __file__ attribute:
import os
import foo.bar
print(os.path.abspath(foo.bar.__file__))
For dynamic imports:
import os
import sys
module_name = 'foo.bar'
__import__(module_name)
print(os.path.abspath(sys.modules[module_name].__file__))
how can i import other external libraries in web2py? is it possible to
load up libs in the static file?
can somebody give me an example?
thanks
peter
If the library is shipped with python, then you can just use import as you would do in regular python script. You can place your import statements into your models, controllers and views, as well as your own python modules (stored in modules folder). For example, I often use traceback module to log stack traces in my controllers:
import traceback
def myaction():
try:
...
except Exception as exc:
logging.error(traceback.format_exc())
return dict(error=str(exc))
If the library is not shipped with python (for example, pyodbc), then you will have to install that library (using distutils or easy_install or pip) so that python can find it and run web2py from the source code: python web2py.py. Then you will be able to use regular import statement as described above. Before you do this make sure you installed the library properly: run python interpreter and type "import library_name". If you don't get any errors you are good to go.
If you have a third party python module or package, you can place it to modules folder and import it as shown below:
mymodule = local_import('module_name')
You can also force web2py to reload the module every time local_import is executed by setting reload option:
mymodule = local_import('module_name', reload=True)
See http://web2py.com/book/default/section/4/18?search=site-packages for more information.
In web2py you import external library as you normally do in Python
import module_name
or
from module_name import object_name
I am not sure what you mean by "in the static file"