If my library has a contrib extra that has dependencies in it (say requests) that I want users to have to install to have access to a CLI API, but I install the contrib extra during my tests in CI how do I use pytest's MonkeyPatch to remove the dependencies during tests to ensure my detection is correct?
For example, if the contrib extra will additionally install requests and so I want users to have to do
$ python -m pip install mylib[contrib]
to then be able to at the command line have a CLI API that would look like
$ mylib contrib myfunction
where myfunction uses the requests dependency
# mylib/src/mylib/cli/contrib.py
import click
try:
import requests
except ModuleNotFoundError:
pass # should probably warn though, but this is just an example
# ...
#click.group(name="contrib")
def cli():
"""
Contrib experimental operations.
"""
#cli.command()
#click.argument("example", default="-")
def myfunction(example):
requests.get(example)
# ...
How do I mock or monkeypatch out requests in my pytest tests so that I can make sure that a user would properly get a warning along with the ModuleNotFoundError if they just do
$ python -m pip install mylib
$ mylib contrib myfunction
? After reading some other questions on the pytest tag I still don't think I understand how to do this, so I'm asking here.
What I ended up doing that worked, and which I had confirmed was a reasonable method thanks to Anthony Sottile, was to mock that the extra dependency (here requests) does not exist by setting it to None in sys.modules and then reloading the module(s) that would require use of requests.
I test that there is an actual complaint that requests doesn't exist to be imported by using caplog.
Here is the test I'm currently using (with the names changed to match my toy example problem in the question above)
import mylib
import sys
import logging
import pytest
from unittest import mock
from importlib import reload
from importlib import import_module
# ...
def test_missing_contrib_extra(caplog):
with mock.patch.dict(sys.modules):
sys.modules["requests"] = None
if "mylib.contrib.utils" in sys.modules:
reload(sys.modules["mylib.contrib.utils"])
else:
import_module("mylib.cli")
with caplog.at_level(logging.ERROR):
# The 2nd and 3rd lines check for an error message that mylib throws
for line in [
"import of requests halted; None in sys.modules",
"Installation of the contrib extra is required to use mylib.contrib.utils.download",
"Please install with: python -m pip install mylib[contrib]",
]:
assert line in caplog.text
caplog.clear()
I should note that this is actually advocated in #Abhyudai's answer to "Test for import of optional dependencies in init.py with pytest: Python 3.5 /3.6 differs in behaviour" which #hoefling linked to above (posted after I had solved this problem but before I got around to posting this).
If people are interested in seeing this in an actual library, c.f. the following two PRs:
pyhf PR #1046
pyhf PR #1119
A note:
Anthony Sottile has warned that
reload() can be kinda iffy -- I'd be careful with it (things which have old references to the old module will live on, sometimes it can introduce new copies of singletons (doubletons? tripletons?)) -- I've tracked down many-a-test-pollution problems to reload()
so I'll revise this answer if I implement a safer alternative.
Related
I'm having a challenging time getting the Python azure-cosmos library to correctly load for the purposes of locally testing a function in VS Code.
The specific error I'm getting (with the file path shortened) is: Exception: ImportError: cannot import name 'exceptions' from 'azure.cosmos' ([shortened]/.venv/lib/python3.8/site-packages/azure/cosmos/__init__.py)
Things I've checked/tried so far:
Check that requirements.txt specifies azure-cosmos
Manually go into python for each of the interpreters available within VS code and ensure I can manually import azure.cosmos
As instructed here, attempt to reinstall the azure-cosmos library using pip3 and ensuring the --pre flag is used.
[Updated] Verified I can successfully import azure.cosmos.cosmos_client as cosmos_client without any errors
Any ideas? Thanks! Below is the relevant section of my code.
import datetime
import logging
import tempfile
import requests
import os
import zipfile
import pandas as pd
import azure.functions as func
from azure.cosmos import exceptions, CosmosClient, PartitionKey
def main(mytimer: func.TimerRequest, calendars: func.Out[func.Document]) -> None:
logging.info("Timer function has initiated.")
This is what you face now:
This is the offcial doc:
https://github.com/Azure-Samples/azure-cosmos-db-python-getting-started
This doc tells you how to solve this problem.
So the solution is to install pre version.(George Chen's solution is right.)
Didn't install the pre version is the root reason, but please notice that, you need to first delete the package. Otherwise, the pre version will not be installed.(Only run install pre will not solve this problem, you need to delete all of the related packages first. And then install the pre package.)
Whether azure.cosmos is needed depends on whether function binding meets your needs, if the binding could do what you want suppose you don't need to use azure.cosmos.
About this import error, I could reproduce this exception, and I check the github solution it have to add a --pre flag.
So my solution is go to task.json under .vscde, add the flag to the command like below.
If you want to get more details about cosmos binding you could refer to this doc:Azure Cosmos DB trigger and bindings
How can I import modules for a Python Azure Function?
import requests
Leads to:
2016-08-16T01:02:02.317 Exception while executing function: Functions.detect_measure. Microsoft.Azure.WebJobs.Script: Traceback (most recent call last):
File "D:\home\site\wwwroot\detect_measure\run.py", line 1, in <module>
import requests
ImportError: No module named requests
Related, where are the modules available documented?
Related question with fully documented answer
Python libraries on Web Job
You need to include a requirements.txt file with your code which lists all the python dependencies of your function
From the docs:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python#python-version-and-package-management
For example, your reqirements.txt file would contain:
requests==2.19.1
Python support is currently experimental for Azure Functions, so documentation isn't very good.
You need to bring your own modules. None are available by default on Azure Functions. You can do this by uploading it via the portal UX or kudu (which is handy for lots of files).
You can leave comments on which packages you'd like, how you'd like to manage you packages here on the tracking issue for "real" Python support - https://github.com/Azure/azure-webjobs-sdk-script/issues/335
Install python packages from the python code itself with the following snippet:
def install(package):
# This function will install a package if it is not present
from importlib import import_module
try:
import_module(package)
except:
from sys import executable as se
from subprocess import check_call
check_call([se,'-m','pip','-q','install',package])
for package in ['requests','hashlib']:
install(package)
Desired libraries mentioned the list gets installed when the azure function is triggered for the first time. for the subsequent triggers, you can comment/ remove the installation code.
As part of my build system, I am using a modified version of a Python package (cogapp). I don't want to install the package because:
I've modified the package and don't want to worry about collision with unmodified versions which may already be installed.
It's nicer if the users of the build system don't need to install extra packages.
However, I'm having problems with using the package if it's not installed. If it is installed, I can run:
python -m cogapp <additional args>
and everything runs as intended.
The package has a __main__.py script:
import sys
from cogapp import Cog
sys.exit(Cog().main(sys.argv))
I tried running this directly, e.g.:
python -m <path>/__main__ <additional_args>
But I get the error:
...
/__main__.py", line 3, in <module>
from cogapp import Cog
ImportError: No module named cogapp
This is probably related to the error I get if I run __init__.py:
from .cogapp import *
The error is:
from .cogapp import *
ValueError: Attempted relative import in non-package
How can I run the package as a package?
EDIT:
I found a fix by removing all the relative imports from cogapp, and removing the -m, i.e. not running as a module. In this instance it's not too bad because it's a small package with only a single directory. However I'm interested in how this should be done in future. There's lots of stuff written around this subject, but no clear answers!
Here's the solution I've come to.
Disclaimer: you are actually installing the package but to a different path than the standard one.
$ mkdir newhome
$ python setup.py install --home=./newhome
$ PYTHONPATH=$PWD/newhome/lib/python <COMMAND_NEEDING_THAT_PACKAGE>
Currently one of my packages requires a JSON parser/encoder, and is designed to use simplejson if available falling back to the json module (in the standard library) if necessary (as benchmarks show simplejson is faster).
However, recently it's been hit or miss as to whether simplejson will install when using zc.buildout - something with the move to github, I believe. Which got me wondering; is it possible to define optional packages in my setup.py file which, if unavailable, won't stop the installation of my package?
optional packages at installation time.
I am assuming you are talking about your setup.py script.
You could change it to have:
# mypackage/setup.py
extras = {
'with_simplejson': ['simplejson>=3.5.3']
}
setup(
...
extras_require=extras,
...)
then you can do either of:
pip install mypackage,
pip install mypackage[with_simplejson]
with the latter installing simplejson>=3.5.3.
Instead of trying to install everything and fallback to a known good version,
you would want to install the subset of packages you know work.
optional packages at execution time.
Once you have two different sets of packages that could be installed, you need
to make sure you can use them if they are available. E.g. for your json import:
try:
# helpful comment saying this should be faster.
import simplejson as json
except ImportError:
import json
Another more complex example:
try:
# xml is dangerous
from defusedxml.cElementTree import parse
except ImportError:
try:
# cElementTree is not available in older python
from xml.cElementTree import parse
except ImportError:
from xml.ElementTree import parse
But you can also find this pattern in some packages:
try:
optional_package = None
import optional.package as optional_package
except ImportError:
pass
...
if optional_package:
# do addtional behavior
AFAIK there is no way to define an optional package and there would be no use to do so. What do you expect when you define an optional package? That it is installed when it is not yet available? (that would somehow make it mandatory)
No, IMHO the correct way to address this is in your imports where you want to use the package. E.g:
try:
from somespecialpackage import someapi as myapi
except ImportError:
from basepackage import theapi as myapi
This of course requires that the two APIs are compatible, but this is the case with simplejson and the standard library json package.
how can i import other external libraries in web2py? is it possible to
load up libs in the static file?
can somebody give me an example?
thanks
peter
If the library is shipped with python, then you can just use import as you would do in regular python script. You can place your import statements into your models, controllers and views, as well as your own python modules (stored in modules folder). For example, I often use traceback module to log stack traces in my controllers:
import traceback
def myaction():
try:
...
except Exception as exc:
logging.error(traceback.format_exc())
return dict(error=str(exc))
If the library is not shipped with python (for example, pyodbc), then you will have to install that library (using distutils or easy_install or pip) so that python can find it and run web2py from the source code: python web2py.py. Then you will be able to use regular import statement as described above. Before you do this make sure you installed the library properly: run python interpreter and type "import library_name". If you don't get any errors you are good to go.
If you have a third party python module or package, you can place it to modules folder and import it as shown below:
mymodule = local_import('module_name')
You can also force web2py to reload the module every time local_import is executed by setting reload option:
mymodule = local_import('module_name', reload=True)
See http://web2py.com/book/default/section/4/18?search=site-packages for more information.
In web2py you import external library as you normally do in Python
import module_name
or
from module_name import object_name
I am not sure what you mean by "in the static file"