I am in the process of deploying to Pypi a Python project, let's call it foobar. I would like to distribute it with a shell command and an IPython magic command. I use Poetry, and the relevant part of my .toml configuration file is:
[tool.poetry.scripts]
foobar = 'foobar.cli:main'
foobar_magic = 'foobar.magic:load_ipython_extension'
After uploading this to TestPypi and installing it with pip, the shell command (foobar) works as expected. However, executing %load_ext foobar_magic in a Jupyter Notebook fails with:
ModuleNotFoundError: No module named 'foobar_magic'
According to the documentation:
You can put your extension modules anywhere you want, as long as they can be imported by Python’s standard import mechanism.
Under the same notebook, I have verified that !foobar and import foobar both work. How can I make foobar_magic be found too?
Moreover, although I'm not there yet, I guess the suffix of the entry point is wrong too. Indeed, the function I specify after the : will be called with no arguments, but the function load_ipython_extension() expects an IPython instance.
So I feel completely lost, and can't find any relevant documentation for deploying IPython Notebook extensions.
Edit 1. %load_ext foobar.magic unexpectedly works, and the magic %foobar does not complain about the arguments. I don't understand why, and why it is %foobar and not %foobar_magic as declared.
Edit 2. the foobar_magic = ... stuff is ignored or useless. Suppressing it has no consequence on %load_ext foobar.magic. I think the latter invocation might be ok. But it's a little annoying not to understand what's going on.
I finally found a workaround:
Delete the line foobar_magic = ... of my .toml.
Move the contents of foobar/magic.py to foobar/__init__.py (originally empty), guarded with the following two lines:
import sys
if "ipykernel" in sys.modules:
# magic stuff
This file being executed each time the module is imported, it is now enough to do (under a notebook):
%load_ext foobar
The guard ensures the magic stuff is executed if and only if foobar is imported from IPython.
This does not answer my original question, and I still do not fully understand how these entry points are supposed to work, but I am happy with the actual result.
Related
Using pycharm with python 3.7. I am using queue.SimpleQueue. The code runs fine, and PyCharm is pointed at the correct interpreter and all that. But with this code:
import queue
Q = queue.SimpleQueue()
I get a warning "Cannot find reference 'SimpleQueue' in 'queue.pyi'".
I do some searching. I hit ctrl-B on the "import queue" statement and it takes me to a file called queue.pyi in the folder helpers/typeshed/stdlib/3/ under the pycharm installation. So apparently instead of the queue.py file in lib/python3.7/ under the python venv, it thinks I'm trying to import this queue.pyi file instead, which I didn't even know existed.
Like I said, the code runs fine, and I can simply add # noinspection PyUnresolvedReferences and the warning goes away, but then the type inferencing and code hints on the variable Q don't work.
Another fix is to instead import _queue and use _queue.SimpleQueue, because apparently in python 3.7 queue.SimpleQueue is implemented in cython and is imported from a cython package _queue. But importing _queue seems hackish and implementation-dependent.
Is there a way to tell PyCharm that import queue means the actual lib/python3.7/queue.py as opposed to whatever helpers/typeshed/stdlib/3/queue.pyi is?
It was fixed in PyCharm 2019.3 https://youtrack.jetbrains.com/issue/PY-31437, could you please try to update?
I’m in the process of writing unit tests for my software.
I wrote some helpers for my tests. For convenience I would like to use them in a jupyter notebook.
When I try to import them inside of the notebook though, I get an error.
from tests import helpers
->
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-12-d8ba72c24738> in <module>
----> 1 from tests.helpers import some_helper
ModuleNotFoundError: No module named 'tests.helpers'
Digging a little, I found out that importing tests actually imports this folder as a module:
'/my_project_path/.venv/lib/python3.8/site-packages/IPython/extensions/tests'
The folder contains test_autoreload.py and test_storemagic.py, which are tests for extensions I use.
Here’s my question, how do I properly manage this conflict? I would like to keep those extension installed, and I would like to keep the name tests for my folder, as it is the convention when working with pytest.
I installed those extensions with pip. Did I miss an option to ignore the tests when installing or something?
Thanks! :)
Not really a solution but some hints here in case they help you:
I was having the same issue today and getting a bit crazy. Not in jupyter but int the ipyton shell. As jupyter uses ipython I guess it will be the same issue.
IPython has a module called tests
within the IPython/extensions folder, and this folder is added is added to the search path when using ipython instead of python.
This are my paths from a python shell (using sys.path):
['',
'/usr/local/lib/python37.zip',
'/usr/local/lib/python3.7',
'/usr/local/lib/python3.7/lib-dynload',
'/usr/local/lib/python3.7/site-packages']
And the ones from an ipython shell:
['/usr/local/bin',
'/usr/local/lib/python37.zip',
'/usr/local/lib/python3.7',
'/usr/local/lib/python3.7/lib-dynload',
'',
'/usr/local/lib/python3.7/site-packages',
'/usr/local/lib/python3.7/site-packages/IPython/extensions',
'/home/dmontaner/.ipython']
So, if your are in iptyhon or in a notebook and you import tests or from tests import something, the interpreter will search in:
'/usr/local/lib/python3.7/site-packages/IPython/extensions'
So far all is as expected. But I also had as you my own tests module in the same directory where my ipython session was running. What should happen then is that my own tests module should be imported first. But it was not and I had the same error as you.
If I would remove the 'IPython/extensions' from the path doing sys.path.remove(.../IPython/extensions') then I was able to import my own module.
I did uninstall ipyton and ipython_genutils and, using python -m pytest in the command line (on a .py file, not a notebook) I was still having a slightly similar issue this time about dash trying to load the IPython. The weird thing is that in my project I am not using dash but flask and dash is not a dependency of pytest anyway I think.
So, I uninstalled ipyton, ipython_genutils, dash and plotly and then I could run pytest importing from my own tests module.
I reinstalled the 4 libraries again and the problem was solved! And now I can import from my tests module even from within a jupyter notebook.
I guess the message is that there is some buggy setup or dependency among all those libraries (and may be some others). Try to reinstall latest versions or recreate your virtual environment in case it helps.
Sorry that I could not figure out what was the exact problem... but I hope this helps you.
The long-term fix for these kind of naming conflicts is to put all your code, including your test code, into a project package, such that you imports look like
import my_project.tests.helpers instead of import tests.helpers.
By choosing a unique name for your project, you define your own namespace and avoid naming conflicts.
I am trying to import a Python module (fiasco). I cloned it from GitHub and everything appeared to be working fine. Importing it works, but when I try to type, for example iron = fiasco.Element('iron', [1e4, 1e6, 1e8]*u.K), I get the error module 'fiasco' has no attribute 'Element'. I am using Spyder's iPython console. This also fails if I start iPython from the terminal, but works if I start python3 from the terminal.
I had done this on two different computers - on one, it worked at first, but started giving me the same error after I restarted the kernel. On the other, it never worked at all.
If it helps: after importing, I tried typing fiasco. When I did this on the computer where it originally worked, the output was <module 'fiasco' from '/Users/shirman/fiasco/fiasco/__init__.py'>. Now, and on the computer it never worked on, it just says <module 'fiasco' (namespace)>. So maybe this has something to do with paths?
Addition: sys.path points to /Users/shirman, and several paths within /Users/shirman/anaconda3. The fiasco folder is in /Users/shirman.
You have inadvertently created a namespace package due to your sys.path setting. Namespace packages are directories without an __init__.py in the Python search path and allow loading of submodules or -packages from different paths (e.g. path1/foo/a.py and path2/foo/b.py can be imported as foo.a and foo.b).
The problem is that import fiasco finds /Users/shirman/fiasco first and imports it as a namespace package. If you set sys.path such that /Users/shirman/fiasco comes before /Users/shirman, the importer finds the actual package /Users/shirman/fiasco/fiasco first.
Namespace packages are a Python 3.3 feature, so either the other machine had a different sys.path setting, a really old Python 3 installation, or you were using Python 2.
pytest appears to be using old source code and failing tests because of it. I'm not sure how to update it.
Test code:
from nba_stats import league
class TestLeaders():
def test_default():
leaders = league.Leaders()
print(leaders)
Source code (league.py):
from nba_stats.nba_api import NbaAPI
from nba_stats import constants
class Leaders:
...
When I run pytest on my parent directory, I get an error that refers to an old import statement.
_____________________________ ERROR collecting test/test_league.py ______________________________
ImportError while importing test module '/home/mfb/src/nba_stats/test/test_league.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test_league.py:1: in <module>
from nba_stats import league
../../../.virtualenvs/nba_stats_dev/lib/python3.6/site-packages/nba_stats/league.py:1: in <module>
from nba_stats import _api_scrape, _get_json
E ImportError: cannot import name '_api_scrape'
I tried resetting my virtualenvironment and also reinstalling my package via pip. What do I need to do to tell it to see the new import statement and why is this happening?
Edit: Deleting my virtual environment completely and then creating a new one seemed to fix it, but it seems to be a recurring issue with any further source code changes. Surely there must be a way to not have to reset my virtualenvironment each time?
Looks like you installed that package (possibly as a dependency through something else if not directly) and also have it cloned locally for development. You can look into local editable installs (https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs), but personally, I prefer to make the test refer directly to the package under which it is being run, since then it can be used "as-is" after cloning it. Do this by modifying sys.path in your test_league.py. Ie., assuming it has a structure with the python code under python/nba_stats, in the parent directory of `test
sys.path = [os.path.join(os.pardir, 'python')] + sys.path
at the top of test_league.py. This puts your local package up front and import will consider it first.
EDIT:
Since you tried and it still did not work (please do make sure that the snippet above does point to the local python package as in the actual structure; the above is just a common one but you may have a different structure), here is how you can see which directories are considered in order, and which are eventually selected:
python -vv -m pytest -svx
You will be able to see if there are spurious directories in sys.path, whether the one tried (as in the snippet above) matches as expected or not, any left-over .pyc files that get picked up, etc.
EDITv2: Since you stated that python -m pytest works, but pytest not, have a look where that pytest executable is coming from with which pytest. Likely it's a system one that refers to a different python then the one in your virtualenv. To see which python it picks up, do:
cat `which python`
and look at the top line.
If that is not the same as what which python gives you (with your desired virtualenv activated), you may have to install pytest for that current virtualenv (python -m pip install pytest).
I have a simple module and a basic def. Module name is example315.py and the def is
def right_justify(s)
print(s)
This works fine when I import example315 and then call example315.right_justify("hello world")
If I change my def to not return anything (in fact I can change it in any way) and then run the function again (AFTER saving my module of course) iit still does the print.
Short of exiting IDLE and starting over I can't seem to get it to work.
Any help appreciated
The module is loaded once per session, you have to re-load it when you change it.
From the Python tutorial on modules:
For efficiency reasons, each module is only imported once per
interpreter session. Therefore, if you change your modules, you must
restart the interpreter – or, if it’s just one module you want to test
interactively, use reload(), e.g. reload(modulename).
The problem you're facing is the fact that IDLE has already imported and built its internal representation of your module. Editing the file on disk won't reflect on the now imported memory-resident version in IDLE. You should be able to get the behavior you're looking for with:
example315 = reload(example315)
And here's some source: Python Docs Source