Cythonized module not available when package is installed - python

I have the following package structure:
setup.py
/cpp_graph_test
fastgraph.pyx
graph.cpp
graph.h
graph.pxd
heap.cpp
heap.h
__init__.py
I've created a setup.py as follows:
from setuptools import Extension, setup
from Cython.Build import cythonize
sourcefiles = ['cpp_graph_test/fastgraph.pyx', 'cpp_graph_test/graph.cpp', 'cpp_graph_test/heap.cpp']
extensions = [
Extension(
name="cpp_graph_test.fastgraph",
sources=sourcefiles,
extra_compile_args=['-O3']
)
]
setup(
name='cpp_graph_test',
packages=['cpp_graph_test'],
ext_modules=cythonize(extensions, language_level=3, include_path=["cpp_graph_test"]),
version='0.0.1'
)
I install, and things seem to go fine...
$ sudo pip3 install .
Processing /home/le_user/Documents/cpp_graph_test
Preparing metadata (setup.py) ... done
Building wheels for collected packages: cpp-graph-test
Building wheel for cpp-graph-test (setup.py) ... done
Created wheel for cpp-graph-test: filename=cpp_graph_test-0.0.1-cp310-cp310-linux_x86_64.whl size=641411 sha256=8e3b20a0bec7a8f5a739deba272289bce370ab99abae02a3787d3f10718b03c9
Stored in directory: /tmp/pip-ephem-wheel-cache-ofm34m2v/wheels/c1/03/6a/f746b1b945b60e93aa67ee67e8e6a2c4537c0a87dbb72ffa34
Successfully built cpp-graph-test
Installing collected packages: cpp-graph-test
Attempting uninstall: cpp-graph-test
Found existing installation: cpp-graph-test 0.0.1
Uninstalling cpp-graph-test-0.0.1:
Successfully uninstalled cpp-graph-test-0.0.1
Successfully installed cpp-graph-test-0.0.1
However, this fails...
$ python3
Python 3.10.4 (main, Apr 2 2022, 09:04:19) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cpp_graph_test.fastgraph import FastGraph
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cpp_graph_test.fastgraph'
I can import the cpp_graph_test package in its entirety, but if I try to use pkgutil to list all the modules in the package, there are none.
Is there something wrong with my setup.py file?
EDIT: When I run the install with pip, the pyx file is translated to C++, and I get binary generated in the build directory, so the Cython build is running, and apparently not creating errors.
EDIT2: While experimenting, I got this error: ImportError: cannot import name 'FastGraph' from 'cpp_graph_test' (/home/le_user/Documents/cpp_graph_test/cpp_graph_test/__init__.py) This error makes it seem like Python is looking at the raw code for the package instead of looking at the actual built, installed package after the pip install. It's like there's some weird symlink somewhere or something...
EDIT3: I can sudo pip3 uninstall cpp_graph_test and it'll tell me "skipping cpp_graph_test as it is not installed." But then I can start a Python shell (from any folder) and say import cpp_graph_test and it'll be successful. Not sure how to uninstall a package that's already uninstalled but that lives on anyway?

Related

Installed a package in Python but cannot import it

I have definitely installed this package locally, called app. However, I am not able to run a Python file where it imports this package and get the ModuleNotFound error.
PS C:\clippd\clippd-meta-functions\app> python -m pip install -e .
Obtaining file:///C:/clippd/clippd-meta-functions/app
Preparing metadata (setup.py) ... done
Installing collected packages: app
Attempting uninstall: app
Found existing installation: app 0.1.0
Uninstalling app-0.1.0:
Successfully uninstalled app-0.1.0
Running setup.py develop for app
Successfully installed app-0.1.0
PS C:\clippd\clippd-meta-functions\app> & C:/Users/rijul/AppData/Local/Microsoft/WindowsApps/python3.10.exe c:/clippd/clippd-meta-functions/app/gamification/complete_flow.py
Traceback (most recent call last):
File "c:\clippd\clippd-meta-functions\app\gamification\complete_flow.py", line 5, in <module>
from app.gamification.streaks.intra_round.ir_class import (
ModuleNotFoundError: No module named 'app'
Ummmm, I think you have different version python

Can't import module installed with pip (anaconda python)

I'm trying to use the ReadIM package with an anaconda environment (anaconda 4.7.12, python 3.8, pip 19.3.1), thus requiring me to use pip install readIM which succeeds, with the package found in conda list and pip list.
I have read a lot of posts talking about using the correct python and pip paths, my python where shows only my anaconda path, where I ran conda install pip beforehand, and then pip -V shows my pip is in my conda path.
I have tried conda update --all and pip uninstall readIM. I also read that it could be an issue with Jupyter notebook, hence in the conda command prompt I tried python then import readIM, which still gave me the same import error.
As advised on the source page, I tried installing by running python setup.py build install/python setup.py install in the source directory, however this gave me an error
cygwin TypeError: '>=' not supported between instances of 'NoneType' and 'str'
I am at a loss, any ideas?
Thanks,
Mustafa.
UPDATE:
Managed to compile it from source successfully, but still get the same error:
(base) C:>python -m pip install ReadIM-0.8.2.tar.gz
Processing c:\readim-0.8.2.tar.gz
Building wheels for collected packages: ReadIM
Building wheel for ReadIM (setup.py) ... done
Created wheel for ReadIM: filename=ReadIM-0.8.2-cp37-cp37m-win_amd64.whl size=219546 sha256=90f8960a6f1f80ae62dc18eab4bcd31fb2a6dfd7da364a5c15fc37e6e2ce0360
Stored in directory: C:\Users\mi4517\AppData\Local\pip\Cache\wheels\d6\a9\11\936e986255027bb654601b322a3431f9bcc3fde72ebb406835
Successfully built ReadIM
Installing collected packages: ReadIM
Successfully installed ReadIM-0.8.2
(base) C:>python
Python 3.7.5 (default, Oct 31 2019, 15:18:51) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
import readIM
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'readIM'>
Following their instructions (python setup.py build install then python setup.py test) yields a successful build but a failed test:
C:\ReadIM-0.8.2>python setup.py test
Traceback (most recent call last):
File "setup.py", line 41, in
assert os.path.isdir(testFileDir)
AssertionError
SOLUTION
It was installing correctly, just that I was importing it incorrectly in Python. I was doing import readIM or import readim when it should be import ReadIM I did not know it was case sensitive!
Thank you for your help, apologies for the trivial mistake on my part.
I had faced a similar issue. With your conda environment activated, type
which pip
Check if it shows the path to pip in your environment. In my case, it was not. It was showing path to some other pip. So pip install was installing in that environment. To fix use the full path of pip, like below
~/anaconda3/envs/my_env/bin/pip install ...
To avoid this issue follow these steps
open Anaconda prompt
type this command conda install -c conda-forge imread
after installation open jupiter notebook.
type import imread then run.
It was installing correctly, just that I was importing it incorrectly in Python. I was doing import readIM or import readim when it should be import ReadIM I did not know it was case sensitive!
Thank you for your help, apologies for the trivial mistake on my part.

No module named 'adbase' - Python3 [duplicate]

This has me pretty confused. I've installed pyad using pip and everything seems fine:
C:\WINDOWS\system32>pip install pyad
Collecting pyad
Using cached pyad-0.5.16.tar.gz
Requirement already satisfied (use --upgrade to upgrade): setuptools in c:\python35\lib\site-packages (from pyad)
Requirement already satisfied (use --upgrade to upgrade): pywin32 in c:\python35\lib\site-packages (from pyad)
Installing collected packages: pyad
Running setup.py install for pyad ... done
Successfully installed pyad-0.5.16
But when I try to use it, I get an error that complains about not finding adbase:
C:\WINDOWS\system32>python
Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyad import aduser
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python35\lib\site-packages\pyad\__init__.py", line 1, in <module>
from adbase import set_defaults as pyad_setdefaults
ImportError: No module named 'adbase'
>>> import pyad
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python35\lib\site-packages\pyad\__init__.py", line 1, in <module>
from adbase import set_defaults as pyad_setdefaults
ImportError: No module named 'adbase'
This is odd, because if I try to uninstall pyad or if I check the site-packages directory, adbase is definitely there:
C:\WINDOWS\system32>pip uninstall pyad
Uninstalling pyad-0.5.16:
c:\python35\lib\site-packages\pyad-0.5.16-py3.5.egg-info
c:\python35\lib\site-packages\pyad\__init__.py
c:\python35\lib\site-packages\pyad\__pycache__\__init__.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\adcomputer.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\addomain.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\adgroup.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\adquery.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\adsearch.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\pyad.cpython-35.pyc
c:\python35\lib\site-packages\pyad\adbase.py
c:\python35\lib\site-packages\pyad\adcomputer.py
c:\python35\lib\site-packages\pyad\adcontainer.py
c:\python35\lib\site-packages\pyad\addomain.py
c:\python35\lib\site-packages\pyad\adgroup.py
c:\python35\lib\site-packages\pyad\adobject.py
c:\python35\lib\site-packages\pyad\adquery.py
c:\python35\lib\site-packages\pyad\adsearch.py
c:\python35\lib\site-packages\pyad\aduser.py
c:\python35\lib\site-packages\pyad\pyad.py
c:\python35\lib\site-packages\pyad\pyadconstants.py
c:\python35\lib\site-packages\pyad\pyadexceptions.py
c:\python35\lib\site-packages\pyad\pyadutils.py
Proceed (y/n)?
pyad directory contents
I'm really not sure what else to try. I've run everything under an elevated command prompt, so it's not a permissions issue. I even tried downloading pyad and installing it using setup.py, but I had the same problem with that. adbase is definitely there, and I can't figure out why Python isn't finding it.
That's a bug on pyad part. They're importing adbase as if it were a standalone module or package, and that's why it does not work. The proper way to fix this would be to change the import to an absolute import from pyad.adbase import ... or relative from .adbase import ....
However, if you check the master branch on Github, you will see that they have actually fixed it. But that's not all, if you check their setup.py you'll see that the version on Github is 0.5.15, while the last version on PyPI, which is the one you have installed, is 0.5.16. Weird.
I suggest you to install the package directly from Github, and that should take care of the problem. To do that, first uninstall pyad and then run
pip install https://github.com/zakird/pyad/archive/master.zip

Migration from distribute to setuptools

I'm trying to convert a project that I had installing fine with distribute over to a newer setuptools based installer. For some reason I can't get setuptools to run at all. When I run my setup.py I get errors from distutils about unsupported options which are all the extension options provided by setuptools. I can't figure out why setuptools isn't taking care of these correctly. This is on a Debian Wheezy system running Python 2.7.
I created a simple test case to demonstrate the problem. It is a standalone script that I want installed as a utility with an executable wrapper script:
foo.py
#!/usr/bin/python
def main():
print 'Foo main() ran'
if __name__ == '__main__':
main()
setup.py
from setuptools import setup
setup(name='foo',
version='1.0',
py_modules = ['foo'],
entry_points = {
'console_scripts': ['foo = foo:main'] # setuptools extension
},
include_package_data = True # another setuptools extension
)
The version of setuptools in the Debian package archive is 0.6.24 which predates the merger with distribute and I would prefer to use something that retains some of the distribute heritage. I've been installing various versions of setuptools with pip. The latest 4.0.1 won't install properly but most anything from 1.4 to 3.8 works:
$ sudo pip install setuptools==3.8
...
$ python
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from setuptools import version
>>> version.__version__
'3.8'
The setuptools package/egg is placed in /usr/local/lib/python2.7/dist-packages which is normal on a Debian system when using pip.
When I try to install with my setup.py I get the following errors:
$ sudo python setup.py install
/usr/lib/python2.7/distutils/dist.py:267:
UserWarning: Unknown distribution option: 'entry_points'
warnings.warn(msg)
/usr/lib/python2.7/distutils/dist.py:267:
UserWarning: Unknown distribution option: 'include_package_data'
warnings.warn(msg)
running install
running build
running build_py
running install_lib
running install_egg_info
Removing /usr/local/lib/python2.7/dist-packages/foo-1.0.egg-info
Writing /usr/local/lib/python2.7/dist-packages/foo-1.0.egg-info
Distutils installs foo.py to /usr/local/lib/python2.7/dist-packages/ but obviously the setuptools generated wrapper script is absent. I would have expected setuptools to do its subclassing/monkeypatching thing to take care of anything not supported by distutils.
I remember setuptools being cranky years ago which is why I settled on using distribute. It's disappointing that it still doesn't "just work". Is there anything obvious that I'm missing? I have a suspicion that this has something to do with Debian's use of the dist-packages directory in place of site-packages but that was never an issue with distribute.
Well it looks like the problem is that setuptools doesn't like being installed into /usr/local/lib/....
I forced pip to put it into /usr/lib/... with:
sudo pip install --install-option="--prefix=/usr/lib/python2.7/dist-packages" \
setuptools==3.8
setup.py installs cleanly after that.

Setuptools setup.py installing when dependencies not satisfied

I have a setup.py that looks a bit (okay, exactly) like this:
#!/usr/bin/env python
from setuptools import setup
import subprocess
import distutils.command.build_py
class BuildWithMake(distutils.command.build_py.build_py):
"""
Build using make.
Then do the default build logic.
"""
def run(self):
# Call make.
subprocess.check_call(["make"])
# Keep installing the Python stuff
distutils.command.build_py.build_py.run(self)
setup(name="jobTree",
version="1.0",
description="Pipeline management software for clusters.",
author="Benedict Paten",
author_email="benedict#soe.ucsc.edu",
url="http://hgwdev.cse.ucsc.edu/~benedict/code/jobTree.html",
packages=["jobTree", "jobTree.src", "jobTree.test", "jobTree.batchSystems",
"jobTree.scriptTree"],
package_dir= {"": ".."},
install_requires=["sonLib"],
# Hook the build command to also build with make
cmdclass={"build_py": BuildWithMake},
# Install all the executable scripts somewhere on the PATH
scripts=["bin/jobTreeKill", "bin/jobTreeStatus",
"bin/scriptTreeTest_Sort.py", "bin/jobTreeRun",
"bin/jobTreeTest_Dependencies.py", "bin/scriptTreeTest_Wrapper.py",
"bin/jobTreeStats", "bin/multijob", "bin/scriptTreeTest_Wrapper2.py"])
It installs the package perfectly fine when run with ./setup.py install. However, it does this whether or not the "sonLib" package is installed, ignoring the dependency.
Is this expected behavior? Should a setup.py install blithely proceed if the dependencies are not installed, leaving it up to pip or whatever to install them beforehand? If not, and setup.py install ought to fail when dependencies are absent, what am I doing wrong?
EDIT: Some version information:
Python 2.7.2 (default, Jan 19 2012, 21:40:50)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> setuptools.__version__
'0.6c12'
>>>
The default install command for Distutils setup doesn't know anything about dependencies. If you are running that, you're right that dependencies will not be checked.
Just going by what you've show in the setup.py, though, you are using Setuptools for the setup function. The Setuptools install command is declared to run easy_install, which in turn does check and download dependencies.
It is possible you are explicitly invoking the Distutils install, by specifying install --single-version-externally-managed.

Categories

Resources