Develop Reference

Python project, installation only works with "pip install -e .", but not "pip install ."

I've made my first python project and uploaded it on GitHub.
Upon installation with pip, I get different errors based on whether, from the root directory, I execute
pip install .
or
pip install -e .
I was able to recreate this error by having a project with these elements:
Dir tree:
.
├── my_package
│   ├── __init__.py
│   ├── main.py
│   └── test.py
└── setup.py
main.py:
import my_package.test
test.py:
print("hello there!")
setup.py
from setuptools import setup, find_packages
setup(
name='my_package',
version='1.0.0',
description='Description of my package',
packages=find_packages(
where="my_package"
),
install_requires=[]
)
This would, after executing
pip install .
python3 my_package/main.py
give this error:
Traceback (most recent call last):
File "./my_package/main.py", line 1, in <module>
import my_package.test
ModuleNotFoundError: No module named 'my_package'
Said error does not appear when main is called by using
pip install -e .
python3 my_package/main.py
giving back the expected "hello there!".
I managed to fix the problem, understanding that the issue was having the main script inside of the package, while importing files from the package itself.
But I still do not understand why it would work anyway when setting up the package/app in development mode...
Question: Why does that happen?

setup.py runs but I get a "ModuleNotFoundError" when trying to import my package

I have a setup.py file. I run it using pip install -e .. The logging suggest my package is successfully installed. But when I run python -c "from mypackage import test_script" I get the error ModuleNotFoundError: No module named 'mypackage'.
My setup.py file looks like:
from setuptools import find_packages, setup
setup(
name="mypackage",
packages=find_package(include=["src", "src.*"]),
version="0.0.1",
description="my pakage",
author="me",
)
My package folder looks like:
src
mypackage
__init__.py
test_script.py
python -c "from src.mypackage import test_script" does work, but I don't want my package name to be preceeded with "src".
Any ideas what is going wrong?

Install own projects as dependency | module name issue

I have multiple projects I am working on several libraries and several clients which require these libraries as dependencies.
library structure
$ pwd
~/Projects/library
$ tree
.
├── api.py
├── __init__.py
└── setup.py
$ cat api.py
import requests
# ...
def process(data):
for record in data:
print(f"Processing {record}")
$ cat __init__.py
from .api import process
$ cat setup.py
from setuptools import find_packages, setup
setup(
name='my_library',
version='1.0.0',
packages=find_packages(),
include_package_data=True,
zip_safe=False,
install_requires=[
'requests',
],
)
I then pushed the code to my private github repo, and now want to install it as a dependency of client
client structure
$ pwd
~/Projects/client
$ tree -a -L 1
.
├── .venv
└── client.py
$ cat client.py
from my_library import process
data = list(range(5))
process(data)
$ . .venv/bin/activate
(.venv) $ pip install git+ssh://git#github.com/USER/library.git
...
Installing collected packages: idna, certifi, urllib3, chardet, requests, my-library
Running setup.py install for my-library ... done
Successfully installed certifi-2019.9.11 chardet-3.0.4 idna-2.8 my-library-1.0.0 requests-2.22.0 urllib3-1.25.3
$ python client.py
Traceback (most recent call last):
File "client.py", line 1, in <module>
from my_library import process
ModuleNotFoundError: No module named 'my_library'
A point that I realized and am thinking it might be related to the question;
The directory (and repo) are named library (single word)
In setup.py the name is my_library (name='my_library') (separated by an underscore)
pip freeze shows it as my-library==1.0.0 (separated with a hyphen)

You are confused between the project name and package/ module name.
Python import system doesn't care about the project name, only pip does.
Python cares about packages and modules but your project doesn't have packages therefore find_packages() doesn't add nothing to your folder.
What you should do is:
Create a folder named my_library under the project folder.
Put __init__.py on this folder
Put you python modules in this folder.
Remove the __init__.py from your project folder.
More info here

Call another setup.py in setup.py

My repository contains my own python module and a submodule to one of its dependencies which has its own setup.py.
I'd like to call the dependency's setupy.py when installing my own lib, how is it possible?
My first attempt:
$ tree
.
├── dependency
│ └── setup.py
└── mylib
└── setup.py
$ cat mylib/setup.py
from setuptools import setup
setup(
name='mylib',
install_requires= ["../dependency"]
# ...
)
$ cd mylib && python setup.py install
error in arbalet_core setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'../depen'"
However install_requires does not accept paths.
My second attempt was to use dependency_links=["../dependency"] with install_requires=["dependency"] however a dependency of the same name already exists in Pypi so setuptools tries to use that version instead of mine.
What's the correct/cleanest way?

A possible solution is to run a custom command before/after the install process.
An example:
from setuptools import setup
from setuptools.command.install import install
import subprocess
class InstallLocalPackage(install):
def run(self):
install.run(self)
subprocess.call(
"python path_to/local_pkg/setup.py install", shell=True
)
setup(
...,
cmdclass={ 'install': InstallLocalPackage }
)

install local package into virtualenv using setuptools

I have a virtualenv with multiple little projects in it. Consider that they are all equal, so my folder structure looks something like this:
categorisation_ml/
categorisation.py
setup.py
__init__.py
nlp/
nlp.py
setup.py
__init__.py
etc/
__init__.py
I want to install both packages into the same virtualenv so that they are both accessible everywhere within the virtualenv.
Using this and this guide, I have created a setup.py script like this (for categorisation in this case):
from setuptools import setup, find_packages
setup(
name = "categorisation",
version = "1.0",
scripts = ['categorisation.py']
)
then, I run python setup.py install , which seems to complete successfully.
When I cd into nlp/, enter python command line and try
import categorisation, I get:
ImportError: No module named categorisation.
What am I missing?

It seems that the package structure and setup.py is off. It should be something like this:
irrelevant_package_name/
__init__.py
setup.py
categorisation_ml/
categorisation.py
__init__.py
nlp/
nlp.py
__init__.py
and then the install script looking like this:
from setuptools import setup, find_packages
setup(
name='package_name',
version='1.0.0',
description='This is a working setup.py',
url='http://somesite.com',
author='Roman',
author_email='roman#somesite.com',
packages=find_packages(),
install_requires=[
'numpy',
],
zip_safe=False
)
Then install it like this:
python setup.py install #(just installs it as is)
python setup.py develop #(Keeps track of changes for development)
If you pip freeze this should come up
package_name==1.0.0
And then in python imports should look like this:
from categorisation_ml import categorisation
from nlp import nlp