Develop Reference

How to install a Python package which is just one SO file

I have an SO file mymodule.cpython-37m-x86_64-linux-gnu.so that I would like to make pip-installable.
My desired end goal is to have my installed package look like this:
% tree /home/.../python3.7/site-packages
/home/.../python3.7/site-packages
├── mymodule-1.0.0.dist-info
└── mymodule.cpython-37m-x86_64-linux-gnu.so
This is what I have tried so far:
% tree .
.
├── mymodule.cpython-37m-x86_64-linux-gnu.so
├── pyproject.toml
└── setup.cfg
# setup.cfg
[options]
py_modules = mymodule
[options.package_data]
* = mymodule.cpython-37m-x86_64-linux-gnu.so
However, when trying to pip install . I cannot seem to get the .so file to be installed into site-packages.
Interestingly, when there is a file named mymodule.py instead, mymodule.py gets installed in the desired location.

Do I need to have a __init__ file if package consists of one file only? [duplicate]

I am trying to learn how to make a Python module available via pip on PyPI. In order to do this, I am testing using the PyPI test site (https://testpypi.python.org/pypi) and have attempted to create a setup.py for the module. My module is a file at the root directory and I cannot get it to be installed successfully. I want to find out how to do this.
Below, I detail the steps I am taking. I suspect that the problem lies in how I have written setup.py.
The anatomy of the repository is as follows:
.
├── examples_1.py
├── LICENSE
├── MANIFEST.in
├── README.rst
├── setup.py
└── supermodule.py
Note that the module is simply the file supermodule.py at the root of the directory. Note also that the file examples_1.py is not to be included in an installation of the module package.
The contents of setup.py are as follows:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import os
import setuptools
def read(*paths):
with open(os.path.join(*paths), "r") as filename:
return filename.read()
def main():
setuptools.setup(
name = "supermodule",
version = "2015.10.30.0820",
description = "super utilities",
long_description = (read("README.rst")),
url = "https://github.com/johndrake1/junk",
author = "John Drake",
author_email = "j.drake#sern.ch",
license = "GPLv3",
package_data = {
"": [
"*.txt",
"*.md",
"*.rst",
"*.py"
]
}
)
if __name__ == "__main__":
main()
I go through the following procedures to register, upload and install the package:
python setup.py register -r https://testpypi.python.org/pypi
python setup.py sdist upload -r https://testpypi.python.org/pypi
sudo pip install -i https://testpypi.python.org/pypi supermodule
In the source distribution, supermodule-2015.10.30.0820.tar.gz, I can see the following directory structure:
.
└── supermodule-2015.10.30.0820
├── LICENSE
├── MANIFEST.in
├── PKG-INFO
├── README.rst
├── setup.cfg
├── setup.py
├── supermodule.egg-info
│   ├── dependency_links.txt
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt
└── supermodule.py
So, it appears that the packaging and uploading works fine and contains the module file supermodule.py that is at the root directory. However, when I install the package, I get the following files installed locally:
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/DESCRIPTION.rst
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/METADATA
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/RECORD
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/WHEEL
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/metadata.json
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/top_level.txt
You can see that the file supermodule.py is not there and it cannot be imported in a Python instance. What should I do to include this file in the installation such that it is importable in Python?
EDIT: Following a suggestion by #DeanFenster, I moved the file supermodule.py to supermodule/__init__.py and changed setup.py to the following:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import os
import setuptools
def read(*paths):
with open(os.path.join(*paths), "r") as filename:
return filename.read()
def main():
setuptools.setup(
name = "supermodule",
version = "2015.10.30.0902",
description = "super utilities",
long_description = (read("README.rst")),
url = "https://github.com/johndrake1/junk",
author = "John Drake",
author_email = "j.drake#sern.ch",
license = "GPLv3",
packages = ["supermodule"]
)
if __name__ == "__main__":
main()
Following registration, upload and installation, this resulted in an installation that made the module importable, with the following files installed locally:
/usr/local/lib/python2.7/dist-packages/supermodule
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info
/usr/local/lib/python2.7/dist-packages/supermodule/__init__.py
/usr/local/lib/python2.7/dist-packages/supermodule/__init__.pyc
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/DESCRIPTION.rst
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/METADATA
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/RECORD
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/WHEEL
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/metadata.json
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/top_level.txt
It's good that this approach works, but I would still like to know how to install the module when it is in the form of a single file.
EDIT: Following a suggestion by #Xk0nSid, I changed setup.py to the following:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import os
import setuptools
def read(*paths):
with open(os.path.join(*paths), "r") as filename:
return filename.read()
def main():
setuptools.setup(
name = "supermodule",
version = "2015.10.30.1001",
description = "super utilities",
long_description = (read("README.rst")),
url = "https://github.com/johndrake1/junk",
author = "John Drake",
author_email = "j.drake#sern.ch",
license = "GPLv3",
py_modules = ["supermodule"],
entry_points = """
[console_scripts]
supermodule = supermodule:supermodule
"""
)
if __name__ == "__main__":
main()
Following registration, upload and installation, this resulted in an installation that made the module importable, with the following files installed locally:
/usr/local/bin/supermodule
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info
/usr/local/lib/python2.7/dist-packages/supermodule.py
/usr/local/lib/python2.7/dist-packages/supermodule.pyc
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/DESCRIPTION.rst
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/METADATA
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/RECORD
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/WHEEL
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/entry_points.txt
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/metadata.json
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/top_level.txt
This approach was successful at handling the single file form of the module.

Try using something like this for a single file. Following is the directory structure:
.
├── example.py
├── LICENSE
├── README.md
└── setup.py
0 directories, 4 files
setup.py
from setuptools import setup
setup(
name='example',
version='0.1.0',
py_modules=['example'],
install_requires=[
'exampledep',
],
entry_points='''
[console_scripts]
example=example:example
''',
)
The above worked for me. This is how example file would look like.
def example():
# Note: You can use sys.argv here
print "Hi! I'm a command written in python."
This can also be imported like so:
import example
example.example()
# or
from example import example
example()
Hope this helps.
Install Requires
The install_requires is used to define the dependencies for your module/application. For e.g in this case example module in dependent on exampledep. So when someone does pip install example, then pip will also install exampledep as it is listed in the dependencies.
Entry Points
This is usually a callable which the end user of package might want to use. This usually a callable and is used for command line. You can look at this question or this doc for more details.

How can I correctly include a path dependency in pyproject.toml?

I have 2 projects structured as below:
/abc-lib
/ abc
/ __init__.py
/ main.py
/ pyproject.toml
/abc-web-api
/ src
/ __init__.py
/ main.py
/ pyproject.toml
I attempted to include abc-lib as a dependency in abc-web-api, thus having a abc-web-api/pyproject.toml as below:
[tool.poetry]
name = "abc-web-api"
version = "0.0.1"
description = "Some description."
authors = ["Someone <someone#example.com>"]
repository = "https://github.com/someone/abc-web-api"
readme = "README.md"
[tool.poetry.scripts]
serve = "src.main:app"
[tool.poetry.dependencies]
python = "~3.6.8"
abc-lib = { path="../abc-lib" }
[tool.poetry.dev-dependencies]
pytest = "^3.10.1"
yapf = "^0.30.0"
flake8 = "^3.8.3"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
When I execute poetry install, I receive the following message:
Package operations: 1 installs, 0 updates, 0 removals
- Installing abc-lib (1.0.0 ../abc-lib)
[ModuleOrPackageNotFound]
No file/folder found for package abc-lib
The version number shown in the "Installing" statement is correct, so I am quite confused about the meaning of [ModuleOrPackageNotFound].
Does anyone know how can I resolve it? Thanks

Your folder structure looks a bit weird. It looks like your prefer the "src" variant. So I would suggest the following:
./
├── abc-lib
│   ├── pyproject.toml
│   └── src
│   └── abc_lib
│   ├── __init__.py
│   └── main.py
└── abc-web-api
├── pyproject.toml
└── src
└── abc_web_api
├── __init__.py
└── main.py
With this pyproject.toml in abc-lib:
[tool.poetry]
name = "abc-lib"
version = "0.1.0"
description = ""
authors = ["Someone <someone#example.com>"]
[tool.poetry.dependencies]
python = "^3.6"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"
And this in abc-web-api:
[tool.poetry]
name = "abc-web-api"
version = "0.1.0"
description = ""
authors = ["Someone <someone#example.com>"]
[tool.poetry.dependencies]
python = "^3.6"
abc-lib = {path = "../abc-lib"}
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"

How to organize a python project with poetry for a command line script

When doing a poetry init I got the following structure:
/packagename
__init__.py
packagename.py
/packagename
/tests
__init__.py
test_packagename.py
pyproject.toml
which is fine for a package but I do not see how to make it fit a command line script.
When I have a script like script.py with the following code structure:
In file script.py:
#!/usr/bin/python3
def main():
print("Ok")
if __name__ == '__main__':
main()
It is not intended to be used as a python module however, it may have dependencies and tests that are interesting to be handle with poetry.
In some example it is shown that with the following poetry syntax:
[tool.poetry.scripts]
cli_script = 'script.py:main'
then one can call the script with:
poetry run cli_script
I am looking for some guideline on howto organize properly my poetry project for such usage.
I have looked for option for poetry init (like poetry init --script) for instance. But it seems that kind of use case was not covered in the new/init poetry options.

By "poetry init" I guess you mean poetry new. But also then your structure looks a bit weird. I would suggest the following structure:
packagename
├── packagename
│   ├── __init__.py
│   └── cli.py
├── tests
│   ├── __init__.py
│   └── test_packagename.py
└── pyproject.toml
The pyproject.toml looks like this:
[tool.poetry]
name = "packagename"
version = "0.1.0"
description = ""
authors = ["finswimmer <finswimmer#example.org>"]
[tool.poetry.scripts]
cli_script = "packagename.cli:main"
[tool.poetry.dependencies]
python = "^3.6"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"
You cli.py as in your example:
#!/usr/bin/python3
def main():
print("Ok")
if __name__ == '__main__':
main()
After a poetry install you can run poetry run cli_script.

Alternatively you can run:
poetry run python3 script.py
where:
script.py is the name of the file having python code.
python3 is the python executable inside the poetry virtual environment. This can be python as well, depending on the python version you have. You may confirm the same using poetry run python3 -V or poetry run python -V, before executing the above command.
This command can be used in crontab as well to schedule simple scripts.

How can a Python module single file be installed using pip and PyPI?

I am trying to learn how to make a Python module available via pip on PyPI. In order to do this, I am testing using the PyPI test site (https://testpypi.python.org/pypi) and have attempted to create a setup.py for the module. My module is a file at the root directory and I cannot get it to be installed successfully. I want to find out how to do this.
Below, I detail the steps I am taking. I suspect that the problem lies in how I have written setup.py.
The anatomy of the repository is as follows:
.
├── examples_1.py
├── LICENSE
├── MANIFEST.in
├── README.rst
├── setup.py
└── supermodule.py
Note that the module is simply the file supermodule.py at the root of the directory. Note also that the file examples_1.py is not to be included in an installation of the module package.
The contents of setup.py are as follows:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import os
import setuptools
def read(*paths):
with open(os.path.join(*paths), "r") as filename:
return filename.read()
def main():
setuptools.setup(
name = "supermodule",
version = "2015.10.30.0820",
description = "super utilities",
long_description = (read("README.rst")),
url = "https://github.com/johndrake1/junk",
author = "John Drake",
author_email = "j.drake#sern.ch",
license = "GPLv3",
package_data = {
"": [
"*.txt",
"*.md",
"*.rst",
"*.py"
]
}
)
if __name__ == "__main__":
main()
I go through the following procedures to register, upload and install the package:
python setup.py register -r https://testpypi.python.org/pypi
python setup.py sdist upload -r https://testpypi.python.org/pypi
sudo pip install -i https://testpypi.python.org/pypi supermodule
In the source distribution, supermodule-2015.10.30.0820.tar.gz, I can see the following directory structure:
.
└── supermodule-2015.10.30.0820
├── LICENSE
├── MANIFEST.in
├── PKG-INFO
├── README.rst
├── setup.cfg
├── setup.py
├── supermodule.egg-info
│   ├── dependency_links.txt
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt
└── supermodule.py
So, it appears that the packaging and uploading works fine and contains the module file supermodule.py that is at the root directory. However, when I install the package, I get the following files installed locally:
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/DESCRIPTION.rst
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/METADATA
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/RECORD
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/WHEEL
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/metadata.json
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0820.dist-info/top_level.txt
You can see that the file supermodule.py is not there and it cannot be imported in a Python instance. What should I do to include this file in the installation such that it is importable in Python?
EDIT: Following a suggestion by #DeanFenster, I moved the file supermodule.py to supermodule/__init__.py and changed setup.py to the following:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import os
import setuptools
def read(*paths):
with open(os.path.join(*paths), "r") as filename:
return filename.read()
def main():
setuptools.setup(
name = "supermodule",
version = "2015.10.30.0902",
description = "super utilities",
long_description = (read("README.rst")),
url = "https://github.com/johndrake1/junk",
author = "John Drake",
author_email = "j.drake#sern.ch",
license = "GPLv3",
packages = ["supermodule"]
)
if __name__ == "__main__":
main()
Following registration, upload and installation, this resulted in an installation that made the module importable, with the following files installed locally:
/usr/local/lib/python2.7/dist-packages/supermodule
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info
/usr/local/lib/python2.7/dist-packages/supermodule/__init__.py
/usr/local/lib/python2.7/dist-packages/supermodule/__init__.pyc
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/DESCRIPTION.rst
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/METADATA
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/RECORD
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/WHEEL
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/metadata.json
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.0902.dist-info/top_level.txt
It's good that this approach works, but I would still like to know how to install the module when it is in the form of a single file.
EDIT: Following a suggestion by #Xk0nSid, I changed setup.py to the following:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import os
import setuptools
def read(*paths):
with open(os.path.join(*paths), "r") as filename:
return filename.read()
def main():
setuptools.setup(
name = "supermodule",
version = "2015.10.30.1001",
description = "super utilities",
long_description = (read("README.rst")),
url = "https://github.com/johndrake1/junk",
author = "John Drake",
author_email = "j.drake#sern.ch",
license = "GPLv3",
py_modules = ["supermodule"],
entry_points = """
[console_scripts]
supermodule = supermodule:supermodule
"""
)
if __name__ == "__main__":
main()
Following registration, upload and installation, this resulted in an installation that made the module importable, with the following files installed locally:
/usr/local/bin/supermodule
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info
/usr/local/lib/python2.7/dist-packages/supermodule.py
/usr/local/lib/python2.7/dist-packages/supermodule.pyc
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/DESCRIPTION.rst
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/METADATA
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/RECORD
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/WHEEL
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/entry_points.txt
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/metadata.json
/usr/local/lib/python2.7/dist-packages/supermodule-2015.10.30.1001.dist-info/top_level.txt
This approach was successful at handling the single file form of the module.

Try using something like this for a single file. Following is the directory structure:
.
├── example.py
├── LICENSE
├── README.md
└── setup.py
0 directories, 4 files
setup.py
from setuptools import setup
setup(
name='example',
version='0.1.0',
py_modules=['example'],
install_requires=[
'exampledep',
],
entry_points='''
[console_scripts]
example=example:example
''',
)
The above worked for me. This is how example file would look like.
def example():
# Note: You can use sys.argv here
print "Hi! I'm a command written in python."
This can also be imported like so:
import example
example.example()
# or
from example import example
example()
Hope this helps.
Install Requires
The install_requires is used to define the dependencies for your module/application. For e.g in this case example module in dependent on exampledep. So when someone does pip install example, then pip will also install exampledep as it is listed in the dependencies.
Entry Points
This is usually a callable which the end user of package might want to use. This usually a callable and is used for command line. You can look at this question or this doc for more details.