Background
My code has the following structure:
.
└── my_package
├── __init__.py
├── classif
│ ├── __init__.py
│ ├── inference.py
│ └── models.py
├── datasets
│ ├── __init__.py
│ └── datasets.py
└── matching
├── __init__.py
├── elastic.py
└── search.py
After a sphinx-quickstart (with autodoc) I ran sphinx-apidoc as follows:
sphinx-apidoc -f -o source my_package -e -M
I now have the following files:
.
├── Makefile
├── build
├── my_package
│ ├── __init__.py
│ ├── __pycache__
│ ├── classif
│ ├── datasets
│ └── matching
└── source
├── _static
├── _templates
├── conf.py
├── index.rst
├── modules.rst
├── my_package.classif.inference.rst
├── my_package.classif.models.rst
├── my_package.classif.rst
├── my_package.datasets.datasets.rst
├── my_package.datasets.rst
├── my_package.matching.elastic.rst
├── my_package.matching.rst
├── my_package.matching.search.rst
└── my_package.rst
I also modified conf.py to add:
import os
import sys
sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('..'))
and added modules to the index.rst:
.. toctree::
:maxdepth: 2
:caption: Contents:
modules
# [more lines]
Now here is the issue
After running make clean ; make html I get the warnings:
/path/to/my_package.classif.rst: WARNING: document isn't included in any toctree
/path/to/my_package.datasets.rst: WARNING: document isn't included in any toctree
/path/to/my_package.matching.rst: WARNING: document isn't included in any toctree
Which makes sense as sphinx-apidoc did not reference them in my_package.rst. How can I solve this?
The issue was actually known -> https://github.com/sphinx-doc/sphinx/issues/4446
Upgrading to sphinx 1.7.6 solved it.
Related
Is there a way to create a requirements.txt file that only contains the modules that my script actually needs?
I usually just do a pip freeze and then remove unused modules.
You can use pipreqs to analyze your project files, and automatically generate a requirements.txt for you.
First step is to install pipreqs. You can do so, by executing the following command in your console:
pip install pipreqs
Then, the only thing you need to do is to execute the command pipreqs, specifying the path where your files are located at. For example, to generate a requirements.txt, based on the modules on the current working directory, you could execute:
pipreqs .
For some other directory:
pipreqs "PATH/TO/YOUR/PROJECT/PYTHON/FILES"
Real World Example
Here's the tree-view of the folder we're going to use as example:
.
├── data
│ └── EXCEL_FILES
│ ├── DIVISION_MAP.xlsx
│ ├── INPUT_CONTAINER.xlsx
│ ├── KITS.xlsx
│ ├── LOADING.xlsx
│ ├── MOQ_CBM_SHIPPING_TYPE.xlsx
│ ├── OUTBOUND_OTM.XLSX
│ ├── PLANT_SOURCE_MAP.xlsx
│ ├── PORT_STATE.xlsx
│ └── SALABLE_STORAGE_LOCATION.xlsx
├── loading
│ ├── __init__.py
│ ├── configs
│ │ ├── Initialization.py
│ │ ├── __init__.py
│ │ ├── config.ini
│ │ └── logconfig.py
│ ├── constants.py
│ ├── read_files.py
│ ├── reports.py
│ └── utils
│ ├── __init__.py
│ ├── date_utils.py
│ ├── file_utils.py
│ └── utils.py
├── logs
│ └── po_opt.log
├── main.py
└── outputs
├── 2022-10-03
├── 2022-10-04
├── 2022-10-05
├── 2022-10-06
├── 2022-11-23
├── 2022-11-24
└── 2022-11-28
Executing the command pipreqs . from the root path I get the following requirements.txt:
❯ pipreqs .
INFO: Successfully saved requirements file in ./requirements.txt
Output:
holidays==0.17.2
numpy==1.20.1
pandas==1.2.4
python_dateutil==2.8.2
six==1.16.0
I'm developing a python package foo. My project structure looks like this:
.
├── foo
│ ├── foo
│ │ ├── bar.py
│ │ ├── foo.py
│ │ ├── __init__.py
│ ├── README.md
│ └── setup.py
├── footest
│ ├── test.py
test.py only has 1 line: import foo
In order for test.py to be able to import the package foo I install it with the command pip3 install -e foo.
Now a new folder called foo.egg-info is created under foo/
.
├── foo
│ ├── foo
│ │ ├── bar.py
│ │ ├── foo.py
│ │ ├── __init__.py
│ ├── foo.egg-info
│ │ ├── dependency_links.txt
│ │ ├── PKG-INFO.txt
│ │ ├── requires.txt
│ │ ├── SOURCES.txt
│ │ ├── top_level.txt
│ ├── README.md
│ └── setup.py
├── footest
│ ├── test.py
What's the purpose of this folder? I tried deleting it and test.py still ran properly. Is is just leftover garbage, similar to the .o files when compiling C projects? If so, is there a way to automatically remove it?
The package.egg-info saves meta data about your installed package like version.
It is used when you for example uninstall it, or "pip list" to see what is installed.
you can open it and take a look.
I need to configure pytest in my django application. But the structure of the directories is different of what I'm used to.
.
├── apps
│ ├── __pycache__
│ ├── admcore
│ ├── ...
│ ├── cauth
│ ├── webservice
│ └── webshell
├── doc
│ └── ...
├── lib
│ └── ...
├── ...
└── project
├── __init__.py
├── globalsettings.py
├── local
│ ├── __init__.py
│ ├── app.uwsgi
│ ├── manage.py
│ ├── settings.py
│ └── wsgi.py
├── mailconf.py
└── urls.py
I created pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE=sgp.local.test_settings
addopts = --nomigrations --cov=. --cov-report=html
and test_settings.py files
#!/usr/bin/env python
import os
import sys
from project import globalsettings
SECRET_KEY = "12341234AFDQFDQSDFAZR123D"
ROOT_PATH = '/usr/local/sgp'
BASEURL_ROOT = ''
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': ':memory:'
}
}
And I put the two files in project directory.
.
├── apps
│ ├── __pycache__
│ ├── admcore
│ ├── ...
│ ├── cauth
│ ├── webservice
│ └── webshell
├── doc
│ └── ...
├── lib
│ └── ...
├── ...
└── project
├── __init__.py
├── globalsettings.py
├── local
│ ├── __init__.py
│ ├── app.uwsgi
│ ├── manage.py
│ ├── pytest.ini
│ ├── settings.py
│ ├── test_settings.py
│ └── wsgi.py
├── mailconf.py
└── urls.py
But it doesn't work as I expected. I have one test in webservice app, but when I run pytest command I got:
platform darwin -- Python 3.7.4, pytest-5.2.0, py-1.8.0, pluggy-0.13.0
Django settings: project.local.test_settings (from environment variable)
rootdir: /project/local, inifile: pytest.ini
plugins: django-3.5.1, cov-2.7.1
collected 0 items
---------- coverage: platform darwin, python 3.7.4-final-0 -----------
Coverage HTML written to dir htmlcov
=========================================================================================================== no tests ran in 0.15s ============================================================================================================
I have no ideia how to configure pytest to this project.
May be lame question for many of you smart people out there, but I am struggling with a simple python package creation.
My dir structure for package is :
address-book/
├── __init__.py
├── dist
│ └── book-0.1.tar.gz
├── address-book
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── person.py
│ └── person.pyc
├── address_book.egg-info
│ ├── PKG-INFO
│ ├── SOURCES.txt
│ ├── dependency_links.txt
│ ├── not-zip-safe
│ └── top_level.txt
├── setup.py
└── tests
├── __init__.py
├── __init__.pyc
├── person_test.py
└── person_test.pyc
The setup.py is as follows:
from setuptools import setup, find_packages
setup(name='address-book',
version='0.1',
description='The funniest joke in the world',
url='http://github.com/storborg/funniest',
author='Address Book',
author_email='flyingcircus#example.com',
license='MIT',
packages=find_packages('.'),
test_suite="tests",
zip_safe=False)
The SOURCES.txt:
setup.py
address-book/__init__.py
address-book/person.py
address_book.egg-info/PKG-INFO
address_book.egg-info/SOURCES.txt
address_book.egg-info/dependency_links.txt
address_book.egg-info/not-zip-safe
address_book.egg-info/top_level.txt
tests/__init__.py
tests/person_test.py
In the person_test.py I am not able to import person.py, what could be the reason?
Solution
In case if someone struggles with the same problem, mine got fixed by not using hyphens as package name. Simple and worked!
It looks like you have address-book and address_book.egg-info. I think it should be address-book.egg-info.
I want to create a pybuilder project with unit tests and packages. As an example, modified simple python app example, with "helloworld" moved to package "hello".
My first instinct was to match the package structure with "main" and "unittest" sources:
+---src
+---main
| \---python
| \---hello
| helloworld.py
| __init__.py
|
\---unittest
\---python
\---hello
helloworld_tests.py
__init__.py
This does not work because of conflicting "hello" package.
BUILD FAILED - 'module' object has no attribute 'helloworld_tests'
I see pybuilder itself just skips the top-level pybuilder package in unittests, but won't do if there are multiple top-level packages.
My second guess would be to create extra top level package for unittests.
\---unittest
\---python
\---tests
| __init__.py
\---hello
helloworld_tests.py
__init__.py
Is there a better solution or established convention how to organize python tests in packages?
Probably nothing really new for the OP, but I just wanted to collect all options that I could come up with in one place:
1) Just append _tests to names of top-level packages
The easiest way to mirror the structure of src/main/python in the src/test/python almost 1:1 would be by simply appending _tests to the names of the top-level packages. For example, if I have only one top-level package rootPkg, then I can add the corresponding rootPkg_tests to the test/ subdirectory:
src
├── main
│ ├── python
│ │ └── rootPkg
│ │ ├── __init__.py
│ │ ├── pkgA
│ │ │ ├── __init__.py
│ │ │ └── modA.py
│ │ └── pkgB
│ │ ├── __init__.py
│ │ └── modB.py
│ └── scripts
│ └── entryPointScript.py
└── test
└── python
└── rootPkg_tests
├── __init__.py
├── pkgA
│ ├── __init__.py
│ └── modA_tests.py
└── pkgB
├── __init__.py
└── modB_tests.py
This seems to work nicely with PyBuilder 0.11.15 and unittest plugin (notice that I've deviated from PyBuilders convention, and put tests in test instead of unittest, you probably shouldn't do this if you intend to use multiple testing frameworks).
2) If there is only one package: do it like PyBuilder
The PyBuilder is itself built with PyBuilder. This is what its source directory looks like (drastically reduced, unnecessary details omitted):
src
├── main
│ ├── python
│ │ └── pybuilder
│ │ ├── __init__.py
│ │ ├── cli.py
│ │ ├── core.py
│ │ └── plugins
│ │ ├── __init__.py
│ │ ├── core_plugin.py
│ │ └── exec_plugin.py
│ └── scripts
│ └── pyb
└── unittest
└── python
├── cli_tests.py
├── core_tests.py
├── plugins
│ ├── exec_plugin_tests.py
│ ├── __init__.py
│ ├── python
│ │ ├── core_plugin_tests.py
│ │ ├── __init__.py
If I understand it correctly, the tree in unittest mirrors the tree in src, but the directory for the top-level package pybuilder is omitted. That's what you have described in your question as first workaround. The drawback is that it doesn't really work if there are multiple top-level packages.
3) Add one additional tests top-level package
That's what you have proposed as a workaround: mirror the tree in main, but wrap everything in an additional tests-package. This works with many top-level packages in /src/main/python and prevents any package name collisions.
I'm not aware of any convention. Upvote one of the comments below if you have an opinion on that matter.