Develop Reference

Building a python egg, first letter is incorrectly removed from root file/folder name

I'm trying to build an egg for my python project using setuptools, however, whenever I build an egg all of the contents are built with the first letter if each file/folder removed.
For example, my parent folder is called dp which gets renamed to p. I.e. when I unzip the egg file, I see a parent folder named p and another folder named GG-INFO (these should be named dp and EGG-INFO respectively). All of the other folders inside folder p are named correctly.
This is an issue because I reference functions in modules within that folder - e.g. from dp.module import function which doesn't work because it complains about not finding the folder dp (which is true since for some reason it's been renamed p).
My setup.py file looks like this:
from setuptools import setup, find_packages
setup(
name="dp",
version="1.0",
author="XXXX",
author_email="XXXX",
description="Data pipeline for XXX algorithm.",
long_description_content_type="text/markdown",
url="XXXX",
packages=find_packages(),
package_data={'': ['*.sql', '*.json', '*.txt']},
include_package_data=True,
classifiers=[
"Programming Language :: Python :: 3"
],
python_requires='>=3.6',
install_requires=['argparse', 'boto3', 'datetime', 'mmlspark', 'pandas', 'pyspark', 'pypandoc', 'scikit-learn',
'numpy', 'googleads', 'mlflow']
)
I've tried renaming the parent directory and the same thing happens. I'm running this via PyCharm (updated to the latest version) on Mac OS Mojave.
Would appreciate any ideas on how to fix this.
Update:
I used a different method to generate the egg which unblocked me, but the issue still remains with the initial method.
Steps to reproduce
Create a new project in Pycharm
Add a setup.py file to root, see above.
Tools -> Run setup.py task -> bdist.egg
Generates an egg. Rename the extension to file_name.zip, unzip the file, check the contents of the folder.
I found that the first letter of the folder names was always missing (i changed the names of the folder and it consistently removed the first letter).
Workaround
Instead of building an egg via Pycharm, i used the command python setup.py bdist_egg in the terminal which created an egg without any issues.
I think this confirms it is a Pycharm bug(?). A colleague managed to intermittently reproduce this bug using Pycharm.

Give the wheel a try.
pip install wheel setuptools pip -U
pip wheel --no-deps --wheel-dir=build .

Use Setuptools with C Extension as part of package

I wrote a C Extension to access an error message for a camera from a proprietary library. The structure is
setup.py
dicamsdk\
|---__init__.py
|---control.py
|---pcoError.c
with setup.py
from setuptools import setup, Extension, find_packages
from dicamsdk.control import __version__
pcoErrorModule = Extension("dicamsdk.pcoError",
sources=["dicamsdk/pcoError.c"],
include_dirs=['C:\Program Files (x86)'
'\Digital Camera Toolbox'
'\Sensicam SDK\include'],
define_macros=[("PCO_ERRT_H_CREATE_OBJECT", None)],
)
setup(
name="pydicamsdk",
platforms=["win-amd64", 'win32'],
license="GNU GPLv3",
ext_modules=[pcoErrorModule],
packages=find_packages(),
version=__version__
)
and the control.py intends to import the compiled C Extension with
from . import pcoError
When I try to build (or install) the package I always receive the error ImportError: cannot import name 'pcoError'.
The only way it seems to work is to comment out the import in control.py and build the C Extension with setup.py build_ext --inplace. Just with the compiled present I can build/install my library.
Is there a solution to be implemented in my setup.py to compile my extension in first place to enable a simple installation?

A more proper solution based on your own answer:
Your version is presumably for the entire project, not just the control module. It is standard to define __version__ in __init__.py. In that case, your import in setup.py would look like from dicamsdk import __version__. This will cause no conflicts, unless you do something foolish like import all your modules automatically from the package root.

The problem was with the import at the beginning of setup.py. The import in line 2 (from dicamsdk.control import __version__) forced the code check of the module.
When removed, the install or build of the packages runs correctly.

How to install data_files of python package into home directory

Here's my setup.py
setup(
name='shipane_sdk',
version='1.0.0.a5',
# ...
data_files=[(os.path.join(os.path.expanduser('~'), '.shipane_sdk', 'config'), ['config/scheduler-example.ini'])],
# ...
)
Packing & Uploading commands:
python setup.py sdist
python setup.py bdist_wheel --universal
twine upload dist/*
Installing command:
pip install shipane_sdk
But, it doesn't install the config/scheduler-example.ini under ~/.shipane_sdk
The pip documents says:
setuptools allows absolute “data_files” paths, and pip honors them as
absolute, when installing from sdist. This is not true when installing
from wheel distributions. Wheels don’t support absolute paths, and
they end up being installed relative to “site-packages”. For
discussion see wheel Issue #92.
Do you know how to do installing from sdist?

There are multiple solutions to this problem and it is all very confusing how inconsistent the packaging tools work. Some time ago I found the following workaround worked for me best with sdist (note that it doesn't work with wheels!):
Instead of using data_files, attach the files to your package using MANIFEST.in, which in your case could look like this:
include config/scheduler-example.ini
Copy the files "manually" to chosen location using this snippet in setup.py:
if 'install' in sys.argv:
from pkg_resources import Requirement, resource_filename
import os
import shutil
# retrieve the temporary path where the package has been extracted to for installation
conf_path_temp = resource_filename(Requirement.parse(APP_NAME), "conf")
# if the config directory tree doesn't exist, create it
if not os.path.exists(CONFIG_PATH):
os.makedirs(CONFIG_PATH)
# copy every file from given location to the specified ``CONFIG_PATH``
for file_name in os.listdir(conf_path_temp):
file_path_full = os.path.join(conf_path_temp, file_name)
if os.path.isfile(file_path_full):
shutil.copy(file_path_full, CONFIG_PATH)
In my case "conf" was the subdirectory in the package that contained my data files and they were supposed to be installed into CONFIG_PATH which was something like /etc/APP_NAME

Python setuptools develop command: "No module named..."

I'm trying to install a python package I've developed using the develop command of setuptools.
[sidenote: There is a bewilderingly vast quantity of information about this on the web (distutils, distutils2, setuptools, distribute). setuptools and develop are, as far as I can tell, the most modern/best practice way to use a piece of code that's in development. Perhaps I am wrong.]
Here's what I did:
(1) I placed an empty __init__.py in the directory with my Python code.
(2) I made a setup.py:
from setuptools import setup, find_packages
setup(name = "STEM_pytools",
version = "0.1",
packages = find_packages(),
author = "Timothy W. Hilton",
author_email = "my#email.address",
description = "visualization and data pre-/post-processing tools for STEM",
license = "",
keywords = "STEM",
url = "")
(3) I ran
python setup.py develop
That seemed to proceed without problems.
However, when I try to use the package, I get:
>>> import STEM_pytools
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named STEM_pytools
The same thing happens with the install command: it's output looks ok, then "No module named STEM_pytools". I'm tearing my hair out. Any suggestions appreciated!

I solved the problem, although I still don't entirely understand why it works now and did not work before. It seems my setup.py and the directory structure of my project were not interacting successfully.
This is the directory structure that worked with my setup.py:
STEMpytools/
setup.py
stem_pytools/
__init__.py
source1.py
source2.py
...
sourceN.py
This directory structure did not work, at least when paired with my setup.py:
STEMpytools/
setup.py
__init__.py
source1.py
source2.py
...
sourceN.py
This explanation helped me a lot: http://bashelton.com/2009/04/setuptools-tutorial/
Now, from the python interpreter, these both work:
import stem_pytools
import stem_pytools.source1
Experimenting on my system suggests it is necessary to place __init__.py and the package source code in a subdirectory one level below the root directory that contains setup.py. I'm not sure from the setuptools and distutils documentation why this is the case.

Distributing pre-built libraries with python modules

I use the following script to distribute a module containing pure python code.
from distutils.core import setup, Extension
import os
setup (name = 'mtester',
version = '0.1',
description = 'Python wrapper for libmtester',
packages=['mtester'],
package_dir={'mtester':'module'},
)
The problem I have is, I modified one of the files that uses an external library (a .so file), which I need to ship along with the existing module. I was suggested to use package_data to include the library. I modified the script to the following.
from distutils.core import setup, Extension
import os
data_dir = os.path.abspath('../lib64/')
setup (name = 'mtester',
version = '0.1',
description = 'Python wrapper for libmtester',
packages=['mtester'],
package_dir={'mtester':'module'},
package_data={'mtester':[data_dir+'mhelper.so']},
)
The problem is, adding package_data did not make any difference. This is not installing the mhelper.so in any location (neither in site-packages nor in site-packages/mtester).
System info: Fedora 10, 64 bit, python 2.5 (Yes it is ancient. But it is our build machine, and it needs to stay that way to maintain backward compatibility)
Any suggestions that would help me resolve this would be well appreciated!

Unfortunately package_data looks for files relative to the top of the package. One fix is to move the helper library under the module dir with the rest of the code:
% mv lib64/mhelper.so module/
Then modify the package_data argument accordingly:
package_data = {'mtester': ['mhelper.so']}
...
Then test:
% python setup.py bdist
% tar tf dist/mtester-0.1.linux-x86_64.tar.gz | grep mhelper
./usr/local/lib/python2.5/dist-packages/mtester/mhelper.so