I want to create a pip package which dependent on some OS specific files:
Let's say there are:
dependency_Windows_x86_64.zip
dependency_Linux_x86_64.zip
dependency_MAC_OS_X.zip
I do not want to include all three archives in a package project, but download them dynamically during the pip install my-package based on user's OS. How can I do that ? Where should I put the code responsible for downloading/unzipping those files ?
My setup.py looks like this:
from setuptools import setup
setup(
name='my-package',
version='0.0.1',
description='Package description',
py_modules=['my_package'],
package_dir={'': 'src'},
classifiers=[
'Intended Audience :: Developers',
'License :: OSI Approved :: MIT License',
'Operating System :: POSIX :: Linux',
'Operating System :: Microsoft :: Windows',
'Operating System :: MacOS',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.7'
],
python_requires='>=3.7'
)
The platform specific dependencies could be kept in separate Python projects (wrappers around data only packages) and then required from the main project like the following:
# setup.cfg
# ...
[options]
install_requires =
my_package_win_amd64 ; platform_system=="Windows" and platform_machine=="x86_64"
my_package_linux-x86_64 ; platform_system=="Linux" and platform_machine=="x86_64"
This approach doesn't depend on setuptools and could be used with other build systems.
First answer is give up and use setuptools. Look at Platform specific dependencies for a good write up of what to do.
Second answer is to make separate packages such as 'mylib-mac', 'mylib-win', 'mylib-linux'.
Third answer is to use the "console_script" approach. This will break. While it generates .exe files on Windows, it has odd failure modes. Also, some users will not be able to dynamically download files because they work from an internal clone of a repository. Randomly running code from the Internet on production can scare people.
Hope this helps!
A solution could be to publish platform specific Python wheels of your project. The platform specific files could be added to the pre-built distributions via a custom setuptools command (probably a sub-command of build, or maybe install).
This is not a full solution, but something like this might be a good start:
#!/usr/bin/env python3
import distutils.command.build
import setuptools
class build_something(setuptools.Command):
user_options = [
('plat-name=', 'p', "platform name to build for"),
]
def initialize_options(self):
self.plat_name = None
def finalize_options(self):
self.set_undefined_options('bdist_wheel', ('plat_name', 'plat_name'))
def run(self):
print(" *** plat_name: {} ***".format(self.plat_name))
print(" *** download the platform specific bits to 'build' ***")
class build(distutils.command.build.build):
sub_commands = [(
'build_something',
None,
)] + distutils.command.build.build.sub_commands
setuptools.setup(
cmdclass={
'build_something': build_something,
'build': build,
},
# ...
)
And then the Python wheels could be built like this:
$ ./setup.py bdist_wheel -p win_amd64
Related
I have an internal tool that we are using at work to pull some data and generate a report. The tool was written in python. I made some changes to the tool, and decided to tidy up the folder structure which seemed a bit messy to me. I probably broke something, but I'm determined to fix it and figure out what the issue is.
The project is named ReportCardLite, and my folder structure is like this:
ReportCardLite root folder
-Folder 1
-Folder 2
-rclite
-----__init.py__
-----A.py
-----B.py
-----C.py
-setup.py
-setup.cfg
-__init__.py
Originally, the author was importing by using the package name, which was odd to me since all the script files were in the same directory. So, in A.py for example, he would say something like "from rclite.B import fun".
I decided to remove the "unnecessary" module name from before all the import statements. Of course, that broke it, but I quickly figured out that I could add a line in my settings.json file to look within the rclite folder for all modules. Now my scripts were importing from one another without the module name, and running fine from within the IDE terminal window.
I next needed to build an executable from this module. The original author had included a setup.py and a setup.cfg file, so I used that to build and install this. But when I run this new executable, I receive errors that modules cannot be found. If I change it back to how it originally was, namely the "rclite.A" qualifier, it runs fine. I've spent hours trying to understand what is going on here, and I'm just out of ideas and cannot find any relevant questions using Google.
Can someone kindly point out which configuration I need to change in order to not have to put "rclite" in front of the import statements? Thanks!
Here is the setup.py script.. didn't see anything that looked promising.
"""ReportCardLite Packaging
See:
https://code.amazon.com/packages/ReportCardLite/trees/mainline
"""
# Always prefer setuptools over distutils
from codecs import open
from os import path
from setuptools import setup, find_packages
from cx_Freeze import setup, Executable
here = path.abspath(path.dirname(__file__))
import sys
# Get the long description from the README file
with open(path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = f.read()
setup(name='ReportCardLite',
# Versions should comply with PEP440. For a discussion on single-sourcing
# the version across setup.py and the project code, see
# https://packaging.python.org/en/latest/single_source_version.html
version='0.9.0',
description='RC.Lite for SIM',
long_description=long_description,
# See https://pypi.python.org/pypi?%3Aaction=list_classifiers
classifiers=[
# How mature is this project? Common values are
# 3 - Alpha
# 4 - Beta
# 5 - Production/Stable
'Development Status :: 3 - Alpha',
# Indicate who your project is intended for
# 'Intended Audience :: Developers',
# 'Topic :: Software Development :: Build Tools',
# Pick your license as you wish (should match "license" above)
# 'License :: OSI Approved :: MIT License',
# Specify the Python versions you support here. In particular, ensure
# that you indicate whether you support Python 2, Python 3 or both.
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6'
],
# You can just specify the packages manually here if your project is
# simple. Or you can use find_packages().
packages=find_packages(exclude=['test']),
#packages = ['rclite'],
# Alternatively, if you want to distribute just a my_module.py, uncomment
# this:
# py_modules=["my_module"],
# List run-time dependencies here. These will be installed by pip when
# your project is installed. For an analysis of "install_requires" vs pip's
# requirements files see:
# https://packaging.python.org/en/latest/requirements.html
install_requires=[
'isoweek',
'jinja2',
'jsonpickle',
'jsonplus',
'markdown',
'pkg-resources',
'premailer',
'python-dateutil',
'pyxdg',
'six'
],
options={
'build_exe': {
'packages': ['lxml', 'asyncio', 'markdown.extensions'],
'include_files': ['templates/', 'webapp/']
},
},
# executables = [Executable("rclite/aiohttp_app.py", base="Win32GUI")],
executables=[Executable(script="rclite/aiohttp_app.py", targetName="run_rclite.exe")],
# List additional groups of dependencies here (e.g. development
# dependencies). You can install these using the following syntax,
# for example:
# $ pip install -e .[dev,test]
extras_require={
'dev': ['wheel'],
'test': ['mock'],
},
# If there are data files included in your packages that need to be
# installed, specify them here. If using Python 2.6 or less, then these
# have to be included in MANIFEST.in as well.
package_data={
# 'sample': ['package_data.dat'],
},
# Although 'package_data' is the preferred approach, in some case you may
# need to place data files outside of your packages. See:
# http://docs.python.org/3.4/distutils/setupscript.html#installing-additional-files # noqa
# In this case, 'data_file' will be installed into '<sys.prefix>/my_data'
# data_files=[('my_data', ['data/data_file'])],
# To provide executable scripts, use entry points in preference to the
# "scripts" keyword. Entry points provide cross-platform support and allow
# pip to create the appropriate form of executable for the target platform.
entry_points={
'console_scripts': [
'rclite = rclite.__main__:main',
'rclite-gui = rclite.aiohttp_app.__main__'
],
},
)
I'm did a python library, it's my first python library
published on pypl and github
The library works very well, but the setup() doesn't.
When I install it by pip install, it dowloand the appfly package but do not install the requirements: Flask,flask_cors, Flask-SocketIO and jsonmerge. So I need install it by myself.
If I install the dependencies myself, it's works very well, but i think it's the wrong way to use a python library right?
here is my setup.py file, I'm doing something wrong?
from setuptools import setup, find_packages
from appfly import __version__ as version
with open('README.md') as readme_file:
readme = readme_file.read()
# with open('HISTORY.md') as history_file:
# history = history_file.read()
requirements = [
'Flask==1.0.2',
'flask_cors==3.0.6',
'Flask-SocketIO==3.0.2',
'jsonmerge==1.5.2'
]
setup(
author="Italo José G. de Oliveira",
author_email='italo.i#live.com',
classifiers=[
'Natural Language :: English',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
],
description="This pkg encapsulate the base flask server configurations",
install_requires=requirements,
license="MIT license",
long_description=readme,
include_package_data=True,
keywords='appfly',
name='appfly',
packages=find_packages(),
url='https://github.com/italojs/appfly',
version=version,
zip_safe=False,
)
The reason for this error is that the setup.py imports from the package. This means that python will try importing the library while processing the setup.py (ie. before any of the dependencies get installed).
Since you are only importing the package to get the version information, this import can be replaced with a different method.
An easy way to do this is to include the version information directly in the setup.py, but the drawback with this is that the version is no longer single sourced.
Other methods involve a bit of work but allow the version information to continue to be single sourced. See https://packaging.python.org/guides/single-sourcing-package-version/ for recommendations. That page has a list of options, some of which may be better suited to your package setup than others. I personally prefer option 3:
Set the value to a __version__ global variable in a dedicated module
in your project (e.g. version.py), then have setup.py read and exec
the value into a variable.
...
Using exec:
version = {}
with open("...sample/version.py") as fp:
exec(fp.read(), version)
# later on we use: version['__version__']
You can also define the version in the __init__.py of your package like:
__version__ = "1.1.0"
Then, instead of importing __version__ in your setup.py, you can read the init.py and exctract the version.
Actually, this is the solution proposed in the official Python guides:
https://packaging.python.org/guides/single-sourcing-package-version/
Another option could be using versioneer package (pandas uses it).
versioneer uses Git tags to create the version value.
Steps
Step 0: Use git tags
Versioneer uses yout Git tags.
Step 1: Install versioneer
pip install versionner
Step 2: Execute versioneer
versioneer install
Step 3: Follow the versioneer instructions
Versioneer will ask you to do some changes in your setup.py and setup.cfg.\
In the setup.py you'll have to add something like:
import versioneer
setup(
version=versioneer.get_version(),
cmdclass=versioneer.get_cmdclass()
)
And in setup.cfg:
[versioneer]
VCS = git
style = pep440
versionfile_source = your_project_name/_version.py
versionfile_build = your_project_name/_version.py
tag_prefix = v
parentdir_prefix =
Note: In the example I set "tag_prefix = v" because I like tagging like: v0.1.0, v1.0.0 and so on
After that, try:
python setup.py version
I'm trying to install a python package in windows 10 using the following setup.py file.
"""Setup file for uhd module"""
from setuptools import setup
setup(name='uhd',
version='3.14.0',
description='Universal Software Radio Peripheral (USRP) Hardware Driver Python API',
classifiers=[
'Development Status :: 4 - Beta',
'License :: OSI Approved :: GNU General Public License v3 (GPLv3)',
'Programming Language :: C++',
'Programming Language :: Python',
'Topic :: System :: Hardware :: Hardware Drivers',
],
keywords='SDR UHD USRP',
author='Ettus Research',
author_email='packages#ettus.com',
url='https://www.ettus.com/',
license='GPLv3',
package_dir={'': 'C:/Users/bcollins/UHD_PY/uhd/host/build/python'},
package_data={'uhd': ['*.so']},
zip_safe=False,
packages=['uhd'],
install_requires=['numpy'])
I execute the script using the command
python setup.py install
I do this from the directory that contains the setup.py file.
This returns the following error
error: package directory 'C:Users\bcollins\UHD_PY\uhd\host\build\python\uhd' does not exist
There is a folder called "uhd" at that location though. The folder contains the __init__.py file
If the script isn't looking for this folder, what is it looking for?
I'm not exactly experienced in this area but my best guess is that its looking for a .so file within the "uhd" folder at that location, but I'm not sure.
I am using python 2.7.
This doesn't answer the original question, but it's how I fixed the same error.
I had:
from setuptools import setup, find_packages
setup(
...
packages=find_packages('src', exclude=['test']),
...
)
I had added the src argument because my packages are located in src, but it turns out find_packages is smart enough on it's own.
Remove the first argument:
from setuptools import setup, find_packages
setup(
...
packages=find_packages(exclude=['test']),
...
)
This was on Python 3.5, but I imagine it applies to most other versions.
package_dir has to be a relative path, not an absolute path. The distutils layer under setuptools tries to reject absolute paths, but the C: confuses it. It ends up converting your path to
C:Users\bcollins\UHD_PY\uhd\host\build\python\uhd
Note the missing backslash between C: and Users. This path is relative to your current working directory on the C drive (windows drive handling is weird), and relative to your working directory, this path is invalid.
I found out that this error can occur when in the python scripts folder (%python_root%\scripts) is not in the environment PATH.
I had this problem, it turned out that you just need to add a slash after your package directory: packages=['uhd'] should be packages=['uhd/'].
I use setuptools and a requirements file to satisfy dependencies for my open source module.
Example setup.py:
from setuptools import setup, find_packages
from pip.req import parse_requirements
# parse_requirements() returns generator of
# pip.req.InstallRequirement objects
install_reqs = parse_requirements('requirements.txt',
session=False)
# reqs is a list of requirement
reqs = [str(ir.req) for ir in install_reqs]
setup(
name='python-symphony',
version='0.1.5',
description='python module for symphony chat',
author='Matt Joyce',
author_email='matt#joyce.nyc',
url='https://github.com/symphonyoss/python-symphony',
license='Apache 2.0',
classifiers=[
'Development Status :: 3 - Alpha',
'Intended Audience :: Developers',
'Topic :: Software Development :: Build Tools',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.4',
],
keywords='symphony chat api python module',
# install dependencies from requirements.txt
install_requires=reqs,
packages=find_packages(),
# bin files / python standalone executable scripts
include_package_data=True,
zip_safe=False,
)
This has worked for some time. However, recently my builds have been breaking. I believe there is a bug in setuptools, possibly related to pygments.
There are two (thus far) identified failure states:
It results in the package being installed as a binary wheel (despite no such package existing in PyPI).
It results in the package installing in site-packages seemingly correctly, but none of the subclasses can be called from the module.
I can get it to work manually by removing the package from site packages and running:
pip install --isolated --no-cache-dir python-symphony
but this results in pygments getting messed up somehow. (I noticed this when trying to debug the module in bpython).
Trying to run this stuff from a venv (wrapped with a Bash script) tends to still fail.
Is anyone aware of a new / recent issue in setuptools that could be responsible for this sort of breakage?
setuptools version: setuptools-33.1.1.dist-info and newer
Ref:
github: https://github.com/symphonyoss/python-symphony
pypi: https://pypi.python.org/pypi/python-symphony/
I am developing a PyQt5 app and making it available via pip install now that pip in python3 can install pyqt5 as dependence. I made an entry point to launch my package, and told setup.py that it's a gui_scripts.
What I would like to do now, is after the person typing pip install package, and the installation is finished, display a message to the person telling that you can now type package in the terminal to load the application. What's the correct way of doing that? Or should I not do this?
If you can ensure that
the package is always installed from a source distribution, not a binary wheel, and
the user uses the -v option for pip install,
you can output text in your setup.py script.
The setup.py is almost a regular Python script.
Just use the print() function at the end of your setup.py file.
In this example the file structure is somedir/setup.py, somedir/test/ and test/__init__.py.
Simple solution
from setuptools import setup
print("Started!")
setup(name='testing',
version='0.1',
description='The simplest setup in the world',
classifiers=[
'Development Status :: 3 - Alpha',
'License :: OSI Approved :: MIT License',
'Programming Language :: Python :: 3.0',
],
keywords='setup',
author='someone',
author_email='someone#example.com',
license='MIT',
packages=['test'],
entry_points={
},
zip_safe=False)
print("Finished!")
Started! running install running bdist_egg running egg_info writing testing.egg-info/PKG-INFO
... ... ...
Processing dependencies for testing==0.1 Finished processing
dependencies for testing==0.1 Finished!
Using setuptools.command.install solution
Also, you can subclass the setuptools.command.install command. Check the difference when you change the order of install.run(self) and os.system("cat testing.egg-info/PKG-INFO") in a clean setup.
from setuptools import setup
from setuptools.command.install import install
import os
class PostInstallCommand(install):
"""Post-installation for installation mode."""
def run(self):
install.run(self)
os.system("cat testing.egg-info/PKG-INFO")
setup(name='testing',
version='0.1',
description='The simplest setup in the world',
classifiers=[
'Development Status :: 3 - Alpha',
'License :: OSI Approved :: MIT License',
'Programming Language :: Python :: 3.0',
],
keywords='setup',
author='someone',
author_email='someone#example.com',
license='MIT',
packages=['test'],
entry_points={
},
cmdclass={
'install': PostInstallCommand,
},
zip_safe=False)
Another option would be the logging module.
rasterio for example uses an approach like this which you could adapt for your needs:
import logging
import sys
logging.basicConfig(stream=sys.stderr, level=logging.INFO)
log = logging.getLogger()
...
log.info("Your message")
sys.stderr will be printed by pip.