Develop Reference

Why does the following error pop up when twine uploading files to pypi

When I execute the following lines and put in my information
twine upload dist/*
The following error pops up
HTTPError: 400 Client Error: The description failed to render in the default format of reStructuredText. See https://pypi.org/help/#description-content-type for more information. for url: https://upload.pypi.org/legacy/
After going to the url, I am not closer to solving the problem.
My setup.py is the following (with blanked out information)
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="quizmaker",
version="0.0.1",
author="my secret name",
author_email="email",
description="secret descripting",
long_description=long_description,
long_description_content_type="text/markdown",
url="the url",
packages=setuptools.find_packages(),
python_requires='>=3.6',
)
If there is any solution to this please let me know. Thank you.

Two possibilities:
You forgot to rebuild the distribution or are uploading an old distribution without long_description_content_type. Make sure you're starting with an empty dist directory, rebuild your distribution and then upload.
You're using an old version of some packaging dependency before long_description_content_type was supported. You need setuptools>=38.6.0, wheel>=0.31.0 and twine>=1.11.0. Upgrade them all with python -m pip install -U setuptools wheel twine and then do #1.

Pypi upload problems: Must be a valid Python identifier

I'm having trouble uploading my package to pypi. I used to be able to just use python setup.py sdist upload -r pypi but this now causes an error:
Upload failed (400): requires: Must be a valid Python identifier.
error: Upload failed (400): requires: Must be a valid Python identifier.
I've tried a few things to get this working but everything has failed with the same error.
I removed the current dist, build and egg folders in my root directory. Then I increased my package version number by 1 micro version. I ensured my ~/.pypirc file is as it should be according to instructions:
[distutils]
index-servers =
pypi
[pypi]
username: c.welsh2
password: ...
and updated pip, twine and setuptools. I create a build using
python setuptools.py bdist_wheel
which created the build in /package_root/dist/* and I try uploading to pypi using
twine upload dist/*
And again I get:
HTTPError: 400 Client Error: requires: Must be a valid Python identifier. for url: https://upload.pypi.org/legacy/
Does anybody know what is causing this problem?
For completeness, here is my setup file:
from distutils.core import setup
import setuptools
#version
MAJOR = 4
MINOR = 0
MICRO = 5
#=======
__version__ = '%d.%d.%d' % (MAJOR, MINOR, MICRO)
setup(
name = 'PyCoTools',
packages = ['PyCoTools'], # this must be the same as the name above
version = __version__,
description = 'A python toolbox for COPASI',
author = 'Ciaran Welsh',
requires=['lxml','argparse','pandas','numpy','scipy','matplotlib.pyplot','scipy','seaborn','sklearn'],
package_data={'PyCoTools':['*.py','Documentation/*.pdf',
'logging_config.conf',
'Documentation/*.html','Licence.txt',
'ReadMe.md',
'Examples/KholodenkoExample/*',
'Examples/BioModelsWorkflowVersion1/*',
'Scripts/*.py',
'Tests/*.py',
'Tests/*.cps',
'PyCoToolsTutorial/*.pickle',
'PyCoToolsTutorial/*.py',
'PyCoToolsTutorial/*.ipynb',
'PyCoToolsTutorial/*.html',
'PyCoToolsTutorial/*.cps']},
author_email = '--<hidden>',
##
url = 'https://pypi.python.org/pypi/PyCoTools',
keywords = ['systems biology','modelling','biological',
'networks','copasi','identifiability analysis','profile likelihood'],
license='GPL4',
install_requires=['pandas','numpy','scipy','matplotlib',
'lxml'],
long_description='''Tools for using Copasi via Python and calculating profile likelihoods. See Github page and documentation for more details''')

Turns out there was quite an unforgiving typo. I couldn't give matplotlib.pyplot to the requires argument since its called matplotlib!

Python setuptools: How can I list a private repository under install_requires?

I am creating a setup.py file for a project which depends on private GitHub repositories. The relevant parts of the file look like this:
from setuptools import setup
setup(name='my_project',
...,
install_requires=[
'public_package',
'other_public_package',
'private_repo_1',
'private_repo_2',
],
dependency_links=[
'https://github.com/my_account/private_repo_1/master/tarball/',
'https://github.com/my_account/private_repo_2/master/tarball/',
],
...,
)
I am using setuptools instead of distutils because the latter does not support the install_requires and dependency_links arguments per this answer.
The above setup file fails to access the private repos with a 404 error - which is to be expected since GitHub returns a 404 to unauthorized requests for a private repository. However, I can't figure out how to make setuptools authenticate.
Here are some things I've tried:
Use git+ssh:// instead of https:// in dependency_links as I would if installing the repo with pip. This fails because setuptools doesn't recognize this protocol ("unknown url type: git+ssh"), though the distribute documentation says it should. Ditto git+https and git+http.
https://<username>:<password>#github.com/... - still get a 404. (This method doesn't work with curl or wget from the command line either - though curl -u <username> <repo_url> -O <output_file_name> does work.)
Upgrading setuptools (0.9.7) and virtualenv (1.10) to the latest versions. Also tried installing distribute though this overview says it was merged back into setuptools. Either way, no dice.
Currently I just have setup.py print out a warning that the private repos must be downloaded separately. This is obviously less than ideal. I feel like there's something obvious that I'm missing, but can't think what it might be. :)
Duplicate-ish question with no answers here.

I was trying to get this to work for installing with pip, but the above was not working for me. From [1] I understood the PEP508 standard should be used, from [2] I retrieved an example which actually does work (at least for my case).
Please note; this is with pip 20.0.2 on Python 3.7.4
setup(
name='<package>',
...
install_requires=[
'<normal_dependency>',
# Private repository
'<dependency_name> # git+ssh://git#github.com/<user>/<repo_name>#<branch>',
# Public repository
'<dependency_name> # git+https://github.com/<user>/<repo_name>#<branch>',
],
)
After specifying my package this way installation works fine (also with -e settings and without the need to specify --process-dependency-links).
References
[1] https://github.com/pypa/pip/issues/4187
[2] https://github.com/pypa/pip/issues/5566

Here's what worked for me:
install_requires=[
'private_package_name==1.1',
],
dependency_links=[
'git+ssh://git#github.com/username/private_repo.git#egg=private_package_name-1.1',
]
Note that you have to have the version number in the egg name, otherwise it will say it can't find the package.

I couldn't find any good documentation on this, but came across the solution mainly through trial & error. Further, installing from pip & setuptools have some subtle differences; but this way should work for both.
GitHub don't (currently, as of August 2016) offer an easy way to get the zip / tarball of private repos. So you need to point setuptools to tell setuptools that you're pointing to a git repo:
from setuptools import setup
import os
# get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/
github_token = os.environ['GITHUB_TOKEN']
setup(
# ...
install_requires='package',
dependency_links = [
'git+https://{github_token}#github.com/user/{package}.git/#{version}#egg={package}-0'
.format(github_token=github_token, package=package, version=master)
]
A couple of notes here:
For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
You need to include some version number (here is 0) at the end of the link, even if there's no package on PyPI. This has to be a actual number, not a word.
You need to preface with git+ to tell setuptools it's to clone the repo, rather than pointing at a zip / tarball
version can be a branch, a tag, or a commit hash
You need to supply --process-dependency-links if installing from pip

I found a (hacky) workaround:
#!/usr/bin/env python
from setuptools import setup
import os
os.system('pip install git+https://github-private.corp.com/user/repo.git#master')
setup( name='original-name'
, ...
, install_requires=['repo'] )
I understand that there are ethical issues with having a system call in a setup script, but I can't think of another way to do this.

Via Tom Hemmes' answer I found this is the only thing that worked for me:
install_requires=[
'<package> # https://github.com/<username>/<package>/archive/<branch_name>.zip']

Using archive URL from github works for me, for public repositories. E.g.
dependency_links = [
'https://github.com/username/reponame/archive/master.zip#egg=eggname-version',
]

With pip 20.1.1, this works for me
install_requires=[ "packson3#https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/138/packson3/dist/packson3-1.0.0.tar.gz"],
in setup.py

Edit: This appears to only work with public github repositories, see comments.
dependency_links=[
'https://github.com/my_account/private_repo_1/tarball/master#egg=private_repo_1',
'https://github.com/my_account/private_repo_2/tarball/master#egg=private_repo_2',
],
Above syntax seems to work for me with setuptools 1.0. At the moment at least the syntax of adding "#egg=project_name-version" to VCS dependencies is documented in the link you gave to distribute documentation.

This work for our scenario:
package is on github in a private repo
we want to install it into site-packages (not into ./src with -e)
being able to use pip install -r requirements.txt
being able to use pip install -e reposdir (or from github), where the dependencies are only specified in requirements.txt
https://github.com/pypa/pip/issues/3610#issuecomment-356687173

Why does PIP convert underscores to dashes

I am trying to install a Pyramid app -- let's say test_app. inside a virtual environment and it is getting installed as test-app (pip freeze output shows it test-app==0.0).
Because of this, I can not import the package.
How should I fix this problem?
More info:
http://mail.python.org/pipermail/distutils-sig/2011-August/017935.html
I am using pip version 1.3.1
setup.py:
import os
from setuptools import setup, find_packages
here = os.path.abspath(os.path.dirname(__file__))
README = open(os.path.join(here, 'README.txt')).read()
CHANGES = open(os.path.join(here, 'CHANGES.txt')).read()
requires = [
'pyramid',
'pyramid_debugtoolbar',
'waitress',
]
setup(name='test_app',
version='0.0',
description='test_app',
long_description=README + '\n\n' + CHANGES,
classifiers=[
"Programming Language :: Python",
"Framework :: Pyramid",
"Topic :: Internet :: WWW/HTTP",
"Topic :: Internet :: WWW/HTTP :: WSGI :: Application",
],
author='',
author_email='',
url='',
keywords='web pyramid pylons',
packages=find_packages(),
include_package_data=True,
zip_safe=False,
install_requires=requires,
tests_require=requires,
test_suite="test_app",
entry_points="""\
[paste.app_factory]
main = test_app:main
""",
)
UPDATE:
to summarize the findings so far:
It is normal that pip reports the package name as test-app.
It is not normal that the egg link is pointing to your virtual env root.
But the fact that the .egg-info file is created inside your virtual env root as well points to develop using that directory as the egg root.
Update 2021
I have now started using Poetry instead of pip for all my new Python projects. It works well for both normal projects and Jupyter notebooks. With its better developer experience for package management all I'd have to do for the above example would be
poetry run xyz
where xyz is a script that I define within the spec file (akin to package.json for npm). I would be able to import my own package as all other packages.

Update 2021
Use Poetry instead of pip.
Original answer:
So, finally after a lot of fiddling around, I've found the solution -- which is annoyingly simple.
I am using virtualenv and am installing the package in the development mode.
I was installing the package from the wrong location. Turns out that the location (directory) from which you run python setup.py develop is indeed the one that goes into the .egg-link file.
You should install the package into the virtual environment FROM the location where your code is.
So, for example, let's say your code resides in '/a/b' and your virtualenv env is in '/x/y/env', then you should install the package like this:
$ cd /a/b
$ /x/y/env/bin/python setup.py develop
This will install the package properly.
Hence, the '-' and '_' issue is not a problem and you should be careful about the location from where you are installing the package in the develop mode.

setup.py examples? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed last month.
The community reviewed whether to reopen this question last month and left it closed:
Original close reason(s) were not resolved
Improve this question
After studying this page:
http://docs.python.org/distutils/builtdist.html
I am hoping to find some setup.py files to study so as to make my own (with the goal of making a fedora rpm file).
Could the s.o. community point me towards some good examples?

Complete walkthrough of writing setup.py scripts here. (with some examples)
If you'd like a real-world example, I could point you towards the setup.py scripts of a couple major projects. Django's is here, pyglet's is here. You can just browse the source of other projects for a file named setup.py for more examples.
These aren't simple examples; the tutorial link I gave has those. These are more complex, but also more practical.

You may find the HitchHiker's Guide to Packaging helpful, even though it is incomplete. I'd start with the Quick Start tutorial. Try also just browsing through Python packages on the Python Package Index. Just download the tarball, unpack it, and have a look at the setup.py file. Or even better, only bother looking through packages that list a public source code repository such as one hosted on GitHub or BitBucket. You're bound to run into one on the front page.
My final suggestion is to just go for it and try making one; don't be afraid to fail. I really didn't understand it until I started making them myself. It's trivial to create a new package on PyPI and just as easy to remove it. So, create a dummy package and play around.

Minimal example
from setuptools import setup, find_packages
setup(
name="foo",
version="1.0",
packages=find_packages(),
)
More info in docs

READ THIS FIRST https://packaging.python.org/en/latest/current.html
Installation Tool Recommendations
Use pip to install Python packages
from PyPI.
Use virtualenv, or pyvenv to isolate application specific dependencies from a shared Python installation.
Use pip wheel to create a cache of wheel distributions, for the purpose of > speeding up subsequent installations.
If you’re looking for management of fully integrated cross-platform software stacks, consider buildout (primarily focused on the web development community) or Hashdist, or conda (both primarily focused on the scientific community).
Packaging Tool Recommendations
Use setuptools to define projects and create Source Distributions.
Use the bdist_wheel setuptools extension available from the wheel project to create wheels. This is especially beneficial, if your project contains binary extensions.
Use twine for uploading distributions to PyPI.
This anwser has aged, and indeed there is a rescue plan for python packaging world called
wheels way
I qoute pythonwheels.com here:
What are wheels?
Wheels are the new standard of python distribution
and are intended to replace eggs. Support is offered in pip >= 1.4 and
setuptools >= 0.8.
Advantages of wheels
Faster installation for pure python and native C extension packages.
Avoids arbitrary code execution for installation. (Avoids setup.py)
Installation of a C extension does not require a compiler on Windows
or OS X.
Allows better caching for testing and continuous
integration.
Creates .pyc files as part of installation to ensure
they match the python interpreter used.
More consistent installs across platforms and machines.
The full story of correct python packaging (and about wheels) is covered at packaging.python.org
conda way
For scientific computing (this is also recommended on packaging.python.org, see above) I would consider using CONDA packaging which can be seen as a 3rd party service build on top of PyPI and pip tools. It also works great on setting up your own version of binstar so I would imagine it can do the trick for sophisticated custom enterprise package management.
Conda can be installed into a user folder (no super user permisssions) and works like magic with
conda install
and powerful virtual env expansion.
eggs way
This option was related to python-distribute.org and is largerly outdated (as well as the site) so let me point you to one of the ready to use yet compact setup.py examples I like:
A very practical example/implementation of mixing scripts and single python files into setup.py is giving here
Even better one from hyperopt
This quote was taken from the guide on the state of setup.py and still applies:
setup.py gone!
distutils gone!
distribute gone!
pip and virtualenv here to stay!
eggs ... gone!
I add one more point (from me)
wheels!
I would recommend to get some understanding of packaging-ecosystem (from the guide pointed by gotgenes) before attempting mindless copy-pasting.
Most of examples out there in the Internet start with
from distutils.core import setup
but this for example does not support building an egg python setup.py bdist_egg (as well as some other old features), which were available in
from setuptools import setup
And the reason is that they are deprecated.
Now according to the guide
Warning
Please use the Distribute package rather than the Setuptools package
because there are problems in this package that can and will not be
fixed.
deprecated setuptools are to be replaced by distutils2, which "will be part of the standard library in Python 3.3". I must say I liked setuptools and eggs and have not yet been completely convinced by convenience of distutils2. It requires
pip install Distutils2
and to install
python -m distutils2.run install
PS
Packaging never was trivial (one learns this by trying to develop a new one), so I assume a lot of things have gone for reason. I just hope this time it will be is done correctly.

I recommend the setup.py of the Python Packaging User Guide's example project.
The Python Packaging User Guide "aims to be the authoritative resource on how to package, publish and install Python distributions using current tools".
As of December 2022 however, the sample project has switched from setup.py to pyproject.toml. For a new project you may want to consider doing the same.

Here is the utility I wrote to generate a simple setup.py file (template) with useful comments and links. I hope, it will be useful.
Installation
sudo pip install setup-py-cli
Usage
To generate setup.py file just type in the terminal.
setup-py
Now setup.py file should occur in the current directory.
Generated setup.py
from distutils.core import setup
from setuptools import find_packages
import os
# User-friendly description from README.md
current_directory = os.path.dirname(os.path.abspath(__file__))
try:
with open(os.path.join(current_directory, 'README.md'), encoding='utf-8') as f:
long_description = f.read()
except Exception:
long_description = ''
setup(
# Name of the package
name=<name of current directory>,
# Packages to include into the distribution
packages=find_packages('.'),
# Start with a small number and increase it with every change you make
# https://semver.org
version='1.0.0',
# Chose a license from here: https://help.github.com/articles/licensing-a-repository
# For example: MIT
license='',
# Short description of your library
description='',
# Long description of your library
long_description = long_description,
long_description_context_type = 'text/markdown',
# Your name
author='',
# Your email
author_email='',
# Either the link to your github or to your website
url='',
# Link from which the project can be downloaded
download_url='',
# List of keyword arguments
keywords=[],
# List of packages to install with this one
install_requires=[],
# https://pypi.org/classifiers/
classifiers=[]
)
Content of the generated setup.py:
automatically fulfilled package name based on the name of the current directory.
some basic fields to fulfill.
clarifying comments and links to useful resources.
automatically inserted description from README.md or an empty string if there is no README.md.
Here is the link to the repository. Fill free to enhance the solution.

Look at this complete example https://github.com/marcindulak/python-mycli of a small python package. It is based on packaging recommendations from https://packaging.python.org/en/latest/distributing.html, uses setup.py with distutils and in addition shows how to create RPM and deb packages.
The project's setup.py is included below (see the repo for the full source):
#!/usr/bin/env python
import os
import sys
from distutils.core import setup
name = "mycli"
rootdir = os.path.abspath(os.path.dirname(__file__))
# Restructured text project description read from file
long_description = open(os.path.join(rootdir, 'README.md')).read()
# Python 2.4 or later needed
if sys.version_info < (2, 4, 0, 'final', 0):
raise SystemExit, 'Python 2.4 or later is required!'
# Build a list of all project modules
packages = []
for dirname, dirnames, filenames in os.walk(name):
if '__init__.py' in filenames:
packages.append(dirname.replace('/', '.'))
package_dir = {name: name}
# Data files used e.g. in tests
package_data = {name: [os.path.join(name, 'tests', 'prt.txt')]}
# The current version number - MSI accepts only version X.X.X
exec(open(os.path.join(name, 'version.py')).read())
# Scripts
scripts = []
for dirname, dirnames, filenames in os.walk('scripts'):
for filename in filenames:
if not filename.endswith('.bat'):
scripts.append(os.path.join(dirname, filename))
# Provide bat executables in the tarball (always for Win)
if 'sdist' in sys.argv or os.name in ['ce', 'nt']:
for s in scripts[:]:
scripts.append(s + '.bat')
# Data_files (e.g. doc) needs (directory, files-in-this-directory) tuples
data_files = []
for dirname, dirnames, filenames in os.walk('doc'):
fileslist = []
for filename in filenames:
fullname = os.path.join(dirname, filename)
fileslist.append(fullname)
data_files.append(('share/' + name + '/' + dirname, fileslist))
setup(name='python-' + name,
version=version, # PEP440
description='mycli - shows some argparse features',
long_description=long_description,
url='https://github.com/marcindulak/python-mycli',
author='Marcin Dulak',
author_email='X.Y#Z.com',
license='ASL',
# https://pypi.python.org/pypi?%3Aaction=list_classifiers
classifiers=[
'Development Status :: 1 - Planning',
'Environment :: Console',
'License :: OSI Approved :: Apache Software License',
'Natural Language :: English',
'Operating System :: OS Independent',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.4',
'Programming Language :: Python :: 2.5',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.2',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
],
keywords='argparse distutils cli unittest RPM spec deb',
packages=packages,
package_dir=package_dir,
package_data=package_data,
scripts=scripts,
data_files=data_files,
)
and and RPM spec file which more or less follows Fedora/EPEL packaging guidelines may look like:
# Failsafe backport of Python2-macros for RHEL <= 6
%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
%{!?python_sitearch: %global python_sitearch %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}
%{!?python_version: %global python_version %(%{__python} -c "import sys; sys.stdout.write(sys.version[:3])")}
%{!?__python2: %global __python2 %{__python}}
%{!?python2_sitelib: %global python2_sitelib %{python_sitelib}}
%{!?python2_sitearch: %global python2_sitearch %{python_sitearch}}
%{!?python2_version: %global python2_version %{python_version}}
%{!?python2_minor_version: %define python2_minor_version %(%{__python} -c "import sys ; print sys.version[2:3]")}
%global upstream_name mycli
Name: python-%{upstream_name}
Version: 0.0.1
Release: 1%{?dist}
Summary: A Python program that demonstrates usage of argparse
%{?el5:Group: Applications/Scientific}
License: ASL 2.0
URL: https://github.com/marcindulak/%{name}
Source0: https://github.com/marcindulak/%{name}/%{name}-%{version}.tar.gz
%{?el5:BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)}
BuildArch: noarch
%if 0%{?suse_version}
BuildRequires: python-devel
%else
BuildRequires: python2-devel
%endif
%description
A Python program that demonstrates usage of argparse.
%prep
%setup -qn %{name}-%{version}
%build
%{__python2} setup.py build
%install
%{?el5:rm -rf $RPM_BUILD_ROOT}
%{__python2} setup.py install --skip-build --prefix=%{_prefix} \
--optimize=1 --root $RPM_BUILD_ROOT
%check
export PYTHONPATH=`pwd`/build/lib
export PATH=`pwd`/build/scripts-%{python2_version}:${PATH}
%if 0%{python2_minor_version} >= 7
%{__python2} -m unittest discover -s %{upstream_name}/tests -p '*.py'
%endif
%clean
%{?el5:rm -rf $RPM_BUILD_ROOT}
%files
%doc LICENSE README.md
%{_bindir}/*
%{python2_sitelib}/%{upstream_name}
%{?!el5:%{python2_sitelib}/*.egg-info}
%changelog
* Wed Jan 14 2015 Marcin Dulak <X.Y#Z.com> - 0.0.1-1
- initial version

Here you will find the simplest possible example of using distutils and setup.py:
https://docs.python.org/2/distutils/introduction.html#distutils-simple-example
This assumes that all your code is in a single file and tells how to package a project containing a single module.