I am trying to find a way to use my Github repo tags for versioning of my package, which should be available for download using something like pip.
The problem is every time I update the package version I have to upload the contents to pypi.
Is there any way to just set the donwload url in pypi point to my github repo, So that when I do something like
pip install -I MySQL_python==1.2.2 and it just install that form the git tag 1.2.2, without me having to upload the version to pypi.
EDIT:(I was not clear enough)
I know about the pip install git+git://blabal way
I am looking for something like I tell pypi that my package is at github.com/bla.git
and the user does pip install bla==1.2 and pip install that from github (with version as tag)
Something like vundle for vim
You could install like this:
pip install -e git+<repo address>#<ref>#egg=<egg name>
where ref could be a commit id, tag name or branch name.
Read the docs.
Related
I've installed a library using the command
pip install git+git://github.com/mozilla/elasticutils.git
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like
-f git+git://github.com/mozilla/elasticutils.git
elasticutils==0.7.dev
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported.
Does anybody have a solution for my problem?
Normally your requirements.txt file would look something like this:
package-one==1.9.4
package-two==3.7.1
package-three==1.0.1
...
To specify a Github repo, you do not need the package-name== convention.
The examples below update package-two using a GitHub repo. The text between # and # denotes the specifics of the package.
Specify commit hash (41b95ec in the context of updated requirements.txt):
package-one==1.9.4
git+https://github.com/path/to/package-two#41b95ec#egg=package-two
package-three==1.0.1
Specify branch name (master):
git+https://github.com/path/to/package-two#master#egg=package-two
Specify tag (0.1):
git+https://github.com/path/to/package-two#0.1#egg=package-two
Specify release (3.7.1):
git+https://github.com/path/to/package-two#releases/tag/v3.7.1#egg=package-two
Note that #egg=package-two is not a comment here, it is to explicitly state the package name
This blog post has some more discussion on the topic.
“Editable” packages syntax can be used in requirements.txt to import packages from a variety of VCS (git, hg, bzr, svn):
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
Also, it is possible to point to particular commit:
-e git://github.com/mozilla/elasticutils.git#000b14389171a9f0d7d713466b32bc649b0bed8e#egg=elasticutils
requirements.txt allows the following ways of specifying a dependency on a package in a git repository as of pip 7.0:1
[-e] git+git://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+https://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+ssh://git.myproject.org/SomeProject#egg=SomeProject
-e git+git#git.myproject.org:SomeProject#egg=SomeProject (deprecated as of Jan 2020)
For Github that means you can do (notice the omitted -e):
git+git://github.com/mozilla/elasticutils.git#egg=elasticutils
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here's my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Documentation
1 https://pip.readthedocs.org/en/stable/reference/pip_install/#git
2 https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support
First, install with git+git or git+https, in any way you know. Example of installing kronok's branch of the brabeion project:
pip install -e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion
Second, use pip freeze > requirements.txt to get the right thing in your requirements.txt. In this case, you will get
-e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion-master
Third, test the result:
pip uninstall brabeion
pip install -r requirements.txt
Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you'd probably want to publish your module in it's own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.
None of these answers worked for me. The only thing that worked was:
git+https://github.com/path_to_my_project.git
No "e", no double "git" and no previous installs necessary.
Github has zip endpoints that in my opinion are preferable to using the git protocol. The advantages are:
You don't have to specify #egg=<project name>
Git doesn't need to be installed in your environment, which is nice for containerized environments
It works much better with pip hashing and caching
The URL structure is easier to remember and more discoverable
You usually want requirements.txt entries to look like this, e.g. without the -e prefix:
https://github.com/org/package/archive/1a58aa586efd4bca37f2cfb9d9348958986aab6c.tar.gz
To install from main branch:
https://github.com/org/package/archive/main.tar.gz
There is also an equivalent .zip endpoint, but it was reported in a comment that always using the .tar.gz endpoint avoids problems with unicode package names.
It seems like this is also a valid format:
gym-tictactoe # git+https://github.com/haje01/gym-tictactoe.git#84e22fc28fe192ba0040bdd56a697f63d3d4a3d5
If you do a pip install "git+https://github.com/haje01/gym-tictactoe.git", then look at what got installed by running pip freeze, you will see the package described in this format and can copy and paste into requirements.txt.
I'm finding that it's kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04's package manager) to actually install the thing I tell it to install. I'm posting this answer to save anyone's time who runs into this problem.
Putting this into a requirements.txt file failed:
git+git://github.com/myname/myrepo.git#my-branch#egg=eggname
By "failed" I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
git+git://github.com/myname/myrepo.git#d27d07c9e862feb939e56d0df19d5733ea7b4f4d#egg=eggname
For private repositories, I found that these two work fine for me:
pip install https://${GITHUB_TOKEN}#github.com/owner/repo/archive/main.tar.gz
Where main.tar.gz refers to the main branch of your repo and can be replaced with other branch names. For more information and using the more recent Github API see here:
pip install https://${GITHUB_TOKEN}#api.github.com/repos/owner/repo/tarball/master
If you have git installed and available, then
pip install git+https://${GITHUB_TOKEN}#github.com/owner/repo.git#main
achieves the same, and it also allows for some more flexibility by appending #branch or #tag or #commit-hash. That approach, however, actually clones the repo into a local temp folder which can take a noticeable amount of time.
You can use the URLs in your requirements.txt, too.
I'm trying to use get_worksheet_by_id function from the gspread package.
I can see the function is available in https://github.com/burnash/gspread/blob/master/gspread/models.py
It's also listed in documentation.
But I it's missing in pip and conda repositories. As a result I'm not able to use it.
https://pypi.org/project/gspread/#files
https://anaconda.org/conda-forge/gspread/files
Not sure where to report it.
As you can see if you look at the blame, the function was only added by this commit, which is from march 2021. The latest version available from pypi and conda-forge is however from february. That is why you don't have if when you install through these channels.
Some suggestions:
You could simply edit the code of the library in your site-packages
Install from the github sources, either by cloning the repo and doing python setup.py install or through python -m pip install git+https://github.com/burnash/gspread
Create an issue on the github repo and ask that the version on conda-forge/pypi is updated to include this feature.
I've installed a library using the command
pip install git+git://github.com/mozilla/elasticutils.git
which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like
-f git+git://github.com/mozilla/elasticutils.git
elasticutils==0.7.dev
in the requirements.txt file, a pip install -r requirements.txt results in the following output:
Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20))
Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: )
No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20))
The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported.
Does anybody have a solution for my problem?
Normally your requirements.txt file would look something like this:
package-one==1.9.4
package-two==3.7.1
package-three==1.0.1
...
To specify a Github repo, you do not need the package-name== convention.
The examples below update package-two using a GitHub repo. The text between # and # denotes the specifics of the package.
Specify commit hash (41b95ec in the context of updated requirements.txt):
package-one==1.9.4
git+https://github.com/path/to/package-two#41b95ec#egg=package-two
package-three==1.0.1
Specify branch name (master):
git+https://github.com/path/to/package-two#master#egg=package-two
Specify tag (0.1):
git+https://github.com/path/to/package-two#0.1#egg=package-two
Specify release (3.7.1):
git+https://github.com/path/to/package-two#releases/tag/v3.7.1#egg=package-two
Note that #egg=package-two is not a comment here, it is to explicitly state the package name
This blog post has some more discussion on the topic.
“Editable” packages syntax can be used in requirements.txt to import packages from a variety of VCS (git, hg, bzr, svn):
-e git://github.com/mozilla/elasticutils.git#egg=elasticutils
Also, it is possible to point to particular commit:
-e git://github.com/mozilla/elasticutils.git#000b14389171a9f0d7d713466b32bc649b0bed8e#egg=elasticutils
requirements.txt allows the following ways of specifying a dependency on a package in a git repository as of pip 7.0:1
[-e] git+git://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+https://git.myproject.org/SomeProject#egg=SomeProject
[-e] git+ssh://git.myproject.org/SomeProject#egg=SomeProject
-e git+git#git.myproject.org:SomeProject#egg=SomeProject (deprecated as of Jan 2020)
For Github that means you can do (notice the omitted -e):
git+git://github.com/mozilla/elasticutils.git#egg=elasticutils
Why the extra answer?
I got somewhat confused by the -e flag in the other answers so here's my clarification:
The -e or --editable flag means that the package is installed in <venv path>/src/SomeProject and thus not in the deeply buried <venv path>/lib/pythonX.X/site-packages/SomeProject it would otherwise be placed in.2
Documentation
1 https://pip.readthedocs.org/en/stable/reference/pip_install/#git
2 https://pip.readthedocs.org/en/stable/reference/pip_install/#vcs-support
First, install with git+git or git+https, in any way you know. Example of installing kronok's branch of the brabeion project:
pip install -e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion
Second, use pip freeze > requirements.txt to get the right thing in your requirements.txt. In this case, you will get
-e git+https://github.com/kronok/brabeion.git#12efe6aa06b85ae5ff725d3033e38f624e0a616f#egg=brabeion-master
Third, test the result:
pip uninstall brabeion
pip install -r requirements.txt
Since pip v1.5, (released Jan 1 2014: CHANGELOG, PR) you may also specify a subdirectory of a git repo to contain your module. The syntax looks like this:
pip install -e git+https://git.repo/some_repo.git#egg=my_subdir_pkg&subdirectory=my_subdir_pkg # install a python package from a repo subdirectory
Note: As a pip module author, ideally you'd probably want to publish your module in it's own top-level repo if you can. Yet this feature is helpful for some pre-existing repos that contain python modules in subdirectories. You might be forced to install them this way if they are not published to pypi too.
None of these answers worked for me. The only thing that worked was:
git+https://github.com/path_to_my_project.git
No "e", no double "git" and no previous installs necessary.
Github has zip endpoints that in my opinion are preferable to using the git protocol. The advantages are:
You don't have to specify #egg=<project name>
Git doesn't need to be installed in your environment, which is nice for containerized environments
It works much better with pip hashing and caching
The URL structure is easier to remember and more discoverable
You usually want requirements.txt entries to look like this, e.g. without the -e prefix:
https://github.com/org/package/archive/1a58aa586efd4bca37f2cfb9d9348958986aab6c.tar.gz
To install from main branch:
https://github.com/org/package/archive/main.tar.gz
There is also an equivalent .zip endpoint, but it was reported in a comment that always using the .tar.gz endpoint avoids problems with unicode package names.
It seems like this is also a valid format:
gym-tictactoe # git+https://github.com/haje01/gym-tictactoe.git#84e22fc28fe192ba0040bdd56a697f63d3d4a3d5
If you do a pip install "git+https://github.com/haje01/gym-tictactoe.git", then look at what got installed by running pip freeze, you will see the package described in this format and can copy and paste into requirements.txt.
I'm finding that it's kind of tricky to get pip3 (v9.0.1, as installed by Ubuntu 18.04's package manager) to actually install the thing I tell it to install. I'm posting this answer to save anyone's time who runs into this problem.
Putting this into a requirements.txt file failed:
git+git://github.com/myname/myrepo.git#my-branch#egg=eggname
By "failed" I mean that while it downloaded the code from Git, it ended up installing the original version of the code, as found on PyPi, instead of the code in the repo on that branch.
However, installing the commmit instead of the branch name works:
git+git://github.com/myname/myrepo.git#d27d07c9e862feb939e56d0df19d5733ea7b4f4d#egg=eggname
For private repositories, I found that these two work fine for me:
pip install https://${GITHUB_TOKEN}#github.com/owner/repo/archive/main.tar.gz
Where main.tar.gz refers to the main branch of your repo and can be replaced with other branch names. For more information and using the more recent Github API see here:
pip install https://${GITHUB_TOKEN}#api.github.com/repos/owner/repo/tarball/master
If you have git installed and available, then
pip install git+https://${GITHUB_TOKEN}#github.com/owner/repo.git#main
achieves the same, and it also allows for some more flexibility by appending #branch or #tag or #commit-hash. That approach, however, actually clones the repo into a local temp folder which can take a noticeable amount of time.
You can use the URLs in your requirements.txt, too.
We have our own internal tool kit that we are packaging into a pip package.
Is there a way we can encrypt the packate itself to protect its contents? (Not the sources so much as some of the config info that is included in a couple of the sources.)
Ideally, we would run something like:
pip install http://www.oursite.com/testpackage/testingkit.tar.gz -decrypt=(crypto key)
I've just started working with setuptools and virtualenv. My package requires the latest python-gearman that is only available from GitHub. The python-gearman version that's on PyPI is an old one. The Github source is setuptools-compatible, i.e. has setup.py, etc. Is there a way to make setuptools download and install the new version instead of looking for it on PyPI and installing the old one?
FYI, the new python-gearman is http://github.com/mtai/python-gearman
The key is to tell easy_install where the package can be downloaded. In this particular case, it can be found at the url http://github.com/mtai/python-gearman/tarball/master. However, that link by itself won't work, because easy_install can't tell just by looking at the URL what it's going to get.
By changing it to http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta instead, easy_install will be able to identify the package name and its version.
The final step is to add the URL to your package's dependency_links, e.g.:
setup(
...
dependency_links = ['http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta']
)
Now, when YOUR package is being installed, easy_install will discover that there is a "gearman 2.0.0beta" available for download from that URL, and happily pick it over the one on PyPI, if you specify "gearman>=2.0.0beta" in your dependencies..
(Normally, the way this sort of thing is done is to include a link on one's PyPI page to the downloadable source; in this case, if the author of the gearman package had included a link like the above, you'd be already set. Typically, people mark the development version with 'myproject-dev' and then people use a requirement of 'myproject>=somever,==dev', so that if there isn't a package of somever or higher, easy_install will try to check out or download the release.)
You'll need to specify --process-dependency-links when using pip. Note that dependency links processing has been deprecated and will be removed in a future release.
You can use the pip install protocol+location[#tag][#egg=Dependency] format to install directly from source using pip.
Git
pip install git+https://github.com/username/repo.git
pip install git+https://github.com/username/repo.git#MyTag
pip install git+https://github.com/username/repo.git#MyTag#egg=ProjectName
Mercurial
pip install hg+https://hg.myproject.org/MyProject/
SVN
pip install svn+svn://svn.myproject.org/svn/MyProject
Bzr
pip install bzr+http://bzr.myproject.org/MyProject/trunk
The following protocols are supported: [+git, +svn, +hg, +bzr]
Versions
#tag lets you specify a specific version/tag to check out.
#egg=name lets you specify what the project is as a dependency for others.
The order must always be #tag#egg=name.
Private Repositories
You can also install from private repositories by changing the protocol to SSH (ssh://) and adding an appropriate user (git#):
git+ssh://git#github.com/username/my_private_repo
You can also install from private repositories with a username / password.
git+https://<username>:<password>#github.com/<user>/<repo>.git
Github provides the ability to create personal OAuth tokens which can be cycled
git+https://<oauth token>:x-oauth-basic#github.com/<user>/<repo>.git
requirements.txt
requirements.txt is used to specify project dependencies:
requirements.txt
package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git
These are not installed automatically with the package and must be installed with the command pip -r requirements.txt.
Including requirements files
Requirements files can include other requirements files:
requirements-docs.txt
sphinx
-r requirements-dev.txt
requirements-dev.txt
some-dev-tool
-r requirements.txt
requirements.txt
package1
package2==1.0.2
package3>=0.0.4
git+https://github.com/username/repo.git
setup.py
Requirements files can install dependencies specified in setup.py with the following command:
-e .
setup.py can also install from repositories using the same syntax as above, but using the dependency_links value as mentioned in this answer.
References:
https://pip.pypa.io/en/latest/user_guide.html#installing-packages
https://pip.pypa.io/en/latest/reference/pip_install.html
As I just had to do the same thing, I found another way to do this as pip's --process-dependency-links are scheduled to be removed in pip 19.0 according to this comment.
pip 18.1 includes the following feature
Allow PEP 508 URL requirements to be used as dependencies.
From the description of PEP 508, the syntax for such URL dependencies looks like:
A minimal URL based lookup:
pip # https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
So in your setup.py it would look like
setup(
...
install_requires = [
...
'python-gearman # https://github.com/mtai/python-gearman/archive/master.zip'
...
]
)
Notice, the link is an archive file and could also be a specific release or branch of a repository as described in this answer. Also, see that answer for working with other repository hosts.
To the best of my knowledge, the easiest way to update the dependency is by using pip install -I . when installing your package from its directory.
Vanilla setuptools does not support downloading directly from a git repository but you can use one of the Download Source links from that page, like:
easy_install http://github.com/mtai/python-gearman/tarball/master