Develop Reference

How to avoid escaping of environmental variables in a python pip requirements file in an automated context

This is actually my first issue here on SOF.
I'm trying to install a requirements.txt file in an automated context like docker cloud.
Also, I'm using my own proxy, linking to the public pip repository while taking some extra ways.
Therefore, I want to use the -i (--index-url) option within the file:
--index-url https://${TEST_NAME}:${TEST_PASSWPRD}#{$URL}/simple
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
# pip-compile --allow-unsafe ./requirements.txt
#
python-dotenv==0.21.0
# via -r ./requirements.txt
However, the curly brackets seems to get escaped when installing and thus it fails.
I've already been digging a lot and I've found no similar topic, beside that the escaping is intended (see here).
Also according to the documentation this should work in this way, not mentioning any escaping.
I do not want to include it in my docker file install, as then a local install will not work.
I am aware of this similar issue, however it does not answer my question.
Can anyone of you point out my error?
Thanks!
Edit: The escaping looks like this:
https://%24%7BTEST_NAME%7D:

Increase Version number if Travis at github was successful

I wrote a simple script in Python.
Now I would like travis to check my code. After travis was successful, the version number should get increased.
Up to now my script has no version number yet. I can store it anywhere where it makes sense for the auto-increment workflow.
How to do this for Python code?
Update
It works now:
run tests
bumpversion
push tag to master
Unfortunately travis does not support "after-all". This means if I want to run the tests for several Python versions, I have no way to bumpversion after the tests of all python versions were successful.
In my case I will check on Python2.7 only until travis resolved this issue: https://github.com/travis-ci/travis-ci/issues/929
Here is my simple script: https://github.com/guettli/compare-with-remote
Solved :-)
It works now:
Developer pushes to github
Travis-CI runs
If all tests are successful bumpversion increases the version
The new version in setup.py get's pushed to the github repo
A new release of the python package gets uploaded to pypi with the tool twine.
I explain the way I do CI with github, travis and pypi here: https://github.com/guettli/github-travis-bumpversion-pypi

If you accept having extra commit for your versioning, you could add this script in continuous_integration/increment_version.py
import os
import pkg_resources
if __name__ == "__main__":
version = pkg_resources.get_distribution("compare_with_remote").version
split_version = version.split('.')
try:
split_version[-1] = str(int(split_version[-1]) + 1)
except ValueError:
# do something about the letters in the last field of version
pass
new_version = '.'.join(split_version)
os.system("sed -i \"s/version='[0-9.]\+'/version='{}'/\" setup.py"
.format(new_version))
os.system("git add -u")
os.system("git commit -m '[ci skip] Increase version to {}'"
.format(new_version))
os.system("git push")
And change your .travis.yml to
after_success:
- python continuous_integration/increment_version.py
I am not sure about how to make the git push part work as it need some testing with the repo rights but I assume that you could probably setup something to allow travis to push in your repo. you could look into that post for instance.
Also note that I used python to perform the git operations but they can be added as extra line in the after_success field:
after_success:
- python continuous_integration/increment_version.py
- git add -u
- git commit -m "[ci skip] version changed"
- git push
I just find it convenient to put the version number in the commit msg.
Also, it is very important to add [ci skip] in the commit message to avoid infinite increment. Maybe it would be safer to trigger the version change on a specific commit msg tag.

Not Python-specific, but this tutorial explains auto-incrementing version numbers by adding .travis.yaml entries which update git tags with each successful build. It seems like a good balance between manual and auto-versioning.
While the tutorial does use npm's package.json for the initial version-check, you could implement a simple equivalent (in Python or otherwise).

Assuming that,
only commits with successful Travis CI builds get merged into the master branch (e.g. using Pull Requests)
the package is always installed with pip from a git repo using, for instance,
pip install git+https://github.com/user/package.git
For auto-incrementing version number, one could simply define the version as the number of commits in the master branch. This can be done with the following lines in setup.py,
minor_version = check_output(['git', 'rev-list',
'--count', 'master']).decode('latin-1').strip()
setup(version='0.{}'.format(minor_version), ...)

Python: Custom package installation not importing module

I'm having a problem with this package that I installed in Python 3.5. After installing it, I try to run requestProxy.py but it won't import any of its own packages. Here's what I did, and what's happening.
I cloned it and created a private repo using these instructions.
I installed in an activated virtualenv, created without using sudo, using:
pip3 install -e HTTP_Proxy_Randomizer
Terminal said it installed ok.
I can find the egg link in my virtualenv's site-packages folder, but when I try to run the main file, it says:
from project.http.requests.parsers.freeproxyParser import freeproxyParser
ImportError: No module named project.http.requests.parsers.freeproxyParser
I had to write a setup.py for the package, which didn't seem to come with its own. I came up with:
setup(name='HTTP_Request_Randomizer',
version='1.0',
description='HTTP Proxy Request Randomizer',
package_dir={'project': 'project','http':'project/http',\
'requests':'project/http/requests','errors':'project/http/requests/errors',\
'parsers':'project/http/requests/parsers','proxy':'project/http/requests/proxy'},
packages=['project','http','requests','errors','parsers','proxy']
Here's the package structure:
pip3 freeze
gives me:
Complete output from command git config --get-regexp remote\..*\.url:
fatal: bad config file line 4 in /home/danny/.gitconfig
----------------------------------------
Error when trying to get requirement for VCS system Command "git config --get-regexp remote\..*\.url" failed with error code 128 in /home/danny/Documents/HTTP_Request_Randomizer, falling back to uneditable format
Could not determine repository location of /home/danny/Documents/HTTP_Request_Randomizer
Django==1.9.7
## !! Could not determine repository location
HTTP-Request-Randomizer==1.0
mysqlclient==1.3.7
So I want to have requestProxy.py install the other necessary packages and not fail at line 1. I'm sure this is a problem with my implementation and not the original author's coding. I was experimenting with this package a couple of weeks ago before I was aware of virtualenvs or pip install -e, and just copied it manually to site-packages. It worked then. Now I understand the concepts to do it more cleanly, but I can't get those to work.
It feels as though I have done something wrong with my git config or with my package_dir structure in setup.py, perhaps?
I've been pythoning for maybe a month and have a lot to learn. I normally find what I need on Stack Overflow without having to bother anyone, but after trying everything with this, I really need some help. Any advice much appreciated.

I figured it out. I was using Ninja IDE, and even though I entered the virtualenv for the project and restarted, it still wasn't recognizing it. I was able to run it from the terminal, and also in Pycharm and Liclipse.

How to add to salt state? (beginner doing Python project using virtualenv)

First, some context: I'm following a web scraping tutorial and am running into (what I believe are) setup-related issues. I keep running into errors with the last step of Part 0 (bottom of page). After creating my virtualenv (using mkvirtualenv), I receive the following error when running the following command (from inside my virtualenv): pip install -r requirements.txt:
[Errono 2] No such file or directory: 'requirements.txt'
Storing debug log for failure in /home/dchaudh/.pip/pip.log
I did some quick googling and came across this solution which seems promising: it suggests that the problem is due to Salt (bc of something called chowning) and that adding no_chown: True to my salt state should fix the issue.
How do I add to the salt state, i.e., what file do I modify? Any help would be appreciated
As you can tell from the tutorial, I'm a relative newbie (even moreso to virtual environments) and I feel like I quickly got in over my head in the process theof trying to find a solution.

Are you actually using Salt? I feel that you found that solution online and attributed your error to Salt. Nothing in the tutorial mentions Salt, so it seems unlikely that you would be using it.
Make sure you are actually in the new-coder/scrape directory before running pip install -r requirements.txt

how to pip uninstall with virtualenv on heroku cedar stack?

I tried to uninstall a module on heroku with:
heroku run bin/python bin/pip uninstall whatever
Pip shows the module in the /app tree then claims to have uinstalled the module, but running the same command again shows it installed in the same location in the /app tree.
Is there a way to get pip uinstall to succeed?
Heroku run instantiates a new dyno and runs the command specified in that dyno only. Dynos are ephemeral which is why the results of the pip uninstall don't stick.

Updated 2013-09-30: the current way to clear the virtualenv seems to specify a different python runtime version in runtime.txt as stated on Github and in the Heroku's devcenter reference.
Be aware that Heroku currently "only endorses and supports the use of Python 2.7.4 and 3.3.2" so unless your application supports both Python 2.7.4 and 3.3.2, you may want to test it with the runtime you'll want to switch to (currently available at http://envy-versions.s3.amazonaws.com/$PYTHON_VERSION.tar.bz2, though it shouldn't be an issue to switch e.g. between 2.7.4 and 2.7.3 in most cases).
Thanks #Jesse for your up-to-date answer and to the commenters who made me aware of the issue.
Was up-to-date in ~november 2012 (I haven't since updated the linked buildpack, my pull request was closed and the CLEAN_VIRTUALENV feature was dropped at some point by the official buildpack):
As David explained, you cannot pip uninstall one package but you can purge and reinstall the whole virtualenv. Use the user-env-compile lab feature with the CLEAN_VIRTUALENV option to purge the virtualenv:
heroku labs:enable user-env-compile
heroku config:add CLEAN_VIRTUALENV=true
Currently this won't work because there is a bug. You'll need to use my fork of the buildpack until this get fixed upstream (pull request was closed) :
heroku config:add BUILDPACK_URL=https://github.com/blaze33/heroku-buildpack-python.git
Now push your new code and you'll notice that the whole virtualenv gets reinstalled.
Andrey's answer no longer works since March 23 2012. The new style virtualenv commit moved the virtual env from /app to /app/.heroku/venv but the purge branch wasn't updated to catch up so that you end up with a virtualenv not being in PYTHONHOME.
To avoid reinstalling everything after each push, disable the option:
heroku labs:disable user-env-compile
heroku config:remove CLEAN_VIRTUALENV BUILDPACK_URL

There is now a simpler way to clear the pip cache. Just change the runtime environment, for example from 'python-2.7.3' to 'python-2.7.2', or vice versa.
To do this add a file called runtime.txt to the root of your repository that contains just the runtime string (as show above) in it.
For this to work you need to have turned on the Heroku labs user-env-compile feature. See https://devcenter.heroku.com/articles/labs-user-env-compile

By default virtualenv is cached between deploys.
To avoid caching of packages you can run:
heroku config:add BUILDPACK_URL=git#github.com:heroku/heroku-buildpack-python.git#purge
That way everything will be built from scratch after you push some changes. To enable the caching just remove the BUILDPACK_URL config variable.
Now to uninstall specific package(s):
Remove the corresponding record(s) from the requirements.txt;
Commit and push the changes.
Thanks to Lincoln from Heroku Support Team for the clarifications.

I've created some fabfile recipes for Maxime and Jesse answers that allow re-installing the requirements with one fab command: https://gist.github.com/littlepea/5096814 (look at the docstrings for explanation and examples).
For Maxime's answer I've created a task 'heroku_clean' (or 'hc'), it'll look something like that:
fab heroku_clean
Or using an alias and specifying a heroku app:
fab hc:app=myapp
For Jesse's answer I've created a task 'heroku_runtime' (or 'hr'), it sets heroku python runtime and commits runtime.txt (also creates it if it didn't exist):
fab heroku_runtime:2.7.2
If runtime version is not passed it will just toggle it between 2.7.2 and 2.7.3, so the easiest way to change and commit the runtime is:
fab hr
Then you can just deploy (push to heroku origin) you app and the virtualenv will be rebuilt. I've also added a 'heroku_deploy' task ('hr') that I use for heroku push and scale that also can be user together with 'heroku_runtime' task. This is my preferred method of deploying and rebuilding virtualenv - everything happens in one command and I can choose when to rebuild it, I don't like to do it every time like Maxime's answer suggest because it can take a long time:
fab hd:runtime=yes
This is an equivalent of:
fab heroku_runtime
fab heroku_deploy