Develop Reference

Which PyPI index does Pip point to?

Ubuntu 20.04.1 LTS | Python 3.8.5 | Pip 20.0.2 (via python3-pip)
I accidentally overwrote the global PyPI index in my laptop with a custom index. (It was meant to be set only within a virtual environment. In case it's useful, I was using AWS CodeArtifact.)
I say this after running pip3 config -v list and seeing global.index-url assigned the other URL as well as ~/.config/.pip/pip.conf listing the same.
In an attempt to set things right, I did the following:
Set pip3's global.index-url to "https://pypi.python.org/pypi", but when I did a test to see if it worked, it stopped halfway while trying to install requests in a virtual environment (using its pip) and said some dependencies were not satisfied.
I then moved ~/.config/.pip to a backup, uninstalled python3-pip and reinstalled the package in the hope that the config would be written anew. While a new ~/config/.pip was not created, I was able to install requests into the virtual environment without any issue.
My confusion stems from the fact that I don't know where pip is picking things up from. None of the files listed below exist!
$ pip config -v list
For variant 'global', will try loading '/etc/xdg/xdg-ubuntu/pip/pip.conf'
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.config/pip/pip.conf'
For variant 'site', will try loading '/home/<user>/test_env/venv/pip.conf'
The output is slightly different when I use pip3 with the virtual environment deactivated.
$ pip3 config -v list
For variant 'global', will try loading '/etc/xdg/xdg-ubuntu/pip/pip.conf'
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/pip.conf'
Could someone please tell me how this all works?

https://pypi.org/simple/ packageX
let me know if its right ?
using
pip install paxckageX --verbose
not sure its right nowadays but here:
Python can't find the file pip.conf
and here
pip user guide Configuration :
Configuration
Config file
pip allows you to set all command line option defaults in a standard ini style config file.
The names and locations of the configuration files vary slightly across platforms. You may have per-user, per-virtualenv or global (shared amongst all users) configuration:
Per-user:
On Unix the default configuration file is: $HOME/.config/pip/pip.conf which respects the XDG_CONFIG_HOME environment variable.
On macOS the configuration file is $HOME/Library/Application Support/pip/pip.conf if directory $HOME/Library/Application Support/pip exists else $HOME/.config/pip/pip.conf.
On Windows the configuration file is %APPDATA%\pip\pip.ini.
There is also a legacy per-user configuration file which is also respected. To find its location:
On Unix and macOS the configuration file is: $HOME/.pip/pip.conf
On Windows the configuration file is: %HOME%\pip\pip.ini
You can set a custom path location for this config file using the environment variable PIP_CONFIG_FILE.
Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf
On Windows the file is: %VIRTUAL_ENV%\pip.ini
Global:
On Unix the file may be located in /etc/pip.conf. Alternatively it may be in a “pip” subdirectory of any of the paths set in the environment variable XDG_CONFIG_DIRS (if it exists), for example /etc/xdg/pip/pip.conf.
On macOS the file is: /Library/Application Support/pip/pip.conf
On Windows XP the file is: C:\Documents and Settings\All Users\Application Data\pip\pip.ini
On Windows 7 and later the file is hidden, but writeable at C:\ProgramData\pip\pip.ini
Global configuration is not supported on Windows Vista.
The global configuration file is shared by all Python installations.
If multiple configuration files are found by pip then they are combined in the following order:
The global file is read
The per-user file is read
The virtualenv-specific file is read
Each file read overrides any values read from previous files, so if the global timeout is specified in both the global file and the per-user file then the latter value will be used.
The names of the settings are derived from the long command line option, e.g. if you want to use a different package index (--index-url) and set the HTTP timeout (--default-timeout) to 60 seconds your config file would look like this:
[global]
timeout = 60
index-url = https://download.zope.org/ppix
Each subcommand can be configured optionally in its own section so that every global setting with the same name will be overridden; e.g. decreasing the timeout to 10 seconds when running the freeze (pip freeze) command and using 60 seconds for all other commands is possible with:
[global]
timeout = 60
[freeze]
timeout = 10
Boolean options like --ignore-installed or --no-dependencies can be set like this:
[install]
ignore-installed = true
no-dependencies = yes
To enable the boolean options --no-compile, --no-warn-script-location and --no-cache-dir, falsy values have to be used:
[global]
no-cache-dir = false
[install]
no-compile = no
no-warn-script-location = false
For options which can be repeated like --verbose and --quiet, a non-negative integer can be used to represent the level to be specified:
[global]
quiet = 0
verbose = 2
It is possible to append values to a section within a configuration file such as the pip.ini file. This is applicable to appending options like --find-links or --trusted-host, which can be written on multiple lines:
[global]
find-links =
http://download.example.com
[install]
find-links =
http://mirror1.example.com
http://mirror2.example.com
trusted-host =
mirror1.example.com
mirror2.example.com
This enables users to add additional values in the order of entry for such command line arguments.
Environment Variables
pip’s command line options can be set with environment variables using the format PIP_<UPPER_LONG_NAME> . Dashes (-) have to be replaced with underscores (_).
For example, to set the default timeout:
Unix/macOS
export PIP_DEFAULT_TIMEOUT=60
Windows
This is the same as passing the option to pip directly:
Unix/macOS
python -m pip --default-timeout=60 [...]
Windows
For command line options which can be repeated, use a space to separate multiple values. For example:
Unix/macOS
export PIP_FIND_LINKS="http://mirror1.example.com http://mirror2.example.com"
Windows
is the same as calling:
Unix/macOS
python -m pip install --find-links=http://mirror1.example.com --find-links=http://mirror2.example.com
Windows
Options that do not take a value, but can be repeated (such as --verbose) can be specified using the number of repetitions, so:
export PIP_VERBOSE=3
is the same as calling:
pip install -vvv
Note
Environment variables set to be empty string will not be treated as false. Please use no, false or 0 instead.
Config Precedence
Command line options have precedence over environment variables, which have precedence over the config file.
Within the config file, command specific sections have precedence over the global section.
Examples:
--host=foo overrides PIP_HOST=foo
PIP_HOST=foo overrides a config file with [global] host = foo
A command specific section in the config file [<command>] host = bar overrides the option with same name in the [global] config file section

Pipenv: Multiple Environments

Right now I'm using virtualenv and just switching over to Pipenv. Today in virtualenv I load in different environment variables and settings depending on whether I'm in development, production, or testingby setting DJANGO_SETTINGS_MODULE to myproject.settings.development, myproject.settings.production, and myproject.settings.testing.
I'm aware that I can set an .env file, but how can I have multiple versions of that .env file?

I'm far from a Python guru, but one solution I can think of would be to create Pipenv scripts that run shell scripts to change the PIPENV_DOTENV_LOCATION and run your startup commands.
Example Pipfile scripts:
[scripts]
development = "./scripts/development.sh"
development.sh Example:
#!/bin/sh
PIPENV_DOTENV_LOCATION=/path/to/.development_env pipenv run python test.py
Then run pipenv run development

You should create different .env files with different prefixes depending on the environment, such as production.env or testing.env. With pipenv, you can use the PIPENV_DONT_LOAD_ENV=1 environment variable to prevent pipenv shell from automatically exporting the .env file and combine this with export $(cat .env | xargs).
export $(cat production.env | xargs) && PIPENV_DONT_LOAD_ENV=1 pipenv shell would configure your environment variables for production and then start a shell in the virtual environment.

Where is the luigi config file?

I have installed luigi by pip command and I would like to change the port for the web UI. I tried to find the config file but I couldn't. Do I need to create one?

You can start luigid with the --port option.
luigid --port 80
Configuration file locations are:
/etc/luigi/luigi.cfg
luigi.cfg (or its legacy name client.cfg) in
your current working directory
LUIGI_CONFIG_PATH environment variable
in increasing order of preference. You do need to create one. e.g.,
[core]
default-scheduler-host=www.example.com
default-scheduler-port=8088

In spite of the documentation saying otherwise, the port configuration from the config file is not used, at least in some versions or some circumstances
Until this is resolved, you should always use the --port option of luigid:
luigid --port 12345
Also see https://github.com/spotify/luigi/issues/2235
For other configuration options a config file should be used. See https://luigi.readthedocs.io/en/stable/configuration.html
For a configuration global to the host you can create a file:
/etc/luigi/luigi.cfg
Make sure it is readable by the user that runs luigid and luig.
Alternatively a local configuration file that will be recognized is
luigi.cfg
which you would have to create in the current working directory.
If you want a custom config file location you could set the environment variable LUIGI_CONFIG_PATH to the full path of your config file.

Environment variable

I have a tox.ini file like this:
[tox]
envlist =
{py27,py34}
[testenv]
setenv =
PYTHONDONTWRITEBYTECODE=1
deps =
-r{toxinidir}/requirements-test.txt
commands =
nosetests -v --stop --with-coverage --cover-package=project
So, I need to pass to my tox env an environment variable from my SO. This variable is private and changes according to the environment.
How I do it?

I found the solution! Tox 2.0 has a settings called passenv
A list of wildcard environment variable names which shall be copied from the tox invocation environment to the test environment. If a specified environment variable doesn’t exist in the tox invocation environment it is ignored. You can use * and ? to match multiple environment variables with one name.
Note that the PATH and PIP_INDEX_URL variables are unconditionally passed down and on Windows SYSTEMROOT, PATHEXT, TEMP and TMP will be passed down as well whereas on unix TMPDIR will be passed down. You can override these variables with the setenv option.

How to set an environment variable in Amazon Elastic Beanstalk (Python)

I have been working on a Django application lately, trying to get it to work with Amazon Elastic Beanstalk.
In my .ebextensions/python.config file, I have set the following:
option_settings:
- namespace: aws:elasticbeanstalk:application:environment
option_name: ProductionBucket
value: s3-bucket-name
- namespace: aws:elasticbeanstalk:application:environment
option_name: ProductionCache
value: memcached-server.site.com:11211
However, whenever I look on the server, no such environment variables are set (and as such, aren't accessible when I try os.getenv('ProductionBucket')
I came across this this page which appears to attempt to document all the namespaces. I've also tried using PARAM1 as the option name, but have had similar results.
How can I set environment variables in Amazon Elastic Beanstalk?
EDIT:
I have also tried adding a command prior to all other commands which would just export an environment variable:
commands:
01_env_vars:
command: "source scripts/env_vars"
... This was also unsuccessful

I was having the same problem.
Believe it or not, you have to commit the .ebextensions directory and all *.config files to version control before you deploy in order for them to show up as environment variables on the server.
In order to keep sensitive information out of version control, you can use a config file like this:
option_settings:
- option_name: API_LOGIN
value: placeholder
- option_name: TRANS_KEY
value: placeholder
- option_name: PROVIDER_ID
value: placeholder
Then edit the configuration in the AWS admin panel (Configuration > Software Configuration > Environment Properties) and update the values there.
You may also find this answer helpful.

Option 1:
You can set environment variables using eb setenv FOO=bar
You can view the environment variables using eb printenv
Option 2:
You can create a config file in your .ebextensions directory, for example 00_environment.config. Then, add your environment variables like this:
option_settings:
- option_name: MY_FIRST_ENV_VAR
value: abc
- option_name: ANOTHER_ENV_VAR
value: 123
However, if you have multiple environments, I have found that it is more useful to set the environment variables directly, using option #1.
I also have found the eb config commands to be helpful: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-config.html
These commands allow you to get, put, list, or delete configuration files on your eb environment.
The command eb config get will save your config, including environment variables, to a local file in .elasticbeanstalk/saved_configs.

I did the following to also get my environment variables that I configure in cloudformation in the non-container phase, eg the regular commands
/opt/elasticbeanstalk/bin/get-config environment | python -c "import json,sys; obj=json.load(sys.stdin); f = open('/tmp/eb_env', 'w'); f.write('\n'.join(map(lambda x: 'export ' + x[0] + '=' + x[1], obj.iteritems())))"
Once you execute this command you will have a file in /tmp/eb_env with all your environment variables. Just execute the following before a command that needs the environment variables
source /tmp/eb_env
Example
source /tmp/eb_env && echo $MY_CUSTOM_ENV
In the config file of elastic beanstalk, it looks like this:
commands:
02-make-sure-we-can-get-our-env-in-the-instance-itself:
command: "/opt/elasticbeanstalk/bin/get-config environment | python -c 'import json,sys; obj=json.load(sys.stdin); f = open(\'/tmp/eb_env\', \'w\'); f.write(\'\n\'.join(map(lambda x: \'export \' + x[0] + \'=\' + x[1], obj.iteritems())))'"

I've checked using a modern (i.e., non legacy) container, and found it under /opt/elasticbeanstalk/deploy/configuration/containerconfiguration as a json file.
The Behaviour seems to be Platform-Dependent: I remember in PHP in particular, it also creates some shell scripts with the values.
Regardless of that, look into /opt/elasticbeanstalk/hooks/configdeploy.
Java case again, it runs this python script, which looks quite handy for you:
https://gist.github.com/19c1e4b718f9a70a4ce1

To set variables on a local run, you can do the following:
eb local setenv CONFIG=dev
eb local run
This also works with Docker MultiContainers, which otherwise will not see your environment.

I know this is an old question but for those who still have the same question like I did here is the solution from AWS documentation: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environments-cfg-softwaresettings.html
To configure environment properties in the Elastic Beanstalk console
Open the Elastic Beanstalk console, and then, in the regions drop-down
list, select your region.
In the navigation pane, choose Environments, and then choose your
environment's name on the list.
In the navigation pane, choose Configuration.
In the Software configuration category, choose Edit.
Under Environment properties, enter key-value pairs.
Choose Apply.