Develop Reference

Enable --no-index by default in pip configuration

Is there a way to set --no-index as default behaviour when installing packages with pip (both for Windows and Unix)?
Even though this configuration seems to work...
# /home/myaccount/.config/pip/pip.conf or C:\Users\MyAccount\AppData\Roaming\pip\pip.ini
[global]
find-urls =
/path/to/folder1
/path/to/folder2
no-index = true
I'm not able to find a reference of it in pip configuration doc.
Am I doing it right?
NOTE: I'm working offline and I don't want to add the extra-burden of a local server repository. I'm just installing offline packages by setting find-urls configuration

Any command-line option can be set using the configuration file, following some simple rules to map the switch to a name in the configuration file.
This is explained in the Naming section of the Configuration chapter:
The names of the settings are derived from the long command line option.
As an example, if you want to use a different package index (--index-url) and set the HTTP timeout (--default-timeout) to 60 seconds, your config file would look like this:
[global]
timeout = 60
index-url = https://download.zope.org/ppix
Since --no-index is a boolean flag switch, the Boolean options section in the same chapter explains how to give such switches a value in the configuration file:
Boolean options like --ignore-installed or --no-dependencies can be set like this:
[install]
ignore-installed = true
no-dependencies = yes
Using no-index = true is exactly how you'd use that setting in a config file. Putting it in the [global] section means it'll apply to all your pip commands.
Because command-line options map directly to names in the config file using these simple rules, the configuration chapter doesn't have to cover every single possible config option.

Which PyPI index does Pip point to?

Ubuntu 20.04.1 LTS | Python 3.8.5 | Pip 20.0.2 (via python3-pip)
I accidentally overwrote the global PyPI index in my laptop with a custom index. (It was meant to be set only within a virtual environment. In case it's useful, I was using AWS CodeArtifact.)
I say this after running pip3 config -v list and seeing global.index-url assigned the other URL as well as ~/.config/.pip/pip.conf listing the same.
In an attempt to set things right, I did the following:
Set pip3's global.index-url to "https://pypi.python.org/pypi", but when I did a test to see if it worked, it stopped halfway while trying to install requests in a virtual environment (using its pip) and said some dependencies were not satisfied.
I then moved ~/.config/.pip to a backup, uninstalled python3-pip and reinstalled the package in the hope that the config would be written anew. While a new ~/config/.pip was not created, I was able to install requests into the virtual environment without any issue.
My confusion stems from the fact that I don't know where pip is picking things up from. None of the files listed below exist!
$ pip config -v list
For variant 'global', will try loading '/etc/xdg/xdg-ubuntu/pip/pip.conf'
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.config/pip/pip.conf'
For variant 'site', will try loading '/home/<user>/test_env/venv/pip.conf'
The output is slightly different when I use pip3 with the virtual environment deactivated.
$ pip3 config -v list
For variant 'global', will try loading '/etc/xdg/xdg-ubuntu/pip/pip.conf'
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/pip.conf'
Could someone please tell me how this all works?

https://pypi.org/simple/ packageX
let me know if its right ?
using
pip install paxckageX --verbose
not sure its right nowadays but here:
Python can't find the file pip.conf
and here
pip user guide Configuration :
Configuration
Config file
pip allows you to set all command line option defaults in a standard ini style config file.
The names and locations of the configuration files vary slightly across platforms. You may have per-user, per-virtualenv or global (shared amongst all users) configuration:
Per-user:
On Unix the default configuration file is: $HOME/.config/pip/pip.conf which respects the XDG_CONFIG_HOME environment variable.
On macOS the configuration file is $HOME/Library/Application Support/pip/pip.conf if directory $HOME/Library/Application Support/pip exists else $HOME/.config/pip/pip.conf.
On Windows the configuration file is %APPDATA%\pip\pip.ini.
There is also a legacy per-user configuration file which is also respected. To find its location:
On Unix and macOS the configuration file is: $HOME/.pip/pip.conf
On Windows the configuration file is: %HOME%\pip\pip.ini
You can set a custom path location for this config file using the environment variable PIP_CONFIG_FILE.
Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf
On Windows the file is: %VIRTUAL_ENV%\pip.ini
Global:
On Unix the file may be located in /etc/pip.conf. Alternatively it may be in a “pip” subdirectory of any of the paths set in the environment variable XDG_CONFIG_DIRS (if it exists), for example /etc/xdg/pip/pip.conf.
On macOS the file is: /Library/Application Support/pip/pip.conf
On Windows XP the file is: C:\Documents and Settings\All Users\Application Data\pip\pip.ini
On Windows 7 and later the file is hidden, but writeable at C:\ProgramData\pip\pip.ini
Global configuration is not supported on Windows Vista.
The global configuration file is shared by all Python installations.
If multiple configuration files are found by pip then they are combined in the following order:
The global file is read
The per-user file is read
The virtualenv-specific file is read
Each file read overrides any values read from previous files, so if the global timeout is specified in both the global file and the per-user file then the latter value will be used.
The names of the settings are derived from the long command line option, e.g. if you want to use a different package index (--index-url) and set the HTTP timeout (--default-timeout) to 60 seconds your config file would look like this:
[global]
timeout = 60
index-url = https://download.zope.org/ppix
Each subcommand can be configured optionally in its own section so that every global setting with the same name will be overridden; e.g. decreasing the timeout to 10 seconds when running the freeze (pip freeze) command and using 60 seconds for all other commands is possible with:
[global]
timeout = 60
[freeze]
timeout = 10
Boolean options like --ignore-installed or --no-dependencies can be set like this:
[install]
ignore-installed = true
no-dependencies = yes
To enable the boolean options --no-compile, --no-warn-script-location and --no-cache-dir, falsy values have to be used:
[global]
no-cache-dir = false
[install]
no-compile = no
no-warn-script-location = false
For options which can be repeated like --verbose and --quiet, a non-negative integer can be used to represent the level to be specified:
[global]
quiet = 0
verbose = 2
It is possible to append values to a section within a configuration file such as the pip.ini file. This is applicable to appending options like --find-links or --trusted-host, which can be written on multiple lines:
[global]
find-links =
http://download.example.com
[install]
find-links =
http://mirror1.example.com
http://mirror2.example.com
trusted-host =
mirror1.example.com
mirror2.example.com
This enables users to add additional values in the order of entry for such command line arguments.
Environment Variables
pip’s command line options can be set with environment variables using the format PIP_<UPPER_LONG_NAME> . Dashes (-) have to be replaced with underscores (_).
For example, to set the default timeout:
Unix/macOS
export PIP_DEFAULT_TIMEOUT=60
Windows
This is the same as passing the option to pip directly:
Unix/macOS
python -m pip --default-timeout=60 [...]
Windows
For command line options which can be repeated, use a space to separate multiple values. For example:
Unix/macOS
export PIP_FIND_LINKS="http://mirror1.example.com http://mirror2.example.com"
Windows
is the same as calling:
Unix/macOS
python -m pip install --find-links=http://mirror1.example.com --find-links=http://mirror2.example.com
Windows
Options that do not take a value, but can be repeated (such as --verbose) can be specified using the number of repetitions, so:
export PIP_VERBOSE=3
is the same as calling:
pip install -vvv
Note
Environment variables set to be empty string will not be treated as false. Please use no, false or 0 instead.
Config Precedence
Command line options have precedence over environment variables, which have precedence over the config file.
Within the config file, command specific sections have precedence over the global section.
Examples:
--host=foo overrides PIP_HOST=foo
PIP_HOST=foo overrides a config file with [global] host = foo
A command specific section in the config file [<command>] host = bar overrides the option with same name in the [global] config file section

Run supervisor in virtual environment

I installed supervisor and gunicorn in my virtual environment (venv).
I am using this tutorial: https://realpython.com/blog/python/kickstarting-flask-on-ubuntu-setup-and-deployment/
I'm confused as to where I should be creating the config file for supervisor as the default etc/supervisor won't apply to me.
The supervisorctl file is in the directory:
/home/giri/venv/py2.7/lib/python2.7/site-packages/supervisor
I noticed this line in the supervisorctl file:
Options:
-c/--configuration -- configuration file path (default /etc/supervisord.conf)
Do I need to manually set this flag each time I run the supervisorctl script or is there another way?
Thanks

As found in the docs (http://supervisord.org/configuration.html):
The Supervisor configuration file is conventionally named
supervisord.conf. It is used by both supervisord and supervisorctl. If
either application is started without the -c option (the option which
is used to tell the application the configuration filename
explicitly), the application will look for a file named
supervisord.conf within the following locations, in the specified
order. It will use the first file it finds.
$CWD/supervisord.conf
$CWD/etc/supervisord.conf
/etc/supervisord.conf
So put the supervisor.conf in your current working directory and you're fine.

How to set an environment variable in Amazon Elastic Beanstalk (Python)

I have been working on a Django application lately, trying to get it to work with Amazon Elastic Beanstalk.
In my .ebextensions/python.config file, I have set the following:
option_settings:
- namespace: aws:elasticbeanstalk:application:environment
option_name: ProductionBucket
value: s3-bucket-name
- namespace: aws:elasticbeanstalk:application:environment
option_name: ProductionCache
value: memcached-server.site.com:11211
However, whenever I look on the server, no such environment variables are set (and as such, aren't accessible when I try os.getenv('ProductionBucket')
I came across this this page which appears to attempt to document all the namespaces. I've also tried using PARAM1 as the option name, but have had similar results.
How can I set environment variables in Amazon Elastic Beanstalk?
EDIT:
I have also tried adding a command prior to all other commands which would just export an environment variable:
commands:
01_env_vars:
command: "source scripts/env_vars"
... This was also unsuccessful

I was having the same problem.
Believe it or not, you have to commit the .ebextensions directory and all *.config files to version control before you deploy in order for them to show up as environment variables on the server.
In order to keep sensitive information out of version control, you can use a config file like this:
option_settings:
- option_name: API_LOGIN
value: placeholder
- option_name: TRANS_KEY
value: placeholder
- option_name: PROVIDER_ID
value: placeholder
Then edit the configuration in the AWS admin panel (Configuration > Software Configuration > Environment Properties) and update the values there.
You may also find this answer helpful.

Option 1:
You can set environment variables using eb setenv FOO=bar
You can view the environment variables using eb printenv
Option 2:
You can create a config file in your .ebextensions directory, for example 00_environment.config. Then, add your environment variables like this:
option_settings:
- option_name: MY_FIRST_ENV_VAR
value: abc
- option_name: ANOTHER_ENV_VAR
value: 123
However, if you have multiple environments, I have found that it is more useful to set the environment variables directly, using option #1.
I also have found the eb config commands to be helpful: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-config.html
These commands allow you to get, put, list, or delete configuration files on your eb environment.
The command eb config get will save your config, including environment variables, to a local file in .elasticbeanstalk/saved_configs.

I did the following to also get my environment variables that I configure in cloudformation in the non-container phase, eg the regular commands
/opt/elasticbeanstalk/bin/get-config environment | python -c "import json,sys; obj=json.load(sys.stdin); f = open('/tmp/eb_env', 'w'); f.write('\n'.join(map(lambda x: 'export ' + x[0] + '=' + x[1], obj.iteritems())))"
Once you execute this command you will have a file in /tmp/eb_env with all your environment variables. Just execute the following before a command that needs the environment variables
source /tmp/eb_env
Example
source /tmp/eb_env && echo $MY_CUSTOM_ENV
In the config file of elastic beanstalk, it looks like this:
commands:
02-make-sure-we-can-get-our-env-in-the-instance-itself:
command: "/opt/elasticbeanstalk/bin/get-config environment | python -c 'import json,sys; obj=json.load(sys.stdin); f = open(\'/tmp/eb_env\', \'w\'); f.write(\'\n\'.join(map(lambda x: \'export \' + x[0] + \'=\' + x[1], obj.iteritems())))'"

I've checked using a modern (i.e., non legacy) container, and found it under /opt/elasticbeanstalk/deploy/configuration/containerconfiguration as a json file.
The Behaviour seems to be Platform-Dependent: I remember in PHP in particular, it also creates some shell scripts with the values.
Regardless of that, look into /opt/elasticbeanstalk/hooks/configdeploy.
Java case again, it runs this python script, which looks quite handy for you:
https://gist.github.com/19c1e4b718f9a70a4ce1

To set variables on a local run, you can do the following:
eb local setenv CONFIG=dev
eb local run
This also works with Docker MultiContainers, which otherwise will not see your environment.

I know this is an old question but for those who still have the same question like I did here is the solution from AWS documentation: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environments-cfg-softwaresettings.html
To configure environment properties in the Elastic Beanstalk console
Open the Elastic Beanstalk console, and then, in the regions drop-down
list, select your region.
In the navigation pane, choose Environments, and then choose your
environment's name on the list.
In the navigation pane, choose Configuration.
In the Software configuration category, choose Edit.
Under Environment properties, enter key-value pairs.
Choose Apply.

What Unix tool to quickly add/remove some text to a Python script?

I'm developing an application using Flask.
I want a quick, automated way to add and remove debug=True to the main function call:
Development:
app.run(debug=True)
Production:
app.run()
For security reasons, as I might expose private/sensitive information about the app if I leave debug mode on "in the wild".
I was thinking of using sed or awk to automate this in a git hook (production version is kept in a bare remote repo that I push to), or including it in a shell script I am going to write to fire up uwsgi and some other "maintenance"-ey tasks that allow the app to be served up properly.
What do you think?

That is not the way to go! My recommendation is to create some configuration Python module (let us say, config.py) with some content such as:
DEBUG = True
Now, in our current code, write this:
import config
app.run(debug=config.DEBUG)
Now, when you run in production, just change DEBUG from True to False. Or you can leave this file unversioned, so the copy of development is different of the copy of production. This is not uncommon since, for example, one does not use the same database connection params both in development and production.
Even if you want to update it automatically, just call sed on the config file with the -i flag. It is way more secure to update just this one file:
$ sed -i.bkp 's/^ *DEBUG *=.*$/DEBUG = False/' config.py

You should set up some environment variable on server. Your script can detect presense of this variable and disable debugging.

You probably should not be using app.run in production (and you definitely don't need it if you are using uwsgi). Instead, use one of the several deployment options discussed in the deployment section of Flask's excellent documentation. (app.run simply calls werkzeug.serving.run_simple which executes Python's included wsgiref server.)
That being said, the correct way to do this is not with a post-deploy edit to your source code but with a server-specific config file that changes your settings as #brandizzi pointed out in his answer.
You can do this in several different ways (Flask has documentation on this too - see Armin's suggestions on configuring from files and handling the development-production switch):
Include both your development and your server's configs in your repository. Use an environmental variable to switch between them:
# your_app.config.develop
DEBUG = True
# your_app.config.production
DEBUG = False
# your_app.app
from flask import Flask
from os import environ
mode = environ.get("YOURAPP_MODE")
mode = "production" if mode is None else "develop"
config = __import__("your_app.config." + mode)
app = Flask("your_app")
app.config.from_object(config)
Store your production configuration in a separate repository along with any other server-specific configurations you may need. Load the config if an environmental variable is set.

I'd use sed:
sed 's/debug=True//'
portable, scriptable, ubiquitous.

You can also use a NOCOMMIT hook (from gitty):
Set this as a pre-commit hook
if git diff --cached | grep NOCOMMIT > /dev/null; then
echo "You tried to commit a line containing NOCOMMIT"
exit 1
fi
exit 0
This will prevent the commit if it contains NOCOMMIT.
You can of course directly replace NOCOMMIT by Debug=True in the hook.