Develop Reference

Which PyPI index does Pip point to?

Ubuntu 20.04.1 LTS | Python 3.8.5 | Pip 20.0.2 (via python3-pip)
I accidentally overwrote the global PyPI index in my laptop with a custom index. (It was meant to be set only within a virtual environment. In case it's useful, I was using AWS CodeArtifact.)
I say this after running pip3 config -v list and seeing global.index-url assigned the other URL as well as ~/.config/.pip/pip.conf listing the same.
In an attempt to set things right, I did the following:
Set pip3's global.index-url to "https://pypi.python.org/pypi", but when I did a test to see if it worked, it stopped halfway while trying to install requests in a virtual environment (using its pip) and said some dependencies were not satisfied.
I then moved ~/.config/.pip to a backup, uninstalled python3-pip and reinstalled the package in the hope that the config would be written anew. While a new ~/config/.pip was not created, I was able to install requests into the virtual environment without any issue.
My confusion stems from the fact that I don't know where pip is picking things up from. None of the files listed below exist!
$ pip config -v list
For variant 'global', will try loading '/etc/xdg/xdg-ubuntu/pip/pip.conf'
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.config/pip/pip.conf'
For variant 'site', will try loading '/home/<user>/test_env/venv/pip.conf'
The output is slightly different when I use pip3 with the virtual environment deactivated.
$ pip3 config -v list
For variant 'global', will try loading '/etc/xdg/xdg-ubuntu/pip/pip.conf'
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/pip.conf'
Could someone please tell me how this all works?

https://pypi.org/simple/ packageX
let me know if its right ?
using
pip install paxckageX --verbose
not sure its right nowadays but here:
Python can't find the file pip.conf
and here
pip user guide Configuration :
Configuration
Config file
pip allows you to set all command line option defaults in a standard ini style config file.
The names and locations of the configuration files vary slightly across platforms. You may have per-user, per-virtualenv or global (shared amongst all users) configuration:
Per-user:
On Unix the default configuration file is: $HOME/.config/pip/pip.conf which respects the XDG_CONFIG_HOME environment variable.
On macOS the configuration file is $HOME/Library/Application Support/pip/pip.conf if directory $HOME/Library/Application Support/pip exists else $HOME/.config/pip/pip.conf.
On Windows the configuration file is %APPDATA%\pip\pip.ini.
There is also a legacy per-user configuration file which is also respected. To find its location:
On Unix and macOS the configuration file is: $HOME/.pip/pip.conf
On Windows the configuration file is: %HOME%\pip\pip.ini
You can set a custom path location for this config file using the environment variable PIP_CONFIG_FILE.
Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf
On Windows the file is: %VIRTUAL_ENV%\pip.ini
Global:
On Unix the file may be located in /etc/pip.conf. Alternatively it may be in a “pip” subdirectory of any of the paths set in the environment variable XDG_CONFIG_DIRS (if it exists), for example /etc/xdg/pip/pip.conf.
On macOS the file is: /Library/Application Support/pip/pip.conf
On Windows XP the file is: C:\Documents and Settings\All Users\Application Data\pip\pip.ini
On Windows 7 and later the file is hidden, but writeable at C:\ProgramData\pip\pip.ini
Global configuration is not supported on Windows Vista.
The global configuration file is shared by all Python installations.
If multiple configuration files are found by pip then they are combined in the following order:
The global file is read
The per-user file is read
The virtualenv-specific file is read
Each file read overrides any values read from previous files, so if the global timeout is specified in both the global file and the per-user file then the latter value will be used.
The names of the settings are derived from the long command line option, e.g. if you want to use a different package index (--index-url) and set the HTTP timeout (--default-timeout) to 60 seconds your config file would look like this:
[global]
timeout = 60
index-url = https://download.zope.org/ppix
Each subcommand can be configured optionally in its own section so that every global setting with the same name will be overridden; e.g. decreasing the timeout to 10 seconds when running the freeze (pip freeze) command and using 60 seconds for all other commands is possible with:
[global]
timeout = 60
[freeze]
timeout = 10
Boolean options like --ignore-installed or --no-dependencies can be set like this:
[install]
ignore-installed = true
no-dependencies = yes
To enable the boolean options --no-compile, --no-warn-script-location and --no-cache-dir, falsy values have to be used:
[global]
no-cache-dir = false
[install]
no-compile = no
no-warn-script-location = false
For options which can be repeated like --verbose and --quiet, a non-negative integer can be used to represent the level to be specified:
[global]
quiet = 0
verbose = 2
It is possible to append values to a section within a configuration file such as the pip.ini file. This is applicable to appending options like --find-links or --trusted-host, which can be written on multiple lines:
[global]
find-links =
http://download.example.com
[install]
find-links =
http://mirror1.example.com
http://mirror2.example.com
trusted-host =
mirror1.example.com
mirror2.example.com
This enables users to add additional values in the order of entry for such command line arguments.
Environment Variables
pip’s command line options can be set with environment variables using the format PIP_<UPPER_LONG_NAME> . Dashes (-) have to be replaced with underscores (_).
For example, to set the default timeout:
Unix/macOS
export PIP_DEFAULT_TIMEOUT=60
Windows
This is the same as passing the option to pip directly:
Unix/macOS
python -m pip --default-timeout=60 [...]
Windows
For command line options which can be repeated, use a space to separate multiple values. For example:
Unix/macOS
export PIP_FIND_LINKS="http://mirror1.example.com http://mirror2.example.com"
Windows
is the same as calling:
Unix/macOS
python -m pip install --find-links=http://mirror1.example.com --find-links=http://mirror2.example.com
Windows
Options that do not take a value, but can be repeated (such as --verbose) can be specified using the number of repetitions, so:
export PIP_VERBOSE=3
is the same as calling:
pip install -vvv
Note
Environment variables set to be empty string will not be treated as false. Please use no, false or 0 instead.
Config Precedence
Command line options have precedence over environment variables, which have precedence over the config file.
Within the config file, command specific sections have precedence over the global section.
Examples:
--host=foo overrides PIP_HOST=foo
PIP_HOST=foo overrides a config file with [global] host = foo
A command specific section in the config file [<command>] host = bar overrides the option with same name in the [global] config file section

Ansible: Change playbooks location

I have all playbooks in /etc/ansible/playbooks and I want to execute them anywhere on the pc
I tried to configure playbook_dir variable in ansible.cfg
[defaults]
playbook_dir = /etc/ansible/playbooks/
and tried to put ANSIBLE_PLAYBOOK_DIR variable in ~/.bashrc
export ANSIBLE_PLAYBOOK_DIR=/etc/ansible/playbooks/
but I only got the same error in both cases:
nor#nor:~$ ansible-playbook test3.yaml
ERROR! the playbook: test3.yaml could not be found
This is my ansible version:
ansible 2.9.7
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/nor/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.7/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.7.3 (default, Oct 7 2019, 12:56:13) [GCC 8.3.0]
Does anyone know the problem and how to solve it?

According to https://manpages.debian.org/testing/ansible/ansible-inventory.1.en.html :
--playbook-dir 'BASEDIR'
Since this tool does not use playbooks, use this as a subsitute playbook directory.This sets the relative path for many features including roles/ group_vars/ etc.
This means that ANSIBLE_PLAYBOOK_DIR is not used as a replacement for specifying the the absolute / relative path to your playbook, but it tells the playbook where it should look for roles, host/group vars , etc.
The goal you're trying to achieve is has no solution on the ansible side, you need to achieve this by configuring your shell profile accordingly.
set the following in your .bashrc file:
export playbooks_dir=/path/to/playbooks
when you call the playbook use ansible-playbook $playbooks_dir/test3.yml

As others have said, ANSIBLE_PLAYBOOK_DIR is for setting the relative directory for roles/, files/, etc. IMHO, it's not terribly useful.
If I understand the op, this is how I accomplish a similar result with all versions of ansible ...
PPWD=$PWD cd /my/playbook/dir && ansible-playbook my_playbook.yml; cd $PPWD
Explained,
PPWD=$PWD is to remember the current/present/previous working directory, then
cd /my/playbook/dir and if that succeeds run ansible-playbook my_playbook.yml (everything is relative from there); regardless, always change back to the previous working directory

PLAYBOOK_DIR says:
"A number of non-playbook CLIs have a --playbook-dir argument; this sets the default value for it."
Unfortunately, there is no hint in the doc what "the non-playbook CLIs" might be. ansible-playbook isn't one of them, obviously.
FWIW. If you're looking for a command-line oriented framework try ansible-runner. For example, export the location of private_data_dir
shell> export ansible_private=/path/to/<private-data-dir>
Then run the playbook
shell> ansible-runner -p playbook.yml run $ansible_private

Where is the luigi config file?

I have installed luigi by pip command and I would like to change the port for the web UI. I tried to find the config file but I couldn't. Do I need to create one?

You can start luigid with the --port option.
luigid --port 80
Configuration file locations are:
/etc/luigi/luigi.cfg
luigi.cfg (or its legacy name client.cfg) in
your current working directory
LUIGI_CONFIG_PATH environment variable
in increasing order of preference. You do need to create one. e.g.,
[core]
default-scheduler-host=www.example.com
default-scheduler-port=8088

In spite of the documentation saying otherwise, the port configuration from the config file is not used, at least in some versions or some circumstances
Until this is resolved, you should always use the --port option of luigid:
luigid --port 12345
Also see https://github.com/spotify/luigi/issues/2235
For other configuration options a config file should be used. See https://luigi.readthedocs.io/en/stable/configuration.html
For a configuration global to the host you can create a file:
/etc/luigi/luigi.cfg
Make sure it is readable by the user that runs luigid and luig.
Alternatively a local configuration file that will be recognized is
luigi.cfg
which you would have to create in the current working directory.
If you want a custom config file location you could set the environment variable LUIGI_CONFIG_PATH to the full path of your config file.

Deploying Django to AWS - WSGIPath refers to a file that does not exist

I've been struggling with getting Django and AWS to work together. I'm following the tutorial here:
https://realpython.com/blog/python/deploying-a-django-app-to-aws-elastic-beanstalk/
I've been following all the tutorial steps, including using the "eb option" command to change the WSGIPath, but I keep getting the error:
"ERROR: Your WSGIPath refers to a file that does not exist."
As far as I can tell I've been doing everything exactly according to the tutorial.
The relevant part of my config file looks like this:
NumProcesses: '1'
NumThreads: '15'
StaticFiles: /static/=static/
WSGIPath: iotd/iotd/wsgi.py
What am I doing wrong?

One thing I found when I encountered this error, is that if your repository is a git repository your .ebextensions folder must be tracked and committed otherwise it will not be picked up properly on eb deploy.

I have read the realpython blog post that you referred to. I would also refer you to the AWS tutorial. It is written for the deployment of a bare bones Django project and it can be found at:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-django.html#python-django-configure-for-eb
I found it useful to work through, and learned a great deal fixing the error you have identified. Of course the fix is related to my own implementation of the tutorial, which I followed step-by-step. I have read other posts that talk to this issue, but the solution stated here was not provided in those posts, as far as I can tell.
An abbreviated version of the tutorial follows to provide some context for the comments made here. This abbreviated version begins after creating/activating the virtual environment, but before its activation.
$ mkdir ed_django_app
$ . venv/Scripts/activate
(venv)[~eb_django_app]$ django-admin startproject django_eb
(venv)[~eb_django_app/django_eb]$ python manage.py migrate
(venv)[~eb_django_app/django_eb]$ python manage.py runserver
(venv)[~eb_django_app]$ pip freeze > requirements.txt
(venv)[~eb_django_app]$ deactivate
[~eb_django_app]$ eb init –region us-east-1
After the "eb init" command the .elasticbeanstalk directory, along with some files, are created in the initialization process. In that directory you will find the config.yml file. Its contents are:
branch-defaults:
default:
environment: eb-django-dev
global:
application_name: eb_django_app
default_ec2_keyname: myec2keyname
default_platform: Python 2.7
default_region: us-east-1
profile: eb-cli
sc: null
The tutorial directs the developer to then create a directory called .ebextensions and create the 01-eb_django.config file:
option_settings:
"aws:elasticbeanstalk:application:environment":
DJANGO_SETTINGS_MODULE: "django_eb.settings"
PYTHONPATH: "/opt/python/current/app/django_eb:$PYTHONPATH"
"aws:elasticbeanstalk:container:python":
WSGIPath: "django_eb/django_eb/wsgi.py"
This is YAML and the indentation matters. At least 1 space indent. In this case there are 2 spaces of indent at each level. The WSGIPath is set correctly. It is important to make sure the directory structure is the same as what is indicated in the tutorial.
In the tutorial the "eb create" command is now issued, and as you noted, the following arises:
ERROR: WSGIPath refers to a file that does not exist
The problem that was identified existed in the config.yml where there is the key-pair for application_name:
global:
application_name: eb_django_app
It was changed to:
global:
application_name: django_eb
This resolved the ERROR for me.

Using eb :
eb config
Go to aws:elasticbeanstalk:container:python: and change WSGIPath from:
application.py
to
mysite/wsgi.py
With "mysite" being your application name ofcourse

possible solution error : Your WSGIPath refers to a file that does not exist
after following this tutorial:
https://realpython.com/deploying-a-django-app-to-aws-elastic-beanstalk/
I got the error when I was uploading my protect to aws.
The step that I forgot was to activate my virtual env and from my there type the command 'eb deploy'
note : this error can also occur in different circumstances

This in django.config file, say your app name is 'yourappname',
worked for me.
option_settings:
aws:elasticbeanstalk:container:python:
WSGIPath: yourappname.wsgi:application

What Unix tool to quickly add/remove some text to a Python script?

I'm developing an application using Flask.
I want a quick, automated way to add and remove debug=True to the main function call:
Development:
app.run(debug=True)
Production:
app.run()
For security reasons, as I might expose private/sensitive information about the app if I leave debug mode on "in the wild".
I was thinking of using sed or awk to automate this in a git hook (production version is kept in a bare remote repo that I push to), or including it in a shell script I am going to write to fire up uwsgi and some other "maintenance"-ey tasks that allow the app to be served up properly.
What do you think?

That is not the way to go! My recommendation is to create some configuration Python module (let us say, config.py) with some content such as:
DEBUG = True
Now, in our current code, write this:
import config
app.run(debug=config.DEBUG)
Now, when you run in production, just change DEBUG from True to False. Or you can leave this file unversioned, so the copy of development is different of the copy of production. This is not uncommon since, for example, one does not use the same database connection params both in development and production.
Even if you want to update it automatically, just call sed on the config file with the -i flag. It is way more secure to update just this one file:
$ sed -i.bkp 's/^ *DEBUG *=.*$/DEBUG = False/' config.py

You should set up some environment variable on server. Your script can detect presense of this variable and disable debugging.

You probably should not be using app.run in production (and you definitely don't need it if you are using uwsgi). Instead, use one of the several deployment options discussed in the deployment section of Flask's excellent documentation. (app.run simply calls werkzeug.serving.run_simple which executes Python's included wsgiref server.)
That being said, the correct way to do this is not with a post-deploy edit to your source code but with a server-specific config file that changes your settings as #brandizzi pointed out in his answer.
You can do this in several different ways (Flask has documentation on this too - see Armin's suggestions on configuring from files and handling the development-production switch):
Include both your development and your server's configs in your repository. Use an environmental variable to switch between them:
# your_app.config.develop
DEBUG = True
# your_app.config.production
DEBUG = False
# your_app.app
from flask import Flask
from os import environ
mode = environ.get("YOURAPP_MODE")
mode = "production" if mode is None else "develop"
config = __import__("your_app.config." + mode)
app = Flask("your_app")
app.config.from_object(config)
Store your production configuration in a separate repository along with any other server-specific configurations you may need. Load the config if an environmental variable is set.

I'd use sed:
sed 's/debug=True//'
portable, scriptable, ubiquitous.

You can also use a NOCOMMIT hook (from gitty):
Set this as a pre-commit hook
if git diff --cached | grep NOCOMMIT > /dev/null; then
echo "You tried to commit a line containing NOCOMMIT"
exit 1
fi
exit 0
This will prevent the commit if it contains NOCOMMIT.
You can of course directly replace NOCOMMIT by Debug=True in the hook.