Develop Reference

Bandit Issue with Pyproject.toml

I'm trying to use pyproject.toml to exclude the venv/ directory. But it is not recognising the option.
[tool.bandit]
exclude = "/venv"
[tool.black]
exclude = "(venv)"
[tool.isort]
profile = "black"
skip = "venv"
balanced_wrapping = true
atomic = true
If I use the CLI option like so:
$ bandit -v -r . --exclude "/venv"
the directory is excluded. But if I just run bandit, it doesn't exclude the directory even though I have it in the pyproject.toml.
My bandit version is: 1.7.1.

exclude did not work for me, so I looked through official docs and found this:
We can specify dirs (and files as well) that we want to exclude in a list format
pyproject.toml:
[tool.bandit]
exclude_dirs = ["venv",]
From this documentation:
"Also you can configure bandit via pyproject.toml file. In this case you would explicitly specify the path to configuration via -c too."
Therefore, CLI option would look like this:
bandit -v -r . -c "pyproject.toml"
(will work without quotes as well)
I've never used bandit before, so if I got your question wrong - please feel free to write back, we will figure that out :D

To exclude directory venv, this command works fine for me :
bandit -r . -x */venv/*

Which PyPI index does Pip point to?

Ubuntu 20.04.1 LTS | Python 3.8.5 | Pip 20.0.2 (via python3-pip)
I accidentally overwrote the global PyPI index in my laptop with a custom index. (It was meant to be set only within a virtual environment. In case it's useful, I was using AWS CodeArtifact.)
I say this after running pip3 config -v list and seeing global.index-url assigned the other URL as well as ~/.config/.pip/pip.conf listing the same.
In an attempt to set things right, I did the following:
Set pip3's global.index-url to "https://pypi.python.org/pypi", but when I did a test to see if it worked, it stopped halfway while trying to install requests in a virtual environment (using its pip) and said some dependencies were not satisfied.
I then moved ~/.config/.pip to a backup, uninstalled python3-pip and reinstalled the package in the hope that the config would be written anew. While a new ~/config/.pip was not created, I was able to install requests into the virtual environment without any issue.
My confusion stems from the fact that I don't know where pip is picking things up from. None of the files listed below exist!
$ pip config -v list
For variant 'global', will try loading '/etc/xdg/xdg-ubuntu/pip/pip.conf'
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.config/pip/pip.conf'
For variant 'site', will try loading '/home/<user>/test_env/venv/pip.conf'
The output is slightly different when I use pip3 with the virtual environment deactivated.
$ pip3 config -v list
For variant 'global', will try loading '/etc/xdg/xdg-ubuntu/pip/pip.conf'
For variant 'global', will try loading '/etc/xdg/pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.pip/pip.conf'
For variant 'user', will try loading '/home/<user>/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/pip.conf'
Could someone please tell me how this all works?

https://pypi.org/simple/ packageX
let me know if its right ?
using
pip install paxckageX --verbose
not sure its right nowadays but here:
Python can't find the file pip.conf
and here
pip user guide Configuration :
Configuration
Config file
pip allows you to set all command line option defaults in a standard ini style config file.
The names and locations of the configuration files vary slightly across platforms. You may have per-user, per-virtualenv or global (shared amongst all users) configuration:
Per-user:
On Unix the default configuration file is: $HOME/.config/pip/pip.conf which respects the XDG_CONFIG_HOME environment variable.
On macOS the configuration file is $HOME/Library/Application Support/pip/pip.conf if directory $HOME/Library/Application Support/pip exists else $HOME/.config/pip/pip.conf.
On Windows the configuration file is %APPDATA%\pip\pip.ini.
There is also a legacy per-user configuration file which is also respected. To find its location:
On Unix and macOS the configuration file is: $HOME/.pip/pip.conf
On Windows the configuration file is: %HOME%\pip\pip.ini
You can set a custom path location for this config file using the environment variable PIP_CONFIG_FILE.
Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf
On Windows the file is: %VIRTUAL_ENV%\pip.ini
Global:
On Unix the file may be located in /etc/pip.conf. Alternatively it may be in a “pip” subdirectory of any of the paths set in the environment variable XDG_CONFIG_DIRS (if it exists), for example /etc/xdg/pip/pip.conf.
On macOS the file is: /Library/Application Support/pip/pip.conf
On Windows XP the file is: C:\Documents and Settings\All Users\Application Data\pip\pip.ini
On Windows 7 and later the file is hidden, but writeable at C:\ProgramData\pip\pip.ini
Global configuration is not supported on Windows Vista.
The global configuration file is shared by all Python installations.
If multiple configuration files are found by pip then they are combined in the following order:
The global file is read
The per-user file is read
The virtualenv-specific file is read
Each file read overrides any values read from previous files, so if the global timeout is specified in both the global file and the per-user file then the latter value will be used.
The names of the settings are derived from the long command line option, e.g. if you want to use a different package index (--index-url) and set the HTTP timeout (--default-timeout) to 60 seconds your config file would look like this:
[global]
timeout = 60
index-url = https://download.zope.org/ppix
Each subcommand can be configured optionally in its own section so that every global setting with the same name will be overridden; e.g. decreasing the timeout to 10 seconds when running the freeze (pip freeze) command and using 60 seconds for all other commands is possible with:
[global]
timeout = 60
[freeze]
timeout = 10
Boolean options like --ignore-installed or --no-dependencies can be set like this:
[install]
ignore-installed = true
no-dependencies = yes
To enable the boolean options --no-compile, --no-warn-script-location and --no-cache-dir, falsy values have to be used:
[global]
no-cache-dir = false
[install]
no-compile = no
no-warn-script-location = false
For options which can be repeated like --verbose and --quiet, a non-negative integer can be used to represent the level to be specified:
[global]
quiet = 0
verbose = 2
It is possible to append values to a section within a configuration file such as the pip.ini file. This is applicable to appending options like --find-links or --trusted-host, which can be written on multiple lines:
[global]
find-links =
http://download.example.com
[install]
find-links =
http://mirror1.example.com
http://mirror2.example.com
trusted-host =
mirror1.example.com
mirror2.example.com
This enables users to add additional values in the order of entry for such command line arguments.
Environment Variables
pip’s command line options can be set with environment variables using the format PIP_<UPPER_LONG_NAME> . Dashes (-) have to be replaced with underscores (_).
For example, to set the default timeout:
Unix/macOS
export PIP_DEFAULT_TIMEOUT=60
Windows
This is the same as passing the option to pip directly:
Unix/macOS
python -m pip --default-timeout=60 [...]
Windows
For command line options which can be repeated, use a space to separate multiple values. For example:
Unix/macOS
export PIP_FIND_LINKS="http://mirror1.example.com http://mirror2.example.com"
Windows
is the same as calling:
Unix/macOS
python -m pip install --find-links=http://mirror1.example.com --find-links=http://mirror2.example.com
Windows
Options that do not take a value, but can be repeated (such as --verbose) can be specified using the number of repetitions, so:
export PIP_VERBOSE=3
is the same as calling:
pip install -vvv
Note
Environment variables set to be empty string will not be treated as false. Please use no, false or 0 instead.
Config Precedence
Command line options have precedence over environment variables, which have precedence over the config file.
Within the config file, command specific sections have precedence over the global section.
Examples:
--host=foo overrides PIP_HOST=foo
PIP_HOST=foo overrides a config file with [global] host = foo
A command specific section in the config file [<command>] host = bar overrides the option with same name in the [global] config file section

pip to chose packages from a private index over PyPi even on name conflicts?

I have set up a devpi server to host my own Python modules. I would like to use pip to install them and for pip to install in preference my modules, not those in PyPi.
So far, I have this:
[global]
timeout = 60
index-url = http://devpi.example.org/root/public/+simple/
trusted-host = devpi.example.org
extra-index-url = http://devpi.example.org/root/pypi/+simple/
Which works fine as long as there is no name conflict. If there is, the highest version wins which is not what I want.
I really do not wish to register empty projects on PyPi just to have the names reserved.
Is there any way to get what I want?

Modified my ~/.config/pip/pip.conf like so:
[global]
timeout = 60
trusted-host = devpi.example.org
index-url = http://devpi.example.org/root/public/+simple/
and made sure that the /root/public inherited from /root/pypi like so:
devpi use http://devpi.example.org/
devpi use http://devpi.example.org/root/public --set-cfg
devpi login root --password="MyS3kre7Pwd"
devpi index /root/public bases=root/pypi # ← vital command.
Running:
devpi list mead --all
gives me what I wanted:
http://devpi.example.org/root/public/+f/d12/59ed3e5cf01ca/mead-0.4.0.dev1.tar.gz
http://devpi.example.org/root/public/+f/046/0f3dee895eb46/mead-0.4.0.dev0.tar.gz
http://devpi.example.org/root/pypi/+f/978/98e728d01b4d3/mead-0.0.11.tar.gz
http://devpi.example.org/root/pypi/+f/45d/b104905aeabc2/mead-0.0.10.tar.gz
http://devpi.example.org/root/pypi/+f/da0/c1b3bf979ca6a/mead-0.0.9.tar.gz
http://devpi.example.org/root/pypi/+f/49c/770889ecd3c7a/mead-0.0.8.tar.gz
http://devpi.example.org/root/pypi/+f/12d/10190b47367e8/mead-0.0.7.tar.gz
http://devpi.example.org/root/pypi/+f/34a/6dd6cd6c52c67/mead-0.0.6.tar.gz
http://devpi.example.org/root/pypi/+f/ba8/0cd76854e2253/mead-0.0.5.tar.gz
http://devpi.example.org/root/pypi/+f/1d7/6c5dd5229333b/mead-0.0.4.tar.gz
http://devpi.example.org/root/pypi/+f/47d/208d9cba5ea4e/mead-0.0.3.tar.gz
http://devpi.example.org/root/pypi/+f/20a/e16978e840e38/mead-0.0.2.tar.gz

How to add passenv to tox.ini without editing the file but by running tox in virtualenv shell nature script in Jenkins behind proxy (python)

I am trying to run python unit tests in jenkins using tox's virtualenv. I am behind a proxy so I need to pass HTTP_PROXY and HTTPS_PROXY to tox, else it has problems with downloading stuff.
I found out that I can edit tox.ini and add passenv=HTTP_PROXY HTTPS_PROXY under [testenv], and than using the Create/Update Text File Plugin I can override the tox.ini(as a build step) whenever Jenkins job fetches the original file from repository. This way I can manually copy content of tox.ini from workspace, add the passenv= line below [testenv] and update the file with the plugin mentioned above. But this is not the proper solution. I don't want to edit the tox.ini file this way, because the file is constantly updated. Using this solution would force me to update the tox.ini content inside the plugin everytime it is changed on the git repository and I want the process of running unit tests to be fully automated. And no, I can't edit the original file on git repository.
So is there a way that I can pass the passenv = HTTP_PROXY HTTPS_PROXY in the Shell nature command? This is how my command in Virtualenv Builder looks like:
pip install -r requirements.txt -r test-requirements.txt
pip install tox
tox --skip-missing-interpreter module/tests/
I want to do something like this:
tox --skip-missing-interpreter --[testenv]passenv=HTTP_PROXY HTTPS_PROXY module/tests
How to solve this?
NOTE:
I think there might be a solution with using the {posargs}, but I see that there is a line in the original tox.ini containing that posargs already: python setup.py testr --testr-args='{posargs}' help...

I thought about a workaround:
Create a build step in Jenkins job, that will execute bash script, that will open the tox.ini find line [testenv] and input one line below passenv = HTTP_PROXY HTTPS_PROXY. That would solve the problem. I am working on this right now but anyway if You guys know a better solution please let me know. cheers
Ok so this is the solution:
Add a build step Execute shell
Input this: sed -i.bak '/\[testenv\]/a passenv = HTTP_PROXY HTTPS_PROXY' tox.ini This will update the tox.ini file (input the desired passenv line under [testenv] and save changes). And create a tox.ini.bak backup file with the original data before sed's change.

What Unix tool to quickly add/remove some text to a Python script?

I'm developing an application using Flask.
I want a quick, automated way to add and remove debug=True to the main function call:
Development:
app.run(debug=True)
Production:
app.run()
For security reasons, as I might expose private/sensitive information about the app if I leave debug mode on "in the wild".
I was thinking of using sed or awk to automate this in a git hook (production version is kept in a bare remote repo that I push to), or including it in a shell script I am going to write to fire up uwsgi and some other "maintenance"-ey tasks that allow the app to be served up properly.
What do you think?

That is not the way to go! My recommendation is to create some configuration Python module (let us say, config.py) with some content such as:
DEBUG = True
Now, in our current code, write this:
import config
app.run(debug=config.DEBUG)
Now, when you run in production, just change DEBUG from True to False. Or you can leave this file unversioned, so the copy of development is different of the copy of production. This is not uncommon since, for example, one does not use the same database connection params both in development and production.
Even if you want to update it automatically, just call sed on the config file with the -i flag. It is way more secure to update just this one file:
$ sed -i.bkp 's/^ *DEBUG *=.*$/DEBUG = False/' config.py

You should set up some environment variable on server. Your script can detect presense of this variable and disable debugging.

You probably should not be using app.run in production (and you definitely don't need it if you are using uwsgi). Instead, use one of the several deployment options discussed in the deployment section of Flask's excellent documentation. (app.run simply calls werkzeug.serving.run_simple which executes Python's included wsgiref server.)
That being said, the correct way to do this is not with a post-deploy edit to your source code but with a server-specific config file that changes your settings as #brandizzi pointed out in his answer.
You can do this in several different ways (Flask has documentation on this too - see Armin's suggestions on configuring from files and handling the development-production switch):
Include both your development and your server's configs in your repository. Use an environmental variable to switch between them:
# your_app.config.develop
DEBUG = True
# your_app.config.production
DEBUG = False
# your_app.app
from flask import Flask
from os import environ
mode = environ.get("YOURAPP_MODE")
mode = "production" if mode is None else "develop"
config = __import__("your_app.config." + mode)
app = Flask("your_app")
app.config.from_object(config)
Store your production configuration in a separate repository along with any other server-specific configurations you may need. Load the config if an environmental variable is set.

I'd use sed:
sed 's/debug=True//'
portable, scriptable, ubiquitous.

You can also use a NOCOMMIT hook (from gitty):
Set this as a pre-commit hook
if git diff --cached | grep NOCOMMIT > /dev/null; then
echo "You tried to commit a line containing NOCOMMIT"
exit 1
fi
exit 0
This will prevent the commit if it contains NOCOMMIT.
You can of course directly replace NOCOMMIT by Debug=True in the hook.