Develop Reference

python - Automated building of executables

I have a GUI program I'm managing, written in Python. For the sake of not having to worry about environments, it's distributed as an executable built with PyInstaller. I can run this build from a function defined in the module as MyModule.build() (because to me it makes more sense to manage that script alongside the project itself).
I want to automate this to some extent, such that when a new release is added on Gitlab, it can be built and attached to the release by a runner. The approach I currently have to this is functional but hacky:
I use the Gitlab API to download the source of the tag for the release. I run python -m pip install -r {requirements_path} and python -m pip install {source_path} in the runner's environment. Then import and run the MyModule.build() function to generate an executable. Which is then uploaded and linked to the release with the Gitlab API.
Obviously the middle section is wanting. What are best approaches for similar projects? Can the package and requirments be installed in a separate venv than the one the runner script it running in?

One workflow would be to push a tag to create your release. The following jobs have a rules: configuration so they only run on tag pipelines.
One job will build the executable file. Another job will create the GitLab release using the file created in the first job.
build:
rules:
- if: "$CI_COMMIT_TAG" # Only run when tags are pushed
image: python:3.9-slim
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache: # https://docs.gitlab.com/ee/ci/caching/#cache-python-dependencies
paths:
- .cache/pip
- venv/
script:
- python -m venv venv
- source venv/bin/activate
- python -m pip install -r requirements.txt # package requirements
- python -m pip install pyinstaller # build requirements
- pyinstaller --onefile --name myapp mypackage/__main__.py
artifacts:
paths:
- dist
create_release:
rules:
- if: "$CI_COMMIT_TAG"
needs: [build]
image: registry.gitlab.com/gitlab-org/release-cli:latest
script: # zip/upload your binary wherever it should be downloaded from
- echo "Uploading release!"
release: # create GitLab release
tag_name: $CI_COMMIT_TAG
name: 'Release of myapp version $CI_COMMIT_TAG'
description: 'Release created using the release-cli.'
assets: # link uploaded asset(s) to the release
- name: 'release-zip'
url: 'https://example.com/downloads/myapp/$CI_COMMIT_TAG/myapp.zip'

Excluding pytest files/marks from pre-commit hooks

Is there a way to exclude pytests marked with pytest.mark from running during the pre-commit hook?
Especially, I'd like to exclude the tests that are marked as integration tests.
The content of a tests looks like this
pytestmark = [pytest.mark.integration, pytest.mark.reporting_api]
### some tests
and the .pre-commit-conifg.yaml pytest configuration is
- repo: local
hooks:
- id: pytest
name: pytest
entry: pytest test/
language: system
pass_filenames: false
types: [python]
stages: [commit, push]

2c: it's not a good idea to run tests as part of pre-commit -- in my experience they're going to be too slow and your contributors may get frustrated and turn off the framework entirely
that said, it should be as simple as adding the arguments you want to either entry or args -- personally I prefer entry when working with repo: local hooks (since there's nothing that would "conventionally" override args)
in your case this would look like:
- repo: local
hooks:
- id: pytest
name: pytest
entry: pytest test/ -m 'not integration and not reporting_api'
language: system
pass_filenames: false
types: [python]
stages: [commit, push]
disclaimer: I created pre-commit and I'm a pytest core dev

How to display scores for meeting --fail-under requirement .py files in .py lint, while using pre-commit?

My current pylint configuration:
Installed via pip (in requirements.txt).
The .pre-commit-config.yaml:
- repo: local
hooks:
name: pylint
entry: pylint
language: system
types: [ python ]
files: ^src/
args:
[
"-rn", # display messages
"--rcfile=.pylintrc",
"--fail-under=8.5"
]
Execution method:
source venv/bin/activate &&\
pip freeze &&\
pre-commit install &&\
pre-commit run --all-files
When all .py files receive score higher than 8.5 then pylint is just passing and do not displaying any message. Is there any method to see all of the communicates even if --fail-under is met? (so we know what some is wrong with the files)

there is a setting which will force the output to always display: verbose: true but it is only intended for debug purposes as it tends to make the output noisy and your contributors will be likely to mentally filter it out as warning noise
using your example:
- repo: local
hooks:
name: pylint
entry: pylint
language: system
types: [ python ]
files: ^src/
args:
[
"-rn", # display messages
"--rcfile=.pylintrc",
"--fail-under=8.5"
]
verbose: true
unrelated, but there's no reason to use args for a repo: local hook since nothing can override it, you can specify those directly in entry:
entry: pylint --rn --rcfile=.pylintrc --fail-under=8.5
disclaimer: I created pre-commit

Is there a Python/Django equivalent to Rails bundler-audit?

I'm fairly new to Django so apologies in advance if this is obvious.
In Rails projects, I use a gem called bundler-audit to check that the patch level of the gems I'm installing don't include security vulnerabilities. Normally, I incorporate running bundler-audit into my CI pipeline so that any time I deploy, I get a warning (and fail) if a gem has a security vulnerability.
Is there a similar system for checking vulnerabilities in Python packages?

After writing out this question, I searched around some more and found Safety, which was exactly what I was looking for.
In case anyone else is setting up CircleCI for a Django project and wants to check their packages for vulnerabilities, here is the configuration I used in my .circleci/config.yml:
version: 2
jobs:
build:
# build and run tests
safety_check:
docker:
- image: circleci/python:3.6.1
steps:
- checkout
- run:
command: |
python3 -m venv env3
. env3/bin/activate
pip install safety
# specify requirements.txt
safety check -r requirements.txt
merge_master:
# merge passing code into master
workflows:
version: 2
test_and_merge:
jobs:
- build:
filters:
branches:
ignore: master
- safety_check:
filters:
branches:
ignore: master
- merge_master:
filters:
branches:
only: develop
requires:
- build
# code is only merged if safety check passes
- safety_check
To check that this works, run pip install insecure-package && pip freeze > requirements.txt then push and watch for Circle to fail.

How to configure pycharm / intellij idea to run tox tests

Is it possible to configure pycharm / intellij idea to run tox tests? I want to test my code against different python versions in separated py environments. I was trying to configure it, but so far I only managed to configure single py.test support.

See this PyCharm issue: PY-9727
Credit goes to Andrey Vlasovskikh at PyCharm
As a workaround, you can create a Python file in your project that imports and launches Tox:
import tox
tox.cmdline()
and then run this file via your project interpreter using its context menu.
You'll get hyperlinks in the console output for stack traces, but nothing more.

I'm afraid it's not supported, PyCharm will use the configured interpreter to run the tests.
You are welcome to submit a feature request.

7 years passed and now you can run Tox inside PyCharm. If you use pytest for testing it will even show Test Result same way like with running test locally by PyCharm.
To run tests in pycharm I use config like this
[tox]
envlist = py3.{7,8},codestyle,flake8,lint
minversion = 3.7
[testenv]
usedevelop = true
deps =
pytest
pytest-cov
commands = pytest --cov=src
[testenv:codestyle]
deps = pycodestyle
commands = pycodestyle src tests
[testenv:flake8]
deps = flake8
commands = flake8 src tests
[testenv:lint]
deps = pylint
commands = pylint src tests --rcfile=.pylintrc
Here I described why it looks like this. Also, there is integration with GitHub CI to run it on every push.