Context
After setting up a conda environment.yml and trying to install it with a python 3.8 version, I am experiencing some difficulties.
Attempts
I tried explicitly specifying the python version at the environment creation command:
conda env create --file environment.yml python=3.8
I tried explicitly including the python version in the environment.yml file:
...
dependencies:
- anaconda
- python=3.8
- conda:
# Run python tests.
- pytest-cov
...
And I tried explicitly installing python 3.8 inside the environment with:
conda activate env_name
conda install python==3.8
Which yields a conflict:
Found conflicts! Looking for incompatible packages.
So I tried determining what the conflict is, by evaluating the 2000 output lines that describe the conflicts. I think the first conflict is most relevant:
UnsatisfiableError: The following specifications were found to be incompatible with a past
explicit spec that is not an explicit spec in this operation (pip):
- python=3.8 -> pip
Environment
The conda environment consists of the following four files:
environment.yml
pyproject.toml
requirements.txt
.pre-commit-config.yaml
Which have contents:
environment.yml:
# This file is to automatically configure your environment. It allows you to
# run the code with a single command without having to install anything
# (extra).
# First run:: conda env create --file environment.yml
# If you change this file, run: conda env update --file environment.yml
# Instructions for this networkx-to-lava-nc repository only. First time usage
# On Ubuntu (this is needed for lava-nc):
# sudo apt upgrade
# sudo apt full-upgrade
# yes | sudo apt install gcc
# Conda configuration settings. (Specify which modules/packages are installed.)
name: networkx-to-lava
channels:
- conda-forge
dependencies:
- python=3.8
- conda:
# Run python tests.
- pytest-cov
# Generate plots.
- matplotlib
# Run graph software quickly.
- networkx
- pip
- pip:
# Run pip install on .tar.gz file in GitHub repository (For lava-nc only).
- https://github.com/lava-nc/lava/releases/download/v0.3.0/lava-nc-0.3.0.tar.gz
# Turns relative import paths into absolute import paths.
- absolufy-imports
# Auto format Python code to make it flake8 compliant.
- autoflake
# Scan Python code for security issues.
- bandit
# Code formatting compliance.
- black
# Correct code misspellings.
- codespell
# Verify percentage of code that has at least 1 test.
- coverage
# Auto formats the Python documentation written in the code.
- docformatter
# Auto generate docstrings.
- flake8
# Auto sort the import statements.
- isort
# Auto format Markdown files.
- mdformat
# Auto check static typing.
- mypy
# Auto generate documentation.
- pdoc3
# Auto check programming style aspects.
- pylint
# Auto generate docstrings.
- pyment
# Identify and remove dead code.
- vulture
# Include GitHub pre-commit hook.
- pre-commit
# TODO: identify exact function(and usage).
# Seems to be an autoformatter like black, but installed using npm instead of pip.
- prettier
# Automatically upgrades Python syntax to the new Python version syntax.
- pyupgrade
# Another static type checker for python like mypy.
- pyright
pyproject.toml:
# This is used to configure the black, isort and mypy such that the packages don't conflict.
# This file is read by the pre-commit program.
[tool.black]
line-length = 79
include = '\.pyi?$'
exclude = '''
/(
\.git
| \.mypy_cache
| build
| dist
)/
'''
[tool.coverage.run]
# Due to a strange bug with xml output of coverage.py not writing the full-path
# of the sources, the full root directory is presented as a source alongside
# the main package. As a result any importable Python file/package needs to be
# included in the omit
source = [
"foo",
".",
]
# Excludes the following directories from the coverage report
omit = [
"tests/*",
"setup.py",
]
[tool.isort]
profile = "black"
[tool.mypy]
ignore_missing_imports = true
[tool.pylint.basic]
bad-names=[]
[tool.pylint.messages_control]
# Example: Disable error on needing a module-level docstring
disable=[
"import-error",
"invalid-name",
"fixme",
]
[tool.pytest.ini_options]
# Runs coverage.py through use of the pytest-cov plugin
# An xml report is generated and results are output to the terminal
addopts = "--cov --cov-report xml:cov.xml --cov-report term"
# Sets the minimum allowed pytest version
minversion = 5.0
# Sets the path where test files are located (Speeds up Test Discovery)
testpaths = ["tests"]
requirements.txt
# This file ensures that the pre-commit service is ran every time you commit.
# Basically it ensures people only push files to GIT that are up to standard.
pre-commit
.pre-commit-config.yaml
# This file specifies which checks are performed by the pre-commit service.
# The pre-commit service prevents people from pushing code to git that is not
# up to standards. # The reason mirrors are used instead of the actual
# repositories for e.g. black and flake8, is because those repositories also
# need to contain a pre-commit hook file, which they often don't by default.
# So to resolve that, a mirror is created that includes such a file.
default_language_version:
python: python3.10 # or python3
repos:
# Test if the python code is formatted according to the Black standard.
- repo: https://github.com/Quantco/pre-commit-mirrors-black
rev: 22.3.0
hooks:
- id: black-conda
args:
- --safe
- --target-version=py36
# Test if the python code is formatted according to the flake8 standard.
- repo: https://github.com/Quantco/pre-commit-mirrors-flake8
rev: 4.0.1
hooks:
- id: flake8-conda
# Test if the import statements are sorted correctly.
- repo: https://github.com/PyCQA/isort
rev: 5.10.1
hooks:
- id: isort
args: ["--profile", "black", --line-length=79]
# Test if the variable typing is correct. (Variable typing is when you say:
# def is_larger(nr: int) -> bool: instead of def is_larger(nr). It makes
# it explicit what type of input and output a function has.
# - repo: https://github.com/python/mypy
- repo: https://github.com/pre-commit/mirrors-mypy
# - repo: https://github.com/a-t-0/mypy
rev: v0.950
hooks:
- id: mypy
# Tests if there are spelling errors in the code.
- repo: https://github.com/codespell-project/codespell
rev: v2.1.0
hooks:
- id: codespell
# Performs static code analysis to check for programming errors.
- repo: local
hooks:
- id: pylint
name: pylint
entry: pylint
language: system
types: [python]
args:
[
"-rn", # Only display messages
"-sn", # Don't display the score
]
# Runs additional tests that are created by the pre-commit software itself.
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.2.0
hooks:
# Check user did not add large files.
- id: check-added-large-files
# Check if `.py` files are written in valid Python syntax.
- id: check-ast
# Require literal syntax when initializing empty or zero Python builtin types.
- id: check-builtin-literals
# Checks if there are filenames that would conflict if case is changed.
- id: check-case-conflict
# Checks if the Python functions have docstrings.
- id: check-docstring-first
# Checks if any `.sh` files have a shebang like #!/bin/bash
- id: check-executables-have-shebangs
# Verifies json format of any `.json` files in repo.
- id: check-json
# Checks if there are any existing merge conflicts caused by the commit.
- id: check-merge-conflict
# Checks for symlinks which do not point to anything.
- id: check-symlinks
# Checks if xml files are formatted correctly.
- id: check-xml
# Checks if .yml files are valid.
- id: check-yaml
# Checks if debugger imports are performed.
- id: debug-statements
# Detects symlinks changed to regular files with content path symlink was pointing to.
- id: destroyed-symlinks
# Checks if you don't accidentally push a private key.
- id: detect-private-key
# Replaces double quoted strings with single quoted strings.
# This is not compatible with Python Black.
#- id: double-quote-string-fixer
# Makes sure files end in a newline and only a newline.
- id: end-of-file-fixer
# Removes UTF-8 byte order marker.
- id: fix-byte-order-marker
# Add <# -*- coding: utf-8 -*-> to the top of python files.
- id: fix-encoding-pragma
# Checks if there are different line endings, like \n and crlf.
- id: mixed-line-ending
# Asserts `.py` files in folder `/test/` (by default:) end in `_test.py`.
- id: name-tests-test
# Override default to check if `.py` files in `/test/` START with `test_`.
args: ['--django']
# Ensures JSON files are properly formatted.
- id: pretty-format-json
# Sorts entries in requirements.txt and removes incorrect pkg-resources entries.
- id: requirements-txt-fixer
# Sorts simple YAML files which consist only of top-level keys.
- id: sort-simple-yaml
# Removes trailing whitespaces at end of lines of .. files.
- id: trailing-whitespace
- repo: https://github.com/PyCQA/autoflake
rev: v1.4
hooks:
- id: autoflake
args: ["--in-place", "--remove-unused-variables", "--remove-all-unused-imports", "--recursive"]
name: AutoFlake
description: "Format with AutoFlake"
stages: [commit]
- repo: https://github.com/PyCQA/bandit
rev: 1.7.4
hooks:
- id: bandit
name: Bandit
stages: [commit]
# Enforces formatting style in Markdown (.md) files.
- repo: https://github.com/executablebooks/mdformat
rev: 0.7.14
hooks:
- id: mdformat
additional_dependencies:
- mdformat-toc
- mdformat-gfm
- mdformat-black
- repo: https://github.com/MarcoGorelli/absolufy-imports
rev: v0.3.1
hooks:
- id: absolufy-imports
files: '^src/.+\.py$'
args: ['--never', '--application-directories', 'src']
- repo: https://github.com/myint/docformatter
rev: v1.4
hooks:
- id: docformatter
- repo: https://github.com/pre-commit/pygrep-hooks
rev: v1.9.0
hooks:
- id: python-use-type-annotations
- id: python-check-blanket-noqa
- id: python-check-blanket-type-ignore
# Updates the syntax of `.py` files to the specified python version.
# It is not compatible with: pre-commit hook: fix-encoding-pragma
# - repo: https://github.com/asottile/pyupgrade
# rev: v2.32.1
# hooks:
# - id: pyupgrade
# args: [--py310-plus]
- repo: https://github.com/markdownlint/markdownlint
rev: v0.11.0
hooks:
- id: markdownlint
Package Conflict Output
conda install python=3.8
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed \
UnsatisfiableError: The following specifications were found to be incompatible with a past
explicit spec that is not an explicit spec in this operation (pip):
- python=3.8 -> pip
The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Package openjpeg conflicts for:
openjpeg
pillow -> openjpeg[version='>=2.3.0,<3.0a0|>=2.4.0,<2.5.0a0']
matplotlib-base -> pillow[version='>=6.2.0'] -> openjpeg[version='>=2.3.0,<3.0a0|>=2.4.0,<2.5.0a0']
Package ncurses conflicts for:
wheel -> python -> ncurses[version='6.0.*|>=6.0,<7.0a0|>=6.1,<7.0a0|>=6.2,<7.0a0|>=6.3,<7.0a0']
krb5 -> libedit[version='>=3.1.20210216,<3.2.0a0'] -> ncurses[version='6.0.*|>=6.1,<7.0a0|>=6.2,<7.0.0a0|>=6.2,<7.0a0|>=6.3,<7.0a0']
pluggy -> python[version='>=3.7,<3.8.0a0'] -> ncurses[version='6.0.*|>=6.0,<7.0a0|>=6.1,<7.0a0|>=6.2,<7.0a0|>=6.3,<7.0a0']
...
...
- tornado -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
- unicodedata2 -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
- xorg-libxau -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
- xorg-libxdmcp -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
- xz -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
- zlib -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
- zstd -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']
Your installed version is: 2.33
Question
How can I determine the conflicting package in this conda environment and/or how can I create the environment using python 3.8?
After specifying the python version as the first dependency, and removing the unneeded elements as suggested by merv, I found a working yaml. I removed anaconda, and the conda channel. Furthermore, I ensured the default_version in the .pre-commit-config.yaml file was set to:
default_language_version:
python: python3.8. # or python3
I also deleted the .mypy_cache folder in the .git repository (even though I think this was not required). And I deleted the directory /home/<username>/.cache/pre-commit before running pre-commit run --all-files. (I thought it was worth mentioning as it is inherent to this environment.yml)
I did not have to specify the python version in the environment creation command. Instead, I ran:
conda env create --file environment.yml
The working environment.yml content is:
# This file is to automatically configure your environment. It allows you to
# run the code with a single command without having to install anything
# (extra).
# First run: conda env create --file environment.yml
# If you change this file, run: conda env update --file environment.yml
# Instructions for this networkx-to-lava-nc repository only. First time usage
# On Ubuntu (this is needed for lava-nc):
# sudo apt upgrade
# sudo apt full-upgrade
# yes | sudo apt install gcc
# Conda configuration settings. (Specify which modules/packages are installed.)
name: nx2lava
channels:
- conda-forge
dependencies:
# Specify specific python version.
- python=3.8
# Run python tests.
- pytest-cov
# Generate plots.
- matplotlib
# Run graph software quickly.
- networkx
- pip
- pip:
# Run pip install on .tar.gz file in GitHub repository (For lava-nc only).
- https://github.com/lava-nc/lava/releases/download/v0.3.0/lava-nc-0.3.0.tar.gz
# Turns relative import paths into absolute import paths.
- absolufy-imports
# Auto format Python code to make it flake8 compliant.
- autoflake
# Scan Python code for security issues.
- bandit
# Code formatting compliance.
- black
# Correct code misspellings.
- codespell
# Verify percentage of code that has at least 1 test.
- coverage
# Auto formats the Python documentation written in the code.
- docformatter
# Auto generate docstrings.
- flake8
# Auto sort the import statements.
- isort
# Auto format Markdown files.
- mdformat
# Auto check static typing.
- mypy
# Auto generate documentation.
- pdoc3
# Auto check programming style aspects.
- pylint
# Auto generate docstrings.
- pyment
# Identify and remove dead code.
- vulture
# Include GitHub pre-commit hook.
- pre-commit
# TODO: identify exact function(and usage).
# Seems to be an autoformatter like black, but installed using npm instead of pip.
- prettier
# Automatically upgrades Python syntax to the new Python version syntax.
- pyupgrade
# Another static type checker for python like mypy.
- pyright
Which returns the following to the python --version command:
Python 3.8.13
Is there a way to exclude pytests marked with pytest.mark from running during the pre-commit hook?
Especially, I'd like to exclude the tests that are marked as integration tests.
The content of a tests looks like this
pytestmark = [pytest.mark.integration, pytest.mark.reporting_api]
### some tests
and the .pre-commit-conifg.yaml pytest configuration is
- repo: local
hooks:
- id: pytest
name: pytest
entry: pytest test/
language: system
pass_filenames: false
types: [python]
stages: [commit, push]
2c: it's not a good idea to run tests as part of pre-commit -- in my experience they're going to be too slow and your contributors may get frustrated and turn off the framework entirely
that said, it should be as simple as adding the arguments you want to either entry or args -- personally I prefer entry when working with repo: local hooks (since there's nothing that would "conventionally" override args)
in your case this would look like:
- repo: local
hooks:
- id: pytest
name: pytest
entry: pytest test/ -m 'not integration and not reporting_api'
language: system
pass_filenames: false
types: [python]
stages: [commit, push]
disclaimer: I created pre-commit and I'm a pytest core dev
Gitlab version is 13.6.6
Gitlab-runner version is 11.2.0
my .gitlab-ci.yml:
image: "python:3.7"
before_script:
- pip install flake8
flake8:
stage: test
script:
- flake8 -max-line-length=79
tags:
- test
The only information obtained from Pipelines is script failure and the output of failed job is No job log. How can I get more detailed error output？
Using artifacts can help you.
image: "python:3.7"
before_script:
- pip install flake8
flake8:
stage: test
script:
- flake8 -max-line-length=79
- cd path/to
tags:
- test
artifacts:
when: on_failure
paths:
- path/to/test.log
The log file can be downloaded via the web interface.
Note:- Using when: on_failure will ensure that test.log will only be collected if the build fails, saving disk space on successful builds.
I am using Github Python application workflow for CI. My application creates a folder to store temporary files. It works perfectly when testing on localhost but it will not let me create a new directory in Github actions. I get the below error:
#classmethod
def save_files(cls, files: list) -> str:
"""
saves a list of files in the "files"
folder in app
:param files: list of FileStorage objects
:return: directory name where files saved
"""
folder = time.strftime("%Y%m%d-%H%M%S")
folder_path = Path(__file__).parent / "files" / folder
os.mkdir(folder_path)
E FileNotFoundError: [Errno 2] No such file or directory: /home/runner/work/DocumentAnalysisTool/DocumentAnalysisTool/app/files/20200430-235749
Here is my workflow pythonapp.yml file:
name: Python application
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up Python 3.8
uses: actions/setup-python#v1
with:
python-version: 3.8
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Lint with flake8
run: |
pip install flake8
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
run: |
pip install pytest
pytest
Thank you in advance
In python project I have pre-commit-config.yaml with black and mypy checkers.
The goal is to before checking for errors, first run black . and mypy ./ to fix possible errors and then run hooks.
- repo: local
hooks:
- id: code-formatter
exclude: .pre-commit-config.yaml
name: Format code
entry: black .
language: python
- id: mypy
exclude: .pre-commit-config.yaml
name: Check types
entry: mypy --scripts-are-modules ./
language: python