I have a travis job that looks like this:
jobs:
include:
- stage: "Unit tests"
language: python
python:
- "3.6"
- "3.7"
install:
- pip install -r requirements.txt
script:
- python -m unittest test.client
I would expect this unit test to run two jobs one for python 3.6 and one for 3.7 however it always only runs for the first version listed. Am I missing something here? I followed the guide from the docs
Thanks
The python versions are not defined within the jobs but on the root level.
python:
- "3.6"
- "3.7"
jobs:
...
I found this out because travis recently introduced a build config validation. It can be found under your build -> View config -> Build config validation
Related
I am trying to build (conda-build) and test (pytest) my python project with help of anaconda and TeamCity.
TeamCity is working fine for Linux, but for Windows I have an error on the testing build step (Linux and Windows use the same meta.yaml file):
failed to get install actions, retrying: exception was:
Unsatisfiable dependencies for platform win-64: {'urlib3==1.26.11=pyhd8ed1ab_0', ... <GIANT LIST OF LIBRARIES>
Important to say that the first step (build packages with help of conda-build) is working
But the second step (test step) is failing with the error above. So i think that the error deals with pytest related libs.
**meta.yaml**
{% set version = "1.1.0" %}
package:
name: myname
version: {{ version }}
source:
path ..
build:
number: 0
skip: True
pin_depends: strict
requirements:
build:
- python >= 3.8,<3.9
run:
- pythonv>=3.8,<3.9
- docker-py
- holidays
- scipy
- requests >=2.23.0
- numpy <1.22.0
- matplotlib >= 3.5.0,<4.0
...
- requests-negotiate-sspi # [win]
test:
requires:
- pytest
- pytest-asyncio
- pytest-cov
- pytz
- coverage
- dateutils
- pytest-xdist
- pytest-benchmark
- teamcity-messages
- pytest-bdd
source_files:
- conftest.py
comands:
- pytest <PATH_TO_FILE>
TeamCity test step looks like this:
conda-build <MY_PATH>\win_64\*.tar.bz2 <MY CHANNELS> --override-channels --test
I have a GUI program I'm managing, written in Python. For the sake of not having to worry about environments, it's distributed as an executable built with PyInstaller. I can run this build from a function defined in the module as MyModule.build() (because to me it makes more sense to manage that script alongside the project itself).
I want to automate this to some extent, such that when a new release is added on Gitlab, it can be built and attached to the release by a runner. The approach I currently have to this is functional but hacky:
I use the Gitlab API to download the source of the tag for the release. I run python -m pip install -r {requirements_path} and python -m pip install {source_path} in the runner's environment. Then import and run the MyModule.build() function to generate an executable. Which is then uploaded and linked to the release with the Gitlab API.
Obviously the middle section is wanting. What are best approaches for similar projects? Can the package and requirments be installed in a separate venv than the one the runner script it running in?
One workflow would be to push a tag to create your release. The following jobs have a rules: configuration so they only run on tag pipelines.
One job will build the executable file. Another job will create the GitLab release using the file created in the first job.
build:
rules:
- if: "$CI_COMMIT_TAG" # Only run when tags are pushed
image: python:3.9-slim
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache: # https://docs.gitlab.com/ee/ci/caching/#cache-python-dependencies
paths:
- .cache/pip
- venv/
script:
- python -m venv venv
- source venv/bin/activate
- python -m pip install -r requirements.txt # package requirements
- python -m pip install pyinstaller # build requirements
- pyinstaller --onefile --name myapp mypackage/__main__.py
artifacts:
paths:
- dist
create_release:
rules:
- if: "$CI_COMMIT_TAG"
needs: [build]
image: registry.gitlab.com/gitlab-org/release-cli:latest
script: # zip/upload your binary wherever it should be downloaded from
- echo "Uploading release!"
release: # create GitLab release
tag_name: $CI_COMMIT_TAG
name: 'Release of myapp version $CI_COMMIT_TAG'
description: 'Release created using the release-cli.'
assets: # link uploaded asset(s) to the release
- name: 'release-zip'
url: 'https://example.com/downloads/myapp/$CI_COMMIT_TAG/myapp.zip'
I have a travis integration in my project with build language as python. I want to integrate postman test which requires Node installation. Should i create a separate build for this? Is there a way to accommodate this in the same build. I tried adding a new env but apparently I was getting the tox error.
This is a broad guideline. Ideally:
The jobs are atomic (independent of setup and run)
You install npm or configure Travis to have npm setup alongside Python
You run your jobs in the order you desire (usually Newman last)
It seems nowdays you can have a language per job in Travis. Check: https://docs.travis-ci.com/user/build-matrix/#using-different-programming-languages-per-job. For example:
dist: xenial
language: php
php:
- '5.6'
jobs:
include:
- language: python
python: 3.8
script:
- python -c "print('Hi from Python!')"
- language: node_js
node_js: 12
script:
- npm i newman -g
- newman run COLLECTION
So this will likely allow you to keep one single build + test run.
I'm fairly new to Django so apologies in advance if this is obvious.
In Rails projects, I use a gem called bundler-audit to check that the patch level of the gems I'm installing don't include security vulnerabilities. Normally, I incorporate running bundler-audit into my CI pipeline so that any time I deploy, I get a warning (and fail) if a gem has a security vulnerability.
Is there a similar system for checking vulnerabilities in Python packages?
After writing out this question, I searched around some more and found Safety, which was exactly what I was looking for.
In case anyone else is setting up CircleCI for a Django project and wants to check their packages for vulnerabilities, here is the configuration I used in my .circleci/config.yml:
version: 2
jobs:
build:
# build and run tests
safety_check:
docker:
- image: circleci/python:3.6.1
steps:
- checkout
- run:
command: |
python3 -m venv env3
. env3/bin/activate
pip install safety
# specify requirements.txt
safety check -r requirements.txt
merge_master:
# merge passing code into master
workflows:
version: 2
test_and_merge:
jobs:
- build:
filters:
branches:
ignore: master
- safety_check:
filters:
branches:
ignore: master
- merge_master:
filters:
branches:
only: develop
requires:
- build
# code is only merged if safety check passes
- safety_check
To check that this works, run pip install insecure-package && pip freeze > requirements.txt then push and watch for Circle to fail.
Is there a way to configure travis-ci to make the Python versions dependent on a certain env var?
Please consider the following travis.yml config:
language: python
python:
- "2.5"
- "2.6"
- "2.7"
env:
- DJANGO=1.3.4
- DJANGO=1.4.2
- DJANGO=https://github.com/django/django/zipball/master
install:
- pip install -q Django==$DJANGO --use-mirrors
- pip install -e . --use-mirrors
script:
- python src/runtests.py
Among Django 1.3 (DJANGO=1.3.4) and 1.4 (DJANGO=1.4.2) i also want to test against the latest development version of Django (DJANGO=https://github.com/django/django/zipball/master), which is basically Django 1.5.
The problem i see is that travis-ci will automatically run the integration against all specified Python versions. Django 1.5 however doesn't support Python 2.5 anymore. Is it possible to omit it for the Django development version so that i get integrations like this only:
DJANGO=1.3.4 --> python "2.5", "2.6", "2.7"
DJANGO=1.4.2 --> python "2.5", "2.6", "2.7"
DJANGO=https://github.com/django/django/zipball/master --> python "2.6", "2.7"
UPDATE:
Here's a link to a live example based on Odi's answer which i've been using successfully for a few months:
https://github.com/deschler/django-modeltranslation/blob/master/.travis.yml
You can specify configurations that you want to exclude from the build matrix (i.e. combinations that you don't want to test).
Add this to your .travis.yml:
matrix:
exclude:
- python: "2.5"
env: DJANGO=https://github.com/django/django/zipball/master
Note: only exact matches will be excluded.
See the build documentation (section The Build Matrix) for further information.