Develop Reference

Unrecognized arguments YAML error when running pytest

I have a github action which runs some unit tests when a push is made to the repository. All the commands in the YAML execute successfully such as installing requirments.txt but then returns the following error when it tries to run the pytest command
python3 -m pytest verify/test.py --ds myapp.settings_pytest.
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --ds myapp.settings_pytest.
Strangely the command runs fine locally so I am confused as to why I only encounter this when it is ran from my YAML file. I am also encountering the same error when the same YAML file runs on my AWS build server.
test.yml
name: Run tests
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
env:
django_secret_key: ${{ secrets.DJANGO_SECRET_KEY }}
jobs:
build:
runs-on: ubuntu-latest
strategy:
max-parallel: 4
matrix:
python-version: [3.8]
steps:
- uses: actions/checkout#v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python#v2
with:
python-version: ${{ matrix.python-version }}
- name: Install Dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run Tests
run: |
export DJANGO_SETTINGS_MODULE="myapp.settings_pytest"
python3 -m pytest verify/test.py --ds myapp.settings_pytest

It has nothing to do with YML - you have to install Diango to get access to --ds options.
pip install Django==4.0.2
Or better - keep it in your requirements.txt

Cache is not being correctly loaded in Github actions

I'm trying to cache the python dependencies of my project. To do that, I have this configuration in my workflow:
- uses: actions/cache#v2
id: cache
with:
path: ~/.cache/pip
key: pip-${{ runner.os }}-${{ hashFiles('**/requirements.txt') }}-${{ hashFiles('**/requirements_dev.txt') }}
restore-keys: pip-${{ runner.os }}
- name: Install apt dependencies
run: |
sudo apt-get update
sudo apt-get install gdal-bin
- name: Install dependencies
if: steps.cache.outputs.cache-hit != 'true'
run: |
pip install --upgrade pip==9.0.1
pip install -r requirements.txt
pip install -r requirements_dev.txt
This works, by 'works' I mean that it loads the cache and skip the 'Install dependencies step' and it restores the ~/.cache/pip directory. The problem is that when I try to run the tests, the following error appears:
File "manage.py", line 7, in <module>
from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
Error: Process completed with exit code 1.
Am I caching the incorrect directory? Or what am I doing wrong?
Note: this project is using python2.7 on Ubuntu 16.04

As it explains here, you can cache the whole virtual environment:
- uses: actions/cache#v2
with:
path: ${{ env.pythonLocation }}
key: ${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}-${{ hashFiles('dev-requirements.txt') }}

Trouble installing Weasyprint & Cairo on Linux Web App

I have a Django app that uses Weasyprint to generate PDF outputs. This works fine on my local development machine.
I am able to successfully deploy to Azure Web Apps, but get the following error message:
2020-11-17T07:34:14.287002623Z OSError: no library called "cairo" was found
2020-11-17T07:34:14.287006223Z no library called "libcairo-2" was found
2020-11-17T07:34:14.287009823Z cannot load library 'libcairo.so.2': libcairo.so.2: cannot open shared
object file: No such file or directory
2020-11-17T07:34:14.287016323Z cannot load library 'libcairo.2.dylib': libcairo.2.dylib: cannot open
shared object file: No such file or directory
2020-11-17T07:34:14.287020123Z cannot load library 'libcairo-2.dll': libcairo-2.dll: cannot open
shared object file: No such file or directory
Per Weasyprint's documentation (https://weasyprint.readthedocs.io/en/stable/install.html#debian-ubuntu), I have attempted to make the reccommended installations via a custom deployment script which looks like such:
jobs:
build:
name: Build and Deploy
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python#v2
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install dependencies
run: |
sudo apt-get install build-essential python3-dev python3-pip python3-setuptools python3-wheel python3-cffi libcairo2 libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf2.0-0 libffi-dev shared-mime-info
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Deploy Web App using GH Action azure/webapps-deploy
uses: azure/webapps-deploy#v2
with:
app-name: ${{ env.AZURE_WEBAPP_NAME }}
publish-profile: ${{ secrets.AZURE_WEBAPP_PUBLISH_PROFILE }}
package: ${{ env.AZURE_WEBAPP_PACKAGE_PATH }}
However, my problem persists and I still receive the same message.
Does anybody have experience installing Weasyprint & Cairo on a Linux-based Web App?
I appreciate any help in advance.
UPDATE
Currently, I am able to deploy using the default deployment script created by Azure (shown below). I am then able to SSH into the deployment machine and manually activate the virtual environment & install the requisite packages. This process works and my application now works as expected.
I'd like to roll this command into the deployment process somehow (either as part of the default script or via a post deployment action).
GITHUB ACTIONS DEPLOYMENT SCRIPT
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#master
- name: Set up Python version
uses: actions/setup-python#v1
with:
python-version: '3.6'
- name: Build using AppService-Build
uses: azure/appservice-build#v2
with:
platform: python
platform-version: '3.6'
- name: 'Deploy to Azure Web App'
uses: azure/webapps-deploy#v2
with:
app-name: {{appname}}
slot-name: {{slotname}}
publish-profile: {{profilename}}
MANUAL VIRTUAL ENV ACTIVATION & INSTALLS
source /home/site/wwwroot/pythonenv3.6/bin/activate
sudo apt-get install {{ additional packages }}

The required dependencies and the things you want to install can be successfully added to the .yml file, but whether it takes effect for your webapp, you still need to test, and specific problems need to be analyzed in detail.
If that doesn't work, it is recommended to install ssh manually.
I add linux command in .yml file to apt-get install xxx.
For more details,
Below is my .yml file. It works fine.
# Docs for the Azure Web Apps Deploy action: https://github.com/Azure/webapps-deploy
# More GitHub Actions for Azure: https://github.com/Azure/actions
name: Build and deploy Python app to Azure Web App - pyodbcInstallENV
on:
push:
branches:
- master
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#master
- name: Set up Python version
uses: actions/setup-python#v1
with:
python-version: '3.8'
- name: Install custom env
run: |
cd /home
sudo apt-get update
sudo apt-get install g++
sudo apt-get install unixodbc-dev
pip install pyodbc
- name: Build using AppService-Build
uses: azure/appservice-build#v2
with:
platform: python
platform-version: '3.8'
- name: 'Deploy to Azure Web App'
uses: azure/webapps-deploy#v2
with:
app-name: 'pyodbcInstallENV'
slot-name: 'production'
publish-profile: ${{ secrets.AzureAppService_PublishProfile_d712769***2017c9521 }}

How to reference a directory in my repo during a GitHub action build?

I have some test data that I use for unit tests with pytest. I set their location with environment variables. Looking at my pytest logs the build sees the environment vars but the locations they reference don't exist. In the GitHub Actions docs the repo should be in /home/runner/Repo/. Below is my folder structure. Anyone see any obvious issues?
Repo/
notebooks/
repo/
__init__.py
tests/
tests_db.hdf5
Sample_Raw/
...
__init__.py
test_obj1.py
test_obj2.py
obj1.py
obj2.py
utils.py
build yaml
name: build-test
on:
push:
branches:
- '*' # all branches for now
jobs:
build-and-run:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest]
python-version: [3.8]
steps:
- uses: actions/checkout#v2
- uses: actions/setup-python#v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Generate coverage report
env:
DB_URL: /home/runner/work/Repo/repo/tests/test_db.hdf5
RAW_FOLDER: /home/runner/work/Repo/repo/tests/Sample_Raw/
run: |
pip install pytest
pip install pytest-cov
pytest --cov=./ --cov-report=xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action#v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./coverage.xml
name: codecov-umbrella

twine command not found (circleci)

I am using circleci to automatically deploy my python package to PyPi when a new tag is being pushed to GitHub. I've been following this tutorial
My workflow is failing with the error home/circleci/project/twitter_sentiment/bin/python: No module named twine
I've tried to make sure twine was installed prior to calling the twine command. I've also tried to call the twine command with python -m twine From my understanding, it seems that twine is not added to the path of the container - which causes the command not found error.
How would I go about solving this error?
config.yml file
version: 2
jobs:
build:
docker:
- image: circleci/python:3.6
working_directory: ~/twitter-sentiment
steps:
- checkout
- run:
name: install dependencies
command: |
python3 -m venv twitter_sentiment
. twitter_sentiment/bin/activate
pip install -r requirements.txt
- save_cache:
paths:
- ./twitter_sentiment
key: v1-dependencies-{{ checksum "setup.py" }}
runLibraryTest:
docker:
- image: circleci/python:3.6
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "setup.py" }}
- run:
name: run twitter_sentiment tests
command: |
python3 -m venv twitter_sentiment
. twitter_sentiment/bin/activate
pip install -r requirements.txt
cd test/
python3 -m unittest test_twitterSentiment
- save_cache:
paths:
- ./twitter_sentiment
key: v1-dependencies-{{ checksum "setup.py" }}
- store_artifacts:
path: test-reports
destination: test-reports
deploy:
docker:
- image: circleci/python:3.6
steps:
- checkout
- restore_cache:
key: v1-dependencies-{{ checksum "setup.py" }}
- run:
name: verify git tag vs version
command: |
python3 -m venv twitter_sentiment
. twitter_sentiment/bin/activate
pip install -r requirements.txt
python3 setup.py verify
- run:
name: create package files
command: |
python3 setup.py sdist bdist_wheel
- run:
name: create .pypirc file
command: |
echo -e "[pypi]" >> .pypirc
echo -e "repository = https://pypi.org/legacy/"
echo -e "username = TeddyCr" >> .pypirc
echo -e " = $PYPI_PASSWORD" >> .pypirc
- run:
name: upload package to pypi server
command: |
python3 -m venv twitter_sentiment
. twitter_sentiment/bin/activate
pip install --user --upgrade twine
python -m twine upload dist/*
- save_cache:
paths:
- ./twitter_sentiment
key: v1-dependencies-{{ checksum "setup.py" }}
workflows:
version: 2
build_and_deploy:
jobs:
- build:
filters:
tags:
only: /.*/
- runLibraryTest:
requires:
- build
filters:
tags:
only: /[0-9]+(\.[0-9]+)*/
branches:
ignore: /.*/
- deploy:
requires:
- runLibraryTest
filters:
tags:
only: /[0-9]+(\.[0-9]+)*/
branches:
ignore: /.*/

You're creating a virtual environment, activating it, and then installing twine outside of it.
Remove --user from pip install --user --upgrade twine