For my tests in azure-pipeline, I want to use a container that I then push to Docker Hub.
Actually, the steps are the following:
Pull image from registry
Do the tests
Push the new image with the new commits in the code in the Registry
The problem: The image Pulled from the registry contains the previous code, not the one that I am testing.
What I want to do:
First, deploy the image with the new code in the Docker registry
Then, steps 1 to 3 mentionned before, so that the image that I pull is the up-to-date.
Here is my actual code:
trigger:
- master
resources:
containers:
- container: moviestr_backend
image: nolwenbrosson/cicd:moviestr_backend-$(SourceBranchName)
ports:
- 5000:5000
- container: backend_mongo
image: mongo
ports:
- 27017:27017
pool:
vmImage: 'ubuntu-latest'
strategy:
matrix:
Python37:
python.version: '3.7'
services:
moviestr_backend: moviestr_backend
backend_mongo: backend_mongo
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'
- script: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -r requirements.dev.txt
pip install pytest-azurepipelines
displayName: 'Install dependencies'
- script: |
python -m pytest
displayName: 'Make Unit tests'
- task: Docker#2
displayName: Login to Docker Hub
inputs:
command: login
containerRegistry: cicd
- task: Docker#2
displayName: Build and Push
inputs:
command: buildAndPush
repository: nolwenbrosson/cicd
tags: |
moviestr_backend-master
- task: Docker#2
displayName: Logout of ACR
inputs:
command: logout
containerRegistry: cicd
The problem is ressources is unique for the whole pipeline, and it will Pull the image at the beginning, not after I build my image with my up-to-date code. So, how can I do?
You could try to sperate your docker build and push task in your scenario.
First docker build your image with changed code, and then docker run your new build image.
Then docker test your image, finally docker push it.
Related
need some help. I generated a report from image as artifact and I want this to be pass to another image, here's the script
.gitlab-ci.yml
variables:
website: "https://test.site/"
scantype: "zap-baseline.py"
report: "owasp-zap-report.html"
stages:
- build
scan:
stage: build
image: owasp/zap2docker-stable
allow_failure: true
before_script:
- mkdir -p /zap/wrk
script:
- pwd
- ls
- $scantype -t $website -m 1 -d -I -r $report
- cp /zap/wrk/$report .
artifacts:
paths: [$report]
upload:
stage: build
dependencies:
- scan
image: ubuntu:20.04
script:
- ls
Output
Running with gitlab-runner 15.1.0 (76984217)
on runner zgjy8gPC
Preparing the "docker" executor
00:04
Using Docker executor with image ubuntu:20.04 ...
Pulling docker image ubuntu:20.04 ...
Using docker image sha256:3bc6e9f30f51d2bbf9307fc9d0bdfc30caa38cf4e3b05a714230f9a9b3381d84 for ubuntu:20.04 with digest ubuntu#sha256:af5efa9c28de78b754777af9b4d850112cad01899a5d37d2617bb94dc63a49aa ...
Preparing environment
00:00
Running on runner-zgjy8gpc-project-1067-concurrent-0 via 1c8189df1d47...
Getting source from Git repository
00:03
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/test/test-owazp-pipeline/.git/
Checking out c90e0ea7 as cli...
Removing owasp-zap-report.html
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
Using docker image sha256:3bc6e9f30f51d2bbf9307fc9d0bdfc30caa38cf4e3b05a714230f9a9b3381d84 for ubuntu:20.04 with digest ubuntu#sha256:af5efa9c28de78b754777af9b4d850112cad01899a5d37d2617bb94dc63a49aa ...
$ ls
Cleaning up project directory and file based variables
00:01
Job succeeded
is there any key that I missed?
Please refer to https://docs.gitlab.com/ee/ci/yaml/#needs and the keyword needs
image : mcr.microsoft.com/dotnet/core/sdk:3.1
.deploy: &deploy
before_script:
- apt-get update -y
script:
- cd source/
- pip install -r requirements.txt
- python build_file.py > swagger.yml
I want to run the build_file.py file and write the output to swagger.yml. So to run the file I need to install python. How can I do that?
You can use a different Docker image for each job, so you can split your deployment stage into multiple jobs. In one use the python:3 image for example to run pip and generate the swagger.yml, then define it as an artifact that will be used by the next jobs.
Example (untested!) snippet:
deploy-swagger:
image: python:3
stage: deploy
script:
- cd source/
- pip install -r requirements.txt
- python build_file.py > swagger.yml
artifacts:
paths:
- source/swagger.yml
deploy-dotnet:
image: mcr.microsoft.com/dotnet/core/sdk:3.1
stage: deploy
dependencies:
- deploy-swagger
script:
- ls -l source/swagger.yml
- ...
You could (probably should) also make the swagger generation be part of previous stage and set an expiration for the artifact. See this blog post for example.
I have two repositories A & B.
Azure Repository A - Contains a python app
Azure Repository B - Contains .yml templates and .py scripts I want to run in the .yml templates
According to the documentations.. I cannot do this because when I expand the template into the calling repository A's pipeline.. it will be like a code directive and just inject the code.. it will not know or care about the .py files in the respoitory.
What are my options without doing all my .py routines as inline ?
Azure Repo A's Pipeline Yaml file
trigger: none
resources:
pipelines:
- pipeline: my_project_a_pipeline
source: trigger_pipeline
trigger:
branches:
include:
- master
repositories:
- repository: template_repo_b
type: git
name: template_repo_b
ref: main
stages:
- template: pipelines/some_template.yml#template_repo_b
parameters:
SOME_PARAM_KEY: "some_param_value"
Azure Repo B's some_template.yml
parameters:
- name: SOME_PARAM_KEY
type: string
stages:
- stage: MyStage
displayName: "SomeStage"
jobs:
- job: "MyJob"
displayName: "MyJob"
steps:
- bash: |
echo Bashing
ls -la
displayName: 'Execute Warmup'
- task: PythonScript#0
inputs:
scriptSource: "filePath"
scriptPath: /SOME_PATH_ON_REPO_B/my_dumb_script.py
script: "my_dumb_script.py"
Is there an option to wire in the .py files into a completely separate repo C... add C to resources of B templates.. and be on my way ?
EDIT:
I can see In Azure templates repository, is there a way to mention repository for a filePath parameter of azure task 'pythonScript'? but then how do I consume the python package.. can I still use the PythonScript task ? sounds like I would then need to call my pip packaged code straight from bash ??
I figured it out.. how to pip install py files in azure devops pipelines.. using azure repositories.. via a template in the same repo
just add a reference to yourself at the top of any template
In the consuming repo
repositories:
- repository: this_template_repo
type: git
name: this_template_repo
ref: master
then add a job, referencing yourself by that name
- job: "PIP_INSTALL_LIBS"
displayName: "pip install libraries to agent"
steps:
- checkout: this_template_repo
path: this_template_repo
- bash: |
python3 -m pip install setuptools
python3 -m pip install -e $(Build.SourcesDirectory)/somepypimodule/src --force-reinstall --no-deps
displayName: 'pip install pip package'
Gitlab version is 13.6.6
Gitlab-runner version is 11.2.0
my .gitlab-ci.yml:
image: "python:3.7"
before_script:
- pip install flake8
flake8:
stage: test
script:
- flake8 -max-line-length=79
tags:
- test
The only information obtained from Pipelines is script failure and the output of failed job is No job log. How can I get more detailed error output？
Using artifacts can help you.
image: "python:3.7"
before_script:
- pip install flake8
flake8:
stage: test
script:
- flake8 -max-line-length=79
- cd path/to
tags:
- test
artifacts:
when: on_failure
paths:
- path/to/test.log
The log file can be downloaded via the web interface.
Note:- Using when: on_failure will ensure that test.log will only be collected if the build fails, saving disk space on successful builds.
I am having some trouble with running my function app in python. When i push the function directly through func azure functionapp publish air-temperature-v2 --no-bundler. This publishes the function directly to portal.azure and the function works as expected. However if i try to commit and push to the Azure repos and it generates its build, everything is successful but when I try to run the function, it gives a module name 'pandas' not found error. It works fine locally & online (using no bundler command). My question is, how can I add the no bundler command in azure python pipeline? My yaml is as follows :
# Python package
# Create and test a Python package on multiple Python versions.
# Add steps that analyze code, save the dist with the build record, publish to a PyPI-compatible index, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/python
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
strategy:
matrix:
Python36:
python.version: '3.6'
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'
displayName: 'Use Python $(python.version)'
- script: |
python -m pip install --upgrade pip
pip install -r requirements.txt
displayName: 'Install dependencies'
- script: python HttpExample/__init__.py
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.SourcesDirectory)'
includeRootFolder: false
archiveType: 'zip'
archiveFile: '$(Build.ArtifactStagingDirectory)/Application$(Build.BuildId).zip'
replaceExistingArchive: true
verbose: # (no value); this input is optional
- task: PublishBuildArtifacts#1
#- script: |
# pip install pytest pytest-azurepipelines
# pytest
# displayName: 'pytest'
# ignore
- task: AzureFunctionApp#1
inputs:
azureSubscription: 'zohair-rg'
appType: 'functionAppLinux'
appName: 'air-temperature-v2'
package: '$(Build.ArtifactStagingDirectory)/Application$(Build.BuildId).zip'
startUpCommand: 'func azure functionapp publish air-temperature-v2 --no-bundler'
I have even tried add the no bundler command as the startup command but it still does not work.
This could be related to azure-function-core-tools version issue, please try following and deploy:
Please update your azure-function-core-tool version to the latest
Please try to deploy your build using below command:
func azure functionapp publish <app_name> --build remote
There was a similar issue raise sometime back, couldn't recall but this fix worked.
Alternatively , have you considere Azure CLI task to deploy azure function , here is a detailed article explaining the Azure CI CD using azure CLi for python.
https://clemenssiebler.com/deploy-azure-functions-python-azure-devops/
Hope it helps.