I've been looking and so far been unable to find a way of validating/linting my Jenkinsfile. At least not by using tox, pycharm or another way outside of visual code for example (I did saw some examples of that, more or less).
Does anyone know of a way to do this? I would like to perform some simple checks, like:
return a warning if an environment variable inside the file is used but isn't declared (so I know I have to check if it is set on server level, for example).
Creating some custom checks would be a huge plus: e.g. if strings, without variables, use single quotes instead of double.
Jenkins can validate, or "lint", a Declarative Pipeline from the command line before actually running it. This can be done using a Jenkins CLI command.
Linting via the CLI with SSH
# ssh (Jenkins CLI)
# JENKINS_SSHD_PORT=[sshd port on controller]
# JENKINS_HOSTNAME=[Jenkins controller hostname]
ssh -p $JENKINS_SSHD_PORT $JENKINS_HOSTNAME declarative-linter < Jenkinsfile
More info available at : Linter
Additional reference: Validate Jenkinsfile
Related
When I tried to debug some Deep Learning projects, I noticed that there are many projects use the shell script as the method of passing parameters. We can easily run these projects by bash run.sh in command line, but how to debug these kind of projects in IDEs like Pycharm?
I have tried to use some methods of python to call system commands like: os.system('bash run.sh') in debugger.py and simply debug the debugger.py. But the breakpoint set by pycharm in this method do not work, and will be directly ignored. What is the reason?
I also try to parse the passed parameters in the shell script and add these parameters in the debug configuration of pycharm like this:
debug configuration
but it seems too troublesome, especially when the incoming parameters are complex.
Is there any elegant solution to debug a python project with shell script as the entry?
Or is it not normative to write too complex parameter processing flow in shell script?
i got a python script in my project. Depending on whether it was run manually or via Jenkins, I need to respond to it differently.
The only solution I came up with so far was to set an environment variable via Jenkins and check this var in the script.
Has anyone had a similar problem and could somehow solve it differently ?
If you only want to avoid setting an env variable, but still prefer using an existing env var you might want to use https://plugins.jenkins.io/envinject/#EnvInjectPlugin-BuildCauses which exposes following variables in the Env variables:
BUILD_CAUSE=MANUALTRIGGER
BUILD_CAUSE_MANUALTRIGGER=true in case of manual trigger. It internally checks which cause(s) triggered the build, which can be many like SCMTriggerCause, TimerTriggerCause etc.
This question is very similar to this one but for PyCharm.
I need to use aws-vault to access AWS resources in my script, but this seems to be impossible to accomplish in PyCharm debugging mode. It gives ability to enter script path, parameters, environment variables and there is also external tools functionality, but neither of these work.
Here is the format that works in shell:
aws-vault exec ${AWS_PROFILE} -- script.py
I thought that I've almost arrived at a solution by using external tools and setting the program to "aws-vault" and its arguments to "exec your-profile -- $FilePath$", but it wants to run the script in $FilePath$, finish and only after completion run the debugged script in PyCharm (which is the same one as the one inserted by $FilePath$).
How it would work for my case is by running needed script in debug mode in conjunction with external tool, so the script would go into arguments of the external tool and run as one command.
There are ways to deal with this by launching PyCharm from command line with aws-vault as a prefix or editing its .desktop file and writing the prefix directly into the Exec field, but the app needs to be restarted when AWS profile has to be changed.
Any ideas would be appreciated, thanks.
I was able to do this by installing the envfile plugin in PyCharm. This plugin can read in a .env file when starting a process. Basically I did the following:
Create a script that generates a .env file, envfile.env and name the script generate.sh
This generate.sh script is a shell script that basically does: aws-vault exec $AWS_PROFILE -- env | grep AWS_ > envfile.env, so all the aws creds are in the envfile.env. Possibly add other environment variables if you need so.
Execute command above at least once.
Install the envfile plugin in pycharm.
In the Run configuration, a new tab appears with 'EnvFile'. In this tab, enable the EnvFile. Add the generated envfile.env (see previous).
Do Tool / External Tools and create an external tool for the generate.sh. This way you can execute the script from PyCharm.
Again in the Run configuration add a Before Launch that executes the External Tool generate.sh.
Warning, the temporary aws-creds are in the plaintext envfile.env.
I know unittests and use write them daily.
They get executed during development and CI.
Now I have test which I would like to ensure on the production system:
PYTHONIOENCODING must be "utf8"
Above I used the verb "test", this means I want to check the state. This question is not about how to do this.
AFAIK the unittest framework can't help me here, since it only gets executed during development and CI.
How to solve this in the python world withou re-inventing the wheel?
Above is only an example. There are several other things next to PYTHONIOENCODING which I would like to check.
Next use case for these checks: Some days ago we had an issue on the production sever. The command-line tool convert gets used and some versions are broken and create wrong results. I would like to write a simple check to ensure that the convert tool on the production server is not broken.
Straightforward approach (Checking)
Put this near the start of the code:
import os
if os.environ.get('PYTHONIOENCODING', '').lower() not in {'utf-8', 'utf8'}:
raise EnvironmentError("Environment variable $PYTHONIOENCODING must be set to 'utf8'")
Alternative solution (Ensuring)
In one of the projects I code for, there's a "startup script", so instead of running python3 main.py, we run this in production:
bash main.sh
whose content is rather simple:
#!/bin/bash
export PYTHONIOENCODING=utf8
exec /usr/bin/env python3 main.py
testinfra
If you want to write and run tests against the deployment infrastructure, you can use the testinfra plugin for pytest. For example, test for a simple requirement of validating an environment variable on target machine could look like:
def test_env_var(host):
assert host.run_expect((0,), 'test "$PYTHONIOENCODING" == "utf8"')
This infrastructure test suite can be developed in a separate project and invoked before the actual deployment takes place (for example, we invoke the infra tests right after the docker image is built; if the tests fail, the image is not uploaded to our private image repository/deployed to prod etc).
I want to write a python script to get list of "jenkins" variables and show them.
I wrote this, but it returned all environment variables instead of jenkins variables.
import os
print os.environ
How can I get them in python script?
Jenkins variables are environment variables. There isn't really anything that separates them.
You can use EnvInject plugin to clean the Jenkins environment of all OS environment variables that are inherited, this way you are left with "just Jenkins environment variables", but careful as this may break your tools as none of the paths will be set.
If you want to view just the Build Parameters, you can use the Jenkins API for each job, or parse the job's XML file
There is a specific set of built-in variable that a base Jenkins install adds to each execution step; here is a list from the reference:
https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-below
As Slav says, these are just like any other environment variable, and if you want to access one you would just ask for it by name e.g., for $WORKSPACE, you would use os.environ['WORKSPACE']
You could enumerate the variables at the url above in a list ['WORKSPACE', ...] and print them that way...