Develop Reference

Using Jenkins variables/parameters in Python Script with os.path.join

I'm trying to learn how to use variables from Jenkins in Python scripts. I've already learned that I need to call the variables, but I'm not sure how to implement them in the case of using os.path.join().
I'm not a developer; I'm a technical writer. This code was written by somebody else. I'm just trying to adapt the Jenkins scripts so they are parameterized so we don't have to modify the Python scripts for every release.
I'm using inline Jenkins python scripts inside a Jenkins job. The Jenkins string parameters are "BranchID" and "BranchIDShort". I've looked through many questions that talk about how you have to establish the variables in the Python script, but with the case of os.path.join(),I'm not sure what to do.
Here is the original code. I added the part where we establish the variables from the Jenkins parameters, but I don't know how to use them in the os.path.join() function.
# Delete previous builds.
import os
import shutil
BranchID = os.getenv("BranchID")
BranchIDshort = os.getenv("BranchIDshort")
print "Delete any output from a previous build."
if os.path.exists(os.path.join("C:\\Doc192CS", "Output")):
shutil.rmtree(os.path.join("C:\\Doc192CS", "Output"))
I expect output like: c:\Doc192CS\Output
I am afraid that if I do the following code:
if os.path.exists(os.path.join("C:\\Doc",BranchIDshort,"CS", "Output")):
shutil.rmtree(os.path.join("C:\\Doc",BranchIDshort,"CS", "Output"))
I'll get: c:\Doc\192\CS\Output.
Is there a way to use the BranchIDshort variable in this context to get the output c:\Doc192CS\Output?

User #Adonis gave the correct solution as a comment. Here is what he said:
Indeed you're right. What you would want to do is rather:
os.path.exists(os.path.join("C:\\","Doc{}CS".format(BranchIDshort),"Output"))
(in short use a format string for the 2nd argument)
So the complete corrected code is:
import os
import shutil
BranchID = os.getenv("BranchID")
BranchIDshort = os.getenv("BranchIDshort")
print "Delete any output from a previous build."
if os.path.exists(os.path.join("C:\\Doc{}CS".format(BranchIDshort), "Output")):
shutil.rmtree(os.path.join("C:\\Doc{}CS".format(BranchIDshort), "Output"))
Thank you, #Adonis!

Python nameError help when importing from another file

I have a function in maya that imports in other functions and creates a shelf with buttons for specific functions. I have a function that has a scriptJob command that works fine. if I import that file in manually and not through the shelf button, but gives a NameError when using the shelf script to run it.
This is an example of the script
myShelf.py file:
import loopingFunction
loopingFunction.runThis()
loopingFunction.py file:
import maya.cmds as mc
def setSettings():
#have some settings set before running this
runThis()
def runThis():
print "yay this ran"
evalDeferred(mc.scriptJob(ro=True,ac=["'someMesh.outMesh',runThis"])
if I run this through the shelf function, I get a runThis nameError is not defined and if I try modifying the scriptJob command to loopingFunction.runThis, I get a nameError loopingFunction is not defined(not sure if using loopingFunction.runThis is even correct, to be honest)
Not sure how I can get around this without having to manually import in the function rather than through the shelf file.

Using string references for callback functions like this often leads to scope problems. (More on that, and why not to use strings, here)
If you pass the function directly to the callback as an object, instead of using a string, it should work correctly as along as you have actually imported the code.
in this case you want an evalDeferred -- do you actually need it? -- so it helps to add a little function around the actual code so that the scriptjob creation actually happens later -- otherwise it will get evaluated before the deferral is scheduled.
def runThis():
print "callback ran"
def do_scriptjob():
cmds.scriptJob(ro=True, ac=('someMesh.outMesh', runThis)
cmds.evalDeferred(do_scriptjob)
In both runThis and do_scriptjob I did not add the parens -- we are letting the evalDeferred and the scriptJob have the function objects to call when they are ready. Using the parens would pass the results of the functions which is not what you want here.
Incidentally it looks like you're trying to create a new copy of the scriptJob inside the scriptJob itself. It'd be better to just drop the runOnce flag and leave the scriptJob lying around -- if something in runThis actually affected the someMesh.outMesh attribute, your maya will go into an infinite loop. I did not change the structure in my example, but I would not recommend writing this kind of self-replicating code if you can possibly avoid it.

You have a problem of nested/maya scope variables
mc.scriptJob(ro=True,ac=["'someMesh.outMesh',runThis"]
This line is a maya command string that is evaluated in the main maya scope (like a global)
As your function have a namespace with the import : 'loopingFunction', you need to enforce it in the string command.
import loopingFunction
loopingFunction.runThis()
You should write
evalDeferred(mc.scriptJob(ro=True,ac=["'someMesh.outMesh',loopingFunction.runThis"])
If you want something more general, you can do :
def runThis(ns=''):
print "yay this ran"
if ns != '':
ns += '.'
evalDeferred(mc.scriptJob(ro=True,ac=["'someMesh.outMesh',{}runThis".format(ns)])
and then run in shelf :
import loopingFunction
loopingFunction.runThis('loopingFunction')
like this you can write any formof namepsaces :
import loopingFunction as loopF
loopF.runThis('loopF')

Accessing Robot Framework global variables from a prerun modifier

I am invoking Robot Framework on a folder with a command like following:
robot --name MyTestSuite --variablefile lib/global_variables.py --variable TARGET_TYPE:FOO --variable IMAGE_TYPE:BAR --prerunmodifier MyCustomModifier.py ./tests
MyCustomModifier.py contains a simple SuiteVisitor class, which includes/excludes tags and does a few other things based on some of the variable values set.
How do I access TARGET_TYPE and IMAGE_TYPE in that class? The method shown here does not work, because I want access to the variables before tests start executing, and therefore I get a RobotNotRunningError with message Cannot access execution context.
After finding this issue report, I tried to downgrade to version 2.9.1 but nothing changed.

None of public API's seem to provide this information but debugging the main code does provide an alternative way of obtaining it. It has to be said that this example code will work with version 3.0.2, but may not work in the future as these are internal functions subject to change. That said, I do think that the approach will remain.
As Robot Framework is an application, it obtains the command line arguments through it's main function: run_cli (when running from command line). This function is filled with the arguments from the system itself and can be obtained throughout every python script via:
import sys
cli_args = sys.argv[1:]
Robot Framework has a function that interprets the commandline argument list and make it into a more readable object:
from robot.run import RobotFramework
import sys
options, arguments = RobotFramework().parse_arguments(sys.argv[1:])
The argument variable is a list where all the variables from the command line are added. An example:
arguments[0] = IMAGE_TYPE:BAR
This should allow you to access the information you need.

Custom buildbot Step class - How to make your own, using remoteCommand/remoteShellCommand

I am trying to write a custom step, that will run few commands, and return a pas or fail based on certain conditions.
So far I was able to subclass ShellCommand; so I can execute a shell command on the slave. Now the next step is to write something that will execute not only one shell command, but various ones, and will analyze the results of these commands and act accordingly.
Altho I've not been successful in this endeavor. So far I am only able to subclass ShellCommand; but this allow me to run just one command.
I have found that ShellCommand uses buildstep.remoteCommand and remoteShellCommand; but my attempts to subclass buildstep.BuildStep has been unsuccessful.
The objective that I want to accomplish is to run a finite amount of python or shell commands (without write a shell script and call it from python; I was able to accomplish that), and analyze the results coming from these operations, so I can dictate if a step pass or fail, and what is logged.
So far this is what I have:
class myclass(build step.BuildStep)
def __init__(self, **kwargs):
buildstep.BuildStep.__init__(self, **kwargs)
def start(self):
cmd=buildstep.RemoteShellCommand({'command':"ls -la"})
self.setupEnvironment(cmd)
d=self.runCommand(cmd)
return d
This will run, and I will get an error on the remoteShellCommand line, saying
exceptions.TypeError: __init__() takes at least 3 arguments (2 given)
I've tried with both remoteCommand and remoteShellCommand, and the result is the same.
Checking init for both, I can't see 3 arguments but just the command, so I am not really sure what is wrong. I even tried to use **kwargs but I get an error saying that kwarg is not defined (there was an example from a blog, that would use kwargs; so I tried and it didn't work anyway).
This is the original documentation for the remoteShellCommand:
[Original Buildbot API documentation][1]
Do you know where I could find an example that actually show me how to accomplish this, or at least how do you actually use remoteCommand/remoteShellCommand? The original docs are a mess, even google returns few results that are even more obscure than the original docs.
Any suggestion is welcome; I've been running in circles for the past 3 days, and have no idea about where to look next.

One way to have a shellcommand execute multiple commands is to pass it a script instead of a command as this example shows:
class CoverageReport(ShellCommand):
def __init__(self):
self.coverage_path = '/var/www/coverage'
command = '''rm -rf htmlcov
coverage html
cp htmlcov/* %s/''' % self.coverage_path
description = 'Generating coverage report'
workdir = 'build/python'
self.url = "https://XXXXXXXX/coverage"
ShellCommand.__init__(self, command=command, description=description, workdir=workdir)
This Shellcommand does three things:
remove old report
Generate a new report
Copy the report to the www folder
Returned value will be the one of equivalent bash script, you can probably play with bash to return whatever you want
The other option is to add multiple steps to your build

Common variables in modules

I have three python files, let's call them master.py, slave1.py and slave2.py. Now slave1.py and slave2.py do not have any functions, but are required to do two different things using the same input (say the variable inp).
What I'd like to do is to call both the slave programs from master, and specify the one input variable inp in master, so I don't have to do it twice. Also so I can change the outputs of both slaves in one master program etc.
I'd like to keep the code of both slave1.py and slave2.py separate, so I can debug them individually if required, but when I try to do
#! /usr/bin/python
# This is master.py
import slave1
import slave2
inp = #some input
both slave1 and slave2 run before I can change the input. As I understand it, the way python imports modules is to execute them first. But is there some way to delay executing them so I can specify the common input? Or any other way to specify the input for both files from one place?
EDIT: slave1 and slave2 perform two different simulations being given a particular initial condition. Since the output of the two are the same, I'd like to display them in a similar manner, as well as have control over which files to write the simulated data to. So I figured importing both of them into a master file was the easiest way to do that.

Write the code in your slave modules as functions, import the functions, then call the functions from master with whatever input you need. If you need to have more stateful information, consider constructing an object.

You can do imports at any time:
inp = #some input
import slave1
import slave2
Note that this is generally considered bad design - you would be better off making the modules contain a function, rather than just having it happen when you import the module.

It looks like the architecture of your program is not really optimal. I think you have two files that execute immediately when you run them with python slave1.py. That is nice for scripting, but when you import them you run in trouble as you have experienced.
Best is to wrap your code in the slave files in a function (as suggested by #sr2222) and call these explicitly from master.py:
slave1.py/ slave2.py
def run(inp):
#your code
master.py
import slave1, slave2
inp = "foo"
slave1.run(inp)
slave2.run(inp)
If you still want to be able to run the slaves independently you could add something like this at the end:
if __name__ == "__main__":
inp = "foo"
run(inp)