Develop Reference

Using Jenkins variables/parameters in Python Script with os.path.join

I'm trying to learn how to use variables from Jenkins in Python scripts. I've already learned that I need to call the variables, but I'm not sure how to implement them in the case of using os.path.join().
I'm not a developer; I'm a technical writer. This code was written by somebody else. I'm just trying to adapt the Jenkins scripts so they are parameterized so we don't have to modify the Python scripts for every release.
I'm using inline Jenkins python scripts inside a Jenkins job. The Jenkins string parameters are "BranchID" and "BranchIDShort". I've looked through many questions that talk about how you have to establish the variables in the Python script, but with the case of os.path.join(),I'm not sure what to do.
Here is the original code. I added the part where we establish the variables from the Jenkins parameters, but I don't know how to use them in the os.path.join() function.
# Delete previous builds.
import os
import shutil
BranchID = os.getenv("BranchID")
BranchIDshort = os.getenv("BranchIDshort")
print "Delete any output from a previous build."
if os.path.exists(os.path.join("C:\\Doc192CS", "Output")):
shutil.rmtree(os.path.join("C:\\Doc192CS", "Output"))
I expect output like: c:\Doc192CS\Output
I am afraid that if I do the following code:
if os.path.exists(os.path.join("C:\\Doc",BranchIDshort,"CS", "Output")):
shutil.rmtree(os.path.join("C:\\Doc",BranchIDshort,"CS", "Output"))
I'll get: c:\Doc\192\CS\Output.
Is there a way to use the BranchIDshort variable in this context to get the output c:\Doc192CS\Output?

User #Adonis gave the correct solution as a comment. Here is what he said:
Indeed you're right. What you would want to do is rather:
os.path.exists(os.path.join("C:\\","Doc{}CS".format(BranchIDshort),"Output"))
(in short use a format string for the 2nd argument)
So the complete corrected code is:
import os
import shutil
BranchID = os.getenv("BranchID")
BranchIDshort = os.getenv("BranchIDshort")
print "Delete any output from a previous build."
if os.path.exists(os.path.join("C:\\Doc{}CS".format(BranchIDshort), "Output")):
shutil.rmtree(os.path.join("C:\\Doc{}CS".format(BranchIDshort), "Output"))
Thank you, #Adonis!

Methods to avoid hard-coding file paths in Python

Working with scientific data, specifically climate data, I am constantly hard-coding paths to data directories in my Python code. Even if I were to write the most extensible code in the world, the hard-coded file paths prevent it from ever being truly portable. I also feel like having information about the file system of your machine coded in your programs could be security issue.
What solutions are out there for handling the configuration of paths in Python to avoid having to code them out explicitly?

One of the solution rely on using configuration files.
You can store all your path in a json file like so :
{
"base_path" : "/home/bob/base_folder",
"low_temp_area_path" : "/home/bob/base/folder/low_temp"
}
and then in your python code, you could just do :
import json
with open("conf.json") as json_conf :
CONF = json.load(json_conf)
and then you can use your path (or any configuration variable you like) like so :
print "The base path is {}".format(CONF["base_path"])

First off its always good practise to add a main function to go with each class to test that class or functions in the file. Along with this you determine the current working directory. This becomes incredibly important when running python from a cron job or from a directory that is not the current working directory. No JSON files or environment variables are then needed and you will obtain interoperation across Mac, RHEL and Debian distributions.
This is how you do it, and it will work on windows also if you use '\' instead of '/' (if that is even necessary, in your case).
if "__main__" == __name__:
workingDirectory = os.path.realpath(sys.argv[0])
As you can see when you run your command, the working directory is calculated if you provide a full path or relative path, meaning it will work in a cron job automatically.
After that if you want to work with data that is stored in the current directory use:
fileName = os.path.join( workingDirectory, './sub-folder-of-current-directory/filename.csv' )
fp = open( fileName,'r')
or in the case of the above working directory (parallel to your project directory):
fileName = os.path.join( workingDirectory, '../folder-at-same-level-as-my-project/filename.csv' )
fp = open( fileName,'r')

I believe there are many ways around this, but here is what I would do:
Create a JSON config file with all the paths I need defined.
For even more portability, I'd have a default path where I look for this config file but also have a command line input to change it.

In my opinion passing arguments from command line would be best solution. You should take a look at argparse . This allows you to create nice way to handle arguments from the command line. for example:
myDataScript.py /home/userName/datasource1/

%USERPROFILE% env variable for python

I am writing a script in Python 2.7.
It needs to be able to go whoever the current users profile in Windows.
This is the variable and function I currently have:
import os
desired_paths = os.path.expanduser('HOME'\"My Documents")
I do have doubts that this expanduser will work though. I tried looking for Windows Env Variables to in Python to hopefully find a list and know what to convert it to. Either such tool doesn't exist or I am just not using the right search terms since I am still pretty new and learning.

You can access environment variables via the os.environ mapping:
import os
print(os.environ['USERPROFILE'])
This will work in Windows. For another OS, you'd need the appropriate environment variable.
Also, the way to concatenate strings in Python is with + signs, so this:
os.path.expanduser('HOME'\"My Documents")
^^^^^^^^^^^^^^^^^^^^^
should probably be something else. But to concatenate paths you should be more careful, and probably want to use something like:
os.sep.join(<your path parts>)
# or
os.path.join(<your path parts>)
(There is a slight distinction between the two)
If you want the My Documents directory of the current user, you might try something like:
docs = os.path.join(os.environ['USERPROFILE'], "My Documents")
Alternatively, using expanduser:
docs = os.path.expanduser(os.sep.join(["~","My Documents"]))
Lastly, to see what environment variables are set, you can do something like:
print(os.environ.keys())
(In reference to finding a list of what environment vars are set)

Going by os.path.expanduser , using a ~ would seem more reliable than using 'HOME'.

How to set and retrieve environment variable in Python

I need to set a environment variables in the python, and I try the commands bellow
import os
basepath = os.putenv('CPATH','/Users/cat/doc')
basepath = os.getenv('CPATH','/Users/cat/doc')
And when I print the varible, they are not set:
print basepath
None
What i'm doing wrong?
Rephrasing the question, I would like to create a base path based on a evironment variable. I'm testing this:
os.environ["CPATH"] = "/Users/cat/doc"
print os.environ["CPATH"]
base_path=os.getenv('C_PATH')
And when I try to print the basepath:
print basepath
it always return
None

Try this one.
os.environ["CPATH"] = "/Users/cat/doc"
Python documentation is quite clear about it.
Calling putenv() directly does not change os.environ, so it’s better to modify os.environ.
Based on the documentation, the os.getenv() is available on most flavors of Unix and Windows. OS X is not listed.
Use rather the snippet below to retrieve the value.
value = os.environ.get('CPATH')

Use os.environ:
os.environ['CPATH'] = '/Users/cat/doc'
print os.environ['CPATH'] # /Users/cat/doc
print os.environ.get('CPATH') # /Users/cat/doc
See the above link for more details:
If the platform supports the putenv() function, this mapping may be
used to modify the environment as well as query the environment.
putenv() will be called automatically when the mapping is modified.
Note Calling putenv() directly does not change os.environ, so it’s
better to modify os.environ.

It's a good practice to restore the environment variables at function completion.
You may need something like the modified_environ context manager describe in this question to restore the environment variables.
Usage example:
with modified_environ(CPATH='/Users/cat/doc'):
call_my_function()

What is the easiest way in python to modify linux config file?

I have some python scripts that configure linux computers. One of the tasks is to modify a configuration file for subversion. This file, ~/.subversion/servers is very simple and looks like this:
# store-passwords = no
# store-plaintext-passwords = no
# store-ssl-client-cert-pp = no
# store-ssl-client-cert-pp-plaintext = no
... lots of other options ...
The task of my script is to find a required option, for example store-plaintext-passwords and to set it to specified value, for example yes. The problem is: the script can run multiple times on same machine, so if it is run first time this option can be just commented, if it is run second time it can be uncommented and set to yes, third run can point out that it is uncommented - but set to no etc. Currently i have a rather complex code that search file for the string, splits it for comment/name/value, uncomments it if needed, changes value if needed and replaces it. Maybe it's an easier way possible?

The ~/.subversion/servers file is in INI format.
So you can use the ConfigParser for implementing whatever you need.
http://docs.python.org/library/configparser.html