Trying to run the wordcount.py example from the data-flow quickstart example via pycharm, and I ran into an issue when parsing the command line arguments that contain environment variables.
When I set the environment variables and run the script in the terminal with the same paramaters it works just fine:
python wordcount.py --input gs://dataflow-samples/shakespeare/kinglear.txt --output gs://%BUCKET%/wordcount/outputs --runner DataflowRunner --project %PROJECT% --temp_location gs://%BUCKET%/tmp/
I tried the following variations of parsing the environment variables - none worked: ${ENV}, $ENV$, %ENV%.
I am working on Windows 10, PyCharm version 2019.3.1.
When you pass environment variables as values for the parameters (e.g., --project, --temp_location) to Pycharm configuration, it takes those variables as "string" values instead of their real values which you set on the first screenshot "User environment variables". I did a quick search for many related threads on StackOverflow but not find a solution so I came up with my own one, that is, using your current settings and replacing the values after parsing arguments in your code:
# Args parsing here
if args.project == "${PROJECT}":
args.project = os.environ.get("PROJECT")
args.temp_location = args.temp_location.replace("${BUCKET}", os.environ.get("BUCKET"))
However, I think you should consider using it when
you work with PyCharm more frequently than Terminal.
there are many arguments referring to environment variables and you have to change their values for different settings.
Hope Pycharm will support this feature soon.
Related
I am trying to retrieve the parameters set from the jenkins build into my python script, but am having trouble. I understand the parameters set from here:
Are set as env variables and all I have to do in python is do:
# Env variables
UPDATE_DATA = os.environ.get('update_data')
ALL_BUILDS = os.environ.get('all_builds')
However I am getting None for those values. When I do an echo of those parameters in my jenkins script before my python script runs, I could see them being printed out correctly. However, for some reason python does not see them. If I go manually into a terminal and export a variable and run my python script, it works.. So I'm completely lost here.
Jenkins server is running on linux. Using python 2.7
You can use the boolean variable like this:
Output:
It seems like when I ran the python script in the Jenkins config (not inside a file within my project) like how #souravatta suggested, it found the env variable. So that means the env variable Jenkins is setting, is on a different instance somehow (even though they are on the same computer, same user). I just did a workaround where I wrote the env variables to a file and then just read that file in my python script.
I have followed tutorial from this site : http://www.nltk.org/howto/twitter.html
Right now my problem is I don't know what are the values in Environment Variables need to be input. I am using Windows 7.
On a Unix-like system (including MacOS), you will set the variable something like this:
export TWITTER="/path/to/your/twitter-files"
Rather than having to give this command each time you start a new session, it's advisable to add it to your shell's configuration file, e.g. to .bashrc.
On a Windows machine, right click on “My Computer” then select Properties > Advanced > Environment Variables > User Variables > New...
I try input the value with path to 'twitter-file'
When I run the code in python-notebook:
from nltk.twitter import Twitter
tw = Twitter()
tw.tweets(keywords='love, hate', limit=10) #sample from the public stream
I got this error :
ValueError: Supply a value to the 'subdir' parameter or set the TWITTER environment variable.
Thank you.
I had the same issue (Under Windows 10.) A little background, to make sure we're on the same page- I'm using Anaconda 2 and running the exact same tutorial you reference in a Jupyter notebook. I set the same user environment variable you did:
environment variable screenshot
At this point I kept coming up with the same ValueError when calling Twitter(). I ran: % env to see exactly what was in my working environment and found that the TWITTER environment variable I had set was missing. I reasoned that Jupyter was pulling the environment variables when the program started.
The Fix- I closed Jupyter and Anaconda 2 (and all related windows.) On restarting Jupyter and again running: % env The TWITTER environment variable was now listed and the examples from the tutorial are returning results:
tw = Twitter()
tw.tweets(keywords='love, hate', limit=10) #sample from the public stream
Tutorial Results screenshot
I apologize for the verbose reply- I hope this helps.
subprocess.run('export FOO=BAR', shell=True)
This simply doesn't work, and I have no idea why.
All I am trying to do I set an environment variable from my python (3.5.1) script, and when I run the above line, nothing happens. No errors are raised, and when I check the environment variable myself, it has not been set.
Other shell commands with subprocess.run() do work, such as ls and pwd, but not export.
.run() was added in Python 3.5 (in case you didn't recognise it), but I have also tried the above line with .call() and .Popen(), with no change in results.
I am aware that I can set environment variables in python with os.environ['FOO'] = "BAR", but I will be using shell commands a lot in my project, and I expect that I will need to string multiple commands together, which will make using export easier than os.environ.
My project will run on Linux, which is what my machine is running on.
It works fine; however, the variable setting only exists in the subprocess. You cannot affect the environment of the local process from a child.
os.environ is the correct solution, as it changes the environment of the local process, and those changes will be inherited by any process started with subprocess.run.
You can also use the env argument to run:
subprocess.run(["cmdname", "arg1", "arg number 2"], env=dict(FOO='BAR', **os.environ))
This runs the command in a modified environment that includes FOO=BAR without modifying the current environment.
Lately I have been doing some interactive work in Python.
My setup is an IPython notebook running on a server that uses a grid engine to manage jobs.
Today I was trying to get an IPython cluster going following an example posted here that uses subprocess.Popen to start a the cluster.
I couldn't get the example to work so I tried opening up the IPython/Jupyter terminal emulator and typing in the ipcluster start command and the cluster started right up!
After playing around with things for a while I realized that if I typed env in the terminal emulator I got a different list of environment variables than when I looked at the os.environ variable in Python. The source of the problem seemed to be that the PATH variables were different.
Now I know that I can change the PATH variable in os.environ, but I'm wondering why it is different in the first place? I know very little about environment variables, so this may be a stupid question, but I would have assumed that a terminal emulator and notebook running on the exact same node in the exact same IPython notebook server would have had the exact same environment variables.
Any insight on why the environment variables in the terminal and notebook might be different will be greatly appreciated.
Update: In case it matters, the server I am working on uses the Univa Grid Engine. Also I have noticed that it seems to make a difference whether I use qrsh or qsub to start the notebook server.
Previously I had been using qsub, but by starting the notebook server with qrsh I eliminate many of the differences between env and os.environ. There are still differences, but much fewer. Still not sure what any of that means though:)
As per manual page for qsub, qsh, qrsh, to propagate current shell environment to the job use the -V option:
-V Available for qsub, qsh, qrsh with command and qalter.
Specifies that all environment variables active within the qsub utility be exported to the context of the job.
All environment variables specified with -v, -V or the DISPLAY variable provided with -display will be exported to the defined JSV instances only optionally when this is
requested explicitly during the job submission verification. (see -jsv option above or find more information concerning JSV in jsv(1))
I'm attempting to run python on a system that doesn't allow me to set environment variables. Is there a commandline flag to python that will set PYTHONHOME? I looked here: http://docs.python.org/release/2.3.5/inst/search-path.html but didn't see anything.
So, hopefully something like this:
python -magical_path_flag /my/python/install test.py
EDIT
Thanks for the responses everyone. I'm embarrassed to say I actually meant PYTHONHOME, not PYTHONPATH. (That's what I deserve for asking a question at 1:30 AM.) I've edited my quesiton.
Here's some more info. I'm trying to get python running on Android. I can run python -V no problem, but if I try and execute a script, I get:
I/ControlActivity(18340): Could not find platform independent libraries <prefix>
I/ControlActivity(18340): Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Unfortunately when using the ProcessBuilder and changing the environment variables on Android, it says that they're not modifiable and throws an exception. I'm able to pass all the command line flags I want, so I was hoping I could set PYTHONHOME that way.
I've tried creating a wrapping shell script which exports PYTHONHOME and then calls python but that didn't work. (Got the same error as before.)
Thanks,
Gabe
You could simply set it in your script -- sys.path is a regular, modifiable list. Something like:
import sys
sys.path.append("/path/to/libraries")
should do the trick
In UNIXy shells, you can set an environment variable just for the duration of one command by prepending the command with the environment variable setting:
$ PYTHONPATH=/my/python/install python test.py
If none of the other answers suit you, you can write a wrapper script that temporarily sets the environment variable then exec's your other script. For example:
python mywrapper.py -magical_path_flag /my/python/install test.py