Airflow is returning an error when trying to run a DAG saying that it can't find an environment variable, which is odd because it's able to find 3 other environment variables that I'm storing as a Python variable. No issues with those variables at all.
I have all 4 variables in ~/.profile and have also done
export var1="varirable1"
export var2="varirable2"
export var3="varirable3"
export var4="varirable4"
Under what user does airflow run? I've done those export commands under sudo as well, so I thought they would be picked up by airflow when it runs the dag
Is it maybe because airflow uses non-login shell? Have you tried putting these lines in : ~/.bashrc instead of ~/.profile ?
As per this answer, the variables should be put in /etc/default/airflow (on Debian/Ubuntu) or /etc/sysconfig/airflow (on Centos/Redhat).
If you are just running a local instance you should be able to use environment variables like you expect. Remember that you need to set them in the shell that is running the webserver and scheduler though. If these are in your .profile, you may need to run source ~/.profile.
Related
The parameters that I need to pass to my Python program use some environment variables.
Example:
This is how I run it on the terminal:
export BUCKET="/tmp/bucket"
python main.py --input $BUCKET/input --output $BUCKET/output
On PyCharm, I created a Run/Debug configuration with an environment variable called BUCKET and passed the following string as parameters: --input $BUCKET/input --output $BUCKET/output.
When PyCharm executes the program, it doesn't pick up the value of BUCKET as /tmp/bucket. It considers $BUCKET to be a string.
I also tried using ${BUCKET} instead of $BUCKET but that doesn't work either.
Is there some way to pass variables?
Note: The reason I want to do this is that I have a large number of parameters in my real code. I have only provided a toy example above. I want to be able to update an environment variable only in one place.
I encountered the same problem a few days back. I found a plugin called EnvFile.
Using this you can have an env file exported before running the script. After installing it, you'll get an extra tab EnvFile in your configuration. Select your environment file there. It is specific to the configuration. Now, every time you run the configuration environment variables will be exported.
My .env file
I have created a virtual environment with pyenv virtualenv 3.5.9 projectname for developing a django project.
How can I set environment variables for my code to use?
I tried to add the environment variable DATABASE_USER in /Users/developer/.pyenv/versions/projectname/bin/activate
like this:
export DATABASE_USER="dbuser"
When I tried to echo $DATABASE_USER an empty string gets printed.
Tried to install zsh-autoenv
And now I can echo $DATABASE_USER and get the value set in the .autoenv.zsh file.
But I can't seem to get the environment variable to be available to my django code:
If I try to os.getenv('DATABASE_USER', '') in the python shell inside the virtualenv, I get ''
What could be wrong? Is the zsh-autoenv variables just available for the zsh shell and not python manage.py shell ?
I was wondering a similar thing, and I stumbled across a reddit thread where someone else had asked the same question, and eventually followed up noting some interesting finds.
As you noticed, pyenv doesn't seem to actually use the bin/activate file. They didn't say what the activation method is, but like you, adding environment variables there yielded no results.
In the end, they wound up installing autoenv, which bills itself as directory-based environments. It allows you to create an .env file in your directory, and when you cd to that directory, it runs the .env file. You can use it for environment variables, or you could add anything else to it.
I noticed on the autoenv page that they say you should probably use direnv instead, as it has better features and is higher quality software. Neither of these are Python or pyenv specific, and if you call your python code from outside of the directory, they may not work. Since you're using pyenv, you're probably running your code from within the directory anyway, so I think there's a good chance either one could work.
I am using cx_oracle library and have to specify the following using linux
export LD_LIBRARY_PATH=/opt/oracle/12.1.0_64bit/product/12.1.0/client_2/lib
export ORACLE_HOME=/opt/oracle/12.1.0_64bit/product/12.1.0/client_2
this makes my code work fine
the issue is this doesn't save across every session so I decided to add it to my script instead like this
os.environ["ORACLE_HOME"] = /opt/oracle/12.1.0_64bit/product/12.1.0/client_2
os.environ["LD_LIBRARY_PATH"] = /opt/oracle/12.1.0_64bit/product/12.1.0/client_2
but its not working. I added that line right after my imports so its the first thing ran, but its not changing the variables.
when I echo those two variables after the script they're still both empty.
On Linux, you need to set LD_LIBRARY_PATH before the process (e.g Python) starts.
Otherwise you can add the env vars to files like ~/.bashrc, or have a wrapper script like p.sh:
#!/bin/sh
export LD_LIBRARY_PATH=/opt/oracle/12.1.0_64bit/product/12.1.0/client_2/lib
export ORACLE_HOME=/opt/oracle/12.1.0_64bit/product/12.1.0/client_2
python $#
Do you need the full client? Do you have other Oracle software installed? If no & no, then consider using Oracle Instant Client and using ldconfig so you don't need to set environment variables, see the installation instructions at https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html#ic_x64_inst
I am aware that you can use os.environ['key'] to access an environment variable, but it appears that this function is localized to environment variables for the signed in user. I need to access the system environment variables. Is this possible?
You cannot do this using os.environ in python. You can, however, use os.system to run a commands such as:
sudo -Hiu $user env | grep $var
to access a variable from another user. This is a bit hacky, but it works. And your options are limited here.
Also -- why do you want to? You should have a process for setting the environment variables before this script is run at all, ideally.
This is a bit of a continuation from my previous question: cx_Oracle does not recognize location of Oracle software installation for installation on Linux.
After I was able to get cx_oracle installed properly, I wanted to set up my environment so the environment variables don't have to be exported every time.
To do this, I wrote a shellscript that included these two export statements:
export ORACLE_HOME=/home/user1/instantclient_12_1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME
And saved this .sh file into the /etc/profile.d/ folder.
When I log into the server again with PuTTY, the echo statements say that the environment variables are there:
# echo $ORACLE_HOME
/home/user1/instantclient_12_1
# echo $LD_LIBRARY_PATH
:/home/user1/instantclient_12_1
But when I run some python code with cx_oracle, I get an error:
ImportError: libclntsh.so.12.1: cannot open shared object file: No such file or directory
The code only runs again when I re-enter the export commands for the environment variables. After I do that, the code using cx_oracle runs fine.
Why don't the environment variables work properly even though they show up when I do the echo command? And how do I get the environment variables to persist properly?
The guides I read say to do it with a shell script in /etc/profile.d/ because it's better to not edit /etc/profile directly.
Update:
I tried adding the two export lines to /etc/profile, but I still get the same problem where the environment variables are there when I echo, but I still get this error when trying to use cx_oracle in python:
ImportError: libclntsh.so.12.1: cannot open shared object file: No such file or directory
Am I missing some key thing about defining environment variables?
Second Update:
I tried initializing the environment with a shell script that I planned to run with the code that calls cx_Oracle:
Contents of StartServer.sh:
export ORACLE_HOME=/home/user1/instantclient_12_1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME
python3 ./UDPDBQuery.pyc
And I try to run the code in the background by doing:
bash StartServer.sh &
But I still run into that same error as before, as if I did not put in the environment variables. It only works if I export the variables myself, and then run the code again. The code also stops running in the background when I log out. I'm still very confused as to why it isn't working.
Are environment variables not usable by cx_oracle unless I manually do the export statement for them?
Alright, I found out that one of the two environment variables was not exporting properly with the .sh file in /etc/profile.d, and doing $LD_LIBRARY_PATH would give me No such file or directorytclient_12_1, but $ORACLE_HOME would give me /home/user1/instantclient_12_1/: is a directory.
The way I solved this was to split the export statements into two separate shell scripts in profile.d.
Everything works now.