Unable to get environment variables using os python - python

I recently created two environment variables in my terminal as shown below
export SPARK_HOME='/opt/spark/'
export HAIL_HOME='/home/ABCD/.pyenv/versions/3.7.2/envs/bio/lib/python3.7/site-packages/hail/'
When I use echo $SPARK_HOME or echo $HAIL_HOME, I am able to see the path as output
But, when I use the below os commands in jupyter notebook
os.getenv('SPARK_HOME') # able to get the output /opt/spark/
os.getenv('HAIL_HOME') # returns no output
I also tried defining the same variables from jupyter as well using os.putenv but even then,I see output only for SPARK_HOME
However, I am able to see in my terminal screen the environment variablesSPARK_HOME and HAIL_HOME using printenv command
Can help me understand what's the problem?

I realized that it doesn't produce output for HAIL_HOME because it is installed in my virtual environment. (see .pyenv which is a hidden folder for my virtual environment`)
However, if anyone can confirm this it's even better

use os.environ.get("SPARK_HOME").

Related

os.environ.get return None, but echo works properly

I have issue with "os.environ.get" command. I exported my variable using:
export MYVAR="123"
Then i used:
echo "$MYVAR"
And it returned my value. But when i am doing in python
print(os.environ.get('MYVAR'))
It returns None. For example:
print(os.environ.get('HOME'))
Works correctly. I tried using bashrc, zhrsc, bash_profile and it never works. I am using Linux Elementary 5.1. How can I solve this problem?
Looks like your export of the environment variable is only limited to the shell you are using.
If you set the environment variable system-wide, it will be available to be read in your python code as well.
Set environment variable in /etc/environment for Unix based system

Run Python script with Automator // why does it only work if I include export PATH=/usr/local/bin:$PATH and what does it mean?

I was trying to run a Python script via Mac's Automator and the command is very straight forward:
"cd /Users/myname/Desktop/project && python3 myprojectapp.py".
However, every time I tried to run it, Automator raised an error such as ModuleNotFoundError. This was however, impossible since I had all libraries (e.g. Pandas) installed and running the command in the Terminal as written above worked flawlessly.
Now, I've read somewhere for a similar problem to just include:
"export PATH=/usr/local/bin:$PATH" before the command and it worked. Now, before I go on with my life, I would like to understand what exactly this extra line does and how it affects Automator to the point of making the script work.
Thank you in advance!
That command basically modifies the environment variable PATH and puts the directory /usr/local/bin before everything that is currently in PATH. However, that command is temporary, and the environment variable PATH is restored when the session closes.
What could be happening is the python you're running in terminal and the python Automator is running are different./usr/local/bin probably contains the same python version as you are using in terminal. Take a look at ~/.bash_profile to see if something similar to export PATH=/usr/local/bin:$PATH is in there.
Another way to check is to type which python in both and see if it points to the same python. You probably have yet another python somewhere in the list of directories in your PATH variable.
It's common to use virtual python environments to keep track of which python is running and to experiment with python without messing with system python. Examples of these include: Anaconda and virtualenv.

How to get path to virtualenv in Windows?

First of all, i'm using bash in windows.
I'm trying to write the correct paths to my virtualenv in VSCode but I must be doing something wrong
Any help will be really appreciated! thank you guys!
"python.pythonPath": "C\\Users\\Angel\\AppData\\Local\\Packages\\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\\LocalState\\rootfs\\home\\aquesada\\.local\\share\\virtualenvs\\videoclub_django_new-coDPKRdg\\bin\\python",
"python.linting.pylintPath": "C\\Users\\Angel\\AppData\\Local\\Packages\\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\\LocalState\\rootfs\\home\\aquesada\\.local\\share\\virtualenvs\\videoclub_django_new-coDPKRdg\\bin\\pylint",
"python.venvPath": "C:\\Users\\Angel\\AppData\\Local\\Packages\\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\\LocalState\\rootfs\\home\\aquesada\\.local\\share\\virtualenvs"
virtual env as environmental variable
A virtual environment will set an environmental variable.
In Windows, you should be able to view that variable via echo %VIRTUAL_ENV%
In your Python code, you can grab that env var via os.environ['VIRTUAL_ENV']
VS Code config
python.venvPath should work. A few more things for you to try:
adjust path formatting: a single backslash as delimiter should work; AFAIK Windows only requires two consecutive backslashes at the start of the path
restart VS Code after updating configuration
use the Command Palette to select the Python interpreter from your virtual environment

Why is Anaconda source activate non-existent?

I successfully created two separate Python environments in Anaconda, yet seem to be unable to activate either one of them. I have tried to read up on this topic as much as possible here on Stackoverflow, yet no solution did resolve my issue. I added information asked for by comments in this question (Anaconda Environment Doesnt activate).
When trying to activate an environment, the console output is -bash: activate: No such file or directory.
The output of which conda is /Users/username/anaconda3/bin/conda.
The output of type source is source is a shell builtin.
When trying which activate, the shell returns nothing.
In my bin folder, I also seem not to have an activate executable, but only one which is called activate-global-python-argcomplete.
Why do I lack the standard activate file and how I could resolve this issue best?
I experience a similar problem. In my case, the problem is related to the use of the tcsh, but activate only supports bash and zsh.
You can check your current used shell with the command
echo $0.
You have to use a compatible shell in order to use the source activate command.
I think you happened to install a buggy version of anaconda which was quickly patched.
conda update conda
should get you back up and running.
Start a new terminal and try again. Alternatively, type hash -r and see if it helps.
Do not do this!!!, this broke my dnf
First I tried to point source to a activate file,
source /usr/lib64/python3.6/venv/scripts/common/activate environment
That appeared to work, but no actual values got updated. I tried the answer by mattexx, but it complained conda wasn't installed, so I used.
conda install conda
I had to have root permission since it was being installed to the root environment, but after that everything appears to work.
That is what broke my fedora installation
tcsh was the problem for me. Changed to bash and all is ok.
In this case, the problem might be that the virtual environment was created in /Users/username/anaconda3/envs/ and hence the activate will be in:
/Users/username/anaconda3/envs/NAME_OF_YOUR_VIRTUAL_ENV/bin/activate
Hence to activate the environment you could run:
source /Users/username/anaconda3/envs/NAME_OF_YOUR_VIRTUAL_ENV/bin/activate
My problem was that file /opt/pycharm/plugins/terminal/.zshrc (using ZSH) contained line source $JEDITERM_SOURCE. That variable contained correct paths as two parameters <path-to-anaconda-activate>/activate <path-to-anaconda>, but apparently they got interpreted as a single parameter: <path-to-anaconda-activate>/activate <path-to-anaconda>.
I solved that prepending eval to that line, making it eval source $JEDITERM_SOURCE. Now, the variable expands correctly.
If you're using bash then make the same change to /opt/pycharm/plugins/terminal/jediterm-bash.in.
I'm using the bash shell inside pycharm and fixed the problem by making this change:
[~/opt/pycharm-community/plugins/terminal]$ diff jediterm-bash.in.old jediterm-bash.in
65c65
< source "$JEDITERM_SOURCE"
---
> eval source "$JEDITERM_SOURCE"
Essentially making the same change recommended above by https://stackoverflow.com/users/1564931/netchkin, but to ~/opt/pycharm-community/plugins/terminal/jediterm-bash.in.
This fixed my existing projects. But any new projects don't seem to try and activate the conda environment at all. It seems as if new projects created in Pycharm 2017.1.2 don't set the JEDITERM_SOURCE variable when starting a session in the terminal plugin.
There is an open issue on this at https://youtrack.jetbrains.com/issue/PY-23417
Looks like there was a big change to the conda activate script in version 4.4.0. See https://conda.io/docs/release-notes.html. If I downgrade conda to 4.3.34 then I can get conda virtual environments to work in the pycharm terminal again.

Linux profile.d environment variables don't work with cx_oracle in Python

This is a bit of a continuation from my previous question: cx_Oracle does not recognize location of Oracle software installation for installation on Linux.
After I was able to get cx_oracle installed properly, I wanted to set up my environment so the environment variables don't have to be exported every time.
To do this, I wrote a shellscript that included these two export statements:
export ORACLE_HOME=/home/user1/instantclient_12_1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME
And saved this .sh file into the /etc/profile.d/ folder.
When I log into the server again with PuTTY, the echo statements say that the environment variables are there:
# echo $ORACLE_HOME
/home/user1/instantclient_12_1
# echo $LD_LIBRARY_PATH
:/home/user1/instantclient_12_1
But when I run some python code with cx_oracle, I get an error:
ImportError: libclntsh.so.12.1: cannot open shared object file: No such file or directory
The code only runs again when I re-enter the export commands for the environment variables. After I do that, the code using cx_oracle runs fine.
Why don't the environment variables work properly even though they show up when I do the echo command? And how do I get the environment variables to persist properly?
The guides I read say to do it with a shell script in /etc/profile.d/ because it's better to not edit /etc/profile directly.
Update:
I tried adding the two export lines to /etc/profile, but I still get the same problem where the environment variables are there when I echo, but I still get this error when trying to use cx_oracle in python:
ImportError: libclntsh.so.12.1: cannot open shared object file: No such file or directory
Am I missing some key thing about defining environment variables?
Second Update:
I tried initializing the environment with a shell script that I planned to run with the code that calls cx_Oracle:
Contents of StartServer.sh:
export ORACLE_HOME=/home/user1/instantclient_12_1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME
python3 ./UDPDBQuery.pyc
And I try to run the code in the background by doing:
bash StartServer.sh &
But I still run into that same error as before, as if I did not put in the environment variables. It only works if I export the variables myself, and then run the code again. The code also stops running in the background when I log out. I'm still very confused as to why it isn't working.
Are environment variables not usable by cx_oracle unless I manually do the export statement for them?
Alright, I found out that one of the two environment variables was not exporting properly with the .sh file in /etc/profile.d, and doing $LD_LIBRARY_PATH would give me No such file or directorytclient_12_1, but $ORACLE_HOME would give me /home/user1/instantclient_12_1/: is a directory.
The way I solved this was to split the export statements into two separate shell scripts in profile.d.
Everything works now.

Categories

Resources