I am looking for best practices for maintaining different python installations (or environments) simultaneously, specifically, with regards to achieving the following 3 objectives/requirements:
Note: I am using Anaconda installation for Python
Requirements
Requirement1 - Jupyter Notebook convenience
All python interpreters (either in different installations or environments) should be available in jupyter notebook in the list of kernels.
Requirement2 - Command line convenience
On the command line it should be easy to specify any python interpreter available on my machine to run a py script, something like: 'py2 test.py' or 'py3_env1 test.py'
Requirement3 - Automation convenience
Run the py scripts using the *.bat files, using any python interpreter available on the machine.
Choices
As implicit above, there are 2 broad choices here:
Choice1 - Single installation, multiple environments
...i.e. maintain a 'single' python installation folder but create multiple environments in it using 'conda'.
Choice2 - Mulitple installations, single environment each
... i.e. maintain different python interpreters in separate installation folders with only the root environment.
If I make any any of the two Choices above, I can satisfy some, but not all 3 requirements.
For instance, if I make Choice1, I can satisfy Requirement1 easily, but not 2 and 3; and with Choice2, I can satisfy Requirement2 and Requirement3 only. Details as below.
Satisfying requirements
Satisfying Requirement1 - Jupyter Notebook convenience
If I make Choice1 (single installation, multiple environments), I can satisfy Requirement1 using the solutions posted here. It suggests registering ipykernels in all of the environments.
With Choice2, is it possible for jupyter notebook to see all kernels spread across many python installations? I couldnt make it work.
Satisfing Requirement2 - Command Line convenience
If I make Choice2, then I can satisfy Requirement2 by using Python Launcher or creating symlinks as suggested here
With Choice1, it doesn't work since Python Launcher doesn't recognise different environments (yet!). So, is there any other way to make it possible? Can I, lets say, use some kind of command alias, something like a short-cut command 'py' that refers to the 'root environment' python.exe file, and another one 'py2' that refers to another environment's python.exe. And if I do that, will calling 'py2' be robust in the sense that it will call correct site-packages folder? (Note, we haven't activated the environment here, so the windows 'PATH' may not point to the correct folders). I would assume that activating the environment is necessary, but I am not sure.
Satisfing Requirement3 - Automation Convenience
If I make Choice2, I can probably satisfy Requirement3 using Python Launcher as well (just like above). Also, not to mention, my script can also use shebang operator to mention which python it wants to invoke. However, I don't want to hardcode the python version in the script and keep the flexibility in the command itself.
If I make Choice1, I thought that a *.bat file like below will work:
conda activate py3_env1
python test.py
deactivate
So, I intend to activate the environment first and then call the py file. However, this doesn't work. Calling this bat file executes just the first line, activates the environment and then nothing happens.
So, in short, I am looking to have a consistent setup that allows me to invoke different python versions from all mediums - notebook, command prompt and automated bat files - very conveniently.
Any guidance or recommendation around best practices are very appreciated.
Related
I'm teaching a beginners python class, the environment is Anaconda, VS Code and git (plus a few extras from a requirements.txt).
For the windows students this runs perfectly, however the mac students have an existing python (2.7) to contend with.
The windows students (i.e. they have a windows computer), their environment when they debug matches their console environment. However the mac students seem to be locked to their 2.7 environment.
I've tried aliasing, as suggested here and here
alias python2='python'
alias python='python3'
alias pip2='pip'
alias pip='pip3'
I've modified the .bash_profile file
echo 'export PATH="/Users/$USER/anaconda3/bin:$PATH"' >>.bash_profile
Both of these seem to work perfectly to modify their Terminal environments, when launched externally to VS Code. Neither seem to do anything to the environment launched from [cmd]+[`].
I've also tried conda activate base in the terminal, which seems to have no effect on a python --version or a which python
They can run things using python 3, but that means that they need to remember that they are different to the other 2/3 of the students. It's othering for them, and more work for me!
The students are doing fine, launching things from their external terminal, but it would streamline things greatly if the environments could be as consistent as possible across the OSs.
Whilst they are complete beginners, they can run a shell script. They currently have one that installs pip requirements and vs code extensions.
Is there a configuration that will keep the terminal in line with the debug env?
In my opinion the best practice is to create Python virtual environments (personally I love using conda environments, especially on Mac where you stuck with unremovable old Python version). Then VSCode will automatically (after installing very powerful Python extension) find all your virtual environments. This way you will teach your students a good practice of handling Python zoo a.k.a. package incompatibilities. Terminal environments settings will be consistent with VSCode, without being dependent on unneeded any more aliases. Obviously, virtual environments are OS independent, so you will be more consistent and remove unnecessary confusion between different students.
The additional bonus of the virtenvs is that you can create one exactly according to your requirements.txt and switch from one to another with a single click (in terminal it takes two commands: deactivate -> activate).
You can read more about how to handle Python virtual environments on VSCode site
Given the aliases are run just once and are not persistent in .bash_profile, python targets the default interpreter rather than the expected conda python3 interpreter.
Try to symlink conda's python3 executable to capture the python namespace
ln -sf /Users/$USER/anaconda3/bin/python3 /Users/$USER/anaconda3/bin/python
This will create or update the symlink. Use the same approach for pip and pip3.
Python in vscode let's you select which interpreter will be used to run the scripts.
It is in settings under "python.pythonPath", just set it to point to the interpreter of choice.
It can be set on a project basis as well (which is how you ensure that a project that has a virtual environment will execute using that interpreter and packages), you just select Workspace in the settings pane and add the desired python interpreter there.
I am trying to use Anaconda and conda environments to allow Python programs for data acquisition* etc. to run from the (Anaconda) command line on Windows. The set up will be that the Python programs are installed to a particular location (cloned from Github), within %PATH% or whichever environment variable is more appropriate.
From an Anaconda command prompt in another directory and a particular conda environment, I want (both myself and other users) to be able run either python test.py <args> or test.py <args> (either solution is acceptable) and have a system wide conda environment run its Python to execute the program. test.py can/will have an appropriate shebang set.
Right now the python test.py calls the correct Python within the active conda environment, but cannot find the test.py program as Python won't search the %PATH% or similar looking for the program. test.py does something (Windows does not complain that the executable can't be found, and I've been playing with the file associations to get this far), but doesn't appear to start Python - a simple print function or raise statement as the only entry in the file does nothing.
I've tried setting file associations in Windows, but this hasn't changed anything. I've copied the py.exe/pyw.exe across to the Anaconda environments, with no change.
Is this something that can be done within Anaconda, or am I going to have to fall back on installing base Python directly and trying to use the launcher mechanism there?
Note that I'm also intending to deploy these programs on Raspbian, so any solutions, including non-Anaconda ones, that will work cross platform there would be worth extra effort on my part.
*these programs have significant usage of library packages for accessing external USB/GPIB/serial/ethernet connected lab equipment and use matplotlib, scipy, etc., hence the desire for a cloneable conda environment as the base environment.
It turns out the correct answer to this is fairly simple, but is fairly hard to find explained well. This might be a little clearer than the other answers I found:
Install the standalone launcher from pylauncher and add #!/usr/bin/env python shebangs to your scripts.
This should register .py files to Python.File and will find your Anaconda Pythons in appropriate environments. If you don't have a non-Anaconda python, it will use the Anaconda base environment (these two facts were the key element I was missing from various other answers around this problem that I had looked at, including the documents on python.org).
If you have a Python from python.org installed, then a standalone command line shell will use that, defaulting to Python 2.x, then Python 3.x. With #!/usr/bin/env python shebang, then a regular command shell will try to use python.org pythons first, then the Anaconda base environment. An Anaconda prompt will use the active environment. #! /usr/bin/env python2 or python3 will try to use python.org pythons only and fail if they are not found.
Installing Python 2.7 from python.org installers (and letting the installer set the file associations) will break pylauncher, and reinstalling will not fix it. Instead, set Computer\HKEY_CLASSES_ROOT\Python.File\Shell\open\command default value to "C:\WINDOWS\py.exe" "%L" %* to revert back to the pylauncher set up (assuming you used the launchwin.* packages to install it).
What I want to do is to create a GUI for my tensorflow program.
However, I can't seem to run my conda tensorflow environment inside my python script.
I have already tried this code
subprocess.call("activate tensorflow", shell=True)
but it still says that it cant find the tensorflow module.
Not in general. A virtualenv has its own copy of the Python interpreter, possibly with a different version than the system-wide (most problematically, one can be Python 2 and the other Python 3). Once you have already started one Python interpreter you cannot hot-swap it with another.
If you know that both versions of the interpreter are the same, then you could try something like adding the virtualenv lib and site-packages directories at the beginning of sys.path before doing anything else, but that's not really clean and even then some things may not work correctly (besides, I think the default virtualenv and the Anaconda virtualenv directory structures are somewhat different). Also, even if this worked, it would definitely not be possible to dance around from one virtualenv to another after you start importing modules (once you import a module, there is no way to "unimport" it, and every future reference will use the one loaded first, no matter what sys.path says).
It would be better to just have a different process to work in the virtualenv. You can run a second interpreter directly from the first one, or call a batch/shell script to do it for you, or replace the first Python script with a batch/shell script, etc.
I need to perform some custom settings for the python interpreter: specifically related to pyspark: the "interpeter" will actually be the spark-submit (aka pyspark) shell script. The intent is to be able to run pyspark jobs within the python console. Running within a Run Configuration would also be just fine: this would be an alternate approach.
I use IJ Ultimate - which has good python support: except well maybe for this particular use case.
Let us compare to pycharm - and specifically an ability to customize the interpreter - including setting local, remote, or virtualenv:
The Intellij Ultimate seems to lack those options: instead it is pointed to the libraries for a python sdk. That will not be sufficent for the given use case:
Here is the dropdown: notice there is no way to add a custom python interpreter.
So is there a way in Intellij to set the interpreter path? I want to set it to $SPARK_HOME/bin/pyspark ?
PyCharm and IntelliJ have the exact same options to add and configure Python code.
PyCharm just makes it easier.
Those windows do not have anything for interpreters
Pretty sure it does... You add interpreters here.
bin/pyspark is not an interpreter, it is a shell script. You just set the regular Python interpreter.You also need to add the Pyspark libraries. (See below)
Then, you configure the environment variables here (Run Configurations) (see those below)
As far as PySpark libraries go, you have to add these (use the full path, not variables)
$SPARK_HOME/python/
$SPARK_HOME/python/lib/py4j-X.X-src.zip
You also need to set these variables in the Edit Configurations window shown
SPARK_HOME = path to spark
PYTHONPATH = path to py4j-X.X-src.zip (also need to append the path to the current python interpreter's directory, I believe)
Ref:
How to link PyCharm with PySpark?
https://github.com/ybenoit/pyspark-ide-starter (it's got some French in it, but it's still readable)
And here's a video of some code running
https://www.youtube.com/watch?v=u-P4keLaBzc
I am trying to run script in bash from python, but I am currently working in a virtualenv, and when my script calls on a specific program, I get "usr/bin/env: luajit: No such file or directory".
When I run the same script inside a separate bash window (outside the virtualenv), it runs perfectly.
I have a feeling it is because this program is not properly being pointed to, but have no idea how to tell my virtualenv to look for that program.. How do I get around this?
I am on Ubuntu 14.04, using python 2
The answer lies in environment variables. PATH should contain the location where your luajit is.
See setting an environment variable in virtualenv for some ways to automatically set environment variables in a virtualenv (but two of the currently suggested ways require wrappers and one - editing a stock script). There's no magic: virtualenv edition by Allison Kaptur describes the (rather simple) magic behind virtualenv's work so you may be able to find an even better place to put the variable assignment in.