Develop Reference

ipython3 does not work in the terminal with python3.7

I recently upgraded from Python3.6 to Python3.7. Since I have upgraded, when I type in ipython3 in the terminal I get an error:
~$ ipython3
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3/dist-packages/IPython/__init__.py", line 48, in <module>
from .core.application import Application
File "/usr/lib/python3/dist-packages/IPython/core/application.py", line 25, in <module>
from IPython.core import release, crashhandler
File "/usr/lib/python3/dist-packages/IPython/core/crashhandler.py", line 28, in <module>
from IPython.core import ultratb
File "/usr/lib/python3/dist-packages/IPython/core/ultratb.py", line 124, in <module>
from IPython.utils import path as util_path
File "/usr/lib/python3/dist-packages/IPython/utils/path.py", line 18, in <module>
from IPython.utils.process import system
File "/usr/lib/python3/dist-packages/IPython/utils/process.py", line 19, in <module>
from ._process_posix import system, getoutput, arg_split, check_pid
File "/usr/lib/python3/dist-packages/IPython/utils/_process_posix.py", line 24, in <module>
import pexpect
File "/usr/lib/python3/dist-packages/pexpect/__init__.py", line 75, in <module>
from .pty_spawn import spawn, spawnu
File "/usr/lib/python3/dist-packages/pexpect/pty_spawn.py", line 14, in <module>
from .spawnbase import SpawnBase
File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 224
def expect(self, pattern, timeout=-1, searchwindowsize=-1, async=False):
^
SyntaxError: invalid syntax
Furthermore I have noticed that my jupyter-notebook does not seem to work with the python3 kernel now as well (I get the kernel dead error).
NOTEs:
ipython and jupyter-notebook are working with fine when I use them with Python2.
I am using Ubuntu 18.04 (Bionic Beaver) although I don't think this is relevant

async is a reserved keyword in python3.7 and an old version of pexpect is using async as variable. The solution is to upgrade pexpect.
For me, it was conflicting with apt-get installed python3, so I had to first uninstall it/them:
sudo apt-get remove python-pexpect python3-pexpect
And then
sudo pip3.7 install --upgrade pexpect

UPDATE Please update your installed packages.
This error for pexpect has been reported and closed already issue
In Python 3.7, async and await are now reserved keywords. This is what is breaking some of your installed packages.
If you do not need the new features in 3.7, roll back to 3.6 and wait until your packages are updated to support the new syntax in 3.7
What's New in Python 3.7

I just solve this problem by upgrading pexpect manually.
Download pexpect4.6 source code from https://github.com/pexpect/pexpect/releases/tag/4.6
Extract the source code, get into the folder and install the pexpect by:
sudo python3.7 setup.py install

As others have noted, having the current version of the pexpect package should solve this. In my case, the python3-pexpect package that I had installed through apt was stuck on an old version and I first had to remove it. Then installing the current pexpect package through pip3 fixed the problem.

This appears to be an issue related to async being keyword in Python 3.7
As given here, updating pipenv might be the solution

async is a reserved word in Python 3.7
You can edit the packages yourself if you feel comfortable doing it.
Here is a shell command that does just that:
sed "s/async/_async/g" "/usr/lib/python3/dist-packages/pexpect/spawnbase.py" > tmp.txt && cat tmp.txt > "/usr/lib/python3/dist-packages/pexpect/spawnbase.py" && rm tmp.txt
Otherwise, you might want to use Python 3.6 while waiting for an update.

jupyter with multiple version of python 3 (3.6 and 3.7) choosing the wrong site-packages

I have been banging my head at my desk for 3 days now and I am not sure what to do about this issue, so please if you have any idea of what is going on let me know.
Issue: When I am running a globally installed (outside of the virtual environment) jupyter notebook with a registered kernel (ipykernel installed in the virtual env) where the python3 -V of the virtual environment is 3.6.4 and the global python3 -V is 3.7.0, the jupyter notebook (is ran from wihin the active venv) crashes when I activate the venv kernel because it is trying to pick up site packages from the global python 3.7.0
[I 11:11:13.221 NotebookApp] Serving notebooks from local directory: /Users/cap/Desktop/Projects/lastresort
[I 11:11:13.221 NotebookApp] 0 active kernels
[I 11:11:13.221 NotebookApp] The Jupyter Notebook is running at:
[I 11:11:13.221 NotebookApp] http://localhost:8888/?token=cc92b513df1e586ce592bfc4fe641b9f2d76fdde480c6f07
[I 11:11:13.221 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 11:11:13.222 NotebookApp]
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://localhost:8888/?token=cc92b513df1e586ce592bfc4fe641b9f2d76fdde480c6f07&token=cc92b513df1e586ce592bfc4fe641b9f2d76fdde480c6f07
[I 11:11:13.438 NotebookApp] Accepting one-time-token-authenticated connection from ::1
[I 11:11:16.385 NotebookApp] Kernel started: 2ad09f31-36f6-48cf-b859-7f78fdb6adb8
[I 11:11:17.014 NotebookApp] Adapting to protocol v5.1 for kernel 2ad09f31-36f6-48cf-b859-7f78fdb6adb8
[I 11:11:19.398 NotebookApp] Starting buffering for 2ad09f31-36f6-48cf-b859-7f78fdb6adb8:cda1e6c165f3482482d80df7d2664e9b
[I 11:11:19.609 NotebookApp] Kernel shutdown: 2ad09f31-36f6-48cf-b859-7f78fdb6adb8
[I 11:11:19.630 NotebookApp] Kernel started: 9b90f202-97bf-43b1-ba50-64d6a1323405
Traceback (most recent call last):
File "/Users/cap/.pyenv/versions/3.6.4/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Users/cap/.pyenv/versions/3.6.4/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/lib/python3.7/site-packages/ipykernel_launcher.py", line 15, in <module>
from ipykernel import kernelapp as app
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/lib/python3.7/site-packages/ipykernel/__init__.py", line 2, in <module>
from .connect import *
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/lib/python3.7/site-packages/ipykernel/connect.py", line 18, in <module>
import jupyter_client
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/lib/python3.7/site-packages/jupyter_client/__init__.py", line 4, in <module>
from .connect import *
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/lib/python3.7/site-packages/jupyter_client/connect.py", line 23, in <module>
import zmq
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/vendor/lib/python3.7/site-packages/zmq/__init__.py", line 47, in <module>
from zmq import backend
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/vendor/lib/python3.7/site-packages/zmq/backend/__init__.py", line 40, in <module>
reraise(*exc_info)
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/vendor/lib/python3.7/site-packages/zmq/utils/sixcerpt.py", line 34, in reraise
raise value
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/vendor/lib/python3.7/site-packages/zmq/backend/__init__.py", line 27, in <module>
_ns = select_backend(first)
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/vendor/lib/python3.7/site-packages/zmq/backend/select.py", line 26, in select_backend
mod = __import__(name, fromlist=public_api)
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/vendor/lib/python3.7/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
from . import (constants, error, message, context,
ImportError: cannot import name 'constants'
As you can see from the few lines above, it is calling the wrong site-packages:
Traceback (most recent call last):
File "/Users/cap/.pyenv/versions/3.6.4/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Users/cap/.pyenv/versions/3.6.4/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/lib/python3.7/site-packages/ipykernel_launcher.py", line 15, in <module>
from ipykernel import kernelapp as app
File "/usr/local/Cellar/jupyter/1.0.0_5/libexec/lib/python3.7/site-packages/ipykernel/__init__.py", line 2, in <module>
This is my setup:
pyenv for managing multiple versions of python
pipenv for virtualenvs (still not super happy with this one)
How I set it up:
Install global python (I don't like messing around with pyenv for my default python, I keep to the most up to date version)
> brew install python3 # This install 3.7.0
Install pyenv
> brew install pyenv
Install python 3.6.4 (I need this for tensorflow)
> pyenv install 3.6.4
Install pipenv (I am not using the pip installer here)
> brew install pipenv
Create a project (I have a WORKON_HOME setup so my .venvs get
created in my ~/.venvs)
> cd Projects
> mkdire test_project
> cd test_project
> pipenv --python 3.6.4 #create the virtual env
Launch the virtual env
> pipenv shell
> python3 -V
> Python 3.6.4 # I have validated in another shell that global is still 3.7
> pyenv which python3
> /Users/cap/.pyenv/versions/3.6.4/bin/python3
Install ipykernel and setup
> pipenv install ipykernel
> ipython kernel install --user --name=lastresort-yyl8tfk8 --display-name "Python (lastresort-yyl8tfk8)"
Check if jupyter sees the new kernel (Note: I am still in the venv)
> jupyter kernelspec list
> lastresort-yyl8tfk8 /Users/cap/Library/Jupyter/kernels/lastresort-yyl8tfk8
Run the jupyter notebook and select the kernel
The jotebook runs fine, but when I get in and select the Python (lastresort-yyl8tfk8), the kernel restarts to load the lastresort-yyl8tfk8 kernel and I get the error from above, where it is getting confused with site-packages from 3.7.
I tried a bunch of things with paths. I also even resorted to ipython profile changes such as in here: iptyhon profile virtual env
I really need some help at least with a direction of how to solve this...
My ultimate goal is simple:
Be able to have multiple versions of python with corresponding ipykernels that I can run from their virtual environments after registering them with jupyter where the jupyter install is global. In this way I can keep my code and notebook in their project folders with venvs for packages. It is pretty simple actually, I just don't know what I am messing up.
Thank you all in advance!
UPDATE:
I just verified that if I remove the global jupyter notebook and install it in the venv, it works properly. I did not expect it not to, but still.
UPDATE:
It seems to be some PYTHONPATH shenanigans but I don't know how to fix it.
UPDATE:
Tried to run using python -m as suggested in comments, but it produced the same results. For some reason when I call the jupyter notebook (installed outside of the virtual environment) with an activated virtual environment (with correctly set python3.6 in ipython in there) it does not call the right 3.6 site packages.
I have seen so many people using that setup. I really don't know where to go from here.

TLDR
For me it worked installing jupyter in the virtual enviroment created by pipenv with the following command:
pipenv run pip install jupyter
Too Short, Wanna Read!
I had the same problem with python 3 versions so I started using pipenv as you show, with a simple Pipfile configuration as follows:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
numpy = "*"
pandas = "*"
matplotlib = "*"
[requires]
python_version = "3.7"
I thought that, as I specified python version to 3.7 in that config file, whenever I wanted to run pipenv run jupyter notebook, the kernel would be running under Python 3.7...
But when imported the librares indicated there from the notebook:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
I got the "non module installed error"
ModuleNotFoundError: No module named 'matplotlib'
And then in the same notebook, I checked in which version was the kernel running with the following code:
import sys
print(sys.version)
The output showed me that it was running under Python 3.6, so in that vesion I didn't have those modules installed, cause pipenv was intalling them for 3.7 version.
So I tried installing again jupyter but this time under the virtual enviroment created by pipenv, using the run command and pip tool:
pipenv run pip install jupyter
It worked for me!

To make pipenv use the Pipfile you want when running it from any path you want, you should add to your ~/.bashrc and ~/.profile the following env variable:
export PIPENV_PIPFILE="<path_to_the_Pipefile_you_want>"
Otherwise, pipenv will try to use a Pipefile from the path where it is being run.

Error setting environment variables in python

I am trying to use Bloomberg python API. I need to set BLPAPI_ROOT environment variable for this. I added,
export BLPAPI_ROOT="/home/user/Downloads/blpapi_cpp_3.6.3.1"
export PATH=$PATH:$BLPAPI_ROOT
to my .bashrc file and ran source .bashrc.
Now, when I open python shell and do,
print os.environ['BLPAPI_ROOT']
it gives me correct output. But when the same this runs inside the setup.py provided, it throws a
Traceback (most recent call last):
File "setup.py", line 27, in <module>
blpapiRoot = os.environ['BLPAPI_ROOT']
File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__
raise KeyError(key)
KeyError: 'BLPAPI_ROOT'
What am I missing here ?
System :
Ubuntu 12.04
Python 2.7

I would try using it the following way:
import os
try:
os.environ['BLPAPI_ROOT'] = "/home/user/Downloads/blpapi_cpp_3.6.3.1"
except EnvironmentError:
sys.exit(1)

This is quite old, but for anyone searching, you can get around this by setting sudo to keep the environmental variable BLPAPI_ROOT, a la keep environmental variables using sudo.
sudo visudo
Then add:
Defaults env_keep +="BLPAPI_ROOT"
You can now run:
sudo python setup.py install
and it should work just fine.

parallel installion of Python 2.7 and 3.3 via Homebrew - pip3 fails

I would like to make the jump and get acquainted with Python 3.
I followed the instructions found here with the installation working flawlessly.
I'm also able to use the provided virtualenv to create enviroments for Python 2 and Python 3 (Followed the instuctions here.). Unfortunalty pip3 fails when no virtualenv is activated. I need to use it to install global modules for python3.
This is the error message:
± |master ✓| → pip3
Traceback (most recent call last):
File "/usr/local/bin/pip3", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/site-packages/distribute-0.6.45-py2.7.egg/pkg_resources.py", line 51
def _bypass_ensure_directory(name, mode=0777):
^
SyntaxError: invalid token
It looks like pip3 is trying to access distribute of python2. Is there any workaround for this?

I was having the same problem as you were and I had
export PYTHONPATH="/usr/local/lib/python2.7/site-packages:$PYTHONPATH"
in my ~/.bash_profile. Removing that line solved the problem for me. If you have that or something like it in your ~/.bashrc or ~/.bash_profile, try removing it.

How do I work around this problem creating a virtualenv environment with a custom-build Python?

I need to run some code on a Linux machine with Python 2.3.4
pre-installed. I'm not on the sudoers list for that machine, so I
built Python 2.6.4 into (a subdirectory in) my home directory. Then I
attempted to use virtualenv (for the first time), but got:
$ Python-2.6.4/python virtualenv/virtualenv.py ENV
New python executable in ENV/bin/python
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Installing setuptools.........
Complete output from command /apps/users/dspitzer/ENV/bin/python -c "#!python
\"\"\"Bootstrap setuptoo...
" /apps/users/dspitzer/virtualen...6.egg:
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
'import site' failed; use -v for traceback
Traceback (most recent call last):
File "<string>", line 67, in <module>
ImportError: No module named md5
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
File "virtualenv/virtualenv.py", line 1488, in <module>
main()
File "virtualenv/virtualenv.py", line 529, in main
use_distribute=options.use_distribute)
File "virtualenv/virtualenv.py", line 619, in create_environment
install_setuptools(py_executable, unzip=unzip_setuptools)
File "virtualenv/virtualenv.py", line 361, in install_setuptools
_install_req(py_executable, unzip)
File "virtualenv/virtualenv.py", line 337, in _install_req
cwd=cwd)
File "virtualenv/virtualenv.py", line 590, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /apps/users/dspitzer/ENV/bin/python -c "#!python
\"\"\"Bootstrap setuptoo...
" /apps/users/dspitzer/virtualen...6.egg failed with error code 1
Should I be setting PYTHONHOME to some value? (I intentionally named
my ENV "ENV" for lack of a better name.)
Not knowing if I can ignore those errors, I tried installing nose
(0.11.1) into my ENV:
$ cd nose-0.11.1/
$ ls
AUTHORS doc/ lgpl.txt nose.egg-info/ selftest.py*
bin/ examples/ MANIFEST.in nosetests.1 setup.cfg
build/ functional_tests/ NEWS PKG-INFO setup.py
CHANGELOG install-rpm.sh* nose/ README.txt unit_tests/
$ ~/ENV/bin/python setup.py install
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Traceback (most recent call last):
File "setup.py", line 1, in <module>
from nose import __version__ as VERSION
File "/apps/users/dspitzer/nose-0.11.1/nose/__init__.py", line 1, in <module>
from nose.core import collector, main, run, run_exit, runmodule
File "/apps/users/dspitzer/nose-0.11.1/nose/core.py", line 3, in <module>
from __future__ import generators
ImportError: No module named __future__
Any advice?

Have you actually run "make install" on your custom python build? Usually you'll want to do something like
./configure --prefix=/path/to/installdir (other options)
make
make install
Note Prefix can be any directory you have full write-permissions to, for example I very often use $HOME/apps on shared-hosting environments.
Then run /path/to/installdir/bin/python, not the one from your build directory. This should create the correct variables, and after that you can install virtualenv. Might be best to install virtualenv using its setup.py:
cd virtualenv_source_dir
/path/to/installdir/bin/python setup.py install
This may require installing setuptools first, using the same method.
Then finally:
# Just to be safe
export PATH="/path/to/installdir/bin:$PATH"
virtualenv ~/ENV
~/ENV/bin/pip install somepackage # (and such)

In addition to Crast's suggestion of making sure you actually installed your custom compiled Python, you should also check that the custom Python can actually find its libraries. This is the hint you're getting with the message about PYTHONHOME. The import errors suggest you need to set in your .bashrc or appropriate shell configuration export PYTHONHOME=/path/to/python_installation.
Additionally, when you are trying to tell virtualenv to use a non-default version of python, you need to use the -p,--python flag, e.g.,
virtualenv --python=/path/to/python_installation/bin/python myenv
See also the related question, "Use different Python version with virtualenv".

I had the same error when trying to install on an existing directory that already had easy_install in lib/python2.6. I had to put a link from lib64/python2.6 to lib/python2.6.
I am not saying my fix is the right fix, rather, I'm pointing to another reason why you might get this error.

Now you can easily install Python as an unpriviledged user using Anaconda: http://continuum.io/downloads
It's similar to this question. Once of the answers details making a new environment, so you don't need to use virtual-env and avoid the occasional gotchas: Installing Anaconda into a Virtual Environment.
conda create -n myenv1 ipython scipy

I don't have enough rep to add this as a comment on #Crast's answer and this question is 4 years old, but this might be useful to someone. In Windows, you have to path out to python.exe, but it seems that in Linux/OS X you just path to the folder. Example:
Windows:
virtualenv -p <PATH TO YOUR DESIRED PYTHON.EXE> venv
Creates a virtual environment in subfolder "venv" in current directory.