NOTE: There have been several EDITs to the question, as per comments. They are indicated below, and separated by lines.
As of now, the only remaining issue seems to be that numpy cannot load, possibly (but not certainly) due to two alternative python 3.8 systems present.
I have updated my msys2 system a couple of months ago.
That apparently included an update from python 3.7 to 3.8, but which left me with two broken pythons: I can start python when it is 3.7, but there are no associated packages, and I cannot start python when it is 3.8, which is the version holding packages.
I do not know what went wrong with that, or what did I do wrong.
I just noticed this now with the first time I mean to use python again after the upgrade.
I will describe here a sequence of steps I followed and what I found.
I will post supporting code below for clarity.
I can start python, but pandas (e.g.) and many other packages are not found in python.
Checking further, /mingw64/lib/python3.7/site-packages is essentially empty (surely emptied when upgrading to 3.8).
Looking for the pandas package, I found I have one version installed.
The pandas version is for python 3.8, surely upgraded from 3.7.
I redirected PYTHONPATH from 3.7 to 3.8
Now I cannot even start python.
EDIT Now I can start python, with some misconfiguration issues (i.e., partially fixed).
Now the question is
How can I fix python3.8, which gives the error below?
ImportError: cannot import name 'open_code' from 'io' (unknown location)
How can I fix python3.8, which gives the problems below?
New problems:
5.1. I should have python pointing to 3.8, and also fix packages.
5.2. Some modules are not found, some other are.
Note: I don't know if Msys2 upgrade breaks python2-pyqt5 has anything to do with this.
Related:
https://github.com/tox-dev/tox/issues/1334
https://github.com/yan12125/python3-android/issues/19
https://python-forum.io/Thread-Fatal-Python-error-init-sys-streams-can-t-initialize-sys-standard-streams-Attribute
TL;DR: Supporting code
pandas not found
$ python
Python 3.7.4 (default, Jul 11 2019, 10:29:54)
[GCC 9.1.0] on msys
Type "help", "copyright", "credits" or "license" for more information.
Reading /home/user1/.pythonrc
readline is in /usr/lib/python3.7/lib-dynload/readline.cpython-37m.dll
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>>
pandas is actually installed
$ pacman -Sl | grep python | grep installed
mingw64 mingw-w64-x86_64-python 3.8.2-1 [installed: 3.8.1-1]
mingw64 mingw-w64-x86_64-python-apipkg 1.5-1 [installed]
...
mingw64 mingw-w64-x86_64-python-pandas 1.0.3-1 [installed: 1.0.1-1]
...
mingw64 mingw-w64-x86_64-python2-setuptools 44.1.0-1 [installed: 42.0.2-1]
msys python 3.7.4-1 [installed]
msys python2 2.7.17-1 [installed]
My pandas version is for python 3.8
$ pacman -Ql mingw-w64-x86_64-python-pandas | head -5
mingw-w64-x86_64-python-pandas /mingw64/
mingw-w64-x86_64-python-pandas /mingw64/lib/
mingw-w64-x86_64-python-pandas /mingw64/lib/python3.8/
mingw-w64-x86_64-python-pandas /mingw64/lib/python3.8/site-packages/
mingw-w64-x86_64-python-pandas /mingw64/lib/python3.8/site-packages/pandas-1.0.1-py3.8.egg-info/
I redirected PYTHONPATH from 3.7 to 3.8
Changed
export PYVERSION="3.7"
export PYTHONDIR2="${MINGW_HOME}/lib/python${PYVERSION}"
export PYTHONPATH="${PYTHONDIR2}:${PYTHONDIR2}/site-packages"
to
export PYVERSION="3.8"
...
Now I cannot even start python.
EDIT: Old problem:
$ python
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Traceback (most recent call last):
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/io.py", line 54, in <module>
ImportError: cannot import name 'open_code' from 'io' (unknown location)
Aborted (core dumped)
New problems:
$ python --version
Python 3.7.4
$ type python
python is hashed (/usr/bin/python)
$ ls /usr/bin/python
/usr/bin/python
$ python3.8
Python 3.8.2 (default, Apr 9 2020, 13:17:39) [GCC 9.3.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Reading C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/home/user1/.pythonrc
Module readline not available.
Traceback (most recent call last):
File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/home/user1/.pythonrc", line 42, in <module>
del os, atexit, readline, rlcompleter, save_history, historyPath
NameError: name 'readline' is not defined
>>> import readline
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
>>> import zipfile
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pandas/__init__.py", line 16, in <module>
raise ImportError(
ImportError: Unable to import required dependencies:
numpy: DLL load failed while importing _ctypes: No se puede encontrar el módulo especificado.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
from . import core
File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/__init__.py", line 106, in <module>
from . import _dtype_ctypes
File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/_dtype_ctypes.py", line 25, in <module>
import _ctypes
ImportError: DLL load failed while importing _ctypes: No se puede encontrar el módulo especificado.
>>> exit()
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'readline'
EDIT #2: Adding info as requested.
I just noticed I don't have pip. That matches the fact that I never installed any package with pip...
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/c/Users/user1/OneDrive/Documents/soft-hard-ware/linux-ubuntu:/c/Users/user1/OneDrive/Documents/soft-hard-ware/linux-ubuntu/rsync:/c/Users/user1/Documents/appls_mydocs/science-math-visualization/gp524-win64-mingw_3/gnuplot/bin:/mingw64/bin
$ echo $PYTHONPATH
/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8:/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages
$ echo $PYTHONSTARTUP
/home/user1/.pythonrc
$ which python3.8
/mingw64/bin/python3.8
$ python3.8 -m pip freeze
C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/bin/python3.8.exe: No module named pip
$ python3.8 -c "import sys; print(sys.builtin_module_names)"
('_abc', '_ast', '_codecs', '_collections', '_functools', '_imp', '_io', '_locale', '_operator', '_signal', '_sre', '_stat', '_string', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', '_winapi', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'msvcrt', 'nt', 'sys', 'time', 'winreg', 'xxsubtype')
EDIT #3:
Posting as requested, plus additional info.
$ cat .pythonrc
import os
print( "Reading " + os.path.realpath(__file__) )
# enable syntax completion
try:
import readline
print( "readline is in " + readline.__file__ )
except ImportError:
print("Module readline not available.")
else:
import rlcompleter
readline.parse_and_bind("tab: complete")
# From https://docs.python.org/2/tutorial/interactive.html
# Add auto-completion and a stored history file of commands to your Python
# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
# bound to the Esc key by default (you can change it - see readline docs).
#
# Store the file in ~/.pystartup, and set an environment variable to point
# to it: "export PYTHONSTARTUP=~/.pystartup" in bash.
import atexit
import os
#import readline
#import rlcompleter
historyPath = os.path.expanduser("~/.pyhistory")
def save_history(historyPath=historyPath):
import readline
readline.write_history_file(historyPath)
if os.path.exists(historyPath):
#import readline
readline.read_history_file(historyPath)
atexit.register(save_history)
del os, atexit, readline, rlcompleter, save_history, historyPath
I don't see why which python3.8 and PYTHONPATH were out of sync:
$ cygpath -w $(which python3.8)
C:\Users\user1\Documents\appls_mydocs\PortableApps\MSYS2Portable\App\msys32\mingw64\bin\python3.8.exe
$ echo $PYTHONPATH
/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8:/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages
$ cygpath -w /c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8
C:\Users\user1\Documents\appls_mydocs\PortableApps\MSYS2Portable\App\msys32\mingw64\lib\python3.8
$ cygpath -w /c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages
C:\Users\user1\Documents\appls_mydocs\PortableApps\MSYS2Portable\App\msys32\mingw64\lib\python3.8\site-packages
$ which python
/usr/bin/python
I seem to have two incomplete/broken python installations (3.7, 3.8).
I don't know what led to an "incomplete" upgrade.
A couple of observations (see code below):
python points to 3.7
readline is available for 3.7 and not for 3.8.
I don't know why.
pandas (and many others) is available for 3.8 and not for 3.7.
Many of them won't import in 3.8 anyway, due to missing dependencies (which I guess are available in 3.7).
I don't know why.
python3.8 reports the location of .pythonrc in Windows format, while 3.7 reports it in Cygwin format.
Is that normal?
Removing PYTHONPATH does not help.
$ unset PYTHONPATH
$ python3.8
Python 3.8.2 (default, Apr 9 2020, 13:17:39) [GCC 9.3.0 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Reading C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/home/user1/.pythonrc
Module readline not available.
Traceback (most recent call last):
File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/home/user1/.pythonrc", line 42, in <module>
del os, atexit, readline, rlcompleter, save_history, historyPath
NameError: name 'readline' is not defined
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pandas/__init__.py", line 16, in <module>
raise ImportError(
ImportError: Unable to import required dependencies:
numpy: DLL load failed while importing _ctypes: No se puede encontrar el módulo especificado.
>>> exit()
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'readline'
$ python
Python 3.7.4 (default, Jul 11 2019, 10:29:54)
[GCC 9.1.0] on msys
Type "help", "copyright", "credits" or "license" for more information.
Reading /home/user1/.pythonrc
readline is in /usr/lib/python3.7/lib-dynload/readline.cpython-37m.dll
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>> exit()
EDIT #4:
I checked for all python packages, and I found something strange. It seems the basic python installation comprises two separate packages, one mingw64 and another msys.
I wouldn't know what does each one do.
$ pacman -Sl | grep "python" | grep "installed"
...
mingw64 mingw-w64-x86_64-python 3.8.2-2 [installed]
...
msys python 3.8.2-1 [installed: 3.7.4-1]
...
On one hand, there was a mismatch in installed versions mingw64 (3.8.2-2) vs msys (3.7.4-1).
On the other hand, the subversions available are not exactly the same for both (3.8.2-2) vs (3.8.2-1).
Anyway, I proceeded upgrading msys python, and this fixed things significantly.
$ pacman -S python
...
$ python
Python 3.8.2 (default, Apr 16 2020, 15:31:48)
[GCC 9.3.0] on msys
Type "help", "copyright", "credits" or "license" for more information.
Reading /home/user1/.pythonrc
Traceback (most recent call last):
File "/home/user1/.pythonrc", line 9, in <module>
import readline
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/readline.py", line 6, in <module>
from pyreadline.rlmain import Readline
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/__init__.py", line 12, in <module>
from . import logger, clipboard, lineeditor, modes, console
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/clipboard/__init__.py", line 13, in <module>
from .win32_clipboard import GetClipboardText, SetClipboardText
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/clipboard/win32_clipboard.py", line 37, in <module>
import ctypes.wintypes as wintypes
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/ctypes/wintypes.py", line 20, in <module>
class VARIANT_BOOL(ctypes._SimpleCData):
ValueError: _type_ 'v' not supported
Failed calling sys.__interactivehook__
Traceback (most recent call last):
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site.py", line 412, in register_readline
import readline
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/readline.py", line 6, in <module>
from pyreadline.rlmain import Readline
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/__init__.py", line 12, in <module>
from . import logger, clipboard, lineeditor, modes, console
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/clipboard/__init__.py", line 13, in <module>
from .win32_clipboard import GetClipboardText, SetClipboardText
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/pyreadline/clipboard/win32_clipboard.py", line 37, in <module>
import ctypes.wintypes as wintypes
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/ctypes/wintypes.py", line 20, in <module>
class VARIANT_BOOL(ctypes._SimpleCData):
ValueError: _type_ 'v' not supported
So I have readline in 3.8 now. But there is another issue that pops up during loading of .pythonrc.
Plus, there is a problem with numpy.
It does not stem from the python version (/usr/bin/python points now to 3.8).
$ python --version
Python 3.8.2
$ python
...
>>> import numpy
Traceback (most recent call last):
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/__init__.py", line 24, in <module>
from . import multiarray
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/multiarray.py", line 14, in <module>
from . import overrides
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
from . import core
File "/c/Users/user1/Documents/appls_mydocs/PortableApps/MSYS2Portable/App/msys32/mingw64/lib/python3.8/site-packages/numpy/core/__init__.py", line 54, in <module>
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
1. Check that you expected to use Python3.8 from "/usr/bin/python.exe",
and that you have no directories in your PATH or PYTHONPATH that can
interfere with the Python and numpy version "1.18.3" you're trying to use.
2. If (1) looks fine, you can open a new issue at
https://github.com/numpy/numpy/issues. Please include details on:
- how you installed Python
- how you installed numpy
- your operating system
- whether or not you have multiple versions of Python installed
- if you built from source, your compiler versions and ideally a build log
- If you're working with a numpy git repository, try `git clean -xdf`
(removes all files not under version control) and rebuild numpy.
Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.
Original error was: No module named 'numpy.core._multiarray_umath'
EDIT #5
As per #a_guest suggestion, and pointing me to "ValueError: _type_ 'v' not supported" error after installing PyReadline, I removed pyreadline, and this problem is gone.
Now the only remaining issue seems to be that numpy cannot load, possibly (but not certainly) due to two alternative python 3.8 systems present.
So the questions now is: Msys2: Two python installations?
The ImportError: cannot import name 'open_code' from 'io' (unknown location) comes from the fact that there are two different versions of Python conflicting with each other. python still points to the old version 3.7 but PYTHONPATH got updated to point to the new 3.8 version. As the documentation of PYTHONPATH states, it becomes prepended to the module search path and hence shadows any builtin modules:
The default search path is installation dependent, but generally begins with prefix/lib/pythonversion (see PYTHONHOME above). It is always appended to PYTHONPATH.
You can reproduce that behavior by creating two different virtual environments and then start one while having PYTHONPATH point to the other. In the following I used Miniconda to create two different environments, py37 and py38, containing a 3.7 and 3.8 installation respectively.
(py37) user#pc:~$ python --version
Python 3.7.6
(py37) user#pc:~$ PYTHONPATH=~/miniconda3/envs/py38/lib/python3.8/ python
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Traceback (most recent call last):
File "/home/user/miniconda3/envs/py38/lib/python3.8/io.py", line 54, in <module>
ImportError: cannot import name 'open_code' from 'io' (unknown location)
Aborted (core dumped)
I just installed Python to start experimenting a bit with Scrapy.
Found a tutorial to first install Miniconda and with this I was able to install Scrapy. Now the problem is that the software I'm trying to run also requires Twisted. So I installed twisted with pip.
Twisted seems to have been installed correctly but I'm still getting an error when running the script that the Twisted module could not be found.
Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
================ RESTART: C:\xampp\htdocs\crawler\crawler.py ================
Traceback (most recent call last):
File "C:\xampp\htdocs\crawler\crawler.py", line 3, in <module>
from scrapy.utils.project import get_project_settings
File "C:\xampp\htdocs\crawler\scrapy\__init__.py", line 43, in <module>
from twisted import version as _txv
ModuleNotFoundError: No module named 'twisted'
>>>
================ RESTART: C:\xampp\htdocs\crawler\crawler.py ================
Traceback (most recent call last):
File "C:\xampp\htdocs\crawler\crawler.py", line 3, in <module>
from scrapy.utils.project import get_project_settings
File "C:\xampp\htdocs\crawler\scrapy\__init__.py", line 43, in <module>
from twisted import version as _txv
ModuleNotFoundError: No module named 'twisted'
>>>
Anybody a suggestion?
Thanks in advance,
You installed twisted for python2. When you use the command pip , you are installing a package for python2. You are using python3 (3.6.3). To install twisted for python3, you have to do:
pip3 install twisted
I hope this helps.
To try if twisted correctly installed, start python interpreter and type
import twisted
If you can't import twisted, you should check if it is in your python path.
Python path can be checked in the python interpreter as
import sys
print sys.path
The directory, where your twisted is installed has to be among those folders.
I have been working on some makefiles which call some python scripts to perform various tasks. Up to now, I had always been makeing on a machine with Ubuntu 16.04 (64 bits) installed. Today, I have tried building on my old 32 bits machine with Lubuntu 16.04 using the same makefile and it fails when calling the Python script.
At first, I thought that my Python version was not new enough, but I also have Python 3.5 installed and I changed the line to python3 in the makefile and it still fails:
python3 /home/morane/Documents/Programming/ConnectX/cxPythonTools/RunUnitTests.py -t /home/morane/bin/tests/unit/cxUnitTests.out -l /home/morane/bin/tests/unit/log/unitTests.log
Traceback (most recent call last):
File "/home/morane/RunUnitTests.py", line 41, in <module>
from subprocess import CREATE_NEW_CONSOLE
ImportError: cannot import name 'CREATE_NEW_CONSOLE'
Makefile:51: recipe for target 'unittests' failed
make: *** [cxbaseunit] Error
To test, I have tried to repeat the error directly in the Python shell:
Python 3.5.2+ (default, Sep 22 2016, 12:18:14)
[GCC 6.2.0 20160927] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> from subprocess import CREATE_NEW_CONSOLE
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'CREATE_NEW_CONSOLE'
and it still fails. Does anyone know whats going on?
Regards
It appears that you're running in Linux. The CREATE_NEW_CONSOLE constant is only available on Windows.
https://docs.python.org/3/library/subprocess.html#subprocess.CREATE_NEW_CONSOLE
Specifically the header of section 17.5.4:
The STARTUPINFO class and following constants are only available on Windows.
I'm running Debian wheezy and have installed mitmproxy from system packages, and pushbullet.py (https://github.com/randomchars/pushbullet.py) using pip, ie.
apt-get install mitmproxy
pip install pushbullet.py
Pushbullet works when I import from python cmdline, like so:
$ python
Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pushbullet import PushBullet
>>> pb=PushBullet(myapikeyremoved)
>>>
However, when I try to import the pushbullet module from within a mitmproxy script, it is unable to find the module.
$ echo "from pushbullet import PushBullet" > mypb.py
$ mitmproxy -s mypb.py
Script load error: Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/libmproxy/script.py", line 48, in load
execfile(path, ns, ns)
File "mypb.py", line 1, in <module>
from pushbullet import PushBullet
ImportError: No module named pushbullet
Now, pip has installed pushbullet.py and its dependencies python-magic, requests, backports.ssl-match-hostname into /usr/local/lib/python2.7/dist-packages/. It seems that cmdline python has found the pip-installed packages, but mitmproxy is not able to for some reason.
So, I've tried to add the path via PYTHONPATH or via system path, but I think I must still be doing something wrong.
$ cat mypb.py
import sys
sys.path.append('/usr/local/lib/python2.7/dist-packages')
from pushbullet import PushBullet
pb=PushBullet(myapikeyremoved)
$ mitmproxy -s mypb.py
Script load error: Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/libmproxy/script.py", line 48, in load execfile(path, ns, ns)
File "mypb.py", line 4, in <module>
pb=PushBullet(myapikeyremoved)
File "/usr/local/lib/python2.7/dist-packages/pushbullet/pushbullet.py", line 26, in __init__
self.refresh()
File "/usr/local/lib/python2.7/dist-packages/pushbullet/pushbullet.py", line 223, in refresh
self._load_devices()
File "/usr/local/lib/python2.7/dist-packages/pushbullet/pushbullet.py", line 32, in _load_devices
resp_dict = resp.json()
TypeError: 'dict' object is not callable
Can anyone shed light on why everything works in via cmdline python and fails when loaded as a script from within mitmproxy, and how to fix it? How should modules installed via pip be loaded?
Thanks to Christian Rapp's comments, I looked further into why mitmproxy's libmproxy was being loaded from /usr/lib/python2.6. It turns out that (at least on the debian wheezy package), the mitmproxy shebang explicitly specifies python2.6:
$ head -1 /usr/bin/mitmproxy
#! /usr/bin/python2.6
That explains why mitmproxy fails to find the pushbullet module; it is looking in the 2.6 directories instead.
I guess that the "TypeError: 'dict' object is not callable" error when I explicitly loaded the module is also because pushbullet relied on some feature of python 2.7.
I have python 2.6 on Windows. I just installed Dropbox API and when I do:
import dropbox
It gives an error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build\bdist.win32\egg\dropbox\__init__.py", line 3, in <module>
# $Id$
File "build\bdist.win32\egg\dropbox\client.py", line 22, in <module>
File "C:\Python26\lib\site-packages\dropbox-2.2.0-py2.6.egg\dropbox\rest.py", line 415
utf8_params = {encode(k): encode(v) for k, v in params.iteritems()}
What can i do?
Reproducible also on Linux:
#> python
Python 2.6.5 (r265:79063, May 6 2011, 16:17:46)
[GCC 4.5.0 20100604 [gcc-4_5-branch revision 160292]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import dropbox
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.6/site-packages/dropbox-2.2.0-py2.6.egg/dropbox/__init__.py", line 3, in <module>
from . import client, rest, session
File "/usr/local/lib/python2.6/site-packages/dropbox-2.2.0-py2.6.egg/dropbox/client.py", line 22, in <module>
from .rest import ErrorResponse, RESTClient, params_to_urlencoded
File "/usr/local/lib/python2.6/site-packages/dropbox-2.2.0-py2.6.egg/dropbox/rest.py", line 415
utf8_params = {encode(k): encode(v) for k, v in params.iteritems()}
^
SyntaxError: invalid syntax
I was desperate to get it working on 2.6 as I can't upgrade to latest versions of python. Finally got it working.
All you need to do is change the dictionary comprehension
From:
utf8_params = {encode(k): encode(v) for k, v in params.iteritems()}
To:
for k, v in params.iteritems():
utf8_params[encode(k)] = encode(v)
And rest of the module worked as usual so far.
I don't have the rep to add a comment so adding as an answer :)
This is definitely related to Python 2.6. The install-script of Dropbox's Python SDK is already throwing syntax errors when you run it with Python 2.6. Strangely, it keeps running and finishes regardless. Dropbox is not very specific about the required minimal version of Python, but the install-script suggests 2.6. Nevertheless, importing the Dropbox module will fail both on Windows and Linux due to a syntax error.
I just installed Python 2.7.9 in parallel to my original Python 2.6.5 environment. With 2.7, the Dropbox SDK installs and loads without any syntax error at at.