Develop Reference

Error with TensorFlow MNIST [duplicate]

Since updating from Homebrew Python 2.7.11 (from 2.7.10) I'm suddenly unable to test register my package on PyPi from the PyCharm IDE console.
Running (as an "External Tool")
python -B setup.py register -r pypitest
I now get
Traceback (most recent call last):
File "setup.py", line 22, in <module>
from setuptools import setup
File "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 12, in <module>
from setuptools.extension import Extension
File "/usr/local/lib/python2.7/site-packages/setuptools/extension.py", line 8, in <module>
from .dist import _get_unpatched
File "/usr/local/lib/python2.7/site-packages/setuptools/dist.py", line 16, in <module>
from setuptools.depends import Require
File "/usr/local/lib/python2.7/site-packages/setuptools/depends.py", line 6, in <module>
from setuptools import compat
File "/usr/local/lib/python2.7/site-packages/setuptools/compat.py", line 17, in <module>
import httplib
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 80, in <module>
import mimetools
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/mimetools.py", line 6, in <module>
import tempfile
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tempfile.py", line 32, in <module>
import io as _io
File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
Referenced from: /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
Expected in: flat namespace
in /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
Process finished with exit code 1
I'm not sure how to proceed. I only get this issue if I execute from within my IDE's console. If I do it directly at the system command line (Terminal on OS X) I have no problems.
OS X 10.11.3; Homebrew Python 2.7.11; PyCharm 5.0.3

tl;dr: Fix this issue by doing one of the following:
type hash -r python, OR
log out and log in.
EDIT: An answer to my related question makes it clear what's happening here. When you install a new version of python, you may need to run hash -r python to tell bash to reset the "cached" location to the python executable.
In my case, I was typing python, which was on my $PATH at /usr/local/bin/python. But bash was still using the old cache location /usr/bin/python. So, the old executable was called, but the new path was provided to python in sys.argv[0]. This means that the old executable was running, but the new sys.executable value caused all the wrong modules to get loaded (including the io module).
I'm having the same problem. I installed python 2.7.11 via an installer from Python.org. Strangely, the issue seems to be related to some subtle difference between how OSX launches python when I invoke it from the shell using the full path vs. using just the word python.
So, for me, this works (invoking python via the full path /usr/local/bin/python):
$ which python
/usr/local/bin/python
$ /usr/local/bin/python -c "import io"
$
... but this doesn't:
$ python -c "import io"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
Referenced from: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
Expected in: flat namespace
in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
So, as a workaround, you can try doing the same thing.
Elsewhere, I've posted a separate question about this puzzling behavior. Maybe somehow merely calling python invokes some strange mix of the 2.7.11 executable with the 2.7.10 dylibs??

According to https://github.com/klen/python-mode/issues/634:
I had the same issue, but successfully fixed. In my case I compiled
python and vim with homebrew, when PYTHON_PATH has been specified and
set to one of my dev environments, where I also had some libraries,
including io. Workaround was simple: open new terminal, make sure that
you do not have custom PYTHON_PATH, uninstall python, uninstall vim.
Reinstall both of them.
and
Problem solved.
Culprit is the update from python 2.7.10 to 2.7.11.
If you are using conda package control, simply run "conda install
python=2.7.10" will solve this problem.
This doesn't give the root cause though. Since this happens with _io, this looks like a bug in python 2.7.11 (unlikely, there would be a world-scale outcry and a prompt fix if it was) or some packaging bug or version mismatch specifically with the homebrew version (and maybe some related ones, too).
Try to import _io in the console and if it succeeds, check if it was loaded from the same path.

Reinstall python.
brew unlink python && brew reinstall python
Secure the path
export PYTHONPATH=$PYTHONPATH:/usr/local/bin/
BACKUP and Change the order of "paths" file.
sudo nano /etc/paths
it seems, the order of paths, it is decisive to run python properly. In my case, the result was:
#sudo nano /etc/paths
/usr/bin
/usr/local/bin
/bin
/usr/sbin
/sbin
On my mac, path is like this.
$ which python
/usr/local/bin/python
Now I can run both:
$ /usr/local/bin/python -c "import io"
$ python -c "import io"

I had the same issue, it is successfully fixed by just replacing the _io.so file.
sudo find / -name _io.so
copy the path of the _io.so file which DOES NOT belong to python-2.7.11. For example, copy the path of _io.so which is under python-2.7.5:
/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
Replace the /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so file with the _io.so that you just found.

This happened to me as well in MacVim. I solved it by making sure :python print(sys.path) is using system Python (e.g. /Library/Python/2.7/...)
Since I installed MacVim via Homebrew, I just did that by:
Spawn a new shell that had which python -> /usr/bin/python. For my case I needed to remove the pyenv line from my .bash_profile. If you installed Python via Homebrew you may want to brew unlink python first
brew reinstall macvim

If your problem is caused by anaconda, it is unnecessary to remove //anaconda directory.
Just open your ~/.bash_profile, find the line
export PATH="//anaconda/bin:$PATH
and comment it out, then restart your terminal session.

Another quick workaround if you don't mind sticking with Python 2.7.10 is to specify the path of the Python interpreter executable that will be used for the virtualenv. On OSX that path is usually /usr/bin/python:
virtualenv venv --python=/usr/bin/python

Can't add comment (?) so this just to share my exp., downgrade to 2.7.10 works fr me.

I got this error after a failed NLTK download, I needed to uninstall anaconda:
sudo rm -rf ~/anaconda
update PATH variable

This happened when I already had tried to create a venv in a folder, and mistakenly was trying to initialize a second one! So I just removed venv directory and re-ran the command. Very likely this is not the answer to this solution, but searching my error brought me here, so it may help some others who are stuck.

I solved this issue by removing the symbolic link that was in /usr/local/bin and copying the actual python binary, that was pointed to by said link, there.

I had the same issue when I tried to use PyCharm. Solved by setting "python interpreter" in project configuration to point to the python virtual env I wanted to use, which was an Anaconda env. Somehow the interpreter path was missing the "anaconda" portion of ~/.../anaconda/.../_io.so. No need to uninstall anaconda.

pdfminer - ImportError: No module named pdfminer.pdfdocument

I am trying to install pdfMiner to work with CollectiveAccess. My host (pair.com) has given me the following information to help in this quest:
When compiling, it will likely be necessary to instruct the
installation to use your account space above, and not try to install
into the operating system directories. Typically, using "--
home=/usr/home/username/pdfminer" at the end of the install command
should allow for that.
I followed this instruction when trying to install.
The result was:
running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /usr/home/username/pdfminer/bin/latin2ascii.py to 755
changing mode of /usr/home/username/pdfminer/bin/pdf2txt.py to 755
changing mode of /usr/home/username/pdfminer/bin/dumppdf.py to 755
running install_egg_info
Removing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
Writing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
I don't see anything wrong with that (I'm very new to python), but when I try to run the sample command $ pdf2txt.py samples/simple1.pdf I get this error:
Traceback (most recent call last): File "pdf2txt.py", line 3, in <module>
from pdfminer.pdfdocument import PDFDocument ImportError: No module named pdfminer.pdfdocument
I'm running python 2.7.3. I can't install from root (shared hosting). The most recent version of pdfminer, which is 2014/03/28.
I've seen some posts on similar issues ("no module named. . . " but nothing exactly the same. The proposed solutions either don't help (such as installing with sudo - not an option; specifying the path for python (which doesn't seem to be the issue), etc.).
Or is this a question for my host? (i.e., something amiss or different about their setup)

I had an error like this:
No module named 'pdfminer.pdfinterp'; 'pdfminer' is not a package
My problem was that I had named my script pdfminer.py which for the reasons that I don't know, Python took it for the original pdfminer package files and tried to compiled it.
I renamed my script to something else, deleted all the *.pyc file and __pycache__ directory and my problem was solved.

use this command worked for me and removed the error
pip install pdfminer.six

Since the package pdfminer is installed to a non-standard/non-default location, Python won't be be able to find it. In order to use it, you will need to add it to your 'pythonpath'. Three ways:
At run time, put this in your script pdf2txt.py:
import sys
# if there are no conflicting packages in the default Python Libs =>
sys.path.append("/usr/home/username/pdfminer")
or
import sys
# to always use your package lib before the system's =>
sys.path.insert(1, "/usr/home/username/pdfminer")
Note: The install path specified with --home is used as the Lib for all packages which you might want to install, not just this one. You should delete that folder and re-install with --
home=/usr/home/username/myPyLibs (or any generic name) so that when you install other packages with that install path, you would only need the one path to add to your local Lib to be able to import them:
import sys
sys.path.insert(1, "/usr/home/username/myPyLibs")
Add it to PYTHONPATH before executing your script:
export PYTHONPATH="${PYTHONPATH}:/usr/home/username/myPyLibs"
And then put that in your ~/.bashrc file (/usr/home/username/.bashrc) or .profile as applicable. This may not work for programs which are not executed from the console.
Create a VirtualEnv and install the packages you need to that.

I have a virtual environment and I had to activate it before I did a pip3 install to have the venv see it.
source ~/venv/bin/activate

Python module development workflow - setup and build [duplicate]

I'm developing my own module in python 2.7. It resides in ~/Development/.../myModule instead of /usr/lib/python2.7/dist-packages or /usr/lib/python2.7/site-packages. The internal structure is:
/project-root-dir
/server
__init__.py
service.py
http.py
/client
__init__.py
client.py
client/client.py includes PyCachedClient class. I'm having import problems:
project-root-dir$ python
Python 2.7.2+ (default, Jul 20 2012, 22:12:53)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import http
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "server/http.py", line 9, in <module>
from client import PyCachedClient
ImportError: cannot import name PyCachedClient
I didn't set PythonPath to include my project-root-dir, therefore when server.http tries to include client.PyCachedClient, it tries to load it from relative path and fails. My question is - how should I set all paths/settings in a good, pythonic way? I know I can run export PYTHONPATH=... in shell each time I open a console and try to run my server, but I guess it's not the best way. If my module was installed via PyPi (or something similar), I'd have it installed in /usr/lib/python... path and it'd be loaded automatically.
I'd appreciate tips on best practices in python module development.

My Python development workflow
This is a basic process to develop Python packages that incorporates what I believe to be the best practices in the community. It's basic - if you're really serious about developing Python packages, there still a bit more to it, and everyone has their own preferences, but it should serve as a template to get started and then learn more about the pieces involved. The basic steps are:
Use virtualenv for isolation
setuptools for creating a installable package and manage dependencies
python setup.py develop to install that package in development mode
virtualenv
First, I would recommend using virtualenv to get an isolated environment to develop your package(s) in. During development, you will need to install, upgrade, downgrade and uninstall dependencies of your package, and you don't want
your development dependencies to pollute your system-wide site-packages
your system-wide site-packages to influence your development environment
version conflicts
Polluting your system-wide site-packages is bad, because any package you install there will be available to all Python applications you installed that use the system Python, even though you just needed that dependency for your small project. And it was just installed in a new version that overrode the one in the system wide site-packages, and is incompatible with ${important_app} that depends on it. You get the idea.
Having your system wide site-packages influence your development environment is bad, because maybe your project depends on a module you already got in the system Python's site-packages. So you forget to properly declare that your project depends on that module, but everything works because it's always there on your local development box. Until you release your package and people try to install it, or push it to production, etc... Developing in a clean environment forces you to properly declare your dependencies.
So, a virtualenv is an isolated environment with its own Python interpreter and module search path. It's based on a Python installation you previously installed, but isolated from it.
To create a virtualenv, install the virtualenv package by installing it to your system wide Python using easy_install or pip:
sudo pip install virtualenv
Notice this will be the only time you install something as root (using sudo), into your global site-packages. Everything after this will happen inside the virtualenv you're about to create.
Now create a virtualenv for developing your package:
cd ~/pyprojects
virtualenv --no-site-packages foobar-env
This will create a directory tree ~/pyprojects/foobar-env, which is your virtualenv.
To activate the virtualenv, cd into it and source the bin/activate script:
~/pyprojects $ cd foobar-env/
~/pyprojects/foobar-env $ . bin/activate
(foobar-env) ~/pyprojects/foobar-env $
Note the leading dot ., that's shorthand for the source shell command. Also note how the prompt changes: (foobar-env) means your inside the activated virtualenv (and always will need to be for the isolation to work). So activate your env every time you open a new terminal tab or SSH session etc..
If you now run python in that activated env, it will actually use ~/pyprojects/foobar-env/bin/python as the interpreter, with its own site-packages and isolated module search path.
A setuptools package
Now for creating your package. Basically you'll want a setuptools package with a setup.py to properly declare your package's metadata and dependencies. You can do this on your own by by following the setuptools documentation, or create a package skeletion using Paster templates. To use Paster templates, install PasteScript into your virtualenv:
pip install PasteScript
Let's create a source directory for our new package to keep things organized (maybe you'll want to split up your project into several packages, or later use dependencies from source):
mkdir src
cd src/
Now for creating your package, do
paster create -t basic_package foobar
and answer all the questions in the interactive interface. Most are optional and can simply be left at the default by pressing ENTER.
This will create a package (or more precisely, a setuptools distribution) called foobar. This is the name that
people will use to install your package using easy_install or pip install foobar
the name other packages will use to depend on yours in setup.py
what it will be called on PyPi
Inside, you almost always create a Python package (as in "a directory with an __init__.py) that's called the same. That's not required, the name of the top level Python package can be any valid package name, but it's a common convention to name it the same as the distribution. And that's why it's important, but not always easy, to keep the two apart. Because the top level python package name is what
people (or you) will use to import your package using import foobar or from foobar import baz
So if you used the paster template, it will already have created that directory for you:
cd foobar/foobar/
Now create your code:
vim models.py
models.py
class Page(object):
"""A dumb object wrapping a webpage.
"""
def __init__(self, content, url):
self.content = content
self.original_url = url
def __repr__(self):
return "<Page retrieved from '%s' (%s bytes)>" % (self.original_url, len(self.content))
And a client.py in the same directory that uses models.py:
client.py
import requests
from foobar.models import Page
url = 'http://www.stackoverflow.com'
response = requests.get(url)
page = Page(response.content, url)
print page
Declare the dependency on the requests module in setup.py:
install_requires=[
# -*- Extra requirements: -*-
'setuptools',
'requests',
],
Version control
src/foobar/ is the directory you'll now want to put under version control:
cd src/foobar/
git init
vim .gitignore
.gitignore
*.egg-info
*.py[co]
git add .
git commit -m 'Create initial package structure.
Installing your package as a development egg
Now it's time to install your package in development mode:
python setup.py develop
This will install the requests dependency and your package as a development egg. So it's linked into your virtualenv's site-packages, but still lives at src/foobar where you can make changes and have them be immediately active in the virtualenv without re-installing your package.
Now for your original question, importing using relative paths: My advice is, don't do it. Now that you've got a proper setuptools package, that's installed and importable, your current working directory shouldn't matter any more. Just do from foobar.models import Page or similar, declaring the fully qualified name where that object lives. That makes your source code much more readable and discoverable, for yourself and other people that read your code.
You can now run your code by doing python client.py from anywhere inside your activated virtualenv. python src/foobar/foobar/client.py works just as fine, your package is properly installed and your working directory doesn't matter any more.
If you want to go one step further, you can even create a setuptools entry point for your CLI scripts. This will create a bin/something script in your virtualenv that you can run from the shell.
setuptools console_scripts entry point
setup.py
entry_points='''
# -*- Entry points: -*-
[console_scripts]
run-fooobar = foobar.main:run_foobar
''',
client.py
def run_client():
# ...
main.py
from foobar.client import run_client
def run_foobar():
run_client()
Re-install your package to activate the entry point:
python setup.py develop
And there you go, bin/run-foo.
Once you (or someone else) installs your package for real, outside the virtualenv, the entry point will be in /usr/local/bin/run-foo or somewhere simiar, where it will automatically be in $PATH.
Further steps
Creating a release of your package and uploading it PyPi, for example using zest.releaser
Keeping a changelog and versioning your package
Learn about declaring dependencies
Learn about Differences between distribute, distutils, setuptools and distutils2
Suggested reading:
The Hitchhiker’s Guide to Packaging
The pip cookbook

So, you have two packages, the first with modules named:
server # server/__init__.py
server.service # server/service.py
server.http # server/http.py
The second with modules names:
client # client/__init__.py
client.client # client/client.py
If you want to assume both packages are in you import path (sys.path), and the class you want is in client/client.py, then in you server you have to do:
from client.client import PyCachedClient
You asked for a symbol out of client, not client.client, and from your description, that isn't where that symbol is defined.
I personally would consider making this one package (ie, putting an __init__.py in the folder one level up, and giving it a suitable python package name), and having client and server be sub-packages of that package. Then (a) you could do relative imports if you wanted to (from ...client.client import something), and (b) your project would be more suitable for redistribution, not putting two very generic package names at the top level of the python module hierarchy.

PyCharm & Pyenv local?

After I broke my Ubuntu precise with a Cython compilation I like to keep the system Python clean. I like to have 2.7.x & 3.4.x besides each other and used Pyenv to have a global default interpreter independent from the system python. Now I also want to define local interpreters on a per project basis, usually done with pyenv local. Unfortunately my favorite IDE PyCharm does only support pyenv global, the local .python-version files are obviously not recognized by PyCharm.
So I tried to use the pyenv global interpreter in PyCharm with ~/.pyenv/bin/python-local-exec which works but unfortunately is already deprecated. When I then tried to create a virtualenv from the pyenv Python 3 global interpreter, it throwed the follwing error:
The following command was executed:
virtualenv.py /home/barrios/code/so/venv
The error output of the command:
pyenv: `python-local-exec' is deprecated and will be removed in the next release.
To upgrade: https://github.com/yyuu/pyenv/wiki/python-local-exec
Failed to import the site module
Traceback (most recent call last):
File "/home/barrios/code/so/venv/lib/python3.4/site.py", line 67, in <module>
import os
File "/home/barrios/code/so/venv/lib/python3.4/os.py", line 634, in <module>
from _collections_abc import MutableMapping
ImportError: No module named '_collections_abc'
Using base prefix '/home/barrios/.pyenv/versions/3.4.0b3'
New python executable in /home/barrios/code/so/venv/bin/python
ERROR: The executable /home/barrios/code/so/venv/bin/python is not functioning
ERROR: It thinks sys.prefix is '/home/barrios/.pyenv/bin' (should be '/home/barrios/code/so/venv')
ERROR: virtualenv is not compatible with this system or executable
As mentioned before, I don't want to install Python 3 into my Ubuntu system folders. Any ideas how to achieve that?
TNX a lot!

I'd suggest to use https://github.com/yyuu/pyenv-virtualenv to create virtualenv for a desired interpreter and then add it as a Python interpreter in PyCharm.

changing virtualenv folder on windows

after a computer fix my python projects dir (windows) changed (say from d: to f:).
now all my virtualenvs are broken. after activating the env the project inside the virtualenv can't find the dependencies and the custom scripts (from the env\scripts folder)won't work
tried running:
virtualenv --relocateble ENV_NAME (with the env name ..)
like in this stackoverflow question and it outputted a lot of lines like:
Script agent\Scripts\deactivate.bat cannot be made relative
and my virtualenv is still broken.
when I manually changed activate.bat set VIRTUAL_ENV to the new path. some scripts work again. but the relocate scripts still doesn't run and most of the scripts are still broken
even running the python interpeter fails with:
Traceback (most recent call last):
File "F:\Python27\learn\agent\agent\lib\site.py", line 677, in <module>
main()
File "F:\Python27\learn\agent\agent\lib\site.py", line 666, in main
aliasmbcs()
File "F:\Python27\learn\agent\agent\lib\site.py", line 506, in aliasmbcs
import locale, codecs
File "F:\Python27\learn\agent\agent\lib\locale.py", line 19, in <module>
import functools
ImportError: No module named functools
is there any way to fix this? HELP
Update: I also changed manually the shebang python interpeter line in all scripts in ENV\Scripts. now all fail with the same python failure as above
Another Update: to #udi the system python path is:
['', 'C:\\dev\\Python27\\lib\\site-packages\\distribute-0.6.37-py2.7.egg', 'C:\\
dev\\Python27\\lib\\site-packages\\pip-1.3.1-py2.7.egg', 'C:\\dev\\Python27\\lib
\\site-packages\\numpy-1.7.1-py2.7-win32.egg', 'C:\\dev\\Python27\\lib\\site-pac
kages\\pandas-0.11.0-py2.7-win32.egg', 'C:\\dev\\Python27\\lib\\site-packages\\p
ytz-2013b-py2.7.egg', 'C:\\dev\\Python27\\lib\\site-packages\\python_dateutil-2.
1-py2.7.egg', 'C:\\dev\\Python27\\lib\\site-packages\\six-1.3.0-py2.7.egg', 'C:\
\dev\\Python27\\lib\\site-packages\\tornado-3.0.1-py2.7.egg', 'C:\\dev\\Python27
\\lib\\site-packages\\pyzmq-13.1.0-py2.7-win32.egg', 'C:\\dev\\Python27\\lib\\si
te-packages\\pygments-1.6-py2.7.egg', 'C:\\Windows\\system32\\python27.zip', 'C:
\\dev\\Python27\\DLLs', 'C:\\dev\\Python27\\lib', 'C:\\dev\\Python27\\lib\\plat-
win', 'C:\\dev\\Python27\\lib\\lib-tk', 'C:\\dev\\Python27', 'C:\\dev\\Python27\
\lib\\site-packages', 'C:\\dev\\Python27\\lib\\site-packages\\setuptools-0.6c11-
py2.7.egg-info']
since I can't run python from the virtualenv, I can't print the python path from there

Correcting python directory path in ENV_FOLDER\Lib\orig-prefix.txt helped me

Seems like your system and local environments create a mix of libraries and binaries from different versions of python.
Chances are you would need to delete Lib, Scripts and Include and start again with virtualenv .. You might be able to save the site-packages folder, but if you have requirements.txt files, you should probably reinstall packages instead (see also: How do I install from a local cache with pip? ).
Anyway, I believe you can create a script that does all this in one step.

I installed both py2 and py3 on my windows 10. And got this error by create virtualenv by using virtualenv xxx directly. After purging folder xxx and reinstalling with virtualenv -p TARGET_PY_EXE xxx everything works smoothly.
Hope this will help multiple python windows users.
By the way, I simply create env variables as PY2 and PY3 instead of adding absolute paths into PATH.