Develop Reference

Running VIM from a Python virtulenv using Python plugins

When I start VIM after activating an Python virtulenv, Python plugins can't find their modules, because they are installed in the system, not in the project's virtualenv.
I am using the aw-watcher-vim plugin, which uses a Python library.
When I create a virtualenv and activate it:
virtualenv -p python3 my_env
. my_env/bin/activate
VIM can't find the module at /usr/local/lib/python3.6/dist-packages/aw_core/ anylonger and prints the following errors:
['Traceback (most recent call last):',
' File "~/.dot-file-repo/neovim/plugged/aw-watcher-vim/plugin/vimwatcher.py", line 6 , in <module>',
' from aw_core.log import setup_logging',
'ModuleNotFoundError: No module named ''aw_core''',
'']
(Slightly reformated for readablity.)
It would be nice to use autocompletion using the code from the virtualenv (via Jedi or so), but let the other plugins find their Python modules. Is there a good solution to this use case?

While some comments suggest to import globally instaled pacckages to virtualenv, I'd suggest to set options in /vimrc to use the system python:
set pythondll=/usr/bin/python
set pythonthreedll=/usr/bin/python3
or whatever your system python is.
The advantage is that you virtual envs are untouched and are used according to your needs.

pdfminer - ImportError: No module named pdfminer.pdfdocument

I am trying to install pdfMiner to work with CollectiveAccess. My host (pair.com) has given me the following information to help in this quest:
When compiling, it will likely be necessary to instruct the
installation to use your account space above, and not try to install
into the operating system directories. Typically, using "--
home=/usr/home/username/pdfminer" at the end of the install command
should allow for that.
I followed this instruction when trying to install.
The result was:
running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /usr/home/username/pdfminer/bin/latin2ascii.py to 755
changing mode of /usr/home/username/pdfminer/bin/pdf2txt.py to 755
changing mode of /usr/home/username/pdfminer/bin/dumppdf.py to 755
running install_egg_info
Removing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
Writing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
I don't see anything wrong with that (I'm very new to python), but when I try to run the sample command $ pdf2txt.py samples/simple1.pdf I get this error:
Traceback (most recent call last): File "pdf2txt.py", line 3, in <module>
from pdfminer.pdfdocument import PDFDocument ImportError: No module named pdfminer.pdfdocument
I'm running python 2.7.3. I can't install from root (shared hosting). The most recent version of pdfminer, which is 2014/03/28.
I've seen some posts on similar issues ("no module named. . . " but nothing exactly the same. The proposed solutions either don't help (such as installing with sudo - not an option; specifying the path for python (which doesn't seem to be the issue), etc.).
Or is this a question for my host? (i.e., something amiss or different about their setup)

I had an error like this:
No module named 'pdfminer.pdfinterp'; 'pdfminer' is not a package
My problem was that I had named my script pdfminer.py which for the reasons that I don't know, Python took it for the original pdfminer package files and tried to compiled it.
I renamed my script to something else, deleted all the *.pyc file and __pycache__ directory and my problem was solved.

use this command worked for me and removed the error
pip install pdfminer.six

Since the package pdfminer is installed to a non-standard/non-default location, Python won't be be able to find it. In order to use it, you will need to add it to your 'pythonpath'. Three ways:
At run time, put this in your script pdf2txt.py:
import sys
# if there are no conflicting packages in the default Python Libs =>
sys.path.append("/usr/home/username/pdfminer")
or
import sys
# to always use your package lib before the system's =>
sys.path.insert(1, "/usr/home/username/pdfminer")
Note: The install path specified with --home is used as the Lib for all packages which you might want to install, not just this one. You should delete that folder and re-install with --
home=/usr/home/username/myPyLibs (or any generic name) so that when you install other packages with that install path, you would only need the one path to add to your local Lib to be able to import them:
import sys
sys.path.insert(1, "/usr/home/username/myPyLibs")
Add it to PYTHONPATH before executing your script:
export PYTHONPATH="${PYTHONPATH}:/usr/home/username/myPyLibs"
And then put that in your ~/.bashrc file (/usr/home/username/.bashrc) or .profile as applicable. This may not work for programs which are not executed from the console.
Create a VirtualEnv and install the packages you need to that.

I have a virtual environment and I had to activate it before I did a pip3 install to have the venv see it.
source ~/venv/bin/activate

Any pyinstaller detailed example about hidden import for psutil?

I want to compile my python code to binary by using pyinstaller, but the hidden import block me. For example, the following code import psutil and print the CPU count:
# example.py
import psutil
print psutil.cpu_count()
And I compile the code:
$ pyinstaller -F example.py --hidden-import=psutil
When I run the output under dist:
ImportError: cannot import name _psutil_linux
Then I tried:
$ pyinstaller -F example.py --hidden-import=_psutil_linux
Still the same error. I have read the pyinstall manual, but I still don't know how to use the hidden import. Is there a detailed example for this? Or at least a example to compile and run my example.py?
ENVs:
OS: Ubuntu 14.04
Python: 2.7.6
pyinstaller: 2.1

Hi hope you're still looking for an answer. Here is how I solved it:
add a file called hook-psutil.py
from PyInstaller.hooks.hookutils import (collect_data_files, collect_submodules)
datas = [('./venv/lib/python2.7/site-packages/psutil/_psutil_linux.so', 'psutil'),
('./venv/lib/python2.7/site-packages/psutil/_psutil_posix.so', 'psutil')]
hiddenimports = collect_submodules('psutil')
And then call pyinstaller --additional-hooks-dir=(the dir contain the above script) script.py

pyinstall is hard to configure, the cx_freeze maybe better, both support windows (you can download the exe directly) and linux. Provide the example.py, In windows, suppose you have install python in the default path (C:\\Python27):
$ python c:\\Python27\\Scripts\\cxfreeze example.py -s --target-dir some_path
the cxfreeze is a python script, you should run it with python, then the build files are under some_path (with a lot of xxx.pyd and xxx.dll).
In Linux, just run:
$ cxfreeze example.py -s --target-dir some_path
and also output a lot of files(xxx.so) under some_path.
The defect of cx_freeze is it would not wrap all libraries to target dir, this means you have to test your build under different environments. If any library missing, just copy them to target dir. A exception case is, for example, if your build your python under Centos 6, but when running under Centos 7, the missing of libc.so.6 will throw, you should compile your python both under Centos 7 and Centos 6.

What worked for me is as follows:
Install python-psutil: sudo apt-get install python-psutil. If you
have a previous installation of the psutil module from other
method, for example through source or easy_install, remove it first.
Run pyinstaller as you do, without the hidden-import option.

still facing the error
Implementation:
1.python program with modules like platform , os , shutil and psutil
when i run the script directly using python its working fine.
2.if i build a binary using pyinstaller. The binary is build successfully. But if i run the binary iam getting the No module named psutil found.I had tried several methods like adding the hidden import and other things. None is working. I trying it almost 2 to 3 days.
Error:
ModuleNotFoundError: No module named 'psutil'
Command used for the creating binary
pyinstaller --hidden-import=['_psutil_linux'] --onefile --clean serverHW.py
i tried --additional-hooks-dir= also not working. When i run the binary im getting module not found error.

How to use docutils to convert .rsts to htmls?

I downloaded a util(cmake,actually) tar,and it's doc was written as .rst files with structure like blow.After googling,I know docutils(a python package) can convert these files to htmls,so installed it ,but I can't find a way to to do it?please tell me,is there any simple command to do it?How?
//doc files structure(generated by ls -l under ubuntu)
command
generator
include
index.rst
manual
module
policy
prop_cache
prop_dir
prop_gbl
prop_sf
prop_test
prop_tgt
release
variable

The docutils documentation tells you what to do. In your case, you want the rst2html.py command, which should have been installed in your PATH when you installed the module. Run
rst2html.py --help
to see what the command-line options are.

Compiling vim with specific version of Python

I'm working on several Python projects who run on various versions of Python. I'm hoping to set up my vim environment to use ropevim, pyflakes, and pylint but I've run into some issues caused by using a single vim (compiled for a specific version of Python which doesn't match the project's Python version).
I'm hoping to build vim into each of my virtualenv directories but I've run into an issue and I can't get it to work. When I try to build vim from source, despite specifying the Python config folder in my virtualenv, the system-wide Python interpreter is always used.
Currently, I have Python 2.6.2 and Python 2.7.1 installed with several virtualenvs created from each version. I'm using Ubuntu 10.04 where the system-default Python is 2.6.5. Every time I compile vim and call :python import sys; print(sys.version) it returns Python 2.6.5.
configure --prefix=/virtualenv/project --enable-pythoninterp=yes --with-python-config-dir=/virtualenv/project/lib/python2.6/config
Results in the following in config.log:
...
configure:5151: checking --enable-pythoninterp argument
configure:5160: result: yes
configure:5165: checking for python
configure:5195: result: /usr/bin/python
...
It should be /virtualenv/project/bin/python. Is there any way to specify the Python interpreter for vim to use?
NOTE: I can confirm that /virtualenv/project/bin appears at the front of PATH environment variable.

I'd recommend building vim against the 2 interpreters, then invoking it using the shell script I provided below to point it to a particular virtualenv.
I was able to build vim against Python 2.7 using the following command (2.7 is installed under $HOME/root):
% LD_LIBRARY_PATH=$HOME/root/lib PATH=$HOME/root/bin:$PATH \
./configure --enable-pythoninterp \
--with-python-config-dir=$HOME/root/lib/python2.7/config \
--prefix=$HOME/vim27
% make install
% $HOME/bin/vim27
:python import sys; print sys.path[:2]
['/home/pat/root/lib/python27.zip', '/home/pat/root/lib/python2.7']
Your virtualenv is actually a thin wrapper around the Python interpreter it was created with -- $HOME/foobar/lib/python2.6/config is a symlink to /usr/lib/python2.6/config.
So if you created it with the system interpreter, VIM will probe for this and ultimately link against the real interpreter, using the system sys.path by default, even though configure will show the virtualenv's path:
% PATH=$HOME/foobar/bin:$PATH ./configure --enable-pythoninterp \
--with-python-config-dir=$HOME/foobar/lib/python2.6/config \
--prefix=$HOME/foobar
..
checking for python... /home/pat/foobar/bin/python
checking Python's configuration directory... (cached) /home/pat/foobar/lib/python2.6/config
..
% make install
% $HOME/foobar/bin/vim
:python import sys; print sys.path[:1]
['/usr/lib/python2.6']
The workaround: Since your system vim is most likely compiled against your system python, you don't need to rebuild vim for each virtualenv: you can just drop a shell script named vim in your virtualenv's bin directory, which extends the PYTHONPATH before calling system vim:
Contents of ~/HOME/foobar/bin/vim:
#!/bin/sh
ROOT=`cd \`dirname $0\`; cd ..; pwd`
PYTHONPATH=$ROOT/lib/python2.6/site-packages /usr/bin/vim $*
When that is invoked, the virtualenv's sys.path is inserted:
% $HOME/foobar/bin/vim
:python import sys; print sys.path[:2]
['/home/pat/foobar/lib/python2.6/site-packages', '/usr/lib/python2.6']

For what it's worth, and no one seems to have answered this here, I had some luck using a command line like the following:
vi_cv_path_python=/usr/bin/python26 ./configure --includedir=/usr/include/python2.6/ --prefix=/home/bcrowder/local --with-features=huge --enable-rubyinterp --enable-pythoninterp --disable-selinux --with-python-config-dir=/usr/lib64/python2.6/config

I would like to give a similar solution to crowder's that works quite well for me.
Imagine you have Python installed in /opt/Python-2.7.5 and that the structure of that folder is
$ tree -d -L 1 /opt/Python-2.7.5/
/opt/Python-2.7.5/
├── bin
├── include
├── lib
└── share
and you would like to build vim with that version of Python. All you need to do is
$ vi_cv_path_python=/opt/Python-2.7.5/bin/python ./configure --enable-pythoninterp --prefix=/SOME/FOLDER
Thus, just by explicitly giving vi_cv_path_python variable to configure the script will deduce everything on it's own (even the config-dir).
This was tested multiple times on vim 7.4+ and lately with vim-7-4-324.

I was having this same issue with 3 different versions of python on my system.
for me the easiest thing was to change my $PATH env variable so that the folder that has the version of python I wanted was (in my case /usr/local/bin) was found before another.

During my compiling vim80, the system python is 2.6, I have another python 2.7 under ~/local/bin, I find that, to make the compiling work:
update $PATH to place my python path ahead
add a soft link, ln -s python python2 ( the configure file would try to locate python config by probing python2 )
make distclean before re-run ./configure to make sure no cached wrong value is picked.