Develop Reference

How do I use libsvm in Python without modifying sys.path?

I installed libsvm in Ubuntu. All I want to do is call from svmutil import * in Python without it giving me an error. However, I see no good documentation on how to configure libsvm to work with python. "Adding libsvm to a path" tells me nothing.
And I don't want to use a package manager. What do I need to do to make libsvm found when running python outside of the original libsvm install folder?
(ykp) yannik#yannik-Inspiron-7580:~/Projects/libsvm$ ls
COPYRIGHT java python svm.o svm-toy
description-pak libsvm.so.2 README svm-predict svm-train
doc-pak Makefile svm.cpp svm-predict.c svm-train.c
FAQ.html Makefile.win svm.def svm-scale tools
heart_scale matlab svm.h svm-scale.c windows
(ykp) yannik#yannik-Inspiron-7580:~/Projects/libsvm$ ls python
commonutil.py Makefile README svmutil.py
description-pak __pycache__ svm.py
I tried doing sudo ln -s libsvm.so.2 /usr/local/lib and adding ~/Projects/libsvm to my PYTHONPATH with no success. Please help!

If ~/Projects/libsvm/python/ is where the module you want to import exists, then you have to add ~/Projects/libsvm/python/ to your PYTHONPATH not ~/Projects/libsvm/. The PYTHONPATH doesn't just automatically recursively search down for python files if that is what you were expecting.

ImportError after cython embed

I can't get otherwise-available modules seen by a compiled python script. How do I need to change the below process in order to accept either venv-based or global modules?
Steps:
$ python3 -m venv sometest
$ cd sometest
$ . bin/activate
(sometest) $ pip3 install PyCrypto Cython
The basic script, using a non-standard module Crypto:
# hello.py
from Crypto.Cipher import AES
import base64
obj = AES.new('This is a key123', AES.MODE_CBC, 'This is an IV456')
msg = "The answer is no"
ciphertext = obj.encrypt(msg)
print(msg)
print(base64.b64encode(ciphertext))
(sometest) $ python3 hello.py
The answer is no
b'1oONZCFWVJKqYEEF4JuL8Q=='
Compiling it:
(sometest) $ cython -3 --embed hello.py
(sometest) $ gcc -Os -I /usr/include/python3.5m -o hello hello.c -lpython3.5m -lpthread -lm -lutil -ldl
(sometest) $ $ ./hello
Traceback (most recent call last):
File "hello.py", line 1, in init hello
from Crypto.Cipher import AES
ImportError: No module named 'Crypto'
I don't think it's a problem with using the venv from a cython-embedded-compiled script: the script works elsewhere in the system without the venv (that is, python3 -c 'from Crypto.Cipher import AES' does not fail).
The process works fine otherwise:
(sometest) $ echo 'print("hello world")' > hello2.py
(sometest) $ cython -3 --embed hello2.py
(sometest) $ gcc -Os -I /usr/include/python3.5m -o hello2 hello2.c -lpython3.5m -lpthread -lm -lutil -ldl
(sometest) $ ./hello2
hello world
System:
(sometest) $ python3 --version
Python 3.5.2
(sometest) $ pip3 freeze
Cython==0.29.11
pkg-resources==0.0.0
pycrypto==2.6.1
(sometest) $ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

Usually, a Python-interpreter isn't "standalone" and in order to work it needs its standard libraries (for example ctypes (compiled) or site.py (interpreted)) and also path to other site-packages (for example numpy) must be set.
Albeit it is possible to make a Python-interpter fully standalone by freezing the py-modules and merging all c-extensions (see for example this SO-post) into the resulting executable, it is easier to provide the needed installation to the embeded interpeter. One can download files needed for a "standard" installation from python-homepage (at least for windows), see also this SO-question).
Sometimes finding standard modules/site packages doesn't work out of the box: one has to help the interpreter by setting Python-path, i.e. by adding <..>/sometest/lib/python3.5/site-packages (sometest being a virtual environment root-folder) to sys.path either programmatically in the pyx-file or by setting PYTHONPATH-environment variable prior to start.
Read on for more gory details and alternative solutions.
This answer is for Linux and Python3 (Python 3.7), the basic idea is the same for Windows/MacOS, but some details might be different.
Because venv is used we have the following alternative to solve the issue:
adding <..>/sometest/lib/python3.5/site-packages (sometest being a virtual environment root-folder) to sys.path either programmatically in the pyx-file or by setting PYTHONPATH-environment variable prior to start.
placing the executable with embeded python in a subdirectory of sometest (e.g. bin or creating an own).
using virtualenv instead of venv.
Note: For the executable with the embeded python, it doesn't play any role whether the virtual environment (or which) is activated or not.
Why does the above solves the issue in your scenario?
The problem is, that the (embeded) Python-interpreter needs to figure out where following things are:
platform independent directory/files, e.g. os.py, argparse.py (mostly everything *.py/ *.pyc). Given sys.prefix, the interpreter can figure out where to find them (i.e. in prefix/lib/pythonX.Y).
platform dependent directory/files, e.g. shared libraries. Given sys.exec_prefix the interpreter can figure out where to find them (e.g. shared libraries can be found in in exec_prefix/lib/pythonX.Y/lib-dynload).
The algorithm can be found here and the search is performed, when Py_Initialize is executed. Once these directories are found, sys.path can be constructed.
However, when using venv, there is a pyvenv.cfg-file next to exe or in the parent directory, which ensures that the right Python-Home is found - a good starting point is the home-key in this file.
If Py_NoSiteFlag is not set, Py_Initialize will utilize site.py (it can be found by the interpreter, because sys.prefix is known) , or more precise site.main(), to add site-packages of the virtual environment to sys.path. While doing so, site.py looks for pyvenv.cfg and parses it. However, local site-packages are added to the python-path only when:
If a file named "pyvenv.cfg" exists one directory above
sys.executable, sys.prefix and sys.exec_prefix are set to that
directory and it is also checked for site-packages (sys.base_prefix
and sys.base_exec_prefix will always be the "real" prefixes of the
Python installation).
In your case pyvenv.cfg is not in the directory above, but in the same as the exe - thus the local site-packages, where the libraries were installed via pip, aren't included. Global site-packages aren't included because pyvenv.cfg has key include-system-site-packages = false. Thus there are no site-packages allowed and the installed libraries cannot be found.
However, moving the exe one directory down, would lead to inclusion of the local site-packages to the path.
There are other scenarios possible, what counts is the location of the executable and not which environment is activated.
A: Executable is somewhere, but not inside a virtual environment
This search heuristic works more or less reliable for installed python-interpreters, but can fall for embeded-interpreters or virtual environments (see this issue for much more information).
If python was installed using usual apt install or similar, then it will be found (due to 4. step in the search algorithm) and the system-installation will be used by the embeded interpreter.
However if files were moved around or python was build from source but not installed, then embeded interperter cannot start up:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
In this case, Py_SetPythonHome or setting environment variable $PYTHONHOME are possible solutions.
B: Executable inside a virtual environment, created with virtualenv
Assuming it is the same Python version for virtual environment and the embeded python (otherwise we have the above case), the emebeded exe will use local side-packages. The home search algorithmus will always find the local home, due to this rule:
Step 3. Try to find prefix and exec_prefix relative to argv0_path, backtracking up the path until it is exhausted. This is the most common step to succeed. Note that if prefix and exec_prefix are
different, exec_prefix is more likely to be found; however if
exec_prefix is a subdirectory of prefix, both will be found.
In this case argv0_path is the path to the exe (there is no pyvenv.cfg file!), and the "landmarks" (lib/python$VERSION/os.py and lib/python$VERSION/lib-dynload) will be found, because they are presented as symlinks in the local-home above the exe.
C: Executable two folders deep inside a venv-environment
Going two and not one folder (where it works) down in a venv-environment results in case A: pyvenv.cfg file isn't read while searching for home (too far above), 'venv`-environments lack symlinks to "landmarkers" (localy only side-packages are present) and such step 3 will fail, with 4. step being the only hope.
Corollary: Embeded Python will not work without a right Python-installation, unless among other possibilities:
the needed files are packed into lib\pythonX.Y\* next to the embeding executable or somewhere above (and there is no pyvenv.cfg around to mess the search up).
or pyvenv.cfg used to point the interpreter to the right location.

Getting error when exporting ipa in development method

When I select IPA distribution method as development and try to export, I am getting this same error for all my projects, and I am not using python anywhere so why I am getting this error. Please help me to solve this error.
I am using Xcode 9.2
ipatool failed with an exception:#<RuntimeError: Couldn't locate python in /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/local/bin /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/libexec /Applications/Xcode.app/Contents/Developer/usr/bin /Applications/Xcode.app/Contents/Developer/usr/local/bin /Applications/Xcode.app/Contents/Developer/Tools /usr/bin /bin /usr/sbin /sbin>
/Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:157:in `locate_tool'
/Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:137:in `unicode_equal?'
/Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:1703:in `block in MakeFileSystemNode'
/Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:1703:in `each'
/Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:1703:in `detect'

I found the problem, thanks to Alper to guiding me in the right direction, when we create or export an IPA Xcode needs python which is located in /usr/bin/python path, and in my case I have installed python but it was not on that location, so I search for python executable file in my system and copy paste that executable file on this /usr/bin/ location, and it work's.

Well it seems that Apple use python and that something has broken. Since the error is in your Xcode.app path, I'd recommend a reinstall of your Xcode to see whether that fixes things.

It happen because XCode can't find path to python binary file.
1) Find your python binary file:
Enter in your terminal:
python
Next enter:
import sys
sys.path
The screen will display the paths to various python directories.
Example:
['', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages']
Copy path to bin/ folder and exit from python:
Just press command+d
2) Copy python binary file to /usr/bin destination:
Enter in your terminal:
cp {INSERT HERE COPIED PATH TO /bin PYTHON DIRECTORY}/python /usr/bin/
Example:
cp /Library/Frameworks/Python.framework/Versions/2.7/bin/python /usr/bin/
After that the issue will fixed!

If you ever have problem with application not locating your python - symlink will do the trick.
For some reason my pythons are installed in /usr/local/bin, while one of my IDEs searches in /usr/bin.
sudo ln -s /usr/local/bin/python /usr/bin/python
fixed this issue.
enjoy!

Windows command line, "python3" pointing to the wrong python3.exe

So, I have Anaconda, OSGeo and Python2.7 installed on my computer.
I'm also using Spyder. In Spyder :
>>> import sys
>>> sys.executable
'C:\\ProgramData\\Anaconda3\\pythonw.exe'
Which is what I want.
However, in the windows command line and powershell :
$ python3
>>> import sys
>>> sys.executable
'C:\\Progra~1\\OSGeo4W\\bin\\python3.exe'
Which is not what I want. I want to use 'C:\\ProgramData\\Anaconda3\\pythonw.exe' (or python.exe, not sure) when using python3 in the command line.
Also :
$ pip3
Fatal error in launcher: Unable to create process using '"'
I don't get why python3 in the windows command line points to OSGeo's version of Python3. Here is my path :
C:\Python27\;C:\Python27\Scripts;C:\ProgramData\Anaconda3;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Git\cmd;C:\Program Files\PuTTY\;C:\Progra~1\OSGeo4W\bin\;C:\Program Files\Microsoft\R Open\R-3.4.0\bin
I also have an environment variable called PYTHONHOME
C:\ProgramData\Anaconda3
Moreover (for completeness of information), I have python 2 installed :
$ python
File "C:\ProgramData\Anaconda3\lib\site.py", line 177
file=sys.stderr)
^
SyntaxError: invalid syntax
($ pip outputs the same thing).
Having python3 and python2.7 both work when using python3 and python (respectively) in the windows command line would be a nice bonus, but it's not really my priority.

There are probably several things you have to take care of:
In general the search order of the Windows PATH is from left to right starting with the system PATH. The first matching element wins. In your case this is correct because the system will search C:\ProgramData\Anaconda3\ first. However in that folder is no executable called python3 by default. On my system I created a simlink pointing to python.exe. On your system you can do it in PowerShell like this:
New-Item -Path C:\ProgramData\Anaconda3\python3.exe -ItemType SymbolicLink -Value C:\ProgramData\Anaconda3\python.exe
pip is located in Scripts\ folder so in your case you have to add C:\ProgramData\Anaconda3\Scripts to your PATH and create the corresponding simlinks again. In this case you have to create two of them because pip.exe is appending its name to the script that is trying to call (i.e. if your exe file is called foo.exe it will try to call foo-script.exe which does not exist) you can create the simlinks in PowerShell with those two commands:
New-Item -Path C:\ProgramData\Anaconda3\Scripts\pip3.exe -ItemType SymbolicLink -Value C:\ProgramData\Anaconda3\Scripts\pip.exe
and
New-Item -Path C:\ProgramData\Anaconda3\Scripts\pip3-script.py -ItemType SymbolicLink -Value C:\ProgramData\Anaconda3\Scripts\pip-script.py
Like this you will be able to use python3 and pip3 from your cmd line. Please check for similar problems with your python2 installation folder.
Hope it helps.

Automatically load a virtualenv when running a script

I have a python script that needs dependencies from a virtualenv. I was wondering if there was some way I could add it to my path and have it auto start it's virtualenv, run and then go back to the system's python.
I've try playing around with autoenv and .env but that doesn't seem to do exactly what I'm looking for. I also thought about changing the shabang to point to the virtualenv path but that seems fragile.

There are two ways to do this:
Put the name of the virtual env python into first line of the script. Like this
#!/your/virtual/env/path/bin/python
Add virtual environment directories to the sys.path. Note that you need to import sys library. Like this
import sys
sys.path.append('/path/to/virtual/env/lib')
If you go with the second option you might need to add multiple paths to the sys.path (site etc). The best way to get it is to run your virtual env python interpreter and fish out the sys.path value. Like this:
/your/virtual/env/bin/python
Python blah blah blah
> import sys
> print sys.path
[ 'blah', 'blah' , 'blah' ]
Copy the value of sys.path into the snippet above.

I'm surprised that nobody has mentioned this yet, but this is why there is a file called activate_this.py in the virtualenv's bin directory. You can pass that to execfile() to alter the module search path for the currently running interpreter.
# doing execfile() on this file will alter the current interpreter's
# environment so you can import libraries in the virtualenv
activate_this_file = "/path/to/virtualenv/bin/activate_this.py"
execfile(activate_this_file, dict(__file__=activate_this_file))
You can put this file at the top of your script to force the script to always run in that virtualenv. Unlike the modifying hashbang, you can use relative path with by doing:
script_directory = os.path.dirname(os.path.abspath(__file__))
activate_this_file = os.path.join(script_directory, '../../relative/path/to/env/bin/activate_this.py')

From the virtualenv documentation:
If you directly run a script or the python interpreter from the
virtualenv’s bin/ directory (e.g. path/to/env/bin/pip or
/path/to/env/bin/python script.py) there’s no need for activation.
So if you just call the python executable in your virtualenv, your virtualenv will be 'active'. So you can create a script like this:
#!/bin/bash
PATH_TO_MY_VENV=/opt/django/ev_scraper/venv/bin
$PATH_TO_MY_VENV/python -c 'import sys; print(sys.version_info)'
python -c 'import sys; print(sys.version_info)'
When I run this script on my system, the two calls to python print what you see below. (Python 3.2.3 is in my virtualenv, and 2.7.3 is my system Python.)
sys.version_info(major=3, minor=2, micro=3, releaselevel='final', serial=0)
sys.version_info(major=2, minor=7, micro=3, releaselevel='final', serial=0)
So any libraries you have installed in your virtualenv will be available when you call $PATH_TO_MY_VENV/python. Calls to your regular system python will of course be unaware of whatever is in the virtualenv.

I think the best answer here is to create a simple script and install it inside your virtualenv. Then you can either directly use the script, or create a symlink, or whatever.
Here's an example:
$ mkdir my-tool
$ cd my-tool
$ mkdir scripts
$ touch setup.py
$ mkdir scripts
$ touch scripts/crunchy-frog
$ chmod +x scripts/crunchy-frog
crunchy-frog
#!/usr/bin/env python
print("Constable Parrot ate one of those!")
setup.py
from setuptools import setup
setup(name="my-cool-tool",
scripts=['scripts/crunchy-frog'],
)
Now:
$ source /path/to/my/env/bin/activate
(env) $ python setup.py develop
(env) $ deactivate
$ cd ~
$ ln -s /path/to/my/env/bin/crunchy-frog crunchy-frog
$ ./crunchy-frog
Constable Parrot ate one of those!
When you install your script (via setup.py install or setup.py develop) then it will replace the first line of the scripts with a shebang line for the env python (which you can verify with $ head /path/to/my/env/bin/crunchy-frog). So whenever you run that particular script, it will use that specific Python env.

Does this help?
import site
site.addsitedir('/path/to/virtualenv/lib/python2.7/site-packages/')

I had this problem before and I made a simple script to look for a virtualenv folder recursively just importing and calling a function:
script_autoenv.py
# -*- coding:utf-8 -*-
import os, site
def locate_env(path, env_name):
"""search for a env directory name in each directory in the path"""
if os.path.isdir(path + "/env"):
env_26_path = '%s/%s/lib/python2.6/site-packages/' % (path, env_name)
env_27_path = '%s/%s/lib/python2.7/site-packages/' % (path, env_name)
if os.path.isdir(env_26_path):
site.addsitedir(env_26_path)
print "Virtualenv 2.6 founding"
elif os.path.isdir(env_27_path):
site.addsitedir(env_27_path)
print "Virtualenv 2.7 founding"
else:
new_path, old_dir = os.path.split(path)
if old_dir:
locate_env(new_path, env_name)
else:
print "No envs found"
You just need to specify the script directory and the env name folder and the script do the rest:
test.py
# -*- coding:utf-8 -*-
import os
import script_autoenv
script_autoenv.locate_env(os.path.realpath(__file__), 'env')
import django
print django.VERSION
I hope it's works for you

The answer may be pipenv (https://pipenv.readthedocs.io/en/latest/).
It will allow you to do something like:
pipenv run python main.py
to run main.py in the python environment with the specified libraries.
You can give it a try here https://rootnroll.com/d/pipenv/
...Maybe is not exactly what you are looking for, but it may be worth taking a look before reinventing it.