Develop Reference

PythonKit can't find PYTHON_LIBRARY for Python3

I'm using PythonKit in my Swift project for MacOS. At the moment I'm using Python 2.7 but from MacOs 12.3 it isn't no more supported so I'm trying to migrate to Python 3 but it doesn't work.
func applicationDidFinishLaunching(_ notification: Notification) {
if #available(OSX 12, *) {
PythonLibrary.useVersion(3)
}
else {
PythonLibrary.useVersion(2)
}
let sys = Python.import("sys")
print("Python \(sys.version_info.major).\(sys.version_info.minor)")
print("Python Version: \(sys.version)")
print("Python Encoding: \(sys.getdefaultencoding().upper())")
sys.path.append(Bundle.main.resourceURL!.absoluteURL.path)
let checker = Python.import("checkLibrary")
_ = Array(checker.check())
}
This is the error message:
PythonKit/PythonLibrary.swift:46: Fatal error: Python library not found. Set the PYTHON_LIBRARY environment variable with the path to a Python library.
The code fail on MacOs 12 on line 9th line (let sys = Python.import("sys")), so I can't interact so sys in any way.
I've already tried to disable sandbox and Hardened Runtime but is seems useless.

I was having the same issue.
where python3
which python3
type -a python3
I could clearly see that Python3 was present using any of the above commands from terminal. Python3 wasnt something that I directly installed, (probably something I added during an install of XCode) but I could see it located at "/usr/bin/python3"
I had already removed the sandbox and disabled the hardened runtime.
Adding the environment variable to XCode did not work and Google ultimately told me to do what had already been done.
Finally, I decided to just perform a fresh install, following the blog as guidance.
https://www.dataquest.io/blog/installing-python-on-mac/#installing-python-mac
https://www.python.org/downloads/macos/ (direct URL for Python download)
After installing, everything worked as expected.
PythonLibrary.useVersion(3)
PythonLibrary.useLibrary(at: "/usr/local/bin/python3")
I could use either of the above methods, without adding the environment variable to the XCode scheme.
Hopefully that helps

Running VIM from a Python virtulenv using Python plugins

When I start VIM after activating an Python virtulenv, Python plugins can't find their modules, because they are installed in the system, not in the project's virtualenv.
I am using the aw-watcher-vim plugin, which uses a Python library.
When I create a virtualenv and activate it:
virtualenv -p python3 my_env
. my_env/bin/activate
VIM can't find the module at /usr/local/lib/python3.6/dist-packages/aw_core/ anylonger and prints the following errors:
['Traceback (most recent call last):',
' File "~/.dot-file-repo/neovim/plugged/aw-watcher-vim/plugin/vimwatcher.py", line 6 , in <module>',
' from aw_core.log import setup_logging',
'ModuleNotFoundError: No module named ''aw_core''',
'']
(Slightly reformated for readablity.)
It would be nice to use autocompletion using the code from the virtualenv (via Jedi or so), but let the other plugins find their Python modules. Is there a good solution to this use case?

While some comments suggest to import globally instaled pacckages to virtualenv, I'd suggest to set options in /vimrc to use the system python:
set pythondll=/usr/bin/python
set pythonthreedll=/usr/bin/python3
or whatever your system python is.
The advantage is that you virtual envs are untouched and are used according to your needs.

ImportError: No module named 'thread'

when I run mitmproxy command in command line, I get the following error.
% mitmproxy
Traceback (most recent call last):
File "/usr/local/bin/mitmproxy", line 7, in <module>
from libmproxy.main import mitmproxy
File "/usr/local/lib/python3.5/site-packages/libmproxy/main.py", line 5, in <module>
import thread
ImportError: No module named 'thread'
I googled this error and found this stackoverflow Q&A page.
pydev importerror: no module named thread, debugging no longer works after pydev upgrade
according to the page above, the error occurs because module "thread" is renamed to "_thread" in python3.
So, I know what's causing this error, but then what?
I don't know what to do now in order to get rid of this error.
I'm new to python. I've just installed Python and pip into my mac OSX as shown below because I want to use mitmproxy.
% which pip
/usr/local/bin/pip
% pip --version
pip 8.1.1 from /usr/local/lib/python3.5/site-packages (python 3.5)
% which python
/usr/bin/python
% which python3
/usr/local/bin/python3
% python --version
Python 2.7.10
% python3 --version
Python 3.5.1
could anyone please tell me what to do now?
Additional Info
As #linusg answered, I created "thread.py" file in "site-packages" directory and pasted the code below in "thread.py"
from _thread import *
__all__ = ("error", "LockType", "start_new_thread", "interrupt_main", "exit", "allocate_lock", "get_ident", "stack_size", "acquire", "release", "locked")
After I did this, "ImportError: No module named 'thread'" disappeared, but now I have another ImportError, which is "import Cookie ImportError: No module named 'Cookie'".
It seems that in Python 3, Cookie module is renamed to http.cookies (stackoverflow.com/questions/3522029/django-mod-python-error).
Now what am I supposed to do?
What I have in "site-packages" directory
% ls /usr/local/lib/python3.5/site-packages (git)-[master]
ConfigArgParse-0.10.0.dist-info/ mitmproxy-0.15.dist-info/
OpenSSL/ netlib/
PIL/ netlib-0.15.1.dist-info/
Pillow-3.0.0.dist-info/ passlib/
PyYAML-3.11.dist-info/ passlib-1.6.5.dist-info/
__pycache__/ pathtools/
_cffi_backend.cpython-35m-darwin.so* pathtools-0.1.2.dist-info/
_markerlib/ pip/
_watchdog_fsevents.cpython-35m-darwin.so* pip-8.1.1.dist-info/
argh/ pkg_resources/
argh-0.26.1.dist-info/ pyOpenSSL-0.15.1.dist-info/
backports/ pyasn1/
backports.ssl_match_hostname-3.5.0.1.dist-info/ pyasn1-0.1.9.dist-info/
blinker/ pycparser/
blinker-1.4.dist-info/ pycparser-2.14.dist-info/
certifi/ pyparsing-2.0.7.dist-info/
certifi-2016.2.28.dist-info/ pyparsing.py
cffi/ pyperclip/
cffi-1.6.0.dist-info/ pyperclip-1.5.27.dist-info/
click/ setuptools/
click-6.2.dist-info/ setuptools-19.4-py3.5.egg-info/
configargparse.py sitecustomize.py
construct/ six-1.10.0.dist-info/
construct-2.5.2.dist-info/ six.py
cryptography/ test/
cryptography-1.1.2.dist-info/ thread.py
easy_install.py tornado/
hpack/ tornado-4.3.dist-info/
hpack-2.0.1.dist-info/ urwid/
html2text/ urwid-1.3.1.dist-info/
html2text-2015.11.4.dist-info/ watchdog/
idna/ watchdog-0.8.3.dist-info/
idna-2.1.dist-info/ wheel/
libmproxy/ wheel-0.26.0-py3.5.egg-info/
lxml/ yaml/
lxml-3.4.4.dist-info/

In Python 3 instead of:
import thread
Do:
import _thread

You are trying to run Python 2 code on Python 3, which will not work.
As of April 2016, mitmproxy only supports Python 2.7. We're actively working to fix that in the next months, but for now you need to use Python 2 or the binaries provided at http://mitmproxy.org.
As of August 2016, the development version of mitmproxy now supports Python 3.5+. The next release (0.18) will be the first one including support for Python 3.5+.
As of January 2017, mitmproxy only supports Python 3.5+.

Go to you site-packages folder, create a file called thread.py and paste this code in it:
from _thread import *
__all__ = ("error", "LockType", "start_new_thread", "interrupt_main", "exit", "allocate_lock", "get_ident", "stack_size", "acquire", "release", "locked")
This creates an 'alias' for the module _thread called thread. While the _thread module is very small, you can use dir() for bigger modules:
# Examle for the Cookies module which was renamed to http.cookies:
# Cookies.py in site-packages
import http.cookies
__all__ = tuple(dir(http.cookies))
Hope this helps!

Easiest solution is to create a virtualenv with python2 and run mitmproxy on this virtualenv
virtualenv -p `which python2` .env
source .env/bin/activate
pip install mitmproxy
.env/bin/mitmproxy

The name of the file saved could be threading, this would give an error as threading is a predefined class in Python. Try changing the name of your file. It would help....

Python module development workflow - setup and build [duplicate]

I'm developing my own module in python 2.7. It resides in ~/Development/.../myModule instead of /usr/lib/python2.7/dist-packages or /usr/lib/python2.7/site-packages. The internal structure is:
/project-root-dir
/server
__init__.py
service.py
http.py
/client
__init__.py
client.py
client/client.py includes PyCachedClient class. I'm having import problems:
project-root-dir$ python
Python 2.7.2+ (default, Jul 20 2012, 22:12:53)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import http
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "server/http.py", line 9, in <module>
from client import PyCachedClient
ImportError: cannot import name PyCachedClient
I didn't set PythonPath to include my project-root-dir, therefore when server.http tries to include client.PyCachedClient, it tries to load it from relative path and fails. My question is - how should I set all paths/settings in a good, pythonic way? I know I can run export PYTHONPATH=... in shell each time I open a console and try to run my server, but I guess it's not the best way. If my module was installed via PyPi (or something similar), I'd have it installed in /usr/lib/python... path and it'd be loaded automatically.
I'd appreciate tips on best practices in python module development.

My Python development workflow
This is a basic process to develop Python packages that incorporates what I believe to be the best practices in the community. It's basic - if you're really serious about developing Python packages, there still a bit more to it, and everyone has their own preferences, but it should serve as a template to get started and then learn more about the pieces involved. The basic steps are:
Use virtualenv for isolation
setuptools for creating a installable package and manage dependencies
python setup.py develop to install that package in development mode
virtualenv
First, I would recommend using virtualenv to get an isolated environment to develop your package(s) in. During development, you will need to install, upgrade, downgrade and uninstall dependencies of your package, and you don't want
your development dependencies to pollute your system-wide site-packages
your system-wide site-packages to influence your development environment
version conflicts
Polluting your system-wide site-packages is bad, because any package you install there will be available to all Python applications you installed that use the system Python, even though you just needed that dependency for your small project. And it was just installed in a new version that overrode the one in the system wide site-packages, and is incompatible with ${important_app} that depends on it. You get the idea.
Having your system wide site-packages influence your development environment is bad, because maybe your project depends on a module you already got in the system Python's site-packages. So you forget to properly declare that your project depends on that module, but everything works because it's always there on your local development box. Until you release your package and people try to install it, or push it to production, etc... Developing in a clean environment forces you to properly declare your dependencies.
So, a virtualenv is an isolated environment with its own Python interpreter and module search path. It's based on a Python installation you previously installed, but isolated from it.
To create a virtualenv, install the virtualenv package by installing it to your system wide Python using easy_install or pip:
sudo pip install virtualenv
Notice this will be the only time you install something as root (using sudo), into your global site-packages. Everything after this will happen inside the virtualenv you're about to create.
Now create a virtualenv for developing your package:
cd ~/pyprojects
virtualenv --no-site-packages foobar-env
This will create a directory tree ~/pyprojects/foobar-env, which is your virtualenv.
To activate the virtualenv, cd into it and source the bin/activate script:
~/pyprojects $ cd foobar-env/
~/pyprojects/foobar-env $ . bin/activate
(foobar-env) ~/pyprojects/foobar-env $
Note the leading dot ., that's shorthand for the source shell command. Also note how the prompt changes: (foobar-env) means your inside the activated virtualenv (and always will need to be for the isolation to work). So activate your env every time you open a new terminal tab or SSH session etc..
If you now run python in that activated env, it will actually use ~/pyprojects/foobar-env/bin/python as the interpreter, with its own site-packages and isolated module search path.
A setuptools package
Now for creating your package. Basically you'll want a setuptools package with a setup.py to properly declare your package's metadata and dependencies. You can do this on your own by by following the setuptools documentation, or create a package skeletion using Paster templates. To use Paster templates, install PasteScript into your virtualenv:
pip install PasteScript
Let's create a source directory for our new package to keep things organized (maybe you'll want to split up your project into several packages, or later use dependencies from source):
mkdir src
cd src/
Now for creating your package, do
paster create -t basic_package foobar
and answer all the questions in the interactive interface. Most are optional and can simply be left at the default by pressing ENTER.
This will create a package (or more precisely, a setuptools distribution) called foobar. This is the name that
people will use to install your package using easy_install or pip install foobar
the name other packages will use to depend on yours in setup.py
what it will be called on PyPi
Inside, you almost always create a Python package (as in "a directory with an __init__.py) that's called the same. That's not required, the name of the top level Python package can be any valid package name, but it's a common convention to name it the same as the distribution. And that's why it's important, but not always easy, to keep the two apart. Because the top level python package name is what
people (or you) will use to import your package using import foobar or from foobar import baz
So if you used the paster template, it will already have created that directory for you:
cd foobar/foobar/
Now create your code:
vim models.py
models.py
class Page(object):
"""A dumb object wrapping a webpage.
"""
def __init__(self, content, url):
self.content = content
self.original_url = url
def __repr__(self):
return "<Page retrieved from '%s' (%s bytes)>" % (self.original_url, len(self.content))
And a client.py in the same directory that uses models.py:
client.py
import requests
from foobar.models import Page
url = 'http://www.stackoverflow.com'
response = requests.get(url)
page = Page(response.content, url)
print page
Declare the dependency on the requests module in setup.py:
install_requires=[
# -*- Extra requirements: -*-
'setuptools',
'requests',
],
Version control
src/foobar/ is the directory you'll now want to put under version control:
cd src/foobar/
git init
vim .gitignore
.gitignore
*.egg-info
*.py[co]
git add .
git commit -m 'Create initial package structure.
Installing your package as a development egg
Now it's time to install your package in development mode:
python setup.py develop
This will install the requests dependency and your package as a development egg. So it's linked into your virtualenv's site-packages, but still lives at src/foobar where you can make changes and have them be immediately active in the virtualenv without re-installing your package.
Now for your original question, importing using relative paths: My advice is, don't do it. Now that you've got a proper setuptools package, that's installed and importable, your current working directory shouldn't matter any more. Just do from foobar.models import Page or similar, declaring the fully qualified name where that object lives. That makes your source code much more readable and discoverable, for yourself and other people that read your code.
You can now run your code by doing python client.py from anywhere inside your activated virtualenv. python src/foobar/foobar/client.py works just as fine, your package is properly installed and your working directory doesn't matter any more.
If you want to go one step further, you can even create a setuptools entry point for your CLI scripts. This will create a bin/something script in your virtualenv that you can run from the shell.
setuptools console_scripts entry point
setup.py
entry_points='''
# -*- Entry points: -*-
[console_scripts]
run-fooobar = foobar.main:run_foobar
''',
client.py
def run_client():
# ...
main.py
from foobar.client import run_client
def run_foobar():
run_client()
Re-install your package to activate the entry point:
python setup.py develop
And there you go, bin/run-foo.
Once you (or someone else) installs your package for real, outside the virtualenv, the entry point will be in /usr/local/bin/run-foo or somewhere simiar, where it will automatically be in $PATH.
Further steps
Creating a release of your package and uploading it PyPi, for example using zest.releaser
Keeping a changelog and versioning your package
Learn about declaring dependencies
Learn about Differences between distribute, distutils, setuptools and distutils2
Suggested reading:
The Hitchhiker’s Guide to Packaging
The pip cookbook

So, you have two packages, the first with modules named:
server # server/__init__.py
server.service # server/service.py
server.http # server/http.py
The second with modules names:
client # client/__init__.py
client.client # client/client.py
If you want to assume both packages are in you import path (sys.path), and the class you want is in client/client.py, then in you server you have to do:
from client.client import PyCachedClient
You asked for a symbol out of client, not client.client, and from your description, that isn't where that symbol is defined.
I personally would consider making this one package (ie, putting an __init__.py in the folder one level up, and giving it a suitable python package name), and having client and server be sub-packages of that package. Then (a) you could do relative imports if you wanted to (from ...client.client import something), and (b) your project would be more suitable for redistribution, not putting two very generic package names at the top level of the python module hierarchy.

Couldn't create working virtual environment for Python 3.4

I installed Python 3.4.2 and Virtualenv 12.0.5 in my Linux Mint 17.1
Then I tried creating:
$ virtualenv venv
And also using --clear and/or -p /usr/bin/python3.4, always getting the messages:
Using base prefix '/usr'
New python executable in venv/bin/python3
Also creating executable in venv/bin/python
ERROR: The executable venv/bin/python3 could not be run: [Errno 13] Permission denied
Another try was:
$ pyvenv-3.4 venv
It gave no errors on creation, but in the venv/bin file the python3.4 is a symbolic link to /usr/local/bin/python3.4. Then when I activate and install any lib using pip or pip3, then try to import it, I get the error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'anymoduledownloaded'
I always used virtualenv in Python 2.X and never got this kind of errors. Any thoughts on what am I doing wrong?
Thanks!!
=======EDITED=======
This is the output of my partitions (fdisk -l):
Device Boot Start End Blocks Id System
/dev/sda1 2048 98707455 49352704 83 Linux
/dev/sda2 303507456 3890644991 1793568768 5 Extended
/dev/sda3 * 98707456 303507455 102400000 7 HPFS/NTFS/exFAT
/dev/sda4 3890644992 3907028991 8192000 82 Linux swap / Solaris
/dev/sda5 303509504 3890644991 1793567744 7 HPFS/NTFS/exFAT`
And also my fstab:
<file system> <mount point> <type> <options> <dump> <pass>
-> was on /dev/sda1 during installation
UUID=a38f9c6d-3cd9-4486-b896-acbc6182ec61 / ext4 errors=remount-ro 0 1
-> swap was on /dev/sda4 during installation
UUID=efad7b53-79a8-4230-8226-9ca90c68ea9d none swap sw 0 0`

Is that a shared partition that you have mounted? Does the shared partition have a different filesystem then the non-shared one you tried upon? If yes, then IMO, that will definitely cause an error since you are making and compiling binaries for python on one filesystem, and so it will not work on another filesystem.
As mentioned in this answer, adding to your /etc/fstab with an entry with exec flag might make it work for you, i.e., you might need to add another entry for the NTFS disk here to make it automount:
<file system> <mount point> <type> <options> <dump> <pass>
-> was on /dev/sdaX during installation
UUID=<uid_of_NTFS> / ntfs auto,user,exec,nodev,rw,errors=remount-ro 0 1

I struggled with this as well so I wrote an ugly bash script to help me with this. The only salient difference between what you do and what I do is on line 133:
/path/to/python/bin/python3.4 /path/to/python/bin/pyvenv /path/to/venv
That is, explicitly name the instance of python and the venv tool. Then
/path/to/venv/bin/pip install django # or whatever
Edit
I installed Linux Mint in a VM to try and build a Python 3.4 virtual environment. Based on the error messages I saw and this answer, I learned that I must do the following to get a complete Python 3.4 build:
apt-get install build-essential libssl-dev openssl
Without this, my Python 3.4 build did not contain pip. Note that you probably want to install readline and other development packages.
Unsolicited Advice
Do not do this as root, create a user dedicated to running your venv
Create a script to create your environment
Check that script into your source code repo
I deleted my python binaries and venvs multiple times and then recreated all of with this script to make sure that my script reproduced my environment and then stripped the identifying information and saved that on github to share it. I should really be using a more formal tool for this like docker/puppet/chef.