Develop Reference

Add a pypi python package to buildroot

I'm trying to integrate python3-functionfs module into buildroot.
I'm able to select it with make menuconfig but when I'm running make the package isn't even downloaded.
The package is available here : functionfs-0.3 pypi page
And the download url here : functionfs-0.3 download link
There is also the github repo here : functionfs git repository
I'm using Buildroot 2017.02 version.
Here is my Config.in file :
config BR2_PACKAGE_PYTHON3_FUNCTIONFS
bool "python3-functionfs"
depends on BR2_PACKAGE_PYTHON3
help
Pythonic API for linux’s functionfs.
functionfs is part of the usb gadget subsystem. Together with usb_gadget’s configfs integration, allows userland to declare and implement an USB device.
https://pypi.python.org/pypi/functionfs
And here is my .mk file :
################################################################################
#
# python3-functionfs
#
################################################################################
PYTHON_FUNCTIONFS_VERSION = 0.3
PYTHON_FUNCTIONFS_SOURCE = functionfs-$(PYTHON_FUNCTIONFS_VERSION).tar.gz
PYTHON_FUNCTIONFS_SITE = https://pypi.python.org/packages/e3/2d/56e0d9ffe0da7c116a6724ee538375689dd59e34dbe1676edf6b66b52be4
PYTHON_FUNCTIONFS_LICENSE = GPLv3+
PYTHON_FUNCTIONFS_LICENSE_FILE = COPYING
PYTHON_FUNCTIONFS_SETUP_TYPE = setuptools
$(eval $(python-package))
The documentation also mentions at 17.8.3. Generating a python-package from a PyPI repository
If the Python package for which you would like to create a Buildroot
package is available on PyPI, you may want to use the scanpypi tool
located in utils/ to automate the process.
You can find the list of existing PyPI packages here.
scanpypi requires Python’s setuptools package to be installed on your
host.
When at the root of your buildroot directory just do :
utils/scanpypi foo bar -o package
This will generate packages python-foo and python-bar in the package
folder if they exist on https://pypi.python.org.
Find the external python modules menu and insert your package inside.
Keep in mind that the items inside a menu should be in alphabetical
order.
Please keep in mind that you’ll most likely have to manually check the
package for any mistakes as there are things that cannot be guessed by
the generator (e.g. dependencies on any of the python core modules
such as BR2_PACKAGE_PYTHON_ZLIB). Also, please take note that the
license and license files are guessed and must be checked. You also
need to manually add the package to the package/Config.in file.
If your Buildroot package is not in the official Buildroot tree but in
a br2-external tree, use the -o flag as follows:
utils/scanpypi foo bar -o other_package_dir
This will generate packages python-foo and python-bar in the
other_package_directory instead of package.
Option -h will list the available options:
utils/scanpypi -h
However I don't have an util/ folder in buildroot main directory.
The script is located at support/scripts/scanpypi but when I'm running it I have the following error :
$ support/scripts/scanpypi functionfs -o package
Traceback (most recent call last):
File "support/scripts/scanpypi", line 47, in <module>
import setuptools
File "/usr/local/lib/python2.7/dist-packages/setuptools/__init__.py", line 11, in <module>
from setuptools.extern.six.moves import filterfalse, map
File "/usr/local/lib/python2.7/dist-packages/setuptools/extern/__init__.py", line 1, in <module>
from pkg_resources.extern import VendorImporter
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 40, in <module>
from pkgutil import get_importer
ImportError: cannot import name get_importer
This can be solved by renaming both support/scripts/pkgutil.py and support/scripts/pkgutil.pyc.
However I would like to understand what's going on when I'm trying to create the package by myself and it doesn't download.
Does someone know why functionfs-0.3.tar.gz is not downloaded when running make ?

Your package is not downloaded because there is a mismatch between your package name and the name of the variables in the .mk files. Basically, you have three things that must match:
The BR2_PACKAGE_<FOO> option in Config.in
The name of the file and directory must be package/<foo>/<foo>.mk
The variables in the .mk files must be named <FOO>_SOMETHING
You didn't say what the name of the .mk file was, but at the very least your option is named BR2_PACKAGE_PYTHON3_FUNCTIONFS while the make variables are named PYTHON_FUNCTIONFS_SOMETHING.
That explains why it isn't downloaded.
Then, regarding the scanpypi script, it is definitely in the utils/ directory in recent versions of Buildroot. It used to be in support/scripts a few releases ago. So basically you're looking at the Buildroot documentation that is online (and matches the latest release), but you're using an older Buildroot version. You can build the Buildroot documentation matching your Buildroot release by running make manual-html.

pdfminer - ImportError: No module named pdfminer.pdfdocument

I am trying to install pdfMiner to work with CollectiveAccess. My host (pair.com) has given me the following information to help in this quest:
When compiling, it will likely be necessary to instruct the
installation to use your account space above, and not try to install
into the operating system directories. Typically, using "--
home=/usr/home/username/pdfminer" at the end of the install command
should allow for that.
I followed this instruction when trying to install.
The result was:
running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /usr/home/username/pdfminer/bin/latin2ascii.py to 755
changing mode of /usr/home/username/pdfminer/bin/pdf2txt.py to 755
changing mode of /usr/home/username/pdfminer/bin/dumppdf.py to 755
running install_egg_info
Removing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
Writing /usr/home/username/pdfminer/lib/python/pdfminer-20140328.egg-info
I don't see anything wrong with that (I'm very new to python), but when I try to run the sample command $ pdf2txt.py samples/simple1.pdf I get this error:
Traceback (most recent call last): File "pdf2txt.py", line 3, in <module>
from pdfminer.pdfdocument import PDFDocument ImportError: No module named pdfminer.pdfdocument
I'm running python 2.7.3. I can't install from root (shared hosting). The most recent version of pdfminer, which is 2014/03/28.
I've seen some posts on similar issues ("no module named. . . " but nothing exactly the same. The proposed solutions either don't help (such as installing with sudo - not an option; specifying the path for python (which doesn't seem to be the issue), etc.).
Or is this a question for my host? (i.e., something amiss or different about their setup)

I had an error like this:
No module named 'pdfminer.pdfinterp'; 'pdfminer' is not a package
My problem was that I had named my script pdfminer.py which for the reasons that I don't know, Python took it for the original pdfminer package files and tried to compiled it.
I renamed my script to something else, deleted all the *.pyc file and __pycache__ directory and my problem was solved.

use this command worked for me and removed the error
pip install pdfminer.six

Since the package pdfminer is installed to a non-standard/non-default location, Python won't be be able to find it. In order to use it, you will need to add it to your 'pythonpath'. Three ways:
At run time, put this in your script pdf2txt.py:
import sys
# if there are no conflicting packages in the default Python Libs =>
sys.path.append("/usr/home/username/pdfminer")
or
import sys
# to always use your package lib before the system's =>
sys.path.insert(1, "/usr/home/username/pdfminer")
Note: The install path specified with --home is used as the Lib for all packages which you might want to install, not just this one. You should delete that folder and re-install with --
home=/usr/home/username/myPyLibs (or any generic name) so that when you install other packages with that install path, you would only need the one path to add to your local Lib to be able to import them:
import sys
sys.path.insert(1, "/usr/home/username/myPyLibs")
Add it to PYTHONPATH before executing your script:
export PYTHONPATH="${PYTHONPATH}:/usr/home/username/myPyLibs"
And then put that in your ~/.bashrc file (/usr/home/username/.bashrc) or .profile as applicable. This may not work for programs which are not executed from the console.
Create a VirtualEnv and install the packages you need to that.

I have a virtual environment and I had to activate it before I did a pip3 install to have the venv see it.
source ~/venv/bin/activate

Python module development workflow - setup and build [duplicate]

I'm developing my own module in python 2.7. It resides in ~/Development/.../myModule instead of /usr/lib/python2.7/dist-packages or /usr/lib/python2.7/site-packages. The internal structure is:
/project-root-dir
/server
__init__.py
service.py
http.py
/client
__init__.py
client.py
client/client.py includes PyCachedClient class. I'm having import problems:
project-root-dir$ python
Python 2.7.2+ (default, Jul 20 2012, 22:12:53)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from server import http
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "server/http.py", line 9, in <module>
from client import PyCachedClient
ImportError: cannot import name PyCachedClient
I didn't set PythonPath to include my project-root-dir, therefore when server.http tries to include client.PyCachedClient, it tries to load it from relative path and fails. My question is - how should I set all paths/settings in a good, pythonic way? I know I can run export PYTHONPATH=... in shell each time I open a console and try to run my server, but I guess it's not the best way. If my module was installed via PyPi (or something similar), I'd have it installed in /usr/lib/python... path and it'd be loaded automatically.
I'd appreciate tips on best practices in python module development.

My Python development workflow
This is a basic process to develop Python packages that incorporates what I believe to be the best practices in the community. It's basic - if you're really serious about developing Python packages, there still a bit more to it, and everyone has their own preferences, but it should serve as a template to get started and then learn more about the pieces involved. The basic steps are:
Use virtualenv for isolation
setuptools for creating a installable package and manage dependencies
python setup.py develop to install that package in development mode
virtualenv
First, I would recommend using virtualenv to get an isolated environment to develop your package(s) in. During development, you will need to install, upgrade, downgrade and uninstall dependencies of your package, and you don't want
your development dependencies to pollute your system-wide site-packages
your system-wide site-packages to influence your development environment
version conflicts
Polluting your system-wide site-packages is bad, because any package you install there will be available to all Python applications you installed that use the system Python, even though you just needed that dependency for your small project. And it was just installed in a new version that overrode the one in the system wide site-packages, and is incompatible with ${important_app} that depends on it. You get the idea.
Having your system wide site-packages influence your development environment is bad, because maybe your project depends on a module you already got in the system Python's site-packages. So you forget to properly declare that your project depends on that module, but everything works because it's always there on your local development box. Until you release your package and people try to install it, or push it to production, etc... Developing in a clean environment forces you to properly declare your dependencies.
So, a virtualenv is an isolated environment with its own Python interpreter and module search path. It's based on a Python installation you previously installed, but isolated from it.
To create a virtualenv, install the virtualenv package by installing it to your system wide Python using easy_install or pip:
sudo pip install virtualenv
Notice this will be the only time you install something as root (using sudo), into your global site-packages. Everything after this will happen inside the virtualenv you're about to create.
Now create a virtualenv for developing your package:
cd ~/pyprojects
virtualenv --no-site-packages foobar-env
This will create a directory tree ~/pyprojects/foobar-env, which is your virtualenv.
To activate the virtualenv, cd into it and source the bin/activate script:
~/pyprojects $ cd foobar-env/
~/pyprojects/foobar-env $ . bin/activate
(foobar-env) ~/pyprojects/foobar-env $
Note the leading dot ., that's shorthand for the source shell command. Also note how the prompt changes: (foobar-env) means your inside the activated virtualenv (and always will need to be for the isolation to work). So activate your env every time you open a new terminal tab or SSH session etc..
If you now run python in that activated env, it will actually use ~/pyprojects/foobar-env/bin/python as the interpreter, with its own site-packages and isolated module search path.
A setuptools package
Now for creating your package. Basically you'll want a setuptools package with a setup.py to properly declare your package's metadata and dependencies. You can do this on your own by by following the setuptools documentation, or create a package skeletion using Paster templates. To use Paster templates, install PasteScript into your virtualenv:
pip install PasteScript
Let's create a source directory for our new package to keep things organized (maybe you'll want to split up your project into several packages, or later use dependencies from source):
mkdir src
cd src/
Now for creating your package, do
paster create -t basic_package foobar
and answer all the questions in the interactive interface. Most are optional and can simply be left at the default by pressing ENTER.
This will create a package (or more precisely, a setuptools distribution) called foobar. This is the name that
people will use to install your package using easy_install or pip install foobar
the name other packages will use to depend on yours in setup.py
what it will be called on PyPi
Inside, you almost always create a Python package (as in "a directory with an __init__.py) that's called the same. That's not required, the name of the top level Python package can be any valid package name, but it's a common convention to name it the same as the distribution. And that's why it's important, but not always easy, to keep the two apart. Because the top level python package name is what
people (or you) will use to import your package using import foobar or from foobar import baz
So if you used the paster template, it will already have created that directory for you:
cd foobar/foobar/
Now create your code:
vim models.py
models.py
class Page(object):
"""A dumb object wrapping a webpage.
"""
def __init__(self, content, url):
self.content = content
self.original_url = url
def __repr__(self):
return "<Page retrieved from '%s' (%s bytes)>" % (self.original_url, len(self.content))
And a client.py in the same directory that uses models.py:
client.py
import requests
from foobar.models import Page
url = 'http://www.stackoverflow.com'
response = requests.get(url)
page = Page(response.content, url)
print page
Declare the dependency on the requests module in setup.py:
install_requires=[
# -*- Extra requirements: -*-
'setuptools',
'requests',
],
Version control
src/foobar/ is the directory you'll now want to put under version control:
cd src/foobar/
git init
vim .gitignore
.gitignore
*.egg-info
*.py[co]
git add .
git commit -m 'Create initial package structure.
Installing your package as a development egg
Now it's time to install your package in development mode:
python setup.py develop
This will install the requests dependency and your package as a development egg. So it's linked into your virtualenv's site-packages, but still lives at src/foobar where you can make changes and have them be immediately active in the virtualenv without re-installing your package.
Now for your original question, importing using relative paths: My advice is, don't do it. Now that you've got a proper setuptools package, that's installed and importable, your current working directory shouldn't matter any more. Just do from foobar.models import Page or similar, declaring the fully qualified name where that object lives. That makes your source code much more readable and discoverable, for yourself and other people that read your code.
You can now run your code by doing python client.py from anywhere inside your activated virtualenv. python src/foobar/foobar/client.py works just as fine, your package is properly installed and your working directory doesn't matter any more.
If you want to go one step further, you can even create a setuptools entry point for your CLI scripts. This will create a bin/something script in your virtualenv that you can run from the shell.
setuptools console_scripts entry point
setup.py
entry_points='''
# -*- Entry points: -*-
[console_scripts]
run-fooobar = foobar.main:run_foobar
''',
client.py
def run_client():
# ...
main.py
from foobar.client import run_client
def run_foobar():
run_client()
Re-install your package to activate the entry point:
python setup.py develop
And there you go, bin/run-foo.
Once you (or someone else) installs your package for real, outside the virtualenv, the entry point will be in /usr/local/bin/run-foo or somewhere simiar, where it will automatically be in $PATH.
Further steps
Creating a release of your package and uploading it PyPi, for example using zest.releaser
Keeping a changelog and versioning your package
Learn about declaring dependencies
Learn about Differences between distribute, distutils, setuptools and distutils2
Suggested reading:
The Hitchhiker’s Guide to Packaging
The pip cookbook

So, you have two packages, the first with modules named:
server # server/__init__.py
server.service # server/service.py
server.http # server/http.py
The second with modules names:
client # client/__init__.py
client.client # client/client.py
If you want to assume both packages are in you import path (sys.path), and the class you want is in client/client.py, then in you server you have to do:
from client.client import PyCachedClient
You asked for a symbol out of client, not client.client, and from your description, that isn't where that symbol is defined.
I personally would consider making this one package (ie, putting an __init__.py in the folder one level up, and giving it a suitable python package name), and having client and server be sub-packages of that package. Then (a) you could do relative imports if you wanted to (from ...client.client import something), and (b) your project would be more suitable for redistribution, not putting two very generic package names at the top level of the python module hierarchy.

ZeroRPC and IronPython

I would really like to use the excellent zerorpc for my project that uses IronPython, but it doesn't seem to be supported.
I have tried downloading the zip of the source for zerorpc and running
"C:\Program Files (x86)\IronPython 2.7\ipy.exe" setup.py install
but I see this along the way:
warning: build_py: byte-compiling is disabled, skipping.
This does not contradict the answer: Fast and scalable RPC between C# and CPython.
My question(s):
Is it possible to get zerorpc running with IronPython and if so, please could you give me a hint of how to do it?
If not, then are there any other equivalent packages that would provide the functionality in IronPython? There's a list here: What is the current choice for doing RPC in Python?, but does anyone know if any of these work with IronPython?
Update 2
Following #PawelJasinski suggestion and his updates to pyzmq ironpython-backend, I have tried the following:
Installed IronPython 2.7.5b2 or later
Installed 32 bit version of zeromq from http://zeromq.org/distro:microsoft-windows
Clone ironpython-backend branch from https://github.com/paweljasinski/pyzmq/tree/ironpython-backend
In pyzmq dir, run ipy.exe setup.py install --user
Clone zerorpc from https://github.com/dotcloud/zerorpc-python
In zerorpc-python, run ipy.exe setup.py install --user
Set the environment variable: set PYZMQ_BACKEND=zmq.backend.ctypes
Attempted to use the 'Cooler' class example from https://github.com/dotcloud/zerorpc-python, running with ipy.exe -X:Frames cooler.py
(cooler.py):
class Cooler(object):
""" Various convenience methods to make things cooler. """
def add_man(self, sentence):
""" End a sentence with ", man!" to make it sound cooler, and
return the result. """
return sentence + ", man!"
def add_42(self, n):
""" Add 42 to an integer argument to make it cooler, and return the
result. """
return n + 42
def boat(self, sentence):
""" Replace a sentence with "I'm on a boat!", and return that,
because it's cooler. """
return "I'm on a boat!"
import zerorpc
s = zerorpc.Server(Cooler())
s.bind("tcp://0.0.0.0:4242")
s.run()
Now I see this error:
Traceback (most recent call last):
File "C:\Users\nlindop\AppData\Roaming\Python\IronPython27\site-packages\zmq\backend\select.py", line 26, in select_backend
File "C:\Users\nlindop\AppData\Roaming\Python\IronPython27\site-packages\zmq\backend\ctypes\__init__.py", line 26, in <module>
File "C:\Users\nlindop\AppData\Roaming\Python\IronPython27\site-packages\zmq\backend\ctypes\constants.py", line 16, in <module>
ImportError: No module named ZMQ

ZeroRPC appears to be pure python and is based on pyzmq. In this case you can try pyzmq ctypes backend for IronPython. https://github.com/paweljasinski/pyzmq/tree/ironpython-backend
use IronPython 2.7.5b2 or newer
install 32-bit version of zmq from http://zeromq.org/distro:microsoft-windows
install pyzmq itself, clone, than ipy.exe setup.py install --user. Install should detect your zmq and pick the right dll
activate the ctypes backend, set the environment variable PYZMQ_BACKEND=zmq.backend.ctypes
UPDATE: ZeroRPC has dependency on gevent which is not available under IronPython, so the above instructions are valid only for pyzmq under IronPython

For the second part of the question. pyro (and its dependency serpent) support IronPython and Jython.
Warning: stay away from IronPython 2.7.5b3 - it has a bug which breaks serpent. 2.7.4 and 2.7.5b2 are ok. Next 2.7.5 has a fix.

How should I handle software packages?

I am trying to install pysqlite and have troubles with that. I found out that the most probable reason of that is missing sqlite headers and I have to install them. My platform: CentOS release 5.3 (Final). I have Python-2.6.2.
I also found out that I need .rpm files. As far as I have them I execute:
rpm -i sqlite3-devel-3.n.n.n.rpm
and everything should be fine.
However, I do not know where to find sqlite3-devel-3.n.n.n.rpm file. Should it already be on my system? I could not locate it with "locate sqlite3-devel-3". Should I download this file? If yes where I can find it and which version should I use? I mean, the .rpm file should be, probably, consistent with the version of sqlite that I have on my computer? If it is the case, how can I find out the version of my sqlite?
If I type "from pysqlite2 import dbapi2 as sqlite" I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named pysqlite2
"yum search pysqlite" gives me the following:
Loaded plugins: fastestmirror
Excluding Packages in global exclude
list Finished
==== Matched: pysqlite ==== python-sqlite.x86_64 : Python bindings
for sqlite.
By the way, I have the following directory:
/home/myname/opt/lib/python2.6/sqlite3
and there I have the following files:
dbapi2.py dbapi2.pyc dbapi2.pyo
dump.py dump.pyc dump.pyo
__init__.py __init__.pyc __init__.pyo test
If I type "import unittest" and then "import sqlite3 as sqlite" I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/myname/opt/lib/python2.6/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import * File "/home/myname/opt/lib/python2.6/sqlite3/dbapi2.py",
line 27, in <module>
from _sqlite3 import * ImportError: No module named _sqlite3
Thank you in advance.

Python 2.6 (and some earlier) include sqlite Python org library ref so you should not need to do this. Just import it and run

You can use buildout to create localized version of your project. This will install all necessary packages without having sudo access to the server.
To give it try, do the following:
mkdir tmp
cd tmp
wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
python bootstrap.py init
vim buildout.cfg
edit buildout.cfg and replace it with following:
[buildout]
parts = sqlite
[sqlite]
recipe = zc.recipe.egg
eggs = pysqlite
interpreter = mypython
Now, run ./bin/buildout to rebuild the project. This will download all of the necessary packages and create a new interpreter for you that you can use test that you can access sqlite.
./bin/buildout
./bin/mypython
>>> import sqlite3
This gives you a controlled environment that you can use to develop inside of.
To learn more about buildout, you can watch videos from pycon 2009 on Setuptools, Distutils and Buildout.
Eggs and Buildout Deployment in Python - Part 1
Eggs and Buildout Deployment in Python - Part 2
Eggs and Buildout Deployment in Python - Part 3
Good luck

Typically, you should install the python sqlite module through yum, something like:
yum install python-sqlite
and then edit your code changing sqlite2 references to sqlite3.
By the way, whenever you read directions to install sqlite3-devel-3.n.n.n.rpm, the n parts are not literal; they're supposed to be replaced with numbers specifying a version of the rpm package.