Develop Reference

python ctypes.CDLL does not load library although location is know

To run one of my script I have to use the shapely python package, but it crashes because it cannot load the libgeos library. It gives me the following error message:
Traceback (most recent call last):
File "download_nl.py", line 2, in <module>
from download_sentinel import sentinel_available, sentinel_download, download_orbits
File "/nfs/home2/gmulder/radar_database/download_sentinel.py", line 14, in <module>
from fastkml import kml
File "/home/gmulder/python_packages/fastkml/__init__.py", line 30, in <module>
from .kml import KML, Document, Folder, Placemark
File "/home/gmulder/python_packages/fastkml/kml.py", line 37, in <module>
from .geometry import Geometry
File "/home/gmulder/python_packages/fastkml/geometry.py", line 23, in <module>
from shapely.geometry import Point, LineString, Polygon
File "/home/gmulder/python_packages/shapely/geometry/__init__.py", line 4, in <module>
from .base import CAP_STYLE, JOIN_STYLE
File "/home/gmulder/python_packages/shapely/geometry/base.py", line 9, in <module>
from shapely.coords import CoordinateSequence
File "/home/gmulder/python_packages/shapely/coords.py", line 8, in <module>
from shapely.geos import lgeos
File "/home/gmulder/python_packages/shapely/geos.py", line 73, in <module>
_lgeos = load_dll('geos_c', fallbacks=['libgeos_c.so.1', 'libgeos_c.so'])
File "/home/gmulder/python_packages/shapely/geos.py", line 68, in load_dll
libname, fallbacks or []))
OSError: Could not find lib geos_c or load any of its variants ['libgeos_c.so.1', 'libgeos_c.so'].
However, the strange thing is that, when I run python interactively, the import goes fine and I do not get any errors. To check whether it is a problem with setting the right paths, I also checked the source. The original code is the following, where the libname gives the library name.
def load_dll(libname, fallbacks=None, mode=DEFAULT_MODE):
lib = find_library(libname)
dll = None
print(sys.version)
dirname = '/hpc/sw/geos-3.5.0/lib'
os.environ['PATH'] = dirname + os.pathsep + os.environ['PATH']
if os.path.exists(os.path.join(dirname, 'libgeos_c.so.1')):
print('library does exist!')
os.chdir(dirname)
if lib is not None:
try:
LOG.debug("Trying `CDLL(%s)`", lib)
dll = CDLL(lib)
except OSError:
LOG.warn("Failed `CDLL(%s)`", lib)
pass
if not dll and fallbacks is not None:
for name in fallbacks:
try:
LOG.debug("Trying `CDLL(%s)`", name)
dll = CDLL(name)
except OSError:
# move on to the next fallback
LOG.warn("Failed `CDLL(%s)`", name)
pass
print(dll)
if dll:
LOG.debug("Library path: %r", lib or name)
LOG.debug("DLL: %r", dll)
return dll
else:
# No shared library was loaded. Raise OSError.
raise OSError(
"Could not find lib {0} or load any of its variants {1}.".format(
libname, fallbacks or []))
To check whether the needed lib file location can be found by the code I added an extra check at line 7, which confirms the library is there. But then, when I try to load it, nothing is loaded, even if I feed the full path to the CDLL function directly.
I also tried to load the library by running python and opening the library interactively (not from a python script), which worked fine. Therefore, I checked the python versions, because I thought that could maybe cause this problem. But the python versions for my script and the interactive session are the same. Can someone help me with this? Likely I missed something, but I have no idea what is causing this problem.

cx_freeze and docx - problems when freezing

I have a simple program that takes input from the user and then does scraping with selenium. Since the user doesn't have Python environment installed I would like to convert it to *.exe. I usually use cx_freeze for that and I have successfully converted .py programs to .exe. At first it was missing some modules (like lxml) but I was able to solve it. Now I think I only have problem with docx package.
This is how I initiate the new document in my program (I guess this is what causes me problems):
doc = Document()
#then I do some stuff to it and add paragraph and in the end...
doc.save('results.docx')
When I run it from python everything works fine but when I convert to exe I get this error:
Traceback (most recent call last):
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module>
exec(code, m.__dict__)
File "tribunalRio.py", line 30, in <module>
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\api.py", line 25, in Document
document_part = Package.open(docx).main_document_part
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\package.py", line 116, in open
pkg_reader = PackageReader.from_file(pkg_file)
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\pkgreader.py", line 32, in from_file
phys_reader = PhysPkgReader(pkg_file)
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\phys_pkg.py", line 31, in __new__
"Package not found at '%s'" % pkg_file
docx.opc.exceptions.PackageNotFoundError: Package not found at 'C:\Users\tyszkap\Dropbox (Dow Jones)\Python Projects\build\exe.win-a
md64-3.4\library.zip\docx\templates\default.docx'
This is my setup.py program:
from cx_Freeze import setup, Executable
executable = Executable( script = "tribunalRio.py" )
# Add certificate to the build
options = {
"build_exe": {'include_files' : ['default.docx'],
'packages' : ["lxml._elementpath", "inspect", "docx", "selenium"]
}
}
setup(
version = "0",
requires = [],
options = options,
executables = [executable])
I thought that explicitly adding default.docx to the package would solve the problem (I have even tried adding it to the library.zip but it gives me even more errors) but it didn't. I have seen this post but I don't know what they mean by:
copying the docx document.py module inside my function (instead of
using Document()
Any ideas? I know that freezing is not the best solution but I really don't want to build a web interface for such a simple program...
EDIT:
I have just tried this solution :
def find_data_file(filename):
if getattr(sys, 'frozen', False):
# The application is frozen
datadir = os.path.dirname(sys.executable)
else:
# The application is not frozen
# Change this bit to match where you store your data files:
datadir = os.path.dirname(__file__)
return os.path.join(datadir, filename)
doc = Document(find_data_file('default.docx'))
but again receive Traceback error (but the file is in this location...):
Traceback (most recent call last):
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module>
exec(code, m.__dict__)
File "tribunalRio.py", line 43, in <module>
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\api.py", line 25, in Document
document_part = Package.open(docx).main_document_part
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\package.py", line 116, in open
pkg_reader = PackageReader.from_file(pkg_file)
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\pkgreader.py", line 32, in from_file
phys_reader = PhysPkgReader(pkg_file)
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\phys_pkg.py", line 31, in __new__
"Package not found at '%s'" % pkg_file
docx.opc.exceptions.PackageNotFoundError: Package not found at 'C:\Users\tyszkap\Dropbox (Dow Jones)\Python Projects\build\exe.win-a
md64-3.4\default.docx'
What am I doing wrong?

I expect you'll find the problem has to do with your freezing operation not placing the default Document() template in the expected location. It's stored as package data in the python-docx package as docx/templates/default.docx (see setup.py here: https://github.com/python-openxml/python-docx/blob/master/setup.py#L37)
I don't know how to fix that in your case, but that's where the problem is it looks like.

I had the same problem and managed to get around it by doing the following. First, I located the default.docx file in the site-packages. Then, I copied it in the same directory as my .py file. I also start the .docx file with Document() which has a docx=... flag, to which I assigned the value: os.path.join(os.getcwd(), 'default.docx') and now it looks like doc = Document(docx=os.path.join(os.getcwd(), 'default.docx')). The final step was to include the file in the freezing process. Et voilà! So far I have no problem.

R packages Rssa is not imported to python with rpy2 importr

When I executed in Python command
rssa = importr('Rssa')
I got eroor
Traceback (most recent call last):
File "<pyshell#3>", line 1, in <module>
rssa = importr('Rssa')
File "C:\Python34\lib\site-packages\rpy2\robjects\packages.py", line 412, in importr
version = version)
File "C:\Python34\lib\site-packages\rpy2\robjects\packages.py", line 178, in __init__
self.__fill_rpy2r__(on_conflict = on_conflict)
File "C:\Python34\lib\site-packages\rpy2\robjects\packages.py", line 280, in __fill_rpy2r__
super(SignatureTranslatedPackage, self).__fill_rpy2r__(on_conflict = on_conflict)
File "C:\Python34\lib\site-packages\rpy2\robjects\packages.py", line 233, in __fill_rpy2r__
rpyobj = conversion.ri2ro(riobj)
File "C:\Python34\lib\functools.py", line 707, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
File "C:\Python34\lib\site-packages\rpy2\robjects\__init__.py", line 101, in _
return SignatureTranslatedFunction(obj)
File "C:\Python34\lib\site-packages\rpy2\robjects\functions.py", line 150, in __init__
raise ValueError("Error: '%s' already in the translation table. This means that the signature of the R function contains the parameters '%s' and/or '%s' <sigh> in multiple copies." %(r_param, r_param, prm_translate[py_param]))
ValueError: Error: '...' already in the translation table. This means that the signature of the R function contains the parameters '...' and/or '...' <sigh> in multiple copies.
Other packages are imported without problem, as example
stats = importr('stats')
tseries = importr('tseries')
forecast = importr('forecast')
I was looking for such problem, but I could not find nothing close. Please, suggest some decision of this problem.

This was caused by bug in Rssa when it tries to take control over 'decompose' function (which is exported from 'stats' package). In particular, in the list of formals '...' is added twice and this is what rpy2 is complaining about.
This will be fixed in the subsequent Rssa releases.
The only workaround is to comment out in R/common.R the following line:
formals(decompose.default) <- c(formals(decompose.default), alist(... = ))
And reinstall Rssa from source. This might be non-trivial on Windows, though R windows builder can help here.

What dependencies do I need for USB programing in python with pyUSB?

I am trying to get the usb.find command to work properly in a python script I'm writing on Angstrom for the Beagleboard.
Here is my code:
#!/usr/bin/env python
import usb.core
import usb.util
import usb.backend.libusb01 as libusb
PYUSB_DEBUG_LEVEL = 'debug'
# find our device
# Bus 002 Device 006: ID 1208:0815
# idVendor 0x1208
# idProduct 0x0815
# dev = usb.core.find(idVendor=0xfffe, idProduct=0x0001)
# iManufacturer 1 TOROBOT.com
dev = usb.core.find(idVendor=0x1208, idProduct=0x0815,
backend=libusb.get_backend() )
I don't know what's missing, but here is what I do know.
When I don't specify the backend, no backend is found. When I do specify the backend "usb.backend.libusb01" I get the following error:
root#beagleboard:~/servo# ./pyServo.py
Traceback (most recent call last):
File "./pyServo.py", line 17, in <module>
dev = usb.core.find(idVendor=0x1208, idProduct=0x0815, backend=libusb.get_backend() )
File "/usr/lib/python2.6/site-packages/usb/core.py", line 854, in find
return _interop._next(device_iter(k, v))
File "/usr/lib/python2.6/site-packages/usb/_interop.py", line 60, in _next
return next(iter)
File "/usr/lib/python2.6/site-packages/usb/core.py", line 821, in device_iter
for dev in backend.enumerate_devices():
File "/usr/lib/python2.6/site-packages/usb/backend/libusb01.py", line 390, in enumerate_devices
_check(_lib.usb_find_busses())
File "/usr/lib/python2.6/ctypes/__init__.py", line 366, in __getattr__
func = self.__getitem__(name)
File "/usr/lib/python2.6/ctypes/__init__.py", line 371, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python: undefined symbol: usb_find_busses
What am I missing so that this will work properly?
Thank you.

Python, pyusb, libusb.
I think that's it.
Well libc, ld-linux.so too of course, but those are in your system by default.
Do nm -D /path-to/libusb.so and see if usb_find_busses symbol is really there. There's always a chance that your libusb is outdated or somehow specially compiled.
Check if you get any log, for example missing libusb.so should produce something like 'Error loading libusb 0.1 backend'

py2exe/pyinstaller and DispatchWithEvents

I have a program that uses the win32com library to control iTunes, but have been having some issues getting it to compile into an executable. The problem seems to revolve around using DispatchWithEvents instead of Dispatch. I've created a very simple program to illustrate my problem:
import win32com.client
win32com.client.gencache.is_readonly = False #From py2exe wiki
class ITunesEvents(object):
def __init__(self): self.comEnabled = True
def OnCOMCallsDisabledEvent(self, reason): self.comEnabled = False
def OnCOMCallsEnabledEvent(self): self.comEnabled = True
# The first line works in the exe, the second doesn't.
itunes = win32com.client.Dispatch("iTunes.Application")
#itunes = win32com.client.DispatchWithEvents("iTunes.Application", ITunesEvents)
lib = getattr(itunes, "LibraryPlaylist")
src = getattr(lib, "Source")
playlists = getattr(src, "Playlists")
print "Found %i playlists." % getattr(playlists, "Count")
Using Dispatch, the program compiles and runs correctly. Using DispatchWithEvents, the program runs fine when called from the command line, but produces the following error when running the exe:
Traceback (most recent call last):
File "sandbox.py", line 16, in <module>
itunes = win32com.client.DispatchWithEvents("iTunes.Application", ITunesEvents)
File "win32com\client\__init__.pyc", line 252, in DispatchWithEvents
File "win32com\client\gencache.pyc", line 520, in EnsureModule
File "win32com\client\gencache.pyc", line 287, in MakeModuleForTypelib
File "win32com\client\makepy.pyc", line 259, in GenerateFromTypeLibSpec
File "win32com\client\gencache.pyc", line 141, in GetGeneratePath
IOError: [Errno 2] No such file or directory: '[distDir]\\library.zip\\win32com\\gen_py\\__init__.py'
I've also tried using PyInstaller, which gives a similar error:
File "<string>", line 16, in <module>
File "[outDir]/win32com.client", line 252, in DispatchWithEvents
File "[outDir]/win32com.client.gencache", line 520, in EnsureModule
File "[outDir]/win32com.client.gencache", line 287, in MakeModuleForTypelib
File "[outDir]/win32com.client.makepy", line 286, in GenerateFromTypeLibSpec
File "[outDir]/win32com.client.gencache", line 550, in AddModuleToCache
File "[outDir]/win32com.client.gencache", line 629, in _GetModule
File "[pyinstallerDir]\iu.py", line 455, in importHook
raise ImportError, "No module named %s" % fqname
ImportError: No module named win32com.gen_py.9E93C96F-CF0D-43F6-8BA8-B807A3370712x0x1x13
I know I can manually add the typelib in my setup.py file, but I'd like to run the code on computers with different versions of iTunes without recompiling so I'd prefer to dynamically create it. If there's no way to do this with the setup/spec, maybe there is another way to load the events? Thanks.
Addition:
Thanks to Ryan, I found I could take the generated py file and after a little digging, was able to come up with the following.
Take the generated py file (from makepy.py) and rename it somewhere like cominterface.py. Then you'll need to do the following to actually create the COM object with event handler.
import cominterface
from types import ClassType
from win32com.client import EventsProxy, _event_setattr_
class ItunesEvents:
'''iTunes events class. See cominterface for details.'''
def OnPlayerPlayEvent(self, t):print "Playing..."
def OnPlayerStopEvent(self, t): print "Stopping..."
itunes = cominterface.iTunesApp()
rClass = ClassType("COMEventClass", (itunes.__class__, itunes.default_source, ItunesEvents), {'__setattr__': _event_setattr_})
instance = rClass(itunes._oleobj_)
itunes.default_source.__init__(instance, instance)
#ItunesEvents.__init__(instance) #Uncomment this line if your events class has __init__.
itunes = EventsProxy(instance)
Then you can go about your business.

I was experiencing the exact same error.
This link put me in the right direction -->
http://www.py2exe.org/index.cgi/UsingEnsureDispatch
however it mentions that :
NB You must ensure that the python...\win32com.client.gen_py dir does not exist
to allow creation of the cache in %temp%
Which was a bit confusing.
What solved it for me was renaming "C:\Python26\Lib\site-packages\win32com\gen_py" to "C:\Python26\Lib\site-packages\win32com\gen_pybak" (when running py2exe)

This is the official way to do it.
(Edit: copied verbatim example from that link above)
import win32com.client
if win32com.client.gencache.is_readonly == True:
#allow gencache to create the cached wrapper objects
win32com.client.gencache.is_readonly = False
# under p2exe the call in gencache to __init__() does not happen
# so we use Rebuild() to force the creation of the gen_py folder
win32com.client.gencache.Rebuild()
# NB You must ensure that the python...\win32com.client.gen_py dir does not exist
# to allow creation of the cache in %temp%
# Use SAPI speech through IDispatch
from win32com.client.gencache import EnsureDispatch
from win32com.client import constants
voice = EnsureDispatch("Sapi.SpVoice", bForDemand=0)
voice.Speak( "Hello World.", constants.SVSFlagsAsync )

Instead of depending on the cache, I'd recommend going into the local cache directory, copying the generated file into your local project file, and naming it something like ITunesInterface.py, and calling to that explicitly. This will make py2exe pull it into your compiled app.