Develop Reference

How to cx Freeze Boto3

I created this simple python program that sends a message to SQS and then retrieves it. It works using python 2.7.11.
import boto3
sqs = boto3.client('sqs')
queue = sqs.get_queue_by_name(QueueName='some-que-name')
queue.send_message(MessageBody='{"phrase": "It\'s the end of the world as we know it" }' )
for message in queue.receive_messages():
print message.body
I then cxFreeze it with this script:
from cx_Freeze import setup, Executable
include_mods = []
excludes = ['tkinter', 'cltk']
buildOptions = dict(packages=[], excludes=excludes, includes=include_mods)
executables = [
Executable('./frozen_boto_3_test.py', 'Console')
]
setup(name='Boto3FrozenTest',
version='1',
description='A test to make sure boto3 is working well when frozen',
options=dict(build_exe=buildOptions),
executables=executables)
I then get this error when I try to run the frozen code
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
exec(code, m.__dict__)
File "./frozen_boto_3_test.py", line 1, in <module>
import boto3
File "/usr/local/lib/python2.7/site-packages/boto3/__init__.py", line 16, in <module>
from boto3.session import Session
File "/usr/local/lib/python2.7/site-packages/boto3/session.py", line 17, in <module>
import botocore.session
File "/usr/local/lib/python2.7/site-packages/botocore/session.py", line 25, in <module>
import botocore.configloader
File "/usr/local/lib/python2.7/site-packages/botocore/configloader.py", line 18, in <module>
from six.moves import configparser
File "/usr/local/lib/python2.7/site-packages/six.py", line 203, in load_module
mod = mod._resolve()
File "/usr/local/lib/python2.7/site-packages/six.py", line 115, in _resolve
return _import_module(self.mod)
File "/usr/local/lib/python2.7/site-packages/six.py", line 82, in _import_module
__import__(name)
ImportError: No module named ConfigParser
In addition to this problem, the library seem to dynamically load services that are not s3, dynamo, or one other service.
Is there a recipe to freezing boto3 ?

The error shows that a hidden (dynamic) import is taking place. If you include the module it is looking for (ConfigParser) in the list of modules you tell cx_Freeze to include, it should work. You may have to do this multiple times.
executables = [cx_Freeze.Executable("MyScript.py")]
includes = ["ConfigParser"]
buildOptions = dict(includes = includes)
cx_Freeze.setup(name, description, options = dict(build_exe = buildOptions),
executables = executables)
Once you have a working program, you can also do this instead of manipulating your specific setup.py. You can add an entry into the cx_Freeze.hooks module that looks something like this:
def load_boto3(finder, module):
finder.IncludeModule("ConfigParser")
Include any others that you discover along the way. Then create a pull request or issue over here:
https://bitbucket.org/anthony_tuininga/cx_freeze

Thanks #Anthony Tuininga
There were a few additional steps that I am posting here:
Added "ConfigParser" and "HTMLParser" as noted by Anthony
from cx_Freeze import setup, Executable
include_mods = ["ConfigParser", "HTMLParser"]
excludes = ['tkinter', 'cltk']
buildOptions = dict(packages=[], excludes=excludes, includes=include_mods)
executables = [
Executable('./frozen_boto_3_test.py', 'Console')
]
setup(name='Boto3FrozenTest',
version='1',
description='A test to make sure boto3 is working well when frozen',
options=dict(build_exe=buildOptions),
executables=executables)
export AWS_DATA_PATH=/usr/local/lib/python2.7/dist-packages/botocore/data
from:
requests.exceptions.SSLError: [Errno 2] No such file or directory
export REQUESTS_CA_BUNDLE=/usr/local/lib/python2.7/dist-packages/botocore/vendored/requests/cacert.pem
Used lower level code to use sqs:
import boto3
sqs = boto3.client('sqs')
queue = sqs.create_queue( QueueName="my-queue" )
sqs.send_message(QueueUrl=queue["QueueUrl"] , MessageBody='{"phrase": "It\'s the end of the world as we know it" }' )
message = sqs.receive_message(QueueUrl=queue["QueueUrl"])
for msg in message['Messages']:
print m
sg['Body']
After this sqs works for the 'hello world' script posted on the OP

cx_freeze and docx - problems when freezing

I have a simple program that takes input from the user and then does scraping with selenium. Since the user doesn't have Python environment installed I would like to convert it to *.exe. I usually use cx_freeze for that and I have successfully converted .py programs to .exe. At first it was missing some modules (like lxml) but I was able to solve it. Now I think I only have problem with docx package.
This is how I initiate the new document in my program (I guess this is what causes me problems):
doc = Document()
#then I do some stuff to it and add paragraph and in the end...
doc.save('results.docx')
When I run it from python everything works fine but when I convert to exe I get this error:
Traceback (most recent call last):
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module>
exec(code, m.__dict__)
File "tribunalRio.py", line 30, in <module>
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\api.py", line 25, in Document
document_part = Package.open(docx).main_document_part
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\package.py", line 116, in open
pkg_reader = PackageReader.from_file(pkg_file)
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\pkgreader.py", line 32, in from_file
phys_reader = PhysPkgReader(pkg_file)
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\phys_pkg.py", line 31, in __new__
"Package not found at '%s'" % pkg_file
docx.opc.exceptions.PackageNotFoundError: Package not found at 'C:\Users\tyszkap\Dropbox (Dow Jones)\Python Projects\build\exe.win-a
md64-3.4\library.zip\docx\templates\default.docx'
This is my setup.py program:
from cx_Freeze import setup, Executable
executable = Executable( script = "tribunalRio.py" )
# Add certificate to the build
options = {
"build_exe": {'include_files' : ['default.docx'],
'packages' : ["lxml._elementpath", "inspect", "docx", "selenium"]
}
}
setup(
version = "0",
requires = [],
options = options,
executables = [executable])
I thought that explicitly adding default.docx to the package would solve the problem (I have even tried adding it to the library.zip but it gives me even more errors) but it didn't. I have seen this post but I don't know what they mean by:
copying the docx document.py module inside my function (instead of
using Document()
Any ideas? I know that freezing is not the best solution but I really don't want to build a web interface for such a simple program...
EDIT:
I have just tried this solution :
def find_data_file(filename):
if getattr(sys, 'frozen', False):
# The application is frozen
datadir = os.path.dirname(sys.executable)
else:
# The application is not frozen
# Change this bit to match where you store your data files:
datadir = os.path.dirname(__file__)
return os.path.join(datadir, filename)
doc = Document(find_data_file('default.docx'))
but again receive Traceback error (but the file is in this location...):
Traceback (most recent call last):
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module>
exec(code, m.__dict__)
File "tribunalRio.py", line 43, in <module>
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\api.py", line 25, in Document
document_part = Package.open(docx).main_document_part
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\package.py", line 116, in open
pkg_reader = PackageReader.from_file(pkg_file)
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\pkgreader.py", line 32, in from_file
phys_reader = PhysPkgReader(pkg_file)
File "C:\Users\tyszkap\AppData\Local\Continuum\Anaconda3\lib\site-packages\docx\opc\phys_pkg.py", line 31, in __new__
"Package not found at '%s'" % pkg_file
docx.opc.exceptions.PackageNotFoundError: Package not found at 'C:\Users\tyszkap\Dropbox (Dow Jones)\Python Projects\build\exe.win-a
md64-3.4\default.docx'
What am I doing wrong?

I expect you'll find the problem has to do with your freezing operation not placing the default Document() template in the expected location. It's stored as package data in the python-docx package as docx/templates/default.docx (see setup.py here: https://github.com/python-openxml/python-docx/blob/master/setup.py#L37)
I don't know how to fix that in your case, but that's where the problem is it looks like.

I had the same problem and managed to get around it by doing the following. First, I located the default.docx file in the site-packages. Then, I copied it in the same directory as my .py file. I also start the .docx file with Document() which has a docx=... flag, to which I assigned the value: os.path.join(os.getcwd(), 'default.docx') and now it looks like doc = Document(docx=os.path.join(os.getcwd(), 'default.docx')). The final step was to include the file in the freezing process. Et voilà! So far I have no problem.

py2exe is giving errors on my os.listdir code

When I run my code from main, it runs perfectly fine, but when I try to build main into an exe using py2exe, it gives this error:
Traceback (most recent call last):
File "main.py", line 118, in <module>
menu.menu.Menu()
File "menu\menu.pyo", line 20, in __init__
File "settingsManager.pyo", line 61, in getSetting
File "settingsManager.pyo", line 148, in __init__
WindowsError: [Error 3] The system cannot find the path specified: 'C:\\Users\\digiholic\\git\\universalSmashSystem\\main.exe\\settings\\rules/*.*'
The line it's referring to is:
for f in os.listdir(os.path.join(os.path.dirname(__file__),'settings','rules')):
It looks like os.listdir is using unix file pathing to find every file, and Windows is having none of that. Is there a way to use listdir in a way that won't blow up everything?

When you're running within the exe you need to check if the module is frozen, the path from __file__ is generally not what you expect when you're within an exe vs the raw python script. You need to access the location with something like this:
import imp, os, sys
def main_is_frozen():
return (hasattr(sys, "frozen") or # new py2exe
hasattr(sys, "importers") # old py2exe
or imp.is_frozen("__main__")) # tools/freeze
def get_main_dir():
if main_is_frozen():
return os.path.dirname(sys.executable)
return os.path.dirname(sys.argv[0])
Source: http://www.py2exe.org/index.cgi/HowToDetermineIfRunningFromExe
You can also check another direction here: http://www.py2exe.org/index.cgi/WhereAmI

VirtualBox Python API - remote access

I'm trying to run virtual machine set on Computer with Windows 7 (my main laptop) from my netbook (Ubuntu system).
On Ubuntu I've prepared python script:
from vboxapi import VirtualBoxManager
import sys
sys.path.append("/home/myLogin/Downloads/sdk/bindings/webservice/python/lib")
mgr = VirtualBoxManager("WEBSERVICE", {'url':'myIP', 'user':'myServerLogin', 'password':'myPassthere'})
vbox = mgr.vbox
name = "Muszelek"
mach = vbox.findMachine(name)
session = mgr.mgr.getSessionObject(vbox)
progress = mach.launchVMProcess(session, "gui", "")
progress.waitForCompletion(-1)
mgr.closeMachineSession(session)
I'm getting error:
Installation problem: check that appropriate libs in place
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/vboxapi/__init__.py", line 981, in __init__
self.vbox = self.platform.getVirtualBox()
File "/usr/local/lib/python2.7/dist-packages/vboxapi/__init__.py", line 856, in getVirtualBox
return self.connect(self.url, self.user, self.password)
File "/usr/local/lib/python2.7/dist-packages/vboxapi/__init__.py", line 910, in connect
self.vbox = self.wsmgr.logon(self.user, self.password)
File "/home/karolinka/Downloads/sdk/bindings/webservice/python/lib/VirtualBox_wrappers.py", line 11790, in logon
req=IWebsessionManager_logonRequestMsg()
NameError: global name 'IWebsessionManager_logonRequestMsg' is not defined
Traceback (most recent call last):
File "vmmach.py", line 5, in <module>
mgr = VirtualBoxManager("WEBSERVICE", {'url':'myIP', 'user':'myComputerLogin', 'password':'myPass'})
File "/usr/local/lib/python2.7/dist-packages/vboxapi/__init__.py", line 985, in __init__
raise ne
NameError: global name 'IWebsessionManager_logonRequestMsg' is not defined
Any idea how could I solve this issue?
Is it problem with libs on my Windows Machine?

I just had the exact same problem on my Mac. The issue is that VirtualBox_wrappers.py imports VirtualBox_client.py (which defines the global name in question) in a try block and continues (pass) if it can not be imported
try:
from VirtualBox_client import *
except:
pass
the issue is that VirtualBox_client.py depends on ZSI, which was not installed, so the import failed. Once I did
easy_install zsi
everything worked as expected.

It seems that the class is not found! Have you tried to look for the webservice api if they are available?
I had a similar issue, I looked in /usr/lib/virtualbox/sdk/bindings and I didn't find the webservice folder only xpcom was in there, so I just included the webservice folder from the sdk and all worked fine.

py2exe/pyinstaller and DispatchWithEvents

I have a program that uses the win32com library to control iTunes, but have been having some issues getting it to compile into an executable. The problem seems to revolve around using DispatchWithEvents instead of Dispatch. I've created a very simple program to illustrate my problem:
import win32com.client
win32com.client.gencache.is_readonly = False #From py2exe wiki
class ITunesEvents(object):
def __init__(self): self.comEnabled = True
def OnCOMCallsDisabledEvent(self, reason): self.comEnabled = False
def OnCOMCallsEnabledEvent(self): self.comEnabled = True
# The first line works in the exe, the second doesn't.
itunes = win32com.client.Dispatch("iTunes.Application")
#itunes = win32com.client.DispatchWithEvents("iTunes.Application", ITunesEvents)
lib = getattr(itunes, "LibraryPlaylist")
src = getattr(lib, "Source")
playlists = getattr(src, "Playlists")
print "Found %i playlists." % getattr(playlists, "Count")
Using Dispatch, the program compiles and runs correctly. Using DispatchWithEvents, the program runs fine when called from the command line, but produces the following error when running the exe:
Traceback (most recent call last):
File "sandbox.py", line 16, in <module>
itunes = win32com.client.DispatchWithEvents("iTunes.Application", ITunesEvents)
File "win32com\client\__init__.pyc", line 252, in DispatchWithEvents
File "win32com\client\gencache.pyc", line 520, in EnsureModule
File "win32com\client\gencache.pyc", line 287, in MakeModuleForTypelib
File "win32com\client\makepy.pyc", line 259, in GenerateFromTypeLibSpec
File "win32com\client\gencache.pyc", line 141, in GetGeneratePath
IOError: [Errno 2] No such file or directory: '[distDir]\\library.zip\\win32com\\gen_py\\__init__.py'
I've also tried using PyInstaller, which gives a similar error:
File "<string>", line 16, in <module>
File "[outDir]/win32com.client", line 252, in DispatchWithEvents
File "[outDir]/win32com.client.gencache", line 520, in EnsureModule
File "[outDir]/win32com.client.gencache", line 287, in MakeModuleForTypelib
File "[outDir]/win32com.client.makepy", line 286, in GenerateFromTypeLibSpec
File "[outDir]/win32com.client.gencache", line 550, in AddModuleToCache
File "[outDir]/win32com.client.gencache", line 629, in _GetModule
File "[pyinstallerDir]\iu.py", line 455, in importHook
raise ImportError, "No module named %s" % fqname
ImportError: No module named win32com.gen_py.9E93C96F-CF0D-43F6-8BA8-B807A3370712x0x1x13
I know I can manually add the typelib in my setup.py file, but I'd like to run the code on computers with different versions of iTunes without recompiling so I'd prefer to dynamically create it. If there's no way to do this with the setup/spec, maybe there is another way to load the events? Thanks.
Addition:
Thanks to Ryan, I found I could take the generated py file and after a little digging, was able to come up with the following.
Take the generated py file (from makepy.py) and rename it somewhere like cominterface.py. Then you'll need to do the following to actually create the COM object with event handler.
import cominterface
from types import ClassType
from win32com.client import EventsProxy, _event_setattr_
class ItunesEvents:
'''iTunes events class. See cominterface for details.'''
def OnPlayerPlayEvent(self, t):print "Playing..."
def OnPlayerStopEvent(self, t): print "Stopping..."
itunes = cominterface.iTunesApp()
rClass = ClassType("COMEventClass", (itunes.__class__, itunes.default_source, ItunesEvents), {'__setattr__': _event_setattr_})
instance = rClass(itunes._oleobj_)
itunes.default_source.__init__(instance, instance)
#ItunesEvents.__init__(instance) #Uncomment this line if your events class has __init__.
itunes = EventsProxy(instance)
Then you can go about your business.

I was experiencing the exact same error.
This link put me in the right direction -->
http://www.py2exe.org/index.cgi/UsingEnsureDispatch
however it mentions that :
NB You must ensure that the python...\win32com.client.gen_py dir does not exist
to allow creation of the cache in %temp%
Which was a bit confusing.
What solved it for me was renaming "C:\Python26\Lib\site-packages\win32com\gen_py" to "C:\Python26\Lib\site-packages\win32com\gen_pybak" (when running py2exe)

This is the official way to do it.
(Edit: copied verbatim example from that link above)
import win32com.client
if win32com.client.gencache.is_readonly == True:
#allow gencache to create the cached wrapper objects
win32com.client.gencache.is_readonly = False
# under p2exe the call in gencache to __init__() does not happen
# so we use Rebuild() to force the creation of the gen_py folder
win32com.client.gencache.Rebuild()
# NB You must ensure that the python...\win32com.client.gen_py dir does not exist
# to allow creation of the cache in %temp%
# Use SAPI speech through IDispatch
from win32com.client.gencache import EnsureDispatch
from win32com.client import constants
voice = EnsureDispatch("Sapi.SpVoice", bForDemand=0)
voice.Speak( "Hello World.", constants.SVSFlagsAsync )

Instead of depending on the cache, I'd recommend going into the local cache directory, copying the generated file into your local project file, and naming it something like ITunesInterface.py, and calling to that explicitly. This will make py2exe pull it into your compiled app.