I am trying to convert a python package to a linux binary (and eventually a windows executable as well) using cx_Freeze. The package has dependency upon multiple egg files, as i understand cx_Freeze doesn't play nice with egg files, so i unzipped the egg files. One of the egg files has a resource string file 'test.resource' in some package 'test.package', to include this resource string file i used -
include_files = ['test/package/test.resource']
Now i see this file is being copied to the target directory along with the binary but when i try to run the binary i get the error - "IOError: [Errno 2] No such file or directory: 'test/package/test.resource'"
The code trying to read the file is doing this:
from pkg_resources import resource_string
strings = resource_string("test.package", "test.resource")
How can i add this resource file so that it's available to the generated binary?
Since the accepted answer is light on details, I thought I'd provide a solution that worked for me. It does not require that you unzip your egg files, but does not preclude it either.
The situation: I have an application that we package with esky using the cx_Freeze freezer, but this information should apply equally well to applications just using cx_Freeze directly. The application makes use of the treq library which includes the following line in treq.__import__:
__version__ = resource_string(__name__, "_version").strip()
I use pkg_resources to determine the filepath of the _version file and manually add it to my zip_includes. We make sure it gets placed into its expected location alongside the rest of the treq library files. In my setup.py file I have:
setup(
name=...,
version=...,
install_requires=[
...,
treq==15.0.0,
...
],
...,
options={
...,
"bdist_esky": {
...,
"freezer_options": {
# This would be "zip_includes" in options if just using cx_Freeze without esky
"zipIncludes": [
# Manually copy the treq _version resource file into our library.zip
(pkg_resources.resource_filename('treq', '_version'), 'treq/_version')
],
"packages": [..., "treq", ...]
...,
...,
...,
...,
)
Bonus: While one does not have to unzip libraries in order for this to work, it seems to be useful enough in general that you might want to always unzip them upon installation (a la python setup.py develop). It's a simple matter of adding the following lines to your application's setup.cfg:
[easy_install]
zip_ok = 0
Then, when you run python setup.py develop, your dependencies are installed into site-packages in their expanded directory form.
In case anyone needs the answer i got it working by using 'zip_includes' instead of 'include_files'
Related
The library I'm working on generates python files according to an executable (which turns ANTLRv4 .g4 files into python files), and I have the following install step:
import os
import subprocess
from setuptools import setup
from setuptools.command.install import install
class AntlrInstallCommand(install):
def run(self):
output_dir = compile_grammar()
print(f"Compiled ANTLRv4 grammar in {output_dir}")
install.run(self)
def compile_grammar():
parser_dir = os.path.join(os.path.dirname(__file__), "my_project/grammar")
subprocess.check_output(
["antlr", "MyGrammar.g4", "-Dlanguage=Python3", "-visitor", "-o", "gen"],
cwd=parser_dir,
)
# The result is created in the subfolder `gen`
return os.path.join(parser_dir, "gen")
setup(
...
install_requires=[
"setuptools",
"antlr4-python3-runtime==4.9.2",
...
],
cmdclass={"install": AntlrInstallCommand},
license="MIT",
python_requires=">=3.6",
)
Which works great if I'm pip install'ing the project on a machine that has antlr installed (since I'm calling it via subprocess).
Ideally, attempting to do this on a machine that doesn't have antlr installed would first install the executable(with the correct version) in either a system directory like /usr/bin, or whatever relevant python bin directory we're working in, but right now it errors out with the following message(which is expected):
running install
error: [Errno 2] No such file or directory: 'antlr'
----------------------------------------
ERROR: Failed building wheel for my_project
I see a couple of solutions each with slight caveats:
sympy uses ANTLR, but it requires the user to install antlr first. See here
setuptools-antlr allows me to download an antlr jar as a giant blob in a python package, and then I can invoke it here. However, the version doesn't match mine (which is 4.9.2).
java2python precompiles the files for me and writes them into the github repo. However, these files are extremely large and are very hard to read as they're autogenerated. If I slightly modify the grammar and don't modify the parser it would also lead to unexpected bugs. As a result, I would like to hide this complexity from the repository as it's tangential to development.
If I can get the right version of the antlr binary and be able to invoke it at install time, that would be optimal. Otherwise I'm okay with picking one of these alternatives. Any suggestions for either case would be appreciated.
Let's say numpy_example.py is:
import numpy as np
a = np.array([1, 2, 3])
print(a)
With Python 2.7.9, generating an executable from this with cxFreeze 4.3.3, with:
"C:\python27\python.exe" "C:\python27\Scripts\cxfreeze" numpy_example.py
--target-dir=.\cxfreeze_x64_py2 --base-name=Win32GUI
--target-name=test.exe --exclude-modules=Tkinter,scipy,Crypto,_ssl,bz2,_yaml
--include-modules=lxml._elementpath
gave 25 files (only .dll, .pyd and .exe files) for a total of 71 MB, with no subfolder:
With Python 3.7.6, generating an executable with cxFreeze 6, with the same command-line command gives a total of 8533 files for a total of 374 MB!
In particular, there is the Lib subfolder containing many libraries unused for this project:
Also, with the Python 2 version, Numpy was packed into 9 .pyd files (and nothing else), whereas here in the Python 3 version it's split in many files and subfolders.
Question: what has changed in cx_freeze, resulting in this less efficient packing?
And how to get a similar packing with cx_freeze for Python 3 than with Python 2?
This seems to be the answer: it's a feature, since cx_freeze 5.0: https://cx-freeze.readthedocs.io/en/latest/releasenotes.html#version-5-0-november-2016
Added support for storing packages in the file system instead of in the zip file. There are a number of packages that assume that they are found in the file system and if found in a zip file instead produce strange errors. The default is now to store packages in the file system but a method is available to place packages in the zip file if they are known to behave properly when placed there. (Issue #73)
Linked: Cx_Freeze build is not including python libraries in zip file
As you pointed out in your own answer, this appears to be the desired functionality. There are other packages out there that can compile to a single executable.
I assume you only want to use cxfreeze, if you know which packages your program doesn't need, you can simply list all of them with the --exclude-modules flag as you have been. You can filter out packages through trial and error by building your application and then renaming the included libraries in the lib directory one by one, adding any unnecessary packages to your excludes list. I was able to reduce the numpy example from 3500~ files to ~1000 this way.
You can utilise the distutils functionality which I found easier when trying to reduce the final package size.
from cx_Freeze
import setup, Executable
import sys
build_exe_options = {
"excludes": ["tkinter", ...your excludes here...],
"optimize": 0
}
setup(
name = "TestProgram",
version = "0.1",
description = "MyDescription",
options = {
"build_exe": build_exe_options
},
executables = [Executable("main.py",
base = ("Win32GUI"
if sys.platform == "win32"
else None))]
)
py setup.py build
Finally, the cxfreeze FAQ has a question about creating a single file, although that's not what you're after, it states to use IExpress to create a self-extracting archive (or 7zip), which might be useful to you if package size is the main concern.
I'm trying to use the CFFI package in Python to create a Python interface for already existing C-code.
I am able to compile a C library by following this blog post. Now I want to make it so that this python library is available without any fancy updates to the sys.path.
I found that maybe creating a distribution through Python's setuptools setup() function would accomplish this and I got it to mostly work by creating a setup.py file as such
import os
import sys
from setuptools import setup, find_packages
os.chdir(os.path.dirname(sys.argv[0]) or ".")
setup(
name="SobelFilterTest",
version="0.1",
description="An example project using Python's CFFI",
packages=find_packages(),
install_requires=["cffi>=1.0.0"],
setup_requires=["cffi>=1.0.0"],
cffi_modules=[
"./src/build_sobel.py:ffi",
"./src/build_file_operations.py:ffi",
],
)
, but I run into this error
build/temp.linux-x86_64-3.5/_sobel.c:492:19: fatal error: sobel.h: No such file or directory
From what I can tell, the problem is that the sobel.h file does not get uploaded into the build folder created by setuptools.setup(). I looked for suggestions of what to do including using Extensions() and writing a MANIFEST.in file, and both seem to add a relative path to the correct header files:
MANIFEST.in
setup.py
SobelFilterTest.egg-info/PKG-INFO
SobelFilterTest.egg-info/SOURCES.txt
SobelFilterTest.egg-info/dependency_links.txt
SobelFilterTest.egg-info/requires.txt
SobelFilterTest.egg-info/top_level.txt
src/file_operations.h
src/macros.h
src/sobel.h
But I still get the same error message. Is there a correct way to go about adding the header file to the build folder? Thanks!
It's actually not pip that is missing the .h file, but rather the compiler (like gcc). Therefore it's not about adding the missing file to setup, but rather make sure that cffi can find it. One way (like mentioned in the comments) is to make it available to the compiler through environment variables, but there is another way.
When setting the source with cffi you can add directories for the compiler like this:
from cffi import FFI
ffibuilder = FFI()
ffibuilder.set_source("<YOUR SOURCE HERE>", include_dirs=["./src"])
# ... Rest of your code
"""
I have converted a python game I designed into an exe. Running the exe itself causes it to flash and then close, meaning an error has occured. Running it from the Command Prompt causes the error as well, but documents it:
Cannot load image: Playfield.png
Couldn't open images\Playfield.png
This is telling me that the load_image block is failing. I have encountered this before when I did not have an images directory.
I attempted to move the images folder to the dist directory. This is the error that shows up:
Traceback (most recent call last):
File "Table_Wars.py", line 728, in <module>
File "Table_Wars.py", line 51, in main
File "Table_Wars.py", line 236, in __init__
File "pygame\__init__.pyc", line 70, in __getattr__
NotImplementedError: font module not available
(ImportError: DLL load failed: The specified module could not be found.)
This is my first time with py2exe, so I'm not really sure what is happening. The raw python file itself, Table_Wars.py, runs as expected.
If it helps, the location for the entire Table_Wars folder is inside a folder called Games, located on my Desktop (C:\Users\Oventoaster\Desktop\Games\Table_Wars). I am running Windows 7 32 bit OS.
On request, here is the output.txt I have generated:
Folder PATH listing for volume OS
Volume serial number is 7659-4C9C
C:\USERS\OVENTOASTER\DESKTOP\GAMES\TABLE_WARS
build
bdist.win32
winexe
bundle-2.7
collect-2.7
ctypes
distutils
email
mime
encodings
logging
multiprocessing
dummy
pygame
threads
unittest
xml
parsers
temp
dist
images
Here is the setup.py I used to convert the file:
from distutils.core import setup
import py2exe
setup(console=['Table_Wars.py'])
EDIT: I have attempted to use the full py2exe example. This will create the exe, but gives the same Cannot load image error. Attempting to put the images folder in the same folder as the exe creates a Runtime Error: The application requested the runtime to terminate it in an unusual way.
The shortened form of the code Slace Diamond suggested prevents py2exe from finding Table_Wars.py:
from cmd:
running py2exe
*** searching for required modules ***
error: Table_Wars.py: No such file or directory.
setup and Table_Wars are in the same directory. If it help, I input the full path to python.exe and setup.py.
EDIT: I seem to be getting closer. I put the images directory within self.extra_datas, and now I am getting this:
Fatal Python error: (segmentation fault)
This application has requested the runtime to terminate it in an unusual way. Please contact the application's suppourt team for more information
When you build a distributable package with py2exe (and py2app for that matter), part of the package environment is to point to a local resource location for files. In your plain unpackaged version, you are referring to a relative "images/" location. For the packaged version, you need to configure your setup.py to include the resources in its own location.
Refer to this doc for very specific info about how to set the data_files option of your package: http://www.py2exe.org/index.cgi/data_files
That page has multiple examples to show both very simple paths, and also a helper function for finding the data and building the data_files list for you.
Here is an example of the simple snippet:
from distutils.core import setup
import py2exe
Mydata_files = [('images', ['c:/path/to/image/image.png'])]
setup(
console=['trypyglet.py.py']
data_files = Mydata_files
options={
"py2exe":{
"unbuffered": True,
"optimize": 2,
"excludes": ["email"]
}
}
)
This closely matches what you are trying to achieve. It is saying that the "image.png" source file should be placed into the "images" directory at the root of the resources location inside the package. This resource root will be your current directory from your python scripts, so you can continue to refer to it as a relative sub directory.
It looks like you've already fixed the image problem by moving the folder into dist. The missing font module, on the other hand, is a known problem between pygame and py2exe. Py2exe doesn't copy some necessary DLLs, so you have to override py2exe's isSystemDLL method, forcing it to include audio and font related DLLs.
If Table_Wars.py is the only module in your project, try running this script with python setup.py py2exe:
from os.path import basename
from distutils.core import setup
import py2exe
origIsSystemDLL = py2exe.build_exe.isSystemDLL
def isSystemDLL(pathname):
if basename(pathname).lower() in ("libogg-0.dll", "sdl_ttf.dll"):
return 0
return origIsSystemDLL(pathname)
py2exe.build_exe.isSystemDLL = isSystemDLL
setup(windows=[{"script": "Table_Wars.py"}],
options={"py2exe": {"dist_dir": "dist"}})
You could also try the example py2exe setup file on the pygame wiki. If neither of them are working, please add the error messages to your question.
I tried running py2exe on a sample project, and it also breaks for me when I use the default pygame font. If you're using the default font, try putting a ttf file in the root of your project and also in the dist folder. You'll have to change the call to pygame.Font in your script as well:
font = pygame.font.Font("SomeFont.ttf", 28)
I have a little problem with setuptools/easy_install; maybe someone could give me a hint what might be the cause of the problem:
To easily distribute one of my python webapps to servers I use setuptools' sdist command to build a tar.gz file which is copied to servers and locally installed using easy_install /path/to/file.tar.gz.
So far this seems to work great. I have listed everything in the MANIFEST.in file like this:
global-include */*.py */*.mo */*.po */*.pot */*.css */*.js */*.png */*.jpg */*.ico */*.woff */*.gif */*.mako */*.cfg
And the resulting tar.gz file does indeed contain all of the files I need.
It gets weird as soon as easy_install tries to actually install it on the remote system. For some reason a directory called locales and a configuration file called migrate.cfg won't get installed. This is odd and I can't find any documentaiton about this, but I guess it's some automatic ignore feature of easy_install?
Is there something like that? And if so, how do I get easy_install to install the locales and migrate.cfg files?
Thanks!
For reference here is the content of my setup.py:
from setuptools import setup, find_packages
requires = ['flup', 'pyramid', 'WebError', 'wtforms', 'webhelpers', 'pil', 'apns', \
'pyramid_beaker', 'sqlalchemy', 'poster', 'boto', 'pypdf', 'sqlalchemy_migrate', \
'Babel']
version_number = execfile('pubserverng/version.py')
setup(
author='Bastian',
author_email='test#domain.com',
url='http://domain.de/',
name = "mywebapp",
install_requires = requires,
version = __version__,
packages = find_packages(),
zip_safe=False,
entry_points = {
'paste.app_factory': [
'pubserverng=pubserverng:main'
]
},
namespace_packages = ['pubserverng'],
message_extractors = { 'pubserverng': [
('**.py', 'python', None),
('templates/**.html', 'mako', None),
('templates/**.mako', 'mako', None),
('static/**', 'ignore', None),
('migrations/**', 'ignore', None),
]
},
)
I hate to answer my own question this quickly, but after some trial and error I found out what the reason behind the missing files was. In fact it was more than one reason:
The SOURCES.txt file was older and included a full list of most files, which resulted in them being bundled correctly.
The MANIFEST.in file was correct, too, so all required files were actually in the .tar.gz archive as expected. The main problem was that a few files simply would not get installed on the target machine.
I had to add include_package_data = True, to my setup.py file. After doing that all files installed as expected.
I'll have to put some research into include_package_data to find out if this weird behavior is documented somewhere. setuptools is a real mess - especially the documentation.
The entire package distribution system in python leaves a lot to be desired. My issues were similar to yours and were eventually solved by using distutils (rather than setuptools) which honored the include_package_data = True setting as expected.
Using distutils allowed me to more or less keep required file list in MANIFEST.inand avoid using the package_data setting where I would have had to duplicate the source list; the draw back is find_packages is not available. Below is my setup.py:
from distutils.core import setup
package = __import__('simplemenu')
setup(name='django-simplemenu',
version=package.get_version(),
url='http://github.com/danielsokolowski/django-simplemenu',
license='BSD',
description=package.__doc__.strip(),
author='Alex Vasi <eee#someuser.com>, Justin Steward <justin+github#justinsteward.com>, Daniel Sokolowski <unemelpmis-ognajd#danols.com>',
author_email='unemelpmis-ognajd#danols.com',
include_package_data=True, # this will read MANIFEST.in during install phase
packages=[
'simplemenu',
'simplemenu.migrations',
'simplemenu.templatetags',
],
# below is no longer needed as we are utilizing MANIFEST.in with include_package_data setting
#package_data={'simplemenu': ['locale/en/LC_MESSAGES/*',
# 'locale/ru/LC_MESSAGES/*']
# },
scripts=[],
requires=[],
)
And here is a MANIFEST.in file:
include LICENSE
include README.rst
recursive-include simplemenu *.py
recursive-include simplemenu/locale *
prune simplemenu/migrations
You need to use the data_files functionality of setup - your files aren't code, so easy_install won't install them by default (it doesn't know where they go).
The upside of this is that these files are added to MANIFEST automatically - you don't need to do any magic to get them there yourself. (In general if a MANIFEST automatically generated by setup.py isn't sufficient, adding them yourself isn't going to magically get them installed.)