I'm creating and packaging a Python (2.7) program to create a Lambda Function on Amazaon WS. The program I have is dependent on the PySha3 package, which is essentially a Python wrapper around the optimized Sha3 C package. Pysha3 imports a C file; the source code of these two files can be seen in this Github Repository.
In order to import a package into the Lambda platform, the whole file and all dependencies must be zipped and uploaded. When I do this and attempt to test the package after the upload I get an error that the Sha3 classes being imported from the C file included in PySha3's package cannot be found. As I result of my own research and troubleshooting I have decided to install a Pysha3 wheel in the root of my program's directory, which I do believe is the right move. However, when I do this I get the error below:
I'm really hoping that if I can get the wheel to install correctly by installing a C compiler and can somehow package the C compiler along with my program, that the Lambda will then work. I found this compiler for Python and I'm hoping that installing it will fix my error locally, and maybe including it in the zip file will fix the error on Lambda AWS.
So, my question is, does my thought process seem valid? As well, any tips/guidance on how to include the compiler in a zipped file so that it will actually be utilized? I'm very new to all of this, so anything will help. Thank you very much!
In order to package a binary for Lambda, you need to build any native code in the correct environment. It appears that you are trying to build on a Windows machine -- even if your build worked, the resulting binary would not be suitable for the Lambda execution environment.
The current supported environment is documented in the Lambda Execution Environment guide, which states also:
If you are using any native binaries in your code, make sure they are compiled in this environment.
Related
I maintain a Python utility that allows bpy to be installable as a Python module. Due to the hugeness of the spurce code, and the length of time it takes to download the libraries, I have chosen to provide this module as a wheel.
Unfortunately, platform differences and Blender runtime expectations makes support for this tricky at times.
Currently, one of my big goals is to get the Blender addon scripts directory to install into the correct location. The directory (simply named after the version of Blender API) has to exist in the same directory as the Python executable.
Unfortunately the way that setuptools works (or at least the way that I have it configured) the 2.79 directory is not always placed as a sibling to the Python executable. It fails on Windows platforms outside of virtual environments.
However, I noticed in setuptools documentation that you can specify eager_resources that supposedly guarantees the location of extracted files.
https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-resource-extraction
https://setuptools.readthedocs.io/en/latest/pkg_resources.html#resource-extraction
There was a lot of hand waving and jargon in the documentation, and 0 examples. I'm really confused as to how to structure my setup.py file in order to guarantee the resource extraction. Currently, I just label the whole 2.79 directory as "scripts" in my setuptools Extension and ship it.
Is there a way to write my setup.py and package my module so as to guarantee the 2.79 directory's location is the same as the currently running python executable when someone runs
py -3.6.8-32 -m pip install bpy
Besides simply "hacking it in"? I was considering writing a install_requires module that would simply move it if possible but that is mangling with the user's file system and kind of hacky. However it's the route I am going to go if this proves impossible.
Here is the original issue for anyone interested.
https://github.com/TylerGubala/blenderpy/issues/13
My build process is identical to the process descsribed in my answer here
https://stackoverflow.com/a/51575996/6767685
Maybe try the data_files option of distutils/setuptools.
You could start by adding data_files=[('mydata', ['setup.py'],)], to your setuptools.setup function call. Build a wheel, then install it and see if you can find mydata/setup.py somewhere in your sys.prefix.
In your case the difficult part will be to compute the actual target directory (mydata in this example). It will depend on the platform (Linux, Windows, etc.), if it's in a virtual environment or not, if it's a global or local install (not actually feasible with wheels currently, see update below) and so on.
Finally of course, check that everything gets removed cleanly on uninstall. It's a bit unnecessary when working with virtual environments, but very important in case of a global installation.
Update
Looks like your use case requires a custom step at install time of your package (since the location of the binary for the Python interpreter relative to sys.prefix can not be known in advance). This can not be done currently with wheels. You have seen it yourself in this discussion.
Knowing this, my recommendation would be to follow the advice from Jan Vlcinsky in his comment for his answer to this question:
Post install script after installing a wheel.
Add an extra setuptools console entry point to your package (let's call it bpyconfigure).
Instruct the users of your package to run it immediately after installing your package (pip install bpy && bpyconfigure).
The purpose of bpyconfigure should be clearly stated (in the documentation and maybe also as a notice shown in the console right after starting bpyconfigure) since it would write into locations of the file system where pip install does not usually write.
bpyconfigure should figure out where is the Python interpreter, and where to write the extra data.
The extra data to write should be packaged as package_data, so that it can be found with pkg_resources.
Of course bpyconfigure --uninstall should be available as well!
I'd like to install the pypoker-eval package. The author provided a nice readme for how to install the package in windows:
pypoker-eval on Windows
install python
open visual studio project and change directories to include python directories and fix the path to the poker-eval project
build the project, there may be an undefined ssize you can typedef as int
locate the newly built dll pypokereval.dll then look at the exports and you will find something similar to init_pokereval_2_4
rename the dll to _pokereval_2_4.pyd (your version numbers) and move it into the test.py directory
edit the .py files so the imported module is _pokereval_2_4 (your version numbers) not a dynamically generated one
To be quite honest though, these instructions really lost me. I've tried doing some research on my own, and I understand that because the library has some C code I need to compile it first. I wasn't able to figure out though what the purpose of changing the directories to include python directories, does this part mean change the working directory? Also is the compiler I use important? I've dabbled some in C but that was with the Code::Blocks IDE.
I think my main problem is I don't really understand what happens when I install a package. Pip has spoiled me and kept this whole process as a block box and now with something like this I feel completely lost. Any recommended reading on installing packages with C dependencies or tips on how I can get this package installed would be much appreciated!
For reference I'm using Python 2.7 on Windows 7.
This is probably a question that has a very easy and straightforward answer, however, despite having a few years programming experience, for some reason I still don't quite get the exact concepts of what it means to "build" and then to "install". I know how to use them and have used them a lot, but have no idea about the exact processes which happen in the background...
I have looked across the web, wikipedia, etc... but there is no one simple answer to it, neither can I find one here.
A good example, which I tried to understand, is adding new modules to python:
http://docs.python.org/2/install/index.html#how-installation-works
It says that "the build command is responsible for putting the files to install into a build directory"
And then for the install command: "After the build command runs (whether you run it explicitly, or the install command does it for you), the work of the install command is relatively simple: all it has to do is copy everything under build/lib (or build/lib.plat) to your chosen installation directory."
So essentially what this is saying is:
1. Copy everything to the build directory and then...
2. Copy everything to the installation directory
There must be a process missing somewhere in the explanation...complilation?
Would appreciate some straightforward not too techy answer but in as much detail as possible :)
Hopefully I am not the only one who doesn't know the detailed answer to this...
Thanks!
Aivoric
Building means compiling the source code to binary in a sandbox location where it won't affect your system if something goes wrong, like a build subdirectory inside the source code directory.
Install means copying the built binaries from the build subdirectory to a place in your system path, where they become easily accessible. This is rarely done by a straight copy command, and it's often done by some package manager that can track the files created and easily uninstall them later.
Usually, a build command does all the compiling and linking needed, but Python is an interpreted language, so if there are only pure Python files in the library, there's no compiling step in the build. Indeed, everything is copied to a build directory, and then copied again to a final location. Only if the library depends on code written in other languages that needs to be compiled you'll have a compiling step.
You want a new chair for your living-room and you want to make it yourself. You browse through a catalog and order a pile of parts. When they arrives at your door, you can't immediately use them. You have to build the chair at your workshop. After a bit of elbow-grease, you can sit down in it. Afterwards, you install the chair in your living-room, in a convenient place to sit down.
The chair is a program you want to use. It arrives at your house as source code. You build it by compiling it into a runnable program. You install it by making it easier to use.
The build and install commands you are refering to come from setup.py file right?
Setup.py (http://docs.python.org/2/distutils/setupscript.html)
This file is created by 3rd party applications / extensions of Python. They are not part of:
Python source code (bunch of c files, etc)
Python libraries that come bundled with Python
When a developer makes a library for python that he wants to share to the world he creates a setup.py file so the library can be installed on any computer that has python. Maybe this is the MISSING STEP
Setup.py sdist
This creates a python module (the tar.gz files). What this does is copy all the files used by the python library into a folder. Creates a setup.py file for the module and archives everything so the library can be built somewhere else.
Setup.py build
This builds the python module back into a library (SPECIFICALLY FOR THIS OS).
As you may know, the computer that the python library originally came from will be different from the library that you are installing on.
It might have a different version of python
It might have a different operating system
It might have a different processor / motherboard / etc
For all the reasons listed above the code will not work on another computer. So setup.py sdist creates a module with only the source files needed to rebuild the library on another computer.
What setup.py does exactly is similar to what a makefile would do. It compiles sources / creates libraries all that stuff.
Now we have a copy of all the files we need in the library and they will work on our computer / operating system.
Setup.py install
Great we have all the files needed. But they won't work. Why? Well they have to be added to Python that's why. This is where install comes in. Now that we have a local copy of the library we need to install it into python so you can use it like so:
import mycustomlibrary
In order to do this we need to do several things including:
Copy files to their library folders in our version of python.
Make sure library can be imported using import command
Run any special install instructions for this library. (seting up paths, etc)
This is the most complicated part of the task. What if our library uses BeautifulSoup? This is not a part of Python Library. We'd have to install it in a way such that our library and any others can use BeautifulSoup without interfering with each other.
Also what if python was installed someplace else? What if it was installed on a server with many users?
Install handles all these problems transparently. What is does is make the library that we just built able to run. All you have to do is use the import command, install handles the rest.
I can't seem to import any of the basic modules located in the "lib-dynload" directory. They are all there, but I get the error: "ImportError: No module named X" when trying to import them.
I checked my sys.path and it includes the directory where all of these modules are located and my PYTHONHOME environment variable is set correctly. I'm at a bit of a loss as to what the problem could be. Some background info: This is cross-compiled from Python 2.6.6 source and installed onto an ARM embedded Linux board with Angstrom.
It did have python on there before, I had tried to bit-bake it into the image but it was missing a lot of stuff. I ended up doing my best to clean the directory tree of anything to do with the previous python before loading on my cross compiled version.
An strace of a simple script that just attempts to import math: http://pastebin.com/3XgJ3nPR
I see no checks in that trace for filenames like math.so or mathmodule.so which might indicate that shared-object modules are turned off entirely — that the version of Python you have compiled cannot load binary modules dynamically.
More: looking over the config.out from my most recent Python build, I see several lines where Python is investigating whether the platform will let it dynamically load binary modules that end in .so:
checking for dlopen... yes
checking DYNLOADFILE... dynload_shlib.o
checking MACHDEP_OBJS... MACHDEP_OBJS
What do these lines say on your cross-compile?
I have recently run across a similar issue building Python 2.7.13, and I believe it is this bug which is being fixed for Python 3 but not ported back to 2. The build process (setup.py) generates a list of modules to build, and then subtracts the list of built-in modules (sys.builtin_module_names); however, setup.py is run (from the Makefile) using python2.7 which in my case picked up the system (Ubuntu) binary rather than the one built, so it subtracts off modules that are built-in for the system python (including operator and collections) but not for the one being built, so they are neither built-in nor built as external modules.
I was able to use a suggestion from the bug and prepend the built python, in the source directory, to the path (and add a symlink from python2.7 -> python). This worked because I was building an x86 python on a multi-arch x64 machine; if you are building for another system like ARM you may need to apply the patch from the bug to get the list of built-in modules from earlier in the build process rather than the host python.
I wrote a Python extension in C, and my python program uses that extension. In order for it to work, I would have to install the extension on the user's system before my program can run. Is there a way to bypass that installation step and somehow just have the extension in my python package? The only compiled part obviously is the extension (since it's in C).
You can avoid having some one to install it independently but you can not avoid installation completely. If his computing platform differs from yours, he will have to build the extension.
What can be done is that you setup a package distribution using distutils. This way the package could be installed or built. You can include the "C" extension in your package.
For some standard platform, you can then provide binary package distribution. The user should have the ability to rebuild, if the binary package is not working out for him.
just put the .d compiled python dll in the same directory as your python script. then you'll be able to import it.