I've found a library which is in C (?), that I want to try from Python:
https://github.com/centaurean/density
I'd like to see if I can store compressed files on disk and decompress them in memory for faster full-file reading times.
I'm new to using external code from Python. How can I create a Python function that uses this library with a little overhead as possible?
(I will work with Windows or Linux)
One way is to compile the library into a dynamic library (.dll on Windows or .so on Linux) and use ctypes (https://docs.python.org/2/library/ctypes.html) to access it.
ctypes provides an easy path into doing exactly this through FFI, with the alternative of using something like SWIG to interface with the python libraries at a much lower layer.
I would recommend generating a little micro-benchmark program to record what the performance is before and after such an experiment, so that you have concrete values to use in choosing if you want to run with ctypes or python-swig code long term over doing something natively in python, which you could trial in c-python or a variant (such as pypy)
Fortunately for you, Python comes with an FFI built-in called ctypes. It works with both Linux and Windows, and there are plenty of tutorials out there to learn how to use it.
Note that you will need to compile the library beforehand into a shared lib (DLL in Windows, SO in Linux).
Related
I have working Python script using scipy and numpy functions and I need to run it on the computer with installed Python but without modules scipy and numpy. How should I do that? Is .pyc the answer or should I do something more complex?
Notes:
I don't want to use py2exe. I am aware of it but it doesn't fit to the problem.
I have read, these questions (What is the difference between .py and .pyc files?, Python pyc files (main file not compiled?)) with obvious connection to this problem but since I am a physicist, not a programmer, I got totally lost.
It is not possible.
A pyc-file is nothing more than a python file compiled into byte-code. It does not contain any modules that this file imports!
Additionally, the numpy module is an extension written in C (and some Python). A substantial piece of it are shared libraries that are loaded into Python at runtime. You need those for numpy to work!
Python first "compiles" a program into bytecode, and then throws this bytecode through an interpreter.
So if your code is all Python code, you would be able to one-time generate the bytecode and then have the Python runtime use this. In fact I've seen projects such as this, where the developer has just looked through the bytecode spec, and implemented a bytecode parsing engine. It's very lightweight, so it's useful for e.g. "Python on a chip" etc.
Problem comes with external libraries not entirely written in Python, (e.g. numpy, scipy).
Python provides a C-API, allowing you to create (using C/C++ code) objects that appear to it as Python objects. This is useful for speeding things up, interacting with hardware, making use of C/C++ libs.
Take a look at Nuitka. If you'll be able to compile your code (not necessarily a possible or easy task), you'll get what you want.
I was wondering if any of you could give me guidance on whether it'd be possible to use some of these TA-LIB functions found here in a python script. I cant find the functions in any other language that I know...
I read this, so there seems to be some level of possibility, however I have little understanding of whats going on in the article since I dont know C at all. Oh and incase you are wondering TA-Lib is ported on python BUT its doesnt really build on mac and most people say they have issues with it.
So essentially, I can't get the whole app to work in swig, I was wondering if I could instead compile the function (not even sure if that makes sense) and use it in a python app (and hopefully some guidance on how to do so).
I believe there are three simple approaches you could take:
SWIG
TA-Lib comes with a Python wrapper that is generated by SWIG. It hasn't been updated in a long time, so is hard-coded to build with Python 2.3. Andy Hawkins wrote some directions to get it to work with newer versions of Python.
Cython
I wrote a TA-Lib python wrapper that uses Cython to wrap all the functions in TA-Lib, and released it on Github. It works really well for me, uses Numpy arrays, is 2-4 times faster, more "pythonic", and easier to install (works on Mac OS X) than the SWIG interface.
Ctypes
If you only need a small number of functions from the library, you can use ctypes to make calls into the TA-Lib library.
If all you need is a single function in a library, yoyu be better of using ctypes - ctypes is a Python module in the standard library that allows to you to perform calls to libraries in native code.
You just have to check on the Python console how to obtain you TA-LIB as a Python object using ctypes, and how to call the function you need. Ctypes converts ints and strings automatically to C for you - you will need to perform some function anotation for other parameter types, though.
The hindrance we have to ship python is the large size of the standard library.
Is there a minimal python distribution or an easy way to pick and choose what we want from
the standard library?
The platform is linux.
If all you want is to get the minimum subset you need (rather than build an exe which would constrain you to Windows systems), use the standard library module modulefinder to list all modules your program requires (you'll get all dependencies, direct and indirect). Then you can zip all the relevant .pyo or .pyc files (depending on whether you run Python with or without the -O flag) and just use that zipfile as your sys.path (plus a directory for all the .pyd or .so native-code dynamic libraries you may need -- those need to live directly in the filesystem to let the OS load them in as needed, can't be loaded directly from a zipfile the way Python bytecode modules can, unfortunately).
Have you looked at py2exe? It provides a way to ship Python programs without requiring a Python installation.
Like Hank Gay and Alex Martelli suggest, you can use py2exe. In addition I would suggest looking into using something like IronPython. Depending on your application, you can use libraries that are built into the .NET framework (or MONO if for Linux). This reduces your shipping size, but adds minimum requirements to your program.
Further, if you are using functions from a library, you can use from module import x instead of doing a wildcard import. This reduces your ship size as well, but maybe not by too much
Hope this helps
I'm trying to create a python script that will disassemble a binary (a Windows exe to be precise) and analyze its code.
I need the ability to take a certain buffer, and extract some sort of struct containing information about the instructions in it.
I've worked with libdisasm in C before, and I found it's interface quite intuitive and comfortable.
The problem is, its Python interface is available only through SWIG, and I can't get it to compile properly under Windows.
At the availability aspect, diStorm provides a nice out-of-the-box interface, but it provides only the Mnemonic of each instruction, and not a binary struct with enumerations defining instruction type and what not.
This is quite uncomfortable for my purpose, and will require a lot of what I see as spent time wrapping the interface to make it fit my needs.
I've also looked at BeaEngine, which does in fact provide the output I need, a struct with binary info concerning each instruction, but its interface is really odd and counter-intuitive, and it crashes pretty much instantly when provided with wrong arguments.
The CTypes sort of ultimate-death-to-your-python crashes.
So, I'd be happy to hear about other solutions, which are a little less time consuming than messing around with djgcc or mingw to make SWIGed libdisasm, or writing an OOP wrapper for diStorm.
If anyone has some guidance as to how to compile SWIGed libdisasm, or better yet, a compiled binary (pyd or dll+py), I'd love to hear/have it. :)
Thanks ahead.
Well, after much meddling around, I managed to compile SWIGed libdisasm!
Unfortunately, it seems to crash python on incorrect (and sometimes correct) usage.
How I did it:
I compiled libdisasm.lib using Visual Studio 6, the only thing you need for this is the source code in whichever libdisasm release you use, and stdint.h and inttypes.h (The Visual C++ compatible version, google it).
I SWIGed the given libdisasm_oop.i file with the following command line
swig -python -shadow -o x86disasm_wrap.c -outdir . libdisasm_oop.i
Used Cygwin to run ./configure in the libdisasm root dir. The only real thing you get from this is config.h
I then created a new DLL project, added x86disasm_wrap.c to it, added the c:\PythonXX\libs and c:\PythonXX\Include folders to the corresponding variables, set to Release configuration (important, either this or do #undef _DEBUG before including python.h).
Also, there is a chance you'll need to fix the path to config.h.
Compiled the DLL project, and named the output _x86disasm.dll.
Place that in the same folder as the SWIG generated x86disasm.py and you're done.
Any suggestions for other, less crashy disasm libs for python?
You might try using ctypes to interface directly with libdisasm instead of going through a SWIG layer. It may be take more development time but AFAIK you should be able to access the underlying functionality using ctypes.
I recommend you look at Pym's disassembly library which is also the backend for Pym's online disassembler.
You can use the distorm library: https://code.google.com/p/distorm/
Here's another build: http://breakingcode.wordpress.com/2009/08/31/using-distorm-with-python-2-6-and-python-3-x-revisited/
There's also BeaEngine: http://www.beaengine.org/
Here's a Windows installer for BeaEngine: http://breakingcode.wordpress.com/2012/04/08/quickpost-installer-for-beaenginepython/
I have a third-party product, a terminal emulator, which provides a DLL that can be linked to a C program to basically automate the driving of this product (send keystrokes, detect what's on the screen and so forth).
I want to drive it from a scripting language (I'm comfortable with Python and slightly less so with Perl) so that we don't have to compile and send out executables to our customers whenever there's a problem found.
We also want the customers to be able to write their own scripts using ours as baselines and they won't entertain the idea of writing and compiling C code.
What's a good way of getting Python/Perl to interface to a Windows DLL. My first thought was to write a server program and have a Python script communicate with it via TCP but there's got to be an easier solution.
One way to call C libraries from Python is to use ctypes:
>>> from ctypes import *
>>> windll.user32.MessageBoxA(None, "Hello world", "ctypes", 0);
In Perl, Win32::API is an easy way to some interfacing to DLLs. There is also Inline::C, if you have access to a compiler and the windows headers.
Perl XSUBs can also create an interface between Perl and C.
In Perl, P5NCI will also do that, at least in some cases. But it seems to me that anything you use that directly manages interfacing with the dll is going to be user-unfriendly, and if you are going to have a user (scriptor?) friendly wrapper, it might as well be an XS module.
I guess I don't see a meaningful distinction between "compile and send out executables" and "compile and send out scripts".
For Python, you could compile an extension which links to the DLL, so that in Python you could just import it like a normal module. You could do this by hand, by using a library like Boost.Python, or by using a tool such as SWIG (which also supports Perl and other scripting languages) to generate a wrapper automatically.
The Python Py_InitModule API function allows you to create a module from c/c++ functions which can then be call from Python.
It takes about a dozen or so lines of c/c++ code to achieve but it is pretty easy code to write:
https://python.readthedocs.org/en/v2.7.2/extending/extending.html#the-module-s-method-table-and-initialization-function
The Zeus editor that I wrote, uses this appoach to allow Zeus macros to be written in Python and it works very well.