Is there any other way to debug swig extensions except for doing
gdb python stuff.py
?
I have wrapped the legacy library libkdtree++ and followed all the swig related memory managemant points (borrowed ref vs. own ref, etc.). But still, I am not sure whether my binding is not eating up memory. It would be helpful to be able to just debug step by step each publicized function: starting from Python then going to via the C glue binding into C space, and returning back.
Is there already such a possibility?
gdb 7.0 supports python scripting. It might help you in this particular case.
Well, for debugging, you use a debugger ;-).
When debugging, it may be a good idea to configure Python with '--with-pydebug' and recompile. It does additional checks then.
If you are looking for memory leaks, there is a simple way:
Run your code over and over in a loop, and look for Python's memory consumption.
Related
I have been wondering if developing Linux kernel modules (drivers) with Python is possible. Is it?
Yes, it is possible:
http://www.kplugs.org/
Although not recommended in production machines, this can be really useful while prototyping your driver.
See here we have certain issues.
We have to understand why Linus Torvalds himself preferred C and Assembly language.C is the only language that won't hinder your performance on raw hardware. The Operating System was designed to use as much minimal resources as possible.
Coming to Python, we already know that it's an interpreted language. So thereby its slow as it runs on a virtual environment.
Yes you can definitely try some. Check this
Instead you can really look forward to filesystem programming and multilevel cache organization and such using python.
No; LKM on Linux have to be compiled down do a specific ELF object code format.
Of course you could make your own hack of Python that does compile down to kernel object code, but as far as I know, at this time there is no such Python publicly available.
I just ran into this SO question and I'm baffled. I'd say I have a fair experience with Python, but only on *nux(-like) OSes and I thought bytecode compilation was a given.
I'm obviously missing something here: was something happening behind the curtain I didn't know about on my OSes, like some configuration defaults ? Is it only on Windows and then, why? Is there any reason why not to compile to bytecode?
Thanks a lot in advance for any opinion, I'm very curious.
Basically, it's offering to pre-compile the Python installation files to bytecode, which would normally happen the first time they're used. As I understand it, it's just intended as a general convenience to avoid that initial compilation on use.
It would potentially save space not compiling all of the files in the standard library, but that would last only as long as you didn't try to use them.
I've seen a few sites talking about injecting DLL's (such as http://www.codeproject.com/KB/DLL/DLL_Injection_tutorial.aspx), but I'm struggling with how to get an EXE to work. any help/tips would be appreciated.
The best way I know how to explain it is "RunPE" where you execute an exe in the memory. Does that help at all?
If you're asking how to inject code into a running Python process, what you want is https://github.com/lmacken/pyrasite .
You can use the Reflective DLL Injector as described here. Metasploit project uses it to load its meterpreter plug-ins. AFAIK this is the only way to inject a DLL, as MS officially does not support "injecting" from memory, only loading from file system.
On a low level, nothing forbids you from allocating a memory region, loading code there, marking it executable.
Note, that none of these techniques are Python specific or even Python related - it is a win32 problem.
What you're talking about is re-implementing UPX in python with more stuff.
Things you would need to do in order to do this:
Change all VirtualAlloc calls to be VirtualAllocEx calls.
Change all Loadlibrary calls to be loadlibraryEX calls.
Implement the relocation fix-ups.
A better approach would probably be tweaking UPX to output a DLL instead of an executable. Then using some python DLL injection code to throw that into another process. You're going to be working with CTypes a lot if you want to do this. Fair warning...
I would recommend this book http://www.amazon.com/Gray-Hat-Python-Programming-Engineers/dp/1593271921 - especially the chapters on writing your own debugger, but it covers the metasploit and other tools as described above.
To inject a shared object (.so, .dll) into any process you can use injector with C, or pyinjector with python/shell.
To inject python code into a running python process, you can use hypno.
I'm trying to create a python script that will disassemble a binary (a Windows exe to be precise) and analyze its code.
I need the ability to take a certain buffer, and extract some sort of struct containing information about the instructions in it.
I've worked with libdisasm in C before, and I found it's interface quite intuitive and comfortable.
The problem is, its Python interface is available only through SWIG, and I can't get it to compile properly under Windows.
At the availability aspect, diStorm provides a nice out-of-the-box interface, but it provides only the Mnemonic of each instruction, and not a binary struct with enumerations defining instruction type and what not.
This is quite uncomfortable for my purpose, and will require a lot of what I see as spent time wrapping the interface to make it fit my needs.
I've also looked at BeaEngine, which does in fact provide the output I need, a struct with binary info concerning each instruction, but its interface is really odd and counter-intuitive, and it crashes pretty much instantly when provided with wrong arguments.
The CTypes sort of ultimate-death-to-your-python crashes.
So, I'd be happy to hear about other solutions, which are a little less time consuming than messing around with djgcc or mingw to make SWIGed libdisasm, or writing an OOP wrapper for diStorm.
If anyone has some guidance as to how to compile SWIGed libdisasm, or better yet, a compiled binary (pyd or dll+py), I'd love to hear/have it. :)
Thanks ahead.
Well, after much meddling around, I managed to compile SWIGed libdisasm!
Unfortunately, it seems to crash python on incorrect (and sometimes correct) usage.
How I did it:
I compiled libdisasm.lib using Visual Studio 6, the only thing you need for this is the source code in whichever libdisasm release you use, and stdint.h and inttypes.h (The Visual C++ compatible version, google it).
I SWIGed the given libdisasm_oop.i file with the following command line
swig -python -shadow -o x86disasm_wrap.c -outdir . libdisasm_oop.i
Used Cygwin to run ./configure in the libdisasm root dir. The only real thing you get from this is config.h
I then created a new DLL project, added x86disasm_wrap.c to it, added the c:\PythonXX\libs and c:\PythonXX\Include folders to the corresponding variables, set to Release configuration (important, either this or do #undef _DEBUG before including python.h).
Also, there is a chance you'll need to fix the path to config.h.
Compiled the DLL project, and named the output _x86disasm.dll.
Place that in the same folder as the SWIG generated x86disasm.py and you're done.
Any suggestions for other, less crashy disasm libs for python?
You might try using ctypes to interface directly with libdisasm instead of going through a SWIG layer. It may be take more development time but AFAIK you should be able to access the underlying functionality using ctypes.
I recommend you look at Pym's disassembly library which is also the backend for Pym's online disassembler.
You can use the distorm library: https://code.google.com/p/distorm/
Here's another build: http://breakingcode.wordpress.com/2009/08/31/using-distorm-with-python-2-6-and-python-3-x-revisited/
There's also BeaEngine: http://www.beaengine.org/
Here's a Windows installer for BeaEngine: http://breakingcode.wordpress.com/2012/04/08/quickpost-installer-for-beaenginepython/
I would like seek some guidance in writing a "process profiler" which runs in kernel mode. I am asking for a kernel mode profiler is because I run loads of applications and I do not want my profiler to be swapped out.
When I said "process profiler" I mean to something that would monitor resource usage by the process. including usage of threads and their statistics.
And I wish to write this in python. Point me to some modules or helpful resource.
Please provide me guidance/suggestion for doing it.
Thanks,
Edit::: Would like to add that currently my interest isto write only for linux. however after i built it i will have to support windows.
It's going to be very difficult to do the process monitoring part in Python, since the python interpreter doesn't run in the kernel.
I suspect there are two easy approaches to this:
use the /proc filesystem if you have one (you don't mention your OS)
Use dtrace if you have dtrace (again, without the OS, who knows.)
Okay, following up after the edit.
First, there's no way you're going to be able to write code that runs in the kernel, in python, and is portable between Linux and Windows. Or at least if you were to, it would be a hack that would live in glory forever.
That said, though, if your purpose is to process Python, there are a lot of Python tools available to get information from the Python interpreter at run time.
If instead your desire is to get process information from other processes in general, you're going to need to examine the options available to you in the various OS APIs. Linux has a /proc filesystem; that's a useful start. I suspect Windows has similar APIs, but I don't know them.
If you have to write kernel code, you'll almost certainly need to write it in C or C++.
don't try and get python running in kernel space!
You would be much better using an existing tool and getting it to spit out XML that can be sucked into Python. I wouldn't want to port the Python interpreter to kernel-mode (it sounds grim writing it).
The /proc option does sound good.
some code code that reads proc information to determine memory usage and such. Should get you going:
http://www.pixelbeat.org/scripts/ps_mem.py reads memory information of processes using Python through /proc/smaps like charlie suggested.
Some of your comments on other answers suggest that you are a relatively inexperienced programmer. Therefore I would strongly suggest that you stay away from kernel programming, as it is very hard even for experienced programmers.
Why would you want to write something that
is a very complex system (just look at existing profiling infrastructures and how complex they are)
can not be done in python (I don't know any kernel that would allow execution of python in kernel mode)
already exists (oprofile on Linux)
have you looked at PSI? (http://www.psychofx.com/psi/)
"PSI is a Python module providing direct access to real-time system and process information. PSI is a Python C extension, providing the most efficient access to system information directly from system calls."
it might give you what you are looking for. .... or at least a starting point.
Edit 2014:
I'd recommend checking out psutil instead:
https://pypi.python.org/pypi/psutil
psutil is actively maintained and has some nifty process monitoring features. PSI seems to be somewhat dead (last release 2009).