I am trying to use ctypes to access a dynamic library in python.
So far, I have created a bunch of structures in python to reflect the structure types in the library. I am now trying to test these structures to see if I have translated everything correctly. I currently have something like the following set up
import ctypes as ct
class MyStruct(ct.Structure)
''' put the actual implementation here '''
pass
if __name__ == "__main__":
mylib = ct.CDLL('/path/to/mylib.dylib')
loadFile = mylib.loadFile
loadFile.restype = ct.Pointer(MyStruct)
parent = MyStruct()
file = b"/path/to/file"
child = loadFile(file, ct.byref(parent))
print(child)
Now, for some reason this segfaults inside the dynamic library when I try to run this code. I am trying to determine where (and why) this happens. I have compiled both the library and my python executable with debug info and am now trying to use gdb to locate the problem. I can start my program and break at a line inside loadFile very nicely
gdb b loading.c:172
gdb r
and gdb breaks. Now, there is another function call (this function is also part of the dynamic library) on line 172 in loading.c that I want to step into. I tried to break inside this other function
gdb b foo.c:636
but gdb keeps giving me
Cannot access memory at address 0x2860
errors. Further, if I just break at line 172 and then use stepi then I can't get any debug info. Is there a way to get into this other function or is it a limitation of gdb when using dynamic libraries.
Related
Probably related to globals and locals in python exec(), Python 2 How to debug code injected by the exec block and How to get local variables updated, when using the `exec` call?
I am trying to develop a test framework for our desktop applications which uses click bot like functions.
My goal was to enable non-programmers to write small scripts which could work as a test script. So my idea is to structure the test scripts by files like:
tests-folder
| -> 01-first-test.py
| -> 02-second-test.py
| ... etc
| -> fixture.py
And then just execute these scripts in alphabetical order. However, I also wanted to have fixtures which would define functions, classes, variables and make them available to the different scripts without having the scripts to import that fixture explicitly. If that works, I could also have that approach for 2 or more directory levels.
I could get it working-ish with some hacking around, but I am not entirely convinced. I have a test_sequence.py which looks like this:
from pathlib import Path
from copy import deepcopy
from my_module.test import Failure
def run_test_sequence(test_dir: str):
error_occured = False
fixture = {
'some_pre_defined_variable': 'this is available in all scripts and fixtures',
'directory_name': test_dir,
}
# Check if fixture.py exists and load that first
fixture_file = Path(dir) / 'fixture.py'
if fixture_file.exists():
with open(fixture_file.absolute(), 'r') as code:
exec(code.read(), fixture, fixture)
# Go over all files in test sequence directory and execute them
for test_file in sorted(Path(test_dir).iterdir()):
if test_file.name == 'fixture.py':
continue
# Make a deepcopy, so scripts cannot influence one another
fixture_copy = deepcopy(fixture)
fixture_copy.update({
'some_other_variable': 'this is available in all scripts but not in fixture'
})
try:
with open(test_file.absolute(), 'r') as code:
exec(code.read(), fixture_locals, fixture_locals)
except Failure:
error_occured = True
return error_occured
This iterates over all files in the directory tests-folder and executes them in order (with fixture.py first). It also makes the local variables, functions and classes from fixture.py available to each test-script.
A test script could then just be arbitrary code that will be executed and if it raises my custom Failure exception, this will be noted as a failed test.
The whole sequence is started with a script that does
from my_module.test_sequence import run_test_sequence
if __name__ == '__main__':
exit(run_test_sequence('tests-folder')
This mostly works.
What it cannot do, and what leaves me unsatisfied with this approach:
I cannot debug the scripts itself. Since the code is loaded as string and then interpreted, breakpoints inside the test scripts are not recognized.
Calling fixture functions behaves weird. When I define a function in fixture.py like:
from my_hello_module import print_hello
def printer():
print_hello()
I will receive a message during execution that print_hello is undefined because the variables/modules/etc. in the scope surrounding printer are lost.
Stacktraces are useless. On failure it shows the stacktrace but of course only shows my line which says `exec(...)' and the insides of that function, but none of the code that has been loaded.
I am sure there are other drawbacks, that I have not found yet, but these are the most annoying ones.
I also tried to find a solution through __import__ but I couldn't get it to inject my custom locals or globals into the imported script.
Is there a solution, that I am too inexperienced to find or another builtin Python function that actually does, what I am trying to do? Or is there no way to achieve this and I should rather have each test-script import the fixture and file/directory names from the test-scripts itself. I want those scripts to have as few dependencies and pythony code as possible. Ideally they are just:
from my_module.test import *
click(x, y, LEFT)
write('admin')
press('tab')
write('password')
press('enter')
if text_on_screen('Login successful'):
succeed('Test successful')
else:
fail('Could not login')
Additional note: I think I had the debugger working when I still used execfile, but it is not available in python3 environments.
I'm trying to compile a usable .dll file from Julia to be used in Python as I've already written a large GUI in Python and need some fast optimization work done. Normally I would just call PyJulia or some "live" call, however this program needs to be compiled to distribute within my research team, so whatever solution I end up with needs to be able to run on its own (without Julia or Python actually installed).
Right now I'm able to create .dll files via PackageCompiler.jl, something I learned from previous posts on StackOverflow, however when trying to run these files in Python via the following code
Julia mock package
module JuliaFunctions
# Pkg.add("BlackBoxOptim")
Base.#ccallable function my_main_function(x::Cfloat,y::Cfloat)::Cfloat
z = 0
for i in 1:x
z += i ^ y
end
return z
end
# function julia_main()
# print("Hello from a compiled executable!")
# end
export my_main_function
end # module
Julia script to use PackageCompiler
# using PackageCompiler
using Pkg
# Pkg.develop(path="JuliaFunctions") # This is how you add a local package
# include("JuliaFunctions/src/JuliaFunctions.jl") # this is how you add a local module
using PackageCompiler
# Pkg.add(path="JuliaFunctions")
#time create_sysimage(:JuliaFunctions, sysimage_path="JuliaFunctions.dll")
Trying to use the resulting .dll in CTypes in Python
import ctypes
from ctypes.util import find_library
from ctypes import *
path = os.path.dirname(os.path.realpath(__file__)) + '\\JuliaFunctions.dll'
# _lib = cdll.LoadLibrary(ctypes.util.find_library(path)) # same error
# hllDll = ctypes.WinDLL(path, winmode=0) # same error
with os.add_dll_directory(os.path.dirname(os.path.realpath(__file__))):
_lib = ctypes.CDLL(path, winmode=0)
I get
OSError: [WinError 127] The specified procedure could not be found
With my current understanding, this means that CTypes found the dll and imported it, but didn't find.. something? I've yet to fully grasp how this behaves.
I've verified the function my_main_function is exported in the .dll file via Nirsoft's DLL Export Viewer. Users from previous similar issues have noted that this sysimage is already callable and should work, but they always add at the end something along the lines of "Note that you will also in general need to initialize the Julia runtime."
What does this mean? Is this even something that can be done independently from the Julia installation? The dev docs in PackageCompiler mention this, however they just mention that julia_main is automatically included in the .dll file and gets called as a sort of launch point. This function is also being exported correctly into the .dll file the above code creates. Below is an image of the Nirsoft export viewer output for reference.
Edit 1
Inexplicably, I've rebuilt this .dll on another machine and made progress. Now, the dll is imported correctly. I'm not sure yet why this worked on a fresh Julia install + Python venv, but I'm going to reinstall them on the other one and update this if anything changes. For anyone encountering this, also note you need to specify the expected output, whatever it may be. In my case this is done by adding (after the import):
_lib.testmethod1.restype = c_double # switched from Cfloat earlier, a lot has changed.
_lib.testmethod1.argtypes = [c_double, c_double] # (defined by ctypes)
The current error is now OSError: exception: access violation writing 0x0000000000000024 when trying to actually use the function, which is specific to Python. Any help on this would also be appreciated.
There is a python script start_test.py.
There is a second python script siple_test.py.
# pseudo code:
start_test.py --calls--> subprocess(python.exe simple_test.py, args_simple_test[])
The python interpreter for both scripts is the same. So instead of opening a new instance, I want to run simple_test.py directly from start_test.py. I need to preserve the sys.args environment. A nice to have would be to actually enter following code section in simple_test.py:
# file: simple_test.py
if __name__ == '__main__':
some_test_function()
Most important is, that the way should be a universal one, not depending on the content of the simple_test.py.
This setup would provide two benefits:
The call is much less resource intensive
The whole stack of simple_test.py can be debugged with pycharm
So, how do I execute the call of a python script, from a python script, without starting a new subprocess?
"Executing a script" is a somewhat blurry term.
Typically the if __name__== "__main__": part does the argument (sys.argv) decoding and then calls a worker function with explicit parameters. For clarity: It should not do anything else, since this additional work can't be called without creating a new process causing all the overhead you are trying to avoid.
You simply bypass that and call this implementing routine directly.
So you end up with start_test.py containing something like:
from simple_test import worker
# ...
worker(typed_arg1, typed_arg2)
I want to pass a constant in a C preprocessor style but with a Python script.
This constant is already declared with AC_DEFINE in my configure.ac file and used in my C program, and now I need to pass it to a Python script too.
I tried with a custom target in my Makefile.am with a sed call to preprocess a specific symbol in my Python script, but it seems dirty-coding to me.
How can I achieve this?
Create a config.py.in with some contents like MYVAR = '''#MYVAR#''' and add it to AC_CONFIG_FILES in your configure.ac. You can then import config in your other Python scripts.
This fulfills much the same function as config.h does for C programs.
I'm not a programmer, so I don't even know in what terms should I ask this. Let's say I've compiled a Python script to have an .exe (I use py2exe to do this). This is the major program. Now, I want to add some extra functionality to it, but I don't want to recompile the entire script with the added functionality. I tried to search something on the web, and I found examples of extending a C++ or other application with Python scripts (like a sort of plugin). But I can't figure out how to do it with an application already written in Python.
I tried this: I wrote major.py (this is the script from where I build the executable) and stuff.py. In major I wrote this:
def generic():
import stuff
while True:
param=input('what did you say? ')
stuff.speak(param)
generic()
And in stuff I wrote this:
def speak(param):
print(param)
Then I created a .exe with py2exe. It works as expected, when I run the program in the command line says "what did you say?" and waits until I type something, then it prints what I typed.
Then, I changed stuff.py with this:
def speak(param):
print('I said '+param)
Hoping that now upon the execution of the .exe created earlier it would print "I said.." plus whatever I typed. Obviously, it didn't work, the program continued to behave like before. So I'm guessing that once I imported stuff and created the .exe file, that import is permanent, not allowing me to change whatever is in stuff. What should I do?
py2exe packs the compiled scripts in the executable.
You need to recreate the executable (which will recompile any changed script) to see the changes take effect.
EDIT following the comments:
You can do it if you dynamically import/reimport the module from inside the executable.
In your main script you do (see code below)
mod, error = loadmodule('mystuff.py')
if mod is not None:
# loading succeeded you can now proceed and do things with it
pass
Of course you have to leave mystuff.py out of the scripts that py2exe packs into the executable. In the above example mystuff.py would be in the same directory as the executable.
The loading code:
def loadmodule(modpath, modname=''):
if not modpath.endswith('.py'):
modpath += '.py'
# generate a random name for the module
if not modname:
modpathbase = os.path.basename(modpath)
modname, _ = os.path.splitext(modpathbase)
version = (sys.version_info[0], sys.version_info[1])
if version < (3, 3):
mod, e = loadmodule2(modpath, modname)
else:
mod, e = loadmodule3(modpath, modname)
return mod, e
def loadmodule2(modpath, modname):
import imp
try:
mod = imp.load_source(modname, modpath)
except Exception as e:
return (None, e)
return (mod, None)
def loadmodule3(modpath, modname):
import importlib.machinery
try:
loader = importlib.machinery.SourceFileLoader(modname, modpath)
mod = loader.load_module()
except Exception as e:
return (None, e)
return (mod, None)
If you run your script from Python, instead of compiling it as an executable, you can make changes and run them without having to recompile. The py2exe is mostly for allowing you to distribute your application to other Windows computers that don't have Python installed. After you have finished developing it, then compile it as an executable so you can run it on other computers.
There is no way to do what you want. py2exe builds a standalone python interpreter (In the file named python.dll) with just the dependencies of your project. Then the .exe file runs your script using that interpreter
I suggest you that if you really need to provide regular upgrades then you should recompile it, or install python in the target machine, or make an updating routine that checks for updates and compiles it in the target machine (With py2exe)