I'm trying to compile a usable .dll file from Julia to be used in Python as I've already written a large GUI in Python and need some fast optimization work done. Normally I would just call PyJulia or some "live" call, however this program needs to be compiled to distribute within my research team, so whatever solution I end up with needs to be able to run on its own (without Julia or Python actually installed).
Right now I'm able to create .dll files via PackageCompiler.jl, something I learned from previous posts on StackOverflow, however when trying to run these files in Python via the following code
Julia mock package
module JuliaFunctions
# Pkg.add("BlackBoxOptim")
Base.#ccallable function my_main_function(x::Cfloat,y::Cfloat)::Cfloat
z = 0
for i in 1:x
z += i ^ y
end
return z
end
# function julia_main()
# print("Hello from a compiled executable!")
# end
export my_main_function
end # module
Julia script to use PackageCompiler
# using PackageCompiler
using Pkg
# Pkg.develop(path="JuliaFunctions") # This is how you add a local package
# include("JuliaFunctions/src/JuliaFunctions.jl") # this is how you add a local module
using PackageCompiler
# Pkg.add(path="JuliaFunctions")
#time create_sysimage(:JuliaFunctions, sysimage_path="JuliaFunctions.dll")
Trying to use the resulting .dll in CTypes in Python
import ctypes
from ctypes.util import find_library
from ctypes import *
path = os.path.dirname(os.path.realpath(__file__)) + '\\JuliaFunctions.dll'
# _lib = cdll.LoadLibrary(ctypes.util.find_library(path)) # same error
# hllDll = ctypes.WinDLL(path, winmode=0) # same error
with os.add_dll_directory(os.path.dirname(os.path.realpath(__file__))):
_lib = ctypes.CDLL(path, winmode=0)
I get
OSError: [WinError 127] The specified procedure could not be found
With my current understanding, this means that CTypes found the dll and imported it, but didn't find.. something? I've yet to fully grasp how this behaves.
I've verified the function my_main_function is exported in the .dll file via Nirsoft's DLL Export Viewer. Users from previous similar issues have noted that this sysimage is already callable and should work, but they always add at the end something along the lines of "Note that you will also in general need to initialize the Julia runtime."
What does this mean? Is this even something that can be done independently from the Julia installation? The dev docs in PackageCompiler mention this, however they just mention that julia_main is automatically included in the .dll file and gets called as a sort of launch point. This function is also being exported correctly into the .dll file the above code creates. Below is an image of the Nirsoft export viewer output for reference.
Edit 1
Inexplicably, I've rebuilt this .dll on another machine and made progress. Now, the dll is imported correctly. I'm not sure yet why this worked on a fresh Julia install + Python venv, but I'm going to reinstall them on the other one and update this if anything changes. For anyone encountering this, also note you need to specify the expected output, whatever it may be. In my case this is done by adding (after the import):
_lib.testmethod1.restype = c_double # switched from Cfloat earlier, a lot has changed.
_lib.testmethod1.argtypes = [c_double, c_double] # (defined by ctypes)
The current error is now OSError: exception: access violation writing 0x0000000000000024 when trying to actually use the function, which is specific to Python. Any help on this would also be appreciated.
Related
When I run the code below, I get different results when I run in .py compared to when I run in .exe using pyinstaller
import win32com.client
import os
ConfigMacroName = "test.xls"
xl=win32com.client.Dispatch("Excel.Application")
Configmacrowb = xl.Workbooks.Open(os.getcwd()+ "\\Completed\\" + ConfigMacroName)
SlotPlansheet = Configmacrowb.Sheets("SlotPlan")
Header = SlotPlansheet.Rows(1)
SOcol = Header.Find('SO', LookAt=1).Column #I used LookAt=1 which is equivalent to LookAt:=xlWhole in excel VBA
SOlinecol = Header.Find('SO Line').Column
print("SO is " + str(SOcol) + "\nSo line is " + str(SOlinecol))
SlotPlansheet = None
Configmacrowb.Close(False)
Configmacrowb = None
xl.Quit()
xl = None
The excel input
The output in .py
The output in .exe
The output in .py file is the correct output I need. If I run it in .exe there will be duplicate variable since they both will refer to column B. For temporary solution I can just loop through the header to check each cell.
But I'm using find() function a lot so I don't know if my other programs are also affected by this inconsistency
Try changing the object creation line to:
xl=win32com.client.gencache.EnsureDispatch(‘Excel.Application’)
In my experience, the win32com.client.Dispatch() function can sometimes cause issues in that it does not guarantee the same result every time it runs. The caller doesn't know if they have an early- or late-bound object. If you have no cached makepy files then you will get a late-bound IDispatch automation interface, but if win32com finds an early-bound interface then it will use it (even if it wasn't your programme that created it). Hence code that ran fine previously may stop working.
Unless you have a good reason to be indifferent, I think it is better to be explicit and choose win32com.client.gencache.EnsureDispatch() or win32com.client.dynamic.Dispatch() for early- or late-binding respectively. I generally choose the EnsureDispatch() route, as it is quicker, enforces case-sensitivity, and gives access to any constants in the type library (eg win32com.client.constants.xlWhole) rather than rely on 'magic' integers.
Also, in the past, I have experienced odd behaviour around indexing (eg this SO question), and this was cured by deleted any gencache wrappers (see below).
Add this line to your debug code:
print('Cache directory:',win32com.client.gencache.GetGeneratePath())
This will tell you where the gencache early-binding python files are being generated, and where win32com.client.Dispatch() will look for any cached wrapper files to attempt early-binding. If you want to clear the cached of generated files just delete the contents of this directory. It will be interesting to see if the OP's two routes have the same directory.
I have written a python code, which takes 3 inputs, and return one output val.
I try to write an excel function, which passes the three inputs to the python function and returns the output.
I have looked into XLwings, but there is so many issues (and the documentation is insanely poor/poorly written) thus it seems useless.
So: is there any other way to call a python function (which takes inputs) from excel?
[SOLVED (ish):]
I managed, after roughly 8 hours of trying, 4 youtube videos and the xlwings homepage, to make it work.
Video for installing: https://training.zoomeranalytics.com/courses/xlwings/lectures/4231276
Video for making a function which takes input and returns output: https://www.youtube.com/watch?v=qn8xGrDuRCg&t=16s
You could try xlOil (disclaimer: I wrote it). The docs are here, but to write a simple three input function, you would install xlOil using:
pip install xloil
xloil install
Then write:
import xloil
#xloil.func
def myfunc(x, y, z):
return x + y * z
Put this code either:
In a py file in the same directory as your spreadsheet, named SpreadsheetName.py
In a py file on your python module path, then edit %APPDATA%\xlOil\xlOil.ini to load it as described in the docs.
Start Excel, open your spreadsheet and the function will be available as myfunc.
You could also try xlSlim (disclaimer I wrote it). The docs are here.
Installation is as simple as downloading and running a Windows installer from here The process is described in the docs and there is a YouTube video https://youtu.be/Zl5QM8rGBC8 It is straightforward, just download and run the installer. The installer is digitally signed and virus scanned.
Then to create the same function as Steve's you would write your function in a Python module:
def myfunc(x,y,z):
return x + y + z
(Note how no changes were made for the function work with xlSlim - this is a defining feature of xlSlim, no decorators or additional Python packages are required.)
Then in Excel use the RegisterPyModule() function to register the module (assuming you saved the module as mymod.py)
=RegisterPyModule("C:\Users\russe\Documents\mymod.py")
The function is now available for use within Excel as myfunc. Any type hints and doc strings are also processed.
I am trying to use ctypes to access a dynamic library in python.
So far, I have created a bunch of structures in python to reflect the structure types in the library. I am now trying to test these structures to see if I have translated everything correctly. I currently have something like the following set up
import ctypes as ct
class MyStruct(ct.Structure)
''' put the actual implementation here '''
pass
if __name__ == "__main__":
mylib = ct.CDLL('/path/to/mylib.dylib')
loadFile = mylib.loadFile
loadFile.restype = ct.Pointer(MyStruct)
parent = MyStruct()
file = b"/path/to/file"
child = loadFile(file, ct.byref(parent))
print(child)
Now, for some reason this segfaults inside the dynamic library when I try to run this code. I am trying to determine where (and why) this happens. I have compiled both the library and my python executable with debug info and am now trying to use gdb to locate the problem. I can start my program and break at a line inside loadFile very nicely
gdb b loading.c:172
gdb r
and gdb breaks. Now, there is another function call (this function is also part of the dynamic library) on line 172 in loading.c that I want to step into. I tried to break inside this other function
gdb b foo.c:636
but gdb keeps giving me
Cannot access memory at address 0x2860
errors. Further, if I just break at line 172 and then use stepi then I can't get any debug info. Is there a way to get into this other function or is it a limitation of gdb when using dynamic libraries.
I'm not a programmer, so I don't even know in what terms should I ask this. Let's say I've compiled a Python script to have an .exe (I use py2exe to do this). This is the major program. Now, I want to add some extra functionality to it, but I don't want to recompile the entire script with the added functionality. I tried to search something on the web, and I found examples of extending a C++ or other application with Python scripts (like a sort of plugin). But I can't figure out how to do it with an application already written in Python.
I tried this: I wrote major.py (this is the script from where I build the executable) and stuff.py. In major I wrote this:
def generic():
import stuff
while True:
param=input('what did you say? ')
stuff.speak(param)
generic()
And in stuff I wrote this:
def speak(param):
print(param)
Then I created a .exe with py2exe. It works as expected, when I run the program in the command line says "what did you say?" and waits until I type something, then it prints what I typed.
Then, I changed stuff.py with this:
def speak(param):
print('I said '+param)
Hoping that now upon the execution of the .exe created earlier it would print "I said.." plus whatever I typed. Obviously, it didn't work, the program continued to behave like before. So I'm guessing that once I imported stuff and created the .exe file, that import is permanent, not allowing me to change whatever is in stuff. What should I do?
py2exe packs the compiled scripts in the executable.
You need to recreate the executable (which will recompile any changed script) to see the changes take effect.
EDIT following the comments:
You can do it if you dynamically import/reimport the module from inside the executable.
In your main script you do (see code below)
mod, error = loadmodule('mystuff.py')
if mod is not None:
# loading succeeded you can now proceed and do things with it
pass
Of course you have to leave mystuff.py out of the scripts that py2exe packs into the executable. In the above example mystuff.py would be in the same directory as the executable.
The loading code:
def loadmodule(modpath, modname=''):
if not modpath.endswith('.py'):
modpath += '.py'
# generate a random name for the module
if not modname:
modpathbase = os.path.basename(modpath)
modname, _ = os.path.splitext(modpathbase)
version = (sys.version_info[0], sys.version_info[1])
if version < (3, 3):
mod, e = loadmodule2(modpath, modname)
else:
mod, e = loadmodule3(modpath, modname)
return mod, e
def loadmodule2(modpath, modname):
import imp
try:
mod = imp.load_source(modname, modpath)
except Exception as e:
return (None, e)
return (mod, None)
def loadmodule3(modpath, modname):
import importlib.machinery
try:
loader = importlib.machinery.SourceFileLoader(modname, modpath)
mod = loader.load_module()
except Exception as e:
return (None, e)
return (mod, None)
If you run your script from Python, instead of compiling it as an executable, you can make changes and run them without having to recompile. The py2exe is mostly for allowing you to distribute your application to other Windows computers that don't have Python installed. After you have finished developing it, then compile it as an executable so you can run it on other computers.
There is no way to do what you want. py2exe builds a standalone python interpreter (In the file named python.dll) with just the dependencies of your project. Then the .exe file runs your script using that interpreter
I suggest you that if you really need to provide regular upgrades then you should recompile it, or install python in the target machine, or make an updating routine that checks for updates and compiles it in the target machine (With py2exe)
I have a pure C module for Python and I'd like to be able to invoke it using the python -m modulename approach. This works fine with modules implemented in Python and one obvious workaround is to add an extra file for that purpose. However I really want to keep things to my one single distributed binary and not add a second file just for this workaround.
I don't care how hacky the solution is.
If you do try to use a C module with -m then you get an error message No code object available for <modulename>.
-m implementation is in runpy._run_module_as_main . Its essence is:
mod_name, loader, code, fname = _get_module_details(mod_name)
<...>
exec code in run_globals
A compiled module has no "code object" accociated with it so the 1st statement fails with ImportError("No code object available for <module>"). You need to extend runpy - specifically, _get_module_details - to make it work for a compiled module. I suggest returning a code object constructed from the aforementioned "import mod; mod.main()":
(python 2.6.1)
code = loader.get_code(mod_name)
if code is None:
+ if loader.etc[2]==imp.C_EXTENSION:
+ code=compile("import %(mod)s; %(mod)s.main()"%{'mod':mod_name},"<extension loader wrapper>","exec")
+ else:
+ raise ImportError("No code object available for %s" % mod_name)
- raise ImportError("No code object available for %s" % mod_name)
filename = _get_filename(loader, mod_name)
(Update: fixed an error in format string)
Now...
C:\Documents and Settings\Пользователь>python -m pythoncom
C:\Documents and Settings\Пользователь>
This still won't work for builtin modules. Again, you'll need to invent some notion of "main code unit" for them.
Update:
I've looked through the internals called from _get_module_details and can say with confidence that they don't even attempt to retrieve a code object from a module of type other than imp.PY_SOURCE, imp.PY_COMPILED or imp.PKG_DIRECTORY . So you have to patch this machinery this way or another for -m to work. Python fails before retrieving anything from your module (it doesn't even check if the dll is a valid module) so you can't do anything by building it in a special way.
Does your requirement of single distributed binary allow for the use of an egg? If so, you could package your module with a __main__.py with your calling code and the usual __init__.py...
If you're really adamant, maybe you could extend pkgutil.ImpLoader.get_code to return something for C modules (e.g., maybe a special __code__ function). To do that, I think you're going to have to actually change it in the Python source. Even then, pkgutil uses exec to execute the code block, so it would have to be Python code anyway.
TL;DR: I think you're euchred. While Python modules have code at the global level that runs at import time, C modules don't; they're mostly just a dict namespace. Thus, running a C module doesn't really make sense from a conceptual standpoint. You need some real Python code to direct the action.
I think that you need to start by making a separate file in Python and getting the -m option to work. Then, turn that Python file into a code object and incorporate it into your binary in such a way that it continues to work.
Look up setuptools in PyPi, download the .egg and take a look at the file. You will see that the first few bytes contain a Python script and these are followed by a .ZIP file bytestream. Something similar may work for you.
There's a brand new thing that may solve your problems easily. I've just learnt about it and it looks preety decent to me: http://code.google.com/p/pts-mini-gpl/wiki/StaticPython