I've written some code in Python using numpy, pandas, scikit-learn. Is it possible to call this Python code from a Julia Program?
I think you can think to three different way to call Python code from Julia, in order from the most low level to the highest level they are:
Use the Foreign Funcall Interface as suggested by #madbird. However you will almost surelly not want to do this, as a package that exploits it, PyCall.jl, already exists;
Use the abovementioned PyCall.jl package to call any python code and/or wrap a python library (well.. "most".. "any" is a dangerous word). Details of this below;
As wrapping a Python library using PyCall is very easy, the most important Python libraries have been already wrapped in Julia, like Pandas in Pandas.jl, Scikit-learn in ScikitLearn.jl, etc. If you need them, just use the corresponding Julia package.
How to use PyCall.jl to call Python code and libraries.
Note: The following is an excerpt from my "Julia Quick Syntax Reference" book (Apress, 2019)
Julia ⇄ Python
The "standard" way to call Python code in Julia is to use the PyCall package.
Some of its nice features are: (a) it can automatically download and install a local copy of Python, private to Julia, in order to avoid messing with version dependency from our "main" Python installation and provide a consistent environment in Linux, Windows and MacOS; (b) it provides automatic conversion between Julia and Python types; (c) it is very simple to use.
Concerning the first point, PyCall by default install the "private" Python environment in Windows and MacOS while it will use the system default Python environment in Linux. +
We can override such behaviour with (from the Julia prompt) ENV["PYTHON"]="blank or /path/to/python"; using Pkg; Pkg.build("PyCall"); where, if the environmental variable is empty, PyCall will install the "private" version of Python.
Given the vast amount of Python libraries, it is no wonder that PyCall is one of the most common Julia packages.
Embed Python code in a Julia program
Embedding Python code in a Julia program is similar to what we saw with C++, except that we don't need (for the most) to wonder about transforming data. We both define and call the Python functions with py"...", and in the function call we can use directly our Julia data:
using PyCall
py"""
def sumMyArgs (i, j):
return i+j
def getNElement (n):
a = [0,1,2,3,4,5,6,7,8,9]
return a[n]
"""
a = py"sumMyArgs"(3,4) # 7
b = py"sumMyArgs"([3,4],[5,6]) # [8,10]
typeof(b) # Array{Int64,1}
c = py"sumMyArgs"([3,4],5) # [8,9]
d = py"getNElement"(1) # 1
Note that we don't need to convert even for complex data like arrays, and the results are converted back to Julia types.
Type conversion is automatic for numeric, boolean, string, IO stream, date/period, and function types, along with tuples, arrays/lists, and dictionaries of these types. Other types are instead converted to the generic PyObject type.
Note from the last line of the previous example that PyCall doesn't attempt index conversion (Python arrays are 0-based while Julia ones are 1-based): calling the python getNElement() function with "1" as argument will retrieve what in Python is the element "1" of the array.
Use Python libraries
Using a Python library is straightforward as well, as shown in the below example that use the ezodf module to create an OpenDocument spreadsheet (a wrapper of ezodf for ODS documents - that internally use PyCall - already exists, OdsIO).
Before attempting to replicate the following code, please be sure that the ezodf module is available to the Python environment you are using in Julia. If this is an independent environment, just follow the Python way to install packages (e.g. with pip). If you are using the "private" Conda environment, you can use the Conda.jl package and type using Conda; Conda.add_channel("conda-forge"); Conda.add("ezodf").
const ez = pyimport("ezodf") # Equiv. of Python `import ezodf as ez`
destDoc = ez.newdoc(doctype="ods", filename="anOdsSheet.ods")
sheet = ez.Sheet("Sheet1", size=(10, 10))
destDoc.sheets.append(sheet)
dcell1 = get(sheet,(2,3)) # Equiv. of Python `dcell1 = sheet[(2,3)]`. This is cell "D3" !
dcell1.set_value("Hello")
get(sheet,"A9").set_value(10.5) # Equiv. of Python `sheet['A9'].set_value(10.5)`
destDoc.backup = false
destDoc.save()
The usage in Julia of the module follows the Python API with few syntax differences.
The module is imported and assigned to a shorter alias, ez.
We can then directly call its functions with the usual Python syntax module.function().
The doc object returned by newdoc is a generic PyObject type. We can then access its attributes and methods with myPyObject.attribute and myPyObject.method() respectively.
In the cases where we can't directly access some indicized values, like sheet[(2,3)] (where the index is a tuple) we can invoke instead the get(object,key) function. +
Finally, note again that index conversion is not automatically implemented: when asking for get(sheet,(2,3)) these are interpreted as Python-based indexes, and cell D3 of the spreadsheet is returned, not B2.
I guess Foreign Funcall Interface is what you're looking for. Here is the example for Julia. More info in PyCall.jl repository.
Related
I'm trying to compile a usable .dll file from Julia to be used in Python as I've already written a large GUI in Python and need some fast optimization work done. Normally I would just call PyJulia or some "live" call, however this program needs to be compiled to distribute within my research team, so whatever solution I end up with needs to be able to run on its own (without Julia or Python actually installed).
Right now I'm able to create .dll files via PackageCompiler.jl, something I learned from previous posts on StackOverflow, however when trying to run these files in Python via the following code
Julia mock package
module JuliaFunctions
# Pkg.add("BlackBoxOptim")
Base.#ccallable function my_main_function(x::Cfloat,y::Cfloat)::Cfloat
z = 0
for i in 1:x
z += i ^ y
end
return z
end
# function julia_main()
# print("Hello from a compiled executable!")
# end
export my_main_function
end # module
Julia script to use PackageCompiler
# using PackageCompiler
using Pkg
# Pkg.develop(path="JuliaFunctions") # This is how you add a local package
# include("JuliaFunctions/src/JuliaFunctions.jl") # this is how you add a local module
using PackageCompiler
# Pkg.add(path="JuliaFunctions")
#time create_sysimage(:JuliaFunctions, sysimage_path="JuliaFunctions.dll")
Trying to use the resulting .dll in CTypes in Python
import ctypes
from ctypes.util import find_library
from ctypes import *
path = os.path.dirname(os.path.realpath(__file__)) + '\\JuliaFunctions.dll'
# _lib = cdll.LoadLibrary(ctypes.util.find_library(path)) # same error
# hllDll = ctypes.WinDLL(path, winmode=0) # same error
with os.add_dll_directory(os.path.dirname(os.path.realpath(__file__))):
_lib = ctypes.CDLL(path, winmode=0)
I get
OSError: [WinError 127] The specified procedure could not be found
With my current understanding, this means that CTypes found the dll and imported it, but didn't find.. something? I've yet to fully grasp how this behaves.
I've verified the function my_main_function is exported in the .dll file via Nirsoft's DLL Export Viewer. Users from previous similar issues have noted that this sysimage is already callable and should work, but they always add at the end something along the lines of "Note that you will also in general need to initialize the Julia runtime."
What does this mean? Is this even something that can be done independently from the Julia installation? The dev docs in PackageCompiler mention this, however they just mention that julia_main is automatically included in the .dll file and gets called as a sort of launch point. This function is also being exported correctly into the .dll file the above code creates. Below is an image of the Nirsoft export viewer output for reference.
Edit 1
Inexplicably, I've rebuilt this .dll on another machine and made progress. Now, the dll is imported correctly. I'm not sure yet why this worked on a fresh Julia install + Python venv, but I'm going to reinstall them on the other one and update this if anything changes. For anyone encountering this, also note you need to specify the expected output, whatever it may be. In my case this is done by adding (after the import):
_lib.testmethod1.restype = c_double # switched from Cfloat earlier, a lot has changed.
_lib.testmethod1.argtypes = [c_double, c_double] # (defined by ctypes)
The current error is now OSError: exception: access violation writing 0x0000000000000024 when trying to actually use the function, which is specific to Python. Any help on this would also be appreciated.
I have a python script which computes an array, and I would like to use this array in Julia.
What is the easiest way to do this?
If you have a long and complicated python script, and you want to call it directly from Julia and have the resulting array returned directly to Julia, the easiest option is probably to wrap your Python script into a Python module, install that module anywhere on your Python path, and import it into Julia with PyCall, using something along the lines of:
using PyCall
my_python_package = pyimport("my_python_package")
foo = my_python_package.my_python_function(some_arguments)
If your script is not so long and complicated, then it may be faster to just call the underlying Python modules directly from Julia, e.g.:
julia> using PyCall
julia> np = pyimport("numpy")
PyObject <module 'numpy' from '/Users/.../python3.7/site-packages/numpy/__init__.py'>
julia> np.random.normal(0,1,1000)
1000-element Array{Float64,1}:
-0.05105327176117878
0.5173117443548936
1.2998060543454042
⋮
-0.022888531778737377
-0.6389562444313613
0.6727725474307601
Note also that you can specify which Python PyCall uses with
julia> ENV["PYTHON"] = "/path/to/my/python/executable"
julia> Pkg.build("PyCall")
and can run arbitrary snippets of Python code from Julia with
julia> pyeval("1+1") # 1+1, but in python
2
julia> py"1+1" # this does the same thing
2
julia> py"""
some longer python snippet including characters
like newlines or \
that I don't want to have to have to escape
""" # also an option
I have written a python code, which takes 3 inputs, and return one output val.
I try to write an excel function, which passes the three inputs to the python function and returns the output.
I have looked into XLwings, but there is so many issues (and the documentation is insanely poor/poorly written) thus it seems useless.
So: is there any other way to call a python function (which takes inputs) from excel?
[SOLVED (ish):]
I managed, after roughly 8 hours of trying, 4 youtube videos and the xlwings homepage, to make it work.
Video for installing: https://training.zoomeranalytics.com/courses/xlwings/lectures/4231276
Video for making a function which takes input and returns output: https://www.youtube.com/watch?v=qn8xGrDuRCg&t=16s
You could try xlOil (disclaimer: I wrote it). The docs are here, but to write a simple three input function, you would install xlOil using:
pip install xloil
xloil install
Then write:
import xloil
#xloil.func
def myfunc(x, y, z):
return x + y * z
Put this code either:
In a py file in the same directory as your spreadsheet, named SpreadsheetName.py
In a py file on your python module path, then edit %APPDATA%\xlOil\xlOil.ini to load it as described in the docs.
Start Excel, open your spreadsheet and the function will be available as myfunc.
You could also try xlSlim (disclaimer I wrote it). The docs are here.
Installation is as simple as downloading and running a Windows installer from here The process is described in the docs and there is a YouTube video https://youtu.be/Zl5QM8rGBC8 It is straightforward, just download and run the installer. The installer is digitally signed and virus scanned.
Then to create the same function as Steve's you would write your function in a Python module:
def myfunc(x,y,z):
return x + y + z
(Note how no changes were made for the function work with xlSlim - this is a defining feature of xlSlim, no decorators or additional Python packages are required.)
Then in Excel use the RegisterPyModule() function to register the module (assuming you saved the module as mymod.py)
=RegisterPyModule("C:\Users\russe\Documents\mymod.py")
The function is now available for use within Excel as myfunc. Any type hints and doc strings are also processed.
I have a pure C module for Python and I'd like to be able to invoke it using the python -m modulename approach. This works fine with modules implemented in Python and one obvious workaround is to add an extra file for that purpose. However I really want to keep things to my one single distributed binary and not add a second file just for this workaround.
I don't care how hacky the solution is.
If you do try to use a C module with -m then you get an error message No code object available for <modulename>.
-m implementation is in runpy._run_module_as_main . Its essence is:
mod_name, loader, code, fname = _get_module_details(mod_name)
<...>
exec code in run_globals
A compiled module has no "code object" accociated with it so the 1st statement fails with ImportError("No code object available for <module>"). You need to extend runpy - specifically, _get_module_details - to make it work for a compiled module. I suggest returning a code object constructed from the aforementioned "import mod; mod.main()":
(python 2.6.1)
code = loader.get_code(mod_name)
if code is None:
+ if loader.etc[2]==imp.C_EXTENSION:
+ code=compile("import %(mod)s; %(mod)s.main()"%{'mod':mod_name},"<extension loader wrapper>","exec")
+ else:
+ raise ImportError("No code object available for %s" % mod_name)
- raise ImportError("No code object available for %s" % mod_name)
filename = _get_filename(loader, mod_name)
(Update: fixed an error in format string)
Now...
C:\Documents and Settings\Пользователь>python -m pythoncom
C:\Documents and Settings\Пользователь>
This still won't work for builtin modules. Again, you'll need to invent some notion of "main code unit" for them.
Update:
I've looked through the internals called from _get_module_details and can say with confidence that they don't even attempt to retrieve a code object from a module of type other than imp.PY_SOURCE, imp.PY_COMPILED or imp.PKG_DIRECTORY . So you have to patch this machinery this way or another for -m to work. Python fails before retrieving anything from your module (it doesn't even check if the dll is a valid module) so you can't do anything by building it in a special way.
Does your requirement of single distributed binary allow for the use of an egg? If so, you could package your module with a __main__.py with your calling code and the usual __init__.py...
If you're really adamant, maybe you could extend pkgutil.ImpLoader.get_code to return something for C modules (e.g., maybe a special __code__ function). To do that, I think you're going to have to actually change it in the Python source. Even then, pkgutil uses exec to execute the code block, so it would have to be Python code anyway.
TL;DR: I think you're euchred. While Python modules have code at the global level that runs at import time, C modules don't; they're mostly just a dict namespace. Thus, running a C module doesn't really make sense from a conceptual standpoint. You need some real Python code to direct the action.
I think that you need to start by making a separate file in Python and getting the -m option to work. Then, turn that Python file into a code object and incorporate it into your binary in such a way that it continues to work.
Look up setuptools in PyPi, download the .egg and take a look at the file. You will see that the first few bytes contain a Python script and these are followed by a .ZIP file bytestream. Something similar may work for you.
There's a brand new thing that may solve your problems easily. I've just learnt about it and it looks preety decent to me: http://code.google.com/p/pts-mini-gpl/wiki/StaticPython
It's not uncommon for an intro programming class to write a Lisp metacircular evaluator. Has there been any attempt at doing this for Python?
Yes, I know that Lisp's structure and syntax lends itself nicely to a metacircular evaluator, etc etc. Python will most likely be more difficult. I am just curious as to whether such an attempt has been made.
For those who don't know what a meta-circular evaluator is, it is an interpreter which is written in the language to be interpreted. For example: a Lisp interpreter written in Lisp, or in our case, a Python interpreter written in Python. For more information, read this chapter from SICP.
As JBernardo said, PyPy is one. However, PyPy's Python interpreter, the meta-circular evaluator that is, is implemented in a statically typed subset of Python called RPython.
You'll be pleased to know that, as of the 1.5 release, PyPy is fully compliant with the official Python 2.7 specification. Even more so: PyPy nearly always beats Python in performance benchmarks.
For more information see PyPy docs and PyPy extra docs.
I think i wrote one here:
"""
Metacircular Python interpreter with macro feature.
By Cees Timmerman, 14aug13.
"""
import re
re_macros = re.compile("^#define (\S+) ([^\r\n]+)", re.MULTILINE)
def meta_python_exec(code):
# Optional meta feature.
macros = re_macros.findall(code)
code = re_macros.sub("", code)
for m in macros:
code = code.replace(m[0], m[1])
# Run the code.
exec(code)
if __name__ == "__main__":
#code = open("metacircular_overflow.py", "r").read() # Causes a stack overflow in Python 3.2.3, but simply raises "RuntimeError: maximum recursion depth exceeded while calling a Python object" in Python 2.7.3.
code = "#define 1 2\r\nprint(1 + 1)"
meta_python_exec(code)