I am writing my thesis application. I need linear programming, but my app is written in Elixir, which is really not the language for such operations. That is why I decided to use Erlport as the Elixir dependency, which is capable of connecting Python code with Elixir. I'm also using Pulp as the python library for the optimization.
Elixir version: 1.10.4,
Erlport version: 0.10.1,
Python version: 3.8.5,
PuLP version: 2.3
I've written such a module for Elixir-Python communication, which leverages the GenServer as the main 'communication hub' between Elixir and Python:
defmodule MyApp.PythonHub do
use GenServer
def start_link(_) do
GenServer.start_link(__MODULE__, nil, name: __MODULE__)
end
def init(_opts) do
path = [:code.priv_dir(:feed), "python"]
|> Path.join() |> to_charlist()
{:ok, pid} = :python.start([{ :python_path, path }, { :python, 'python3' }])
{:ok, pid}
end
def handle_call({:call_function, module, function_name, arguments}, _sender, pid) do
result = :python.call(pid, module, function_name, arguments)
{:reply, result, pid}
end
def call_python_function(file_name, function_name, arguments) do
GenServer.call(__MODULE__, {:call_function, file_name, function_name, arguments}, 10_000)
end
end
The GenServer module is calling python file, which contains such a function:
def calculate_meal_4(products_json, diet_json, lower_boundary, upper_boundary, enhance):
from pulp import LpMinimize, LpProblem, LpStatus, lpSum, LpVariable, value
import json
products_dictionary = json.loads(products_json)
print(products_dictionary)
diets_dictionary = json.loads(diet_json)
print(diets_dictionary)
model = LpProblem(name="diet-minimization", sense=LpMinimize)
# ... products setup ...
x = LpVariable("prod_1_100g", lower_boundary, upper_boundary)
y = LpVariable("prod_2_100g", lower_boundary, upper_boundary)
z = LpVariable("prod_3_100g", lower_boundary, upper_boundary)
w = LpVariable("prod_4_100g", lower_boundary, upper_boundary)
optimization_function = # ... optimization function setup ...
model += # ... optimization boundary function setup ...
model += optimization_function
print(model)
solved_model = model.solve()
print(value(model.objective))
return [value(x), value(y), value(z), value(w)]
The call to the GenServer itself looks like that:
PythonHub.call_python_function(:diets, python_function, [products_json, meal_statistics_json, #min_portion, #max_portion, #macro_enhancement])
where python_function is :calculate_meal_4 and products_json and meal_statistic_json are jsons containing required data.
While calling calculate_meal_4 via python3 diets.py, which launches the python script above with some example, but real (taken from the app), data everything works fine - I've got the minimized result in almost no time. The problem occurs while calling the python script via Elixir Erlport. Looking at the printed outputs I can tell that it seems working until
solved_model = model.solve()
is called. Then the script seems to freeze and GenServer finally reaches the timeout on GenServer.call function.
I've tested also the call on a simple python test file:
def pass_var(a):
print(a)
return [a, a, a]
and it worked fine.
That is why I am really consterned right now and I am looking for any advices. Shamefully I found nothing yet.
Hmm, it might be that calling an external solver freezes the process.
Given that you can execute bash scripts using elixir, you can easily change the python script to be command line executable (I recommend click). Then, you can write the output to a .json or .csv file and read it back in with Elixir when you're done.
#click.group()
def cli():
pass
#cli.command()
#click.argument('products_json', help='your array of products')
#click.argument('diet_json', help='your dietary wishes')
#click.option('--lower-bound', default=0, help='your minimum number of desired calories')
#click.option('--upper-bound', default=100, help='your maximum number of desired calories')
#click.option('--enhance', default=False, help="whether you'd like to experience our enhanced experience")
def calculate_meal_4(products_json, diet_json, lower_boundary, upper_boundary, enhance):
pass
if __name__ == '__main__':
cli()
which you can then call using python3 my_file.py <products_json> <diet_json> ... et cetera.
You can even validate the JSON and then return the parsed data directly.
Related
I'm working on this project where I am using Python + Lupa to run Lua code.
I want to run untrusted Lua code (as a string) within my Python script using lupa.LuaRuntime().eval(). I've looked around to see what I need to do to restrict what this Lua code has access to.
I stumbled upon this old post (https://stackoverflow.com/a/17455485) that shows you how to do it in Lua 5.1 using setfenv():
import lupa
L = lupa.LuaRuntime()
sandbox = L.eval("{}")
setfenv = L.eval("setfenv")
sandbox.print = L.globals().print
sandbox.math = L.globals().math
sandbox.string = L.globals().string
sandbox.foobar = foobar
# etc...
setfenv(0, sandbox)
L.execute("os.execute('rm -rf *')")
the setfenv function doesn't exist in Lua 5.4, which I'm using.
How do you do this in more modern versions of Lua?
I've tried to create a new function (sb_func()) within load() and then call it, but it does nothing
import lupa
L = lupa.LuaRuntime()
sandbox = L.eval("{}")
load = L.eval("load")
sandbox.math = L.globals().math
sandbox.print = L.globals().print
sb_func = load("function sb_func() print('test') return nil end","","t",sandbox)
sb_func
I want to use Luigi to manage workflows in Openstack. I am new to Luigi. For the starter, I just want to authenticate myself to Openstack and then fetch image list, flavor list etc using Luigi. Any help will be appreciable.
I am not good with python but I tried below code. I am also not able to list images. Error: glanceclient.exc.HTTPNotFound: The resource could not be found. (HTTP 404)
import luigi
import os_client_config
import glanceclient.v2.client as glclient
from luigi.mock import MockFile
import sys
import os
def get_credentials():
d = {}
d['username'] = 'X'
d['password'] = 'X'
d['auth_url'] = 'X'
d['tenant_name'] = 'X'
d['endpoint'] = 'X'
return d
class LookupOpenstack(luigi.Task):
d =[]
def requires(self):
pass
def output(self):
gc = glclient.Client(**get_credentials())
images = gc.images.list()
print("images", images)
for i in images:
print(i)
return MockFile("images", mirror_on_stderr=True)
def run(self):
pass
if __name__ == '__main__':
luigi.run(["--local-scheduler"], LookupOpenstack())
The general approach to this is just write python code to perform the tasks you want using the OpenStack API. https://docs.openstack.org/user-guide/sdk.html It looks like the error you are getting is addressed on the OpenStack site. https://ask.openstack.org/en/question/90071/glanceclientexchttpnotfound-the-resource-could-not-be-found-http-404/
You would then just wrap this code in luigi Tasks as appropriate- there's nothing special about doing with this OpenStack, except that you must define the output() of your luigi tasks to match up with an output that indicates the task is done. Right now it looks like the work is being done in the output() method, which should be in the run() method, the output method should just be what to look for to indicate that the run() method is complete so it doesn't run() when required by another task if it is already done.
It's really impossible to say more without understanding more details of your workflow.
I am trying to use Python with Elixir and I wrote the following functional code (you can find the repo I'm building here: https://github.com/arthurcolle/elixir_with_erlport)
defmodule Snake do
use Application
def start(_type, _args) do
import Supervisor.Spec, warn: false
children = [
# Define workers and child supervisors to be supervised
# worker(Snake.Worker, [arg1, arg2, arg3]),
]
opts = [strategy: :one_for_one, name: Snake.Supervisor]
Supervisor.start_link(children, opts)
end
def py do
{:ok, pp} = :python.start()
:python.call(pp, :__builtin__, :print, ["hey there"])
end
end
I can run iex -S mix run, then type in Snake.py, and I will get this output:
"hey there"
:undefined
Okay, great.
Then I try to make it print out the current version of Python by swapping out the two lines above with:
{:ok, pp} = :python.start()
:python.call(pp, :sys, :version, [])
But when I run it, it gives me this arity error
** (FunctionClauseError) no function clause matching in :erlport.call/5
src/erlport.erl:87: :erlport.call(#PID<0.108.0>, :sys, 'version.__str__', [], [])
Which doesn't make any sense to me because my call only is a :erlport.call/4, with one single list at the end (not 2 as it is saying).
{:ok, pp} = :python.start_link()
:python.call(pp, :sys, String.to_atom("version.__str__"), [])
I've got an application I'm currently working on for our company. Its currently built around Python's Cmd module, and features tab-completion for a number of tasks.
For some reason however, the Tab completion only currently works on one machine in the building - running the scripts from other machines doesn't allow the tab completion.
Here's the offending code parts:
def populate_jobs_list():
global avail_jobs
avail_jobs = os.walk(rootDir()).next()[1]
print avail_jobs
...
def complete_job(self, text, line, start_index, end_index):
global avail_jobs
populate_jobs_list()
if text:
return [
jobs for jobs in avail_jobs
if jobs.startswith(text)
]
else:
return avail_jobs
def do_job(self, args):
pass
split_args = args.rsplit()
os.environ['JOB'] = args
job_dir = os.path.join( rootDir(), os.getenv('JOB'))
os.environ['JOB_PROPS'] = (job_dir + '\\job_format.opm')
if not os.path.isdir(job_dir):
print 'Job does not exist. Try again.'
return
else:
print('Jobbed into: ' + os.getenv('JOB'))
return
populate_jobs_list()
prompt = outPrompt()
prompt.prompt = '\> '
prompt.cmdloop('Loading...')
Am I missing something obvious here? Just to clarify, on machine A, the tab completion works as intended. When its run on any other machine in the building, it fails to complete.
Check if the environment variable PYTHONSTARTUP is set properly. It should point to a script which in turn needs to do sth like this:
try:
import readline
except ImportError:
sys.stdout.write("No readline module found, no tab completion available.\n")
else:
import rlcompleter
readline.parse_and_bind('tab: complete')
Maybe (some part of) this is only done properly on the one working machine?
Maybe the readline module is available only on the one working machine?
How do I check whether the screen is off due to the Energy Saver settings in System Preferences under Mac/Python?
Quick and dirty solution: call ioreg and parse the output.
import subprocess
import re
POWER_MGMT_RE = re.compile(r'IOPowerManagement.*{(.*)}')
def display_status():
output = subprocess.check_output(
'ioreg -w 0 -c IODisplayWrangler -r IODisplayWrangler'.split())
status = POWER_MGMT_RE.search(output).group(1)
return dict((k[1:-1], v) for (k, v) in (x.split('=') for x in
status.split(',')))
In my computer, the value for CurrentPowerState is 4 when the screen is on and 1 when the screen is off.
Better solution: use ctypes to get that information directly from IOKit.
The only way i can think off is by using OSX pmset Power Management CML Tool
DESCRIPTION
pmset changes and reads power management settings such as idle sleep timing, wake on administrative
access, automatic restart on power loss, etc.
Refer to the following link, it will provide a great deal of information that should aid you in accomplishing exactly what you are looking for.
http://managingamac.blogspot.com/2012/12/power-assertions-in-python.html
I will include the code provided by the link for "saving and documentation" purposes:
#!/usr/bin/python
import ctypes
import CoreFoundation
import objc
import subprocess
import time
def SetUpIOFramework():
# load the IOKit library
framework = ctypes.cdll.LoadLibrary(
'/System/Library/Frameworks/IOKit.framework/IOKit')
# declare parameters as described in IOPMLib.h
framework.IOPMAssertionCreateWithName.argtypes = [
ctypes.c_void_p, # CFStringRef
ctypes.c_uint32, # IOPMAssertionLevel
ctypes.c_void_p, # CFStringRef
ctypes.POINTER(ctypes.c_uint32)] # IOPMAssertionID
framework.IOPMAssertionRelease.argtypes = [
ctypes.c_uint32] # IOPMAssertionID
return framework
def StringToCFString(string):
# we'll need to convert our strings before use
return objc.pyobjc_id(
CoreFoundation.CFStringCreateWithCString(
None, string,
CoreFoundation.kCFStringEncodingASCII).nsstring())
def AssertionCreateWithName(framework, a_type,
a_level, a_reason):
# this method will create an assertion using the IOKit library
# several parameters
a_id = ctypes.c_uint32(0)
a_type = StringToCFString(a_type)
a_reason = StringToCFString(a_reason)
a_error = framework.IOPMAssertionCreateWithName(
a_type, a_level, a_reason, ctypes.byref(a_id))
# we get back a 0 or stderr, along with a unique c_uint
# representing the assertion ID so we can release it later
return a_error, a_id
def AssertionRelease(framework, assertion_id):
# releasing the assertion is easy, and also returns a 0 on
# success, or stderr otherwise
return framework.IOPMAssertionRelease(assertion_id)
def main():
# let's create a no idle assertion for 30 seconds
no_idle = 'NoIdleSleepAssertion'
reason = 'Test of Pythonic power assertions'
# first, we'll need the IOKit framework
framework = SetUpIOFramework()
# next, create the assertion and save the ID!
ret, a_id = AssertionCreateWithName(framework, no_idle, 255, reason)
print '\n\nCreating power assertion: status %s, id %s\n\n' % (ret, a_id)
# subprocess a call to pmset to verify the assertion worked
subprocess.call(['pmset', '-g', 'assertions'])
time.sleep(5)
# finally, release the assertion of the ID we saved earlier
AssertionRelease(framework, a_id)
print '\n\nReleasing power assertion: id %s\n\n' % a_id
# verify the assertion has been removed
subprocess.call(['pmset', '-g', 'assertions'])
if __name__ == '__main__':
main()
https://opensource.apple.com/source/PowerManagement/PowerManagement-211/pmset/pmset.c
The code relies on IOPMLib, which functions to make assertions, schedule power events, measure thermals, and more.
https://developer.apple.com/documentation/iokit/iopmlib_h
To call these functions through Python, we must go through the IOKit Framework.
https://developer.apple.com/library/archive/documentation/DeviceDrivers/Conceptual/IOKitFundamentals/Introduction/Introduction.html
In order for us to manipulate C data types in Python, we'll use a foreign function interface called ctypes.
http://python.net/crew/theller/ctypes/
Here's the wrapper the author describe's on the page; written by Michael Lynn. The code i posted from the Author's link above is a rewrite of this code to make it more understandable.
https://github.com/pudquick/pypmset/blob/master/pypmset.py