I want to build a library in python for scientific computing. Precisely for natural coordinates mechanics. I want to think a bit before jumping into the code. This post asks for any advice, bad or good ideas. I'm not a pro in python, and I'm sure. There can be experts around.
First, I want my library to be usable with numpy (numeric computing) and casadi (symbolic computing). To be as simple as possible, I'd like the library to be loadable:
import my_package_numpy # option 1
import my_package_casadi # option 2
Second, I want each class or each function and method to work both with these two types of objects ndarrays (numpy) and MX (casadi).
my_var_np = np.array([2,2]) # numeric
my var_mx = MX.sym("var",[2,1]) # symbolic
from my_package_numpy import HelloWorld # option 1
hw = HelloWorld(my_var_np)
from my_package_casadi import HelloWorld # option 2
hw = HelloWorld(my_var_mx)
Third, I want to be able to switch to know when the user uses one or another backend in specific methods. because maths operations may be defined differently. (Is there a global variable available to know that ?)
class HelloWorld:
def __init__(value):
self.value = value
if numpy:
self.transpose_value = value.transpose()
elif casadi:
self.transpose_value = transpose(value)
I know the if are not well written, I don't know where to get this value.
This library could grow fast. I want to avoid as much as possible copied code and things to be written only once.
Any help, advice, or comment, would be appreciated.
I'm trying to build a package with two math backends. And I expect it to be as simple as possible for the user. but I don't know how to build the backend
Maybe you could use an if condition to check the type of the value like this (if the value is a numpy array then use numpy):
class HelloWorld:
def __init__(value):
if type(value) == np.ndarray:
numpy = True
self.value = value
if numpy:
self.transpose_value = value.transpose()
elif casadi:
self.transpose_value = transpose(value)
I don't know if this is the right way, but maybe it works for you :)
Related
So what I'm doing is dynamically generating a function that returns the output of a mathematical equation f(x,y) = z like so:
def compile(self) -> callable[[tuple(float)], float]:
# user input is something like: xsin(y) + 1 = xcos(y)
# input is tokenised and syntax is analysed
# tokens converted to 'eval-able' python code
# this gives: axes[0]*np.sin(axes[1])+1-axes[0]*np.cos(axes[1])
# this is stored in evalString
def func(axes: tuple(float)) -> float:
v = type(self).variables # user-defined variables
f = type(self).functions # user-defined functions
return eval(evalString)
return func
Is there a better way to generate a function like this? I have been told to avoid using exec() due to security issues, but have also been told that eval() isn't much better. I believe that most ways of executing potentially malicious code should be ruled out within the compilation stages, but I am aware that this may not be true and will require further testing.
I am also wondering if using eval() is the best/most efficient way to approach to this or if I should do/use something else?
Many thanks!
I am developing a GUI application (in the civil engineering context) with python3 and QT and want to display an equation in three different ways:
symbolic: sigma=N/A
with values: sigma=200kN/20cm²
as a result: sigma=10kN/cm²
The layout of the equation and the order of symbols has to be the same for both (1) and (2), but i only want to enter the equation once in my sourcecode. I searched a lot, this is the best i could get:
class myfancy_equation():
def __init__(self):
self.a = Symbol('a')
self.b = Symbol('b',commutative=False)
self.x = Symbol('x')
self.expr = (self.b * self.a)/self.x
def mlatex(self):
return latex(self.expr)
def mevaluate(self,a_in,b_in,x_in):
unev = self.expr.subs({self.a:a_in,self.b:b_in,self.x:x_in})
symb_names = dict()
symb_names[self.a] = str(a_in)
symb_names[self.b] = str(b_in)
symb_names[self.x] = str(x_in)
# unev_latex = latex(self.expr.subs({self.a:a_in,self.b:b_in,self.x:x_in}))
unev_latex = latex(self.expr,symbol_names=symb_names)
ev = self.expr.subs({self.a:a_in,self.b:b_in,self.x:x_in}).evalf()
return unev,unev_latex,ev
mfe = myfancy_equation()
lat = mfe.mlatex()
un,unl,ev = mfe.mevaluate(5,7,3)
print("Original, unevaluated, evaluated:")
print( lat,"=",unl,'=',ev)
I have read that sympy was not primarly developed for displaying equations, but the result is hardly readable (and unpredictable) for more complex equations. i tried playing around with the "commutative" parameter, but always end up with a mixed equation like this:
http://www.oberthanner.at/render.png
am i missing the point or is it just impossible with sympy?
btw: i encountered a different behaviour of the commutative parameter when using python2.
commutative=False will only mark that one symbol as non-commutative. SymPy will put the commuting part (in this case, everything else) first, and follow it by the non-commuting symbols in the correct order.
But you shouldn't use that anyhow. It will not give you what you want (e.g., you'll get a*b**-1 instead of a/b if a and b are noncommutative, since a*b**-1 and b**-1*a are different).
I would recommend just getting the latex for the individual parts that you want, and piecing them together in the order you want using string formatting.
Alternately, you can write your own printer that orders things the way you want. See http://docs.sympy.org/latest/modules/printing.html if you are interested in taking that route, and you should also read the source code for the existing printer, since you'll probably want to just take what is already there and modify it a little. This method is good if you want to be more general, and the basic string concatenation gets too messy. If the example you showed is as complicated as it will get, it may be overkill, but if you need to support potentially arbitrarily complicated expressions, it may be better to do it this way.
If you decide to take that second route and need help writing a custom printer, feel free to ask here or on the SymPy mailing list.
I want to search surfs in all images in a given directory and save their keypoints and descriptors for future use. I decided to use pickle as shown below:
#!/usr/bin/env python
import os
import pickle
import cv2
class Frame:
def __init__(self, filename):
surf = cv2.SURF(500, 4, 2, True)
self.filename = filename
self.keypoints, self.descriptors = surf.detect(cv2.imread(filename, cv2.CV_LOAD_IMAGE_GRAYSCALE), None, False)
if __name__ == '__main__':
Fdb = open('db.dat', 'wb')
base_path = "img/"
frame_base = []
for filename in os.listdir(base_path):
frame_base.append(Frame(base_path+filename))
print filename
pickle.dump(frame_base,Fdb,-1)
Fdb.close()
When I try to execute, I get a following error:
File "src/pickle_test.py", line 23, in <module>
pickle.dump(frame_base,Fdb,-1)
...
pickle.PicklingError: Can't pickle <type 'cv2.KeyPoint'>: it's not the same object as cv2.KeyPoint
Does anybody know, what does it mean and how to fix it? I am using Python 2.6 and Opencv 2.3.1
Thank you a lot
The problem is that you cannot dump cv2.KeyPoint to a pickle file. I had the same issue, and managed to work around it by essentially serializing and deserializing the keypoints myself before dumping them with Pickle.
So represent every keypoint and its descriptor with a tuple:
temp = (point.pt, point.size, point.angle, point.response, point.octave,
point.class_id, desc)
Append all these points to some list that you then dump with Pickle.
Then when you want to retrieve the data again, load all the data with Pickle:
temp_feature = cv2.KeyPoint(x=point[0][0],y=point[0][1],_size=point[1], _angle=point[2],
_response=point[3], _octave=point[4], _class_id=point[5])
temp_descriptor = point[6]
Create a cv2.KeyPoint from this data using the above code, and you can then use these points to construct a list of features.
I suspect there is a neater way to do this, but the above works fine (and fast) for me. You might have to play around with your data format a bit, as my features are stored in format-specific lists. I tried to present the above using my idea at its generic base. I hope that this may help you.
Part of the issue is cv2.KeyPoint is a function in python that returns a cv2.KeyPoint object. Pickle is getting confused because, literally, "<type 'cv2.KeyPoint'> [is] not the same object as cv2.KeyPoint". That is, cv2.KeyPoint is a function object, while the type was cv2.KeyPoint. Why OpenCV is like that, I can only make guesses at unless I go digging. I have a feeling it has something to do with it being a wrapper around a C/C++ library.
Python does give you the ability to fix this yourself. I found the inspiration on this post about pickling methods of classes.
I actually use this clip of code, highly modified from the original in the post
import copyreg
import cv2
def _pickle_keypoints(point):
return cv2.KeyPoint, (*point.pt, point.size, point.angle,
point.response, point.octave, point.class_id)
copyreg.pickle(cv2.KeyPoint().__class__, _pickle_keypoints)
Key points of note:
In Python 2, you need to use copy_reg instead of copyreg and point.pt[0], point.pt[1] instead of *point.pt.
You can't directly access the cv2.KeyPoint class for some reason, so you make a temporary object and use that.
The copyreg patching will use the otherwise problematic cv2.KeyPoint function as I have specified in the output of _pickle_keypoints when unpickling, so we don't need to implement an unpickling routine.
And to be nauseatingly complete, cv2::KeyPoint::KeyPoint is an overloaded function in C++, but in Python, this isn't exactly a thing. Whereas in the C++, there's a function that takes the point for the first argument, in Python, it would try to interpret that as an int instead. The * unrolls the point into two arguments, x and y to match the only int argument constructor.
I had been using casper's excellent solution until I realized this was possible.
A similar solution to the one provided by Poik. Just call this once before pickling.
def patch_Keypoint_pickiling(self):
# Create the bundling between class and arguments to save for Keypoint class
# See : https://stackoverflow.com/questions/50337569/pickle-exception-for-cv2-boost-when-using-multiprocessing/50394788#50394788
def _pickle_keypoint(keypoint): # : cv2.KeyPoint
return cv2.KeyPoint, (
keypoint.pt[0],
keypoint.pt[1],
keypoint.size,
keypoint.angle,
keypoint.response,
keypoint.octave,
keypoint.class_id,
)
# C++ Constructor, notice order of arguments :
# KeyPoint (float x, float y, float _size, float _angle=-1, float _response=0, int _octave=0, int _class_id=-1)
# Apply the bundling to pickle
copyreg.pickle(cv2.KeyPoint().__class__, _pickle_keypoint)
More than for the code, this is for the incredibly clear explanation available there : https://stackoverflow.com/a/50394788/11094914
Please note that if you want to expand this idea to other "unpickable" class of openCV, you only need to build a similar function to "_pickle_keypoint". Be sure that you store attributes in the same order as the constructor. You can consider copying the C++ constructor, even in Python, as I did. Mostly C++ and Python constructors seems not to differ too much.
I has issue with the "pt" tuple. However, a C++ constructor exists for X and Y separated coordinates, and thus, allow this fix/workaround.
I've installed ioapiTools, a python module to manage ioapi format files. The module is supposed to handle file and perform operations on them, including basic arithmetic operations. But something is wrong and when I try to, say, multiply an array by a float or an integer, the result is a zero-valued array (both the array and the float/integer are different from zero).
The module in question creates a temporary variable using cdms2 according to the following syntax:
import cdms2 as cdms, cdtime, MV2 as MV, cdutil
import numpy as N
..........
def __mul__(self, other):
"""
Wrapper around cdms tvariable multiply
"""
tmpVar = cdms.tvariable.TransientVariable.__mul__(self,other)
iotmpVar = createVariable(tmpVar, self.ioM, id = self.id,\
attributes=self.attributes, copyFlag = False)
return iotmpVar
But the variable returns nothing but zeros.
Any ideas?
I tried to use ioapiTools, and latest version i found was 0.3.2 from http://www2-pcmdi.llnl.gov/Members/azubrow/ioapiTools/download-source-file .
unfortunately, the code doesn't seem to catchup with evolution of cdat, which now recommend using numpy instead of Numeric. automated translation tool may be resolving some problems, but not all. For example, the class iovar (defined in ioapiTools.py:2103) now needs to have _____new_____ method, as it is a subclass of numpy masked array (i dont know how things are in Numeric). With that, i seems to have _____mul_____ working. i couldn't reproduce your problem though, because i couldn't even get an instance of iovar without having _____new_____ method defined.
i can pass what i got to you if you still need one, but i am sure there are more problems hiding... let me know if you need it though.
Here's how I am declaring constants and using them across different Python classes:
# project/constants.py
GOOD = 1
BAD = 2
AWFUL = 3
# project/question.py
from constants import AWFUL, BAD, GOOD
class Question:
def __init__(self):
...
Is the above a good way to store and use contant values? I realise that after a while, the constants file can get pretty big and I could explicitly be importing 10+ of those constants in any given file.
why not just use
import constants
def use_my_constants():
print constants.GOOD, constants.BAD, constants.AWFUL
From the python zen:
Namespaces are good. Lets do more of those!
EDIT: Except, when you do quote, you should include a reference and check it, because as others have pointed out, it should read:
Namespaces are one honking great idea -- let's do more of those!
This time, I actually copied it from the source: PEP 20 -- The Zen of Python
You also have the option, if the constants are tied to a particular class and used privately within that class of making them specific to that class:
class Foo(object):
GOOD = 0
BAD = 1
WTF = -1
def __init__(self...
and away you go.
You have a few options:
Do it the way you're doing right now with a file of constants
Use a flat file and parse it once, pass a dictionary / class around
Query off to a database
From an overhead point of view, 1 and 2 are about the same. As for your question about importing specific constants, it's much easier to use one of the following conventions:
import constants; some_func(constants.AWFUL)
from constants import *; some_func(AWFUL)
Try to look at the following links(one link contains a receipt from active state of how to implement a real constants):
Importing a long list of constants to a Python file
Create constants using a "settings" module?
Can I prevent modifying an object in Python?
I use a function as immutable constant, like this:
def test_immutable_constant():
def GOOD():
return 1
good_value = GOOD()
good_value = 2
assert GOOD() == 1
assert good_value == 2