So I have a function which creates a dynamic model. I accomplish this in a way very similar to AuditTrail (see django wiki).
Sample of code is here:
https://gist.github.com/0212845ae00891efe555
Is there any way I can make a dynamically-generated class pickle-able? Ideally something thats not a crazy monkeypatch/hack?
I am aware of the problem where pickle can't store a generated or dynamic class. I solved this by rigging in my dynamic type into the modules dict like so:
new_class = type(name, (models.Model,), attrs)
mod = sys.modules[new_class.__module__]
mod.__dict__[new_class.__name__] = new_class
It's FAR from a clean or elegant solution, so if someone can think of a more django-friendly way to make this happen, I am all ears. However, the above code does work.
The reason there aren't answers for this is because the answer is likely hackish. I don't think you can unpickle an object in Python without knowing the structure of the class on the receiving end without some sort of hackish solution. A big reason pickle doesn't support it is probably because it's a fantastic way to introduce malicious code into your application.
http://www.mofeel.net/871-comp-lang-python/2898.aspx explains a bit why dynamically created classes can't be unpickled.
In every case, I've either just serialized a dictionary of the attributes of the object using the dict method, or just figured out some awful work around. I hope you come up with something better.
Good Luck!
Related
I am trying to insert a collection of objects in a flask api. We use marshmallow for deserializing. My endpoint looks like this:
#blp.arguments(SomeSchemas, location='json', as_kwargs=True)
#blp.response(200, SomeSchemas)
def post(self, some_schemas: SomeSchemas) -> dict:
The schema is a simple schema like this:
class SomeSchemas(ma.Schema):
schemas = ma.fields.List(ma.fields.Nested(SomeSchema))
class SomeSchema(ma.Schema):
a = ma.fields.String()
b = ma.fields.Integer()
When i post to the endpoint, I do get a list of the correct data, but it comes in the form of dicts, instead of being correctly translated into the object.
I have also tried explicitly using a list of objects (List[SomeSchema], SomeSchema(many=True), etc.) but I can not seem to figure it out.
I assume this is a very common use case (providing a list of arguments) and that I am missing an obvious solution, but I can't seem to find any reference as to how to do this correctly. To be clear, I am looking for the correct way to call the endpoint with a list (or some other collection type, it does not matter) and have said list be correctly deserialized and with the correct object type.
Disclaimer: flask-smorest maintainer speaking.
I don't think the issue is related to the fact that the input is a list.
IIUC, your problem is that you're gettings dicts, rather than objects, injected in the view function. This is the default marshmallow behaviour. It can be overridden in marshmallow by using a post_load hook to actually instantiate the object.
I generally don't do that. In practice I find it better to instantiate objects in the view function. For instance, in a PUT resource, I prefer to instantiate the existing item from DB then update it with new data. In this case it is better to have the new data as dict than object.
There may not be a single truth, here, it could be a matter of opinion, but while the idea of having the object instantiated in the decorator and passed to the view is appealing, it might be a bit of an abusive shortcut.
I realize this answer is of the "no idea but you shouldn't do it anyway" type. Just saying maybe you shouldn't struggle to achieve this.
This said, I'd be surprised if it worked with non-list / non-nested inputs and I'd wonder why it doesn't work specifically in this case.
This question already has answers here:
How to make a class JSON serializable
(41 answers)
Closed 6 months ago.
I'm noting that the methods I am looking at to serialize a variable into JSON in python don't really seem to handle it all that well, and for my purpose I just want to quickly dump an objects contents into a string format so I can pick out what I actually want to write custom code to handle. I want to be able to dump the main fields at the very least of any class I pass the python serializer and really if its worth the name this should work.
So take the following code:
import json
c = SomeClass()
#causes an error if any field in someclass has another class instance.
json.dumps(c)
leads to..
TypeError: Object of type {Type} is not JSON serializable
Are there any modules other people have used that would solve my problem ? I really don't see how there would not be. Or maybe one might explain how to circumvent this error ?
The goal is to simply get some output to look at. If I wrote a recursion loop in c# using reflection, excepting circular references, it wouldn't be difficult, so I cannot imagine python users have never tackled this exact issue and I'm not satisfied with the answers that I have seen in older posts which seem to suggest a lot of custom tinkering for something seems to be designed in spirit to just dump any old object's contents out.
I don't even need complex traversal is the funny part, though it would be nice. I just need a dump of the property values which are primitive types in many cases. I know this is possible because the debugger does it.
Additionally I looked at one of the methods given indicating to use default lambda to specify how the json serializer should descend into the object:
json.dumps(o, default=lambda k: k.__dict__)
and the object does not contain the standard dict member.
in the end I just ended up writing a class to do this.
edit:
Here use this now you can one way serialize a class structure with this nifty little bit of code that I added to address my problem with f**** discord.py !
end edit
There is no fire and forget option that would disentangle a mass of information.
The way of creating this solution would be to manage seperate lists of subclasses to make sure not to recurse until a stackoverflow is reached.
The slots_ can be used with getattr(o,name) when hasattr(o,'dict') is False.
But the answer is you'd have to create a solution that basically does the job that the json serializer should be doing and cut out circular reference by determining the unique complex types and writing them in seperate tabular entries in the json file and replacing them in the referencing classes with ids.
That way you could cross reference these objects while glancing at them.
However the short answer is no. Python does not offer an out of the box way of doing this and all the provided answers encountered thus far only solve a single use-case or scenario, and do not create a incorporated solution to the problem which the above mentioned algorithm WOULD by NORMALIZING the class data into unique elements.
I made a wrapper, and recently I've reflected about what is a poor design decision in my opinion, however, I'm not sure what would be the better alternative.
Currently, my wrapper works like this (names changed):
import wrapper
api = wrapper.Api(params...)
# fetch fetches data from the API
objects = wrapper.model.fetch(api, params...)
# get_special might be only relevant to a certain object type
other_objects = wrapper.other_model.get_special(api, params...)
Obviously, passing the API as a parameter isn't very oop-y, and regardless it just looks bad. I considered just putting all of those classes in the API class, but I don't think that's a good idea, either. What would be a better design choice?
Thanks in advance :D
As per request, I'm adding more details.
My application is a wrapper for the website launchlibrary.net, which provides data about rocket launches. It's available here - https://github.com/Plutoberth/python-launch-library
Basically, I'm not sure how to implement this in a way that is both tidy and doesn't require you to pass the API each time. My ideal way would be like this, but I think that would require adding those functions in the API, which would make it quite huge.
import launchlibrary as ll # the wrapper
api = ll.Api(params..)
launches = api.Launch(params...)
I guess that this is more of a design question than an implementation one. Implementing that piece of code would be trivial, but it'd make my API class huge.
How can i implement structure array like matlab in python ?
matlab code :
cluster.c=[]
cluster.indiv=[]
Although you can do this in Python (as I explain below), it might not be the best or most pythonic approach. For other users that have to look at your code (including yourself in 3 months) this syntax is extremely confusing. Think for example about how this deals with name conflicts, undefined values and iterating over properties.
Instead consider storing the data in a data structure that is better suited for this such as a dictionary. Then you can just store everything in
cluster = {'c':[],'indiv':[]}
Imitating Matlab in a bad way:
You can assign properties to any mutable objects in python.
If you need an object just for data storage, then you can define a custom class without any functionality in the following way:
class CustomStruct():
pass
Then you can have
struct=CustomStruct()
struct.c=[]
and change or request properties of the class in this way.
Better approach:
If you really want to store these things as properties of an object, then it might be best to define the variables in the init of that class.
class BetterStruct():
def __init__(self):
self.c=[]
self.indiv=[]
In this way, users looking at your code can immediately understand the expected values, and you can guarantee that they are initalised in a proper fashion.
Allowing data control
If you want to verify the data when it is stored, or if it has to be calculated once the user requests it (instead of storing it constantly), then consider using Python property decorators
I was wondering if it possible to use the attrs library to convert nested JSONs to Python class instances so that I can access attributes in that JSON via dot notation (object.attribute.nested_attribute).
My JSONs have a fixed schema, and I would be fine with having to define the classes for that schema manually, but I'm not sure if it would be possible to turn the JSON into the nested class structure without having to instantiate every nested object individually. I'm basically looking for a fromdict() function that knows (based on the keys) which class to turn a JSON object into.
(I also know that there are other ways to build 'DotDicts', but these seem always a bit hacky to me and would probably need thorough testing to verify that they work correctly.)
The attrs wiki currently has two serialization libraries:
cattrs
and related.
With cattrs being maintained by one of attrs’ most prolific contributors.
I know that some people mention integrations with other systems too. At this point it's unlikely that attrs will grow an own solution since the externally developed look pretty good.