Why can't I pickle this python class using pickle? [duplicate] - python

Python docs mention this word a lot and I want to know what it means.

It simply means it can be serialized by the pickle module. For a basic explanation of this, see What can be pickled and unpickled?. Pickling Class Instances provides more details, and shows how classes can customize the process.

Things that are usually not pickable are, for example, sockets, file(handler)s, database connections, and so on. Everything that's build up (recursively) from basic python types (dicts, lists, primitives, objects, object references, even circular) can be pickled by default.
You can implement custom pickling code that will, for example, store the configuration of a database connection and restore it afterwards, but you will need special, custom logic for this.
All of this makes pickling a lot more powerful than xml, json and yaml (but definitely not as readable)

These are all great answers, but for anyone who's new to programming and still confused here's the simple answer:
Pickling an object is making it so you can store it as it currently is, long term (to often to hard disk). A bit like Saving in a video game.
So anything that's actively changing (like a live connection to a database) can't be stored directly (though you could probably figure out a way to store the information needed to create a new connection, and that you could pickle)
Bonus definition: Serializing is packaging it in a form that can be handed off to another program. Unserializing it is unpacking something you got sent so that you can use it

Pickling is the process in which the objects in python are converted into simple binary representation that can be used to write that object in a text file which can be stored. This is done to store the python objects and is also called as serialization. You can infer from this what de-serialization or unpickling means.
So when we say an object is picklable it means that the object can be serialized using the pickle module of python.

Related

How to store and read common data in redis from python and rust?

I have some data being stored in redis cache which will be read by my application in Rust. The data is being stored by python. Whenever I am storing a string or an array, it stores it in a weird form which I was not able to read into Rust. Vice versa, I want to write from Rust and be able to read it in python.
Using django shell:
In [0]: cache.set("test","abc")
In [1]: cache.get("test")
Out[1]:'abc'
Using redis-cli:
127.0.0.1:6379> GET :1:test
"\x80\x04\x95\a\x00\x00\x00\x00\x00\x00\x00\x8c\x03abc\x94."
Output from Rust:
Err(Invalid UTF-8)
Rust code read data using redis-rs library:
let client = redis::Client::open("redis://127.0.0.1:6379")?;
let mut con = client.get_connection()?;
let q:Result<String, redis::RedisError> = con.get(":1:test");
println!("{:?}",q);
I want to be able to read a string or array into Rust as it was written in Python and vice-versa.
Also, data in one key will only be ever written by either Rust or Python, not both.
This question is not a duplicate of this as that deals specifically for accent encoding, however, I want to solve my problem for arrays as well. Moreover, the value being set in redis by django for a string is not simply the UTF encoding for the string.
Ah, the joys of trying to throw data across environments. The thing you're being bitten by right now is called Pickle and is the default serializer of django-redis. What a serializer does in this case (in python) is the transformation of your data between python and redis so you can store it, regardless of the type, but more importantly so you can retrieve it with the type it came in.
The python side
Obviously, if you had infinite time and effort, you could rewrite pickle in rust and you'd then be able to read this format. I'm pretty sure you have neither, and depending on the data you're storing, you might not even want to do so.
Instead, what I'm going to suggest is to change the serializer from pickle to json. The description of what to change in the config is located at https://django-redis-cache.readthedocs.io/en/latest/advanced_configuration.html#pluggable-serializers , and in particular, I'm pretty sure the class name you want to use is django_redis.serializers.JSONSerializer.
This comes with drawbacks. In particular, there will be some object types you will no longer be able to store on the python side, but if you do really intend to read data on the rust side, this should not concern you.
Sven Marnach mentioned in one of the comments that the serde-pickle crate exists. I have not used it myself, but it does look promising and might save you a ton of interop work if it does function.
The rust side
To read stuff, now that every key is going to be json, you'll be decoding types with either serde or miniserde. This should be pretty straightforward; do bear in mind that you will not get native types out of this; instead, you'll get members of the serde::Value enum (Boolean, Number, Object, etc), which you will then have to filter through.
Edit your question to indicate what you are trying to store, and I'll happily expand on how to do this on here!

Is there any difference between Pickling and Serialization?

I have came across these two terms so often while reading about python objects. However, there is a confusion between pickling and serialization since at one place I read
The pickle module implements an algorithm for turning an arbitrary
Python object into a series of bytes. This process is also called
serializing” the object.
If serializing and pickling is same process, why use different terms for them?
You are misreading the article. Pickling and serialisation are not synonymous, nor does the text claim them to be.
Paraphrasing slighly, the text says this:
This module implements an algorithm for turning an object into a series of bytes. This process is also called serializing the object.
I removed the module name, pickle, deliberately. The module implements a process, an algorithm, and that process is commonly known as serialisation.
There are other implementations of that process. You could use JSON or XML to serialise data to text. There is also the marshal module. Other languages have other serialization formats; the R language has one, so does Java. Etc.
See the WikiPedia article on the subject:
In computer science, in the context of data storage, serialization is the process of translating data structures or object state into a format that can be stored (for example, in a file or memory buffer, or transmitted across a network connection link) and reconstructed later in the same or another computer environment.
Python picked the name pickle because it modelled the process on how this was handled in Modula-3, where it was also called pickling. See Pickles: Why are they called that?
in python pickle refers to a module that provides (a specific) serialization of python objects.
serialization itself is a more general term. python objects can also be serialized into json for example.
https://en.wikipedia.org/wiki/Serialization

PyObjC: How can one use NSCoding to implement python pickling?

Title says it all. It seems like it ought be possible (somehow) to implement python-side pickling for PyObjC objects whose Objective-C classes implement NSCoding without re-implementing everything from scratch. That said, while value-semantic members would probably be straightforward, by-reference object graphs and conditional coding might be tricky. How might you get the two sides to "collaborate" on the object graph parts?
PyObjC does support writing Python objects to a (keyed) archive (that is, any object that can be pickled implements NSCoding).
That’s probably the easiest way to serialize arbitrary graphs of Python and Objective-C objects.
As I wrote in the comments for another answer I ran into problems when trying to find a way to implement pickle support for any object that implements NSCoding due to incompatibilities in how NSArchiver and pickle traverse the object graph (IIRC primarily when restoring the archive).
Shouldn't it be pretty straightforward?
On pickling, call encodeWithCoder on the object using an NSArchiver or something. Have pickle store that string.
On unpickling, use NSUnarchiver to create an NSObject from the pickled string.

Storing unpicklabe pygame.Surface objects in external files

So I've got a problem - I'm writing a game prototype in Python, using Pygame, and I want to save my games. All of the game-related data is in three instances of certain classes, and I want to save these three instances to a file. However, I've tried pickling these instances, and it doesn't work. Instead, I get "TypeError: can't pickle Surface objects". This is a problem, because I want to store Surface objects.
I'm open to any alternatives to pickling that there may be, using any other kind of data type. The important thing is that these instances get stored, and that their data is then retrievable later on. So what can I do to overcome this issue? Please keep in mind, I'm not a very experienced programmer, having learned Python in my spare time a year ago, and I can't write much of any other language, though I'm slowly learning C++.
The basic point of pickling is that you should be able to serialise the object somehow. An SDL surface is an in memory object holding a lot of run time state. Trying to serialise it is not totally sensible.
What you should do is to decouple the state of your game from the rendering components so that you can serialise just those (pickling or whatever).
It's like trying to save the state of a video by somehow saving the memory buffers holding the decoded video. This will not work. Instead, how you save it is to serialise the location of the video file and the time offset. Then you can continue playback upon load the next time you restore your application.
Reading http://docs.python.org/library/pickle.html#pickle-protocol, you need to either have Surface objects export a reduce method or use the copy_reg module to tell pickle how to handle that data as documented in http://docs.python.org/library/copy_reg.html#module-copy_reg.
Either way what pickle needs is a function that will turn a blob it can't handle into (some_class, [arguments here]). And then when you unpickle it will construct a new thing of that class with those arguments.

What does it mean for an object to be picklable (or pickle-able)?

Python docs mention this word a lot and I want to know what it means.
It simply means it can be serialized by the pickle module. For a basic explanation of this, see What can be pickled and unpickled?. Pickling Class Instances provides more details, and shows how classes can customize the process.
Things that are usually not pickable are, for example, sockets, file(handler)s, database connections, and so on. Everything that's build up (recursively) from basic python types (dicts, lists, primitives, objects, object references, even circular) can be pickled by default.
You can implement custom pickling code that will, for example, store the configuration of a database connection and restore it afterwards, but you will need special, custom logic for this.
All of this makes pickling a lot more powerful than xml, json and yaml (but definitely not as readable)
These are all great answers, but for anyone who's new to programming and still confused here's the simple answer:
Pickling an object is making it so you can store it as it currently is, long term (to often to hard disk). A bit like Saving in a video game.
So anything that's actively changing (like a live connection to a database) can't be stored directly (though you could probably figure out a way to store the information needed to create a new connection, and that you could pickle)
Bonus definition: Serializing is packaging it in a form that can be handed off to another program. Unserializing it is unpacking something you got sent so that you can use it
Pickling is the process in which the objects in python are converted into simple binary representation that can be used to write that object in a text file which can be stored. This is done to store the python objects and is also called as serialization. You can infer from this what de-serialization or unpickling means.
So when we say an object is picklable it means that the object can be serialized using the pickle module of python.

Categories

Resources