This question already has answers here:
Is it possible to dereference variable id's?
(4 answers)
Closed 3 years ago.
Let's say I have an id of a Python object, which I retrieved by doing id(thing). How do I find thing again by the id number I was given?
If the object is still there, this can be done by ctypes:
import ctypes
a = "hello world"
print ctypes.cast(id(a), ctypes.py_object).value
output:
hello world
If you don't know whether the object is still there, this is a recipe for undefined behavior and weird crashes or worse, so be careful.
You'll probably want to consider implementing it another way. Are you aware of the weakref module?
(Edited) The Python weakref module lets you keep references, dictionary references, and proxies to objects without having those references count in the reference counter. They're like symbolic links.
You can use the gc module to get all the objects currently tracked by the Python garbage collector.
import gc
def objects_by_id(id_):
for obj in gc.get_objects():
if id(obj) == id_:
return obj
raise Exception("No found")
Short answer, you can't.
Long answer, you can maintain a dict for mapping IDs to objects, or look the ID up by exhaustive search of gc.get_objects(), but this will create one of two problems: either the dict's reference will keep the object alive and prevent GC, or (if it's a WeakValue dict or you use gc.get_objects()) the ID may be deallocated and reused for a completely different object.
Basically, if you're trying to do this, you probably need to do something differently.
Just mentioning this module for completeness. This code by Bill Bumgarner includes a C extension to do what you want without looping throughout every object in existence.
The code for the function is quite straightforward. Every Python object is represented in C by a pointer to a PyObject struct. Because id(x) is just the memory address of this struct, we can retrieve the Python object just by treating x as a pointer to a PyObject, then calling Py_INCREF to tell the garbage collector that we're creating a new reference to the object.
static PyObject *
di_di(PyObject *self, PyObject *args)
{
PyObject *obj;
if (!PyArg_ParseTuple(args, "l:di", &obj))
return NULL;
Py_INCREF(obj);
return obj;
}
If the original object no longer exists then the result is undefined. It may crash, but it could also return a reference to a new object that's taken the location of the old one in memory.
eGenix mxTools library does provide such a function, although marked as "expert-only": mx.Tools.makeref(id)
This will do:
a = 0
id_a = id(a)
variables = {**locals(), **globals()}
for var in variables:
exec('var_id=id(%s)'%var)
if var_id == id_a:
exec('the_variable=%s'%var)
print(the_variable)
print(id(the_variable))
But I suggest implementing a more decent way.
Related
Suppose I have a Rust struct like this
struct X{...}
struct Y{
x:X
}
I'd like to be able to write python code that accesses X through Y
y = Y()
y.x.some_method()
What would be the best way to implement it in PyO3? Currently I made two wrapper classes
#[pyclass]
struct XWrapper{
x:X
}
#[pyclass]
struct YWrapper{
y:Y
}
#[pymethods]
impl YWrapper{
#[getter]
pub fn x(&self)->XWrapper{
XWrapper{x:self.y.clone()}
}
}
However, this requires clone(). I'd rather want to return reference. Of course I know that if X was a pyclass, then I could easily return PyRef to it. But the problem is that X and Y come from a Rust library and I cannot nilly-wily add #[pyclass] to them.
I don't think what you say is possible without some rejigging of the interface:
Your XWrapper owns the x and your Y owns its x as well. That means creating an XWrapper will always involve a clone (or a new).
Could we change XWrapper so that it merely contains a reference to an x? Not really, because that would require giving XWrapper a lifetime annotation, and PyO3 afaik doesn't allow pyclasses with lifetime annotation. Makes sense, because passing an object to python puts it on the python heap, at which point rust loses control over the object.
So what can we do?
Some thoughts: Do you really need to expose the composition structure of y to the python module? Just because that's the way it's organized within Rust doesn't mean it needs to be that way in Python. Your YWrapper could provide methods to the python interface that behind the scenes forward the request to the x instance:
#[pymethods]
impl YWrapper{
pub fn some_method(&self) {
self.y.x.some_method();
}
}
This would also be a welcome sight to strict adherents of the Law of Demeter ;)
I'm trying to think of other clever ways. Depending on some of the details of how y.x is accessed and modified by the methods of y itself, it might be possible to add a field x: XWrapper to the YWrapper. Then you create the XWrapper (including a clone of y.x) once when YWrapper is created, and from then on you can return references to that XWrapper in your pub fn x. Of course that becomes much more cumbersome when x gets frequently changed and updated via the methods of y...
In a way, this demonstrates the clash between Python's ref-counted object model and Rust's ownership object model. Rust enforces that you can't arbitrarily mess with objects unless you're their owner.
It is indeed possible to share objects and return them or mutate them. Whatever, just as in Python. What Lagerbaer suggested works and it's actually pretty ideal for small codes. However, if the number of methods increases there will be A LOT of repeating and boilerplate needed (and worse, folds every time you increase the depth of your nesting).
I have no idea if this is something that we are supposed to do. But from what I understood, the way to do it is using Py. God wish I had a habit of reading the docs thoroughly before experimenting.
In https://docs.rs/pyo3/latest/pyo3/#the-gil-independent-types in the MAIN PAGE of the doc says:
When wrapped in Py<...>, like with Py or Py, Python objects no longer have a limited lifetime which makes them easier to store in structs and pass between functions. However, you cannot do much with them without a Python<'py> token, for which you’d need to reacquire the GIL.
A Py is "A GIL-independent reference to an object allocated on the Python heap." https://docs.rs/pyo3/latest/pyo3/prelude/struct.Py.html
In other words, to return pyclass objects, we need to wrap it like Py<pyclass_struct_name>.
Your example is too complicated and to be honest I don't even understand what you are trying to do but here is an alternative version which suits my own usecase more closely. Since this is basically one of the only results that pops in Google I see it fit to paste it here even if it is not an exact response to the example provided above.
So here we go...
Suppose we have a Rust struct X and we cannot modify the lib as you mentioned. We need an XWrapper (let's call it PyX) pyclass to hold it.
So we define them here:
// in lib.rs
pub struct X {}
// in py_bindings.rs
#[pyclass]
struct PyX{
value: Py<X>,
}
impl_new_for!(PyX);
Then for the usage, all we have to do is to initialize the object with a GIL lock (assuming in the init of the XWrapper) and then define a getter for it. THE IMPORTANT NOTE HERE IS THAT YOU CALL clone_ref ON IT AND DO NOT RETURN THE OBJECT.
This is basically a nested class system afterwards and the nested object is immutable (has interior mutability tho) so it's a fantastic way to nest your code as well.
In the example below, I used my needed X as a PyX in yet another wrapper called Api.
#[pyclass]
struct Api {
x: PyX,
}
#[pymethods]
impl Api {
#[new]
fn __new__() -> PyResult<Self> {
Python::with_gil(|py| {
Ok(Self {
x: Py::new(
py,
PyX::new(),
),
})
}
}
#[getter(x)]
fn x(&mut self, py: Python) -> Py<Network> {
self.x.clone_ref(py)
}
}
Piggybacking on this question, say i have a container for a weakreference:
import weakref
class Foo(object):
a = lambda *_: None
def __init__(self, a):
self.a = weakref.ref(a, self._remove)
def _remove(self, *args):
self.__del__(self)
class Bar(object):
pass
>>> bar = Bar()
>>> foo = Foo(bar)
>>> del bar
>>> foo
<__main__.Foo object at 0x...>
I thought of storing the Foo instance in a static WeakKeyDictionary container, with the a attribute as a key, and using weakref.proxy of the instance everywhere--but that seems...inefficient. What's the best way to make it so that the Foo instance deletes itself when its reference to a dies?
You can't. I just spent some time digging through the Python source and ctypes documentation to ironically show how one might really delete (aka Py_DECREF until deallocated) an object until I gave up. The point is, you don't really want to do this. Python manages its own memory for a reason. Sure, it gives you access to things like weak references, but in no case will Python break a strong reference.
What you are proposing is to have an object reach into the environments of every bit of code loaded into the Python interpreter to rip out any references to itself. weakref has to rip out references too, but it only has to remove the references from the weakref object; it doesn't have to touch the object holding a reference to the weakref. To remove a reference in the way you propose would be at least invasive and most likely impossible.
To see why it would be impossible, consider how one might write a Python module in C that defines a type. Each instance of the object is going to hold some PyObject pointers to things it cares about. Some of these might be exposed to Python through properties, while others might remain internal. Suppose one of these internal references referenced one of your Foo objects. For it to 'delete' itself, it would have to reach into our C type and NULL out the reference. But to Python code, the C struct defining the object is opaque. If you dug into it with ctypes, you could inspect the bytes, but who's to know whether some sequence of bytes is a pointer to your object or an int that just happens to have the same value as the address of your object? You can't, at least without knowing implementation details of that type. And you can't handle every case, because someone can add another case just by importing another module written in C. You can't anticipate everything.
So what can you do? If you're deadset on doing something like this, you can mimic weakref's interface. Basically, make a new class that holds a reference to your class; to avoid ambiguity, I'll call this a fakeref. When it's called, it returns the instance of your class. Your class holds weak references1 to all of its fakerefs. Whenever your Foo class wants to delete itself, loop over the fakerefs, Noneing out the references to the Foo. Voilà; your class can 'delete' itself as desired and all of the fakerefs will now return None. But just as with weakrefs, storing the result of a call will make it a strong reference again, and your class will not be able to delete itself in the manner you desire.
All this said, I don't think you've presented a good enough case for why this is necessary. All you've said is that "there's no reason for it to stay in memory". Well, there is: it needs to be there for the objects that reference it. If, at some point in time, it becomes useless, then your objects shouldn't be holding a reference to it. When the objects referencing it don't care about it any more, they should remove those references. Then Python will clean it up with no further intervention on your part.
1 If you don't want to rely on weak references, your fakeref can implement __del__ and remove itself from your Foo instance it holds a reference to (if not None).
I am trying to write a wrapper to a native library in Linux. Problem is this:
definition in c:
int mymethod(mystruct* ptr)
in python:
_lib.mymethod.argtypes = (ctypes.POINTER(mystruct),)
_lib.mymethod.restype = ctypes.c_int
s = mystruct()
_lib.mymethod(ctypes.byref(s))
# raises: expected LP_mystruct instance instead of pointer to mystruct
_lib.mymethod(ctypes.pointer(s))
# raises expected LP_mystruct instance instead of LP_mystruct
errors. How to pass a structure as a pointer to a native method ?
Thanks.
Mete
The problem is that the higher level "POINTER" from ctypes is, in Python, a different object than "a generic pointer" (ctypes.CArgObject by ctypes.byref)which is returned or a single number representing a memory address (which is what is returned by ctype's adrresof) - you can either annotate your function to receive a `ctypes.c_voidp and call it with _lib.mymethod(ctypes.addressof(a)) instead -
Or if you want to work on the stronged-typed side to avoid errors that would crash Python (a type error raises a Python exception instead - a wrong parameter passed to a C unction would cause a segmentation fault on the Python interpreter itself), you have to create a variable to hold the new "type" which is a POINTER to your structure - and then create an instance of this type with the address of your structure:
mystruct_pointer = ctypes.POINTER(mystruct)
_lib.mymethod.argtypes = (mystruct_pointer,)
_lib.mymethod.restype = ctypes.c_int
s = mystruct()
_lib.mymethod(mystruct_pointer.from_address(ctypes.addressof(s)))
(I know that this is an old question, but I think the accepted answer is an unnecessary workaround, so I want to leave this here for posterity.)
Actually ctypes should explicitly support using byref() to pass a pointer like that:
ctypes exports the byref() function which is used to pass parameters by reference. The same effect can be achieved with the pointer() function, although pointer() does a lot more work since it constructs a real pointer object, so it is faster to use byref() if you don’t need the pointer object in Python itself.
The likely cause of this is that you have defined your struct in more than one place (e.g. in different modules) - if the argtypes assignment sees one definition and the function call sees the other, this confusing error arises. In other words, ctypes tries to match two mystruct types that are (probably) identical in contents, and have the exact same name, but they are not the same type. As long as the base struct type is a single type object, it doesn't matter if you construct a pointer to it using pointer(), byref() or POINTER()() - ctypes will detect that the underlying (pointed-to) type is the same.
To verify if this is the case, try assert(_lib.mymethod.argtypes[0]._type_ == type(s)) right before calling the external function.
I've read the documentation for the Python C-API, and even written a few extension modules. However, I'm still a bit unclear on the exact semantics when it comes to returning Python objects from a C function.
The limited examples in the Python docs usually show a C function which returns the result of Py_BuildValue. Now, Py_BuildValue returns a New Reference, and transfers ownership of this reference over to the interpreter. So, can I extrapolate from this that it is a general rule that any object returned to Python must be a new reference, and that returning an object from a C function is the same as transferring ownership of the object over to the interpreter?
If so, what about cases where you return an object that is already owned by something? For example, suppose you write a C function which takes in a PyObject* which is a tuple, and you call PyTuple_GetItem on it and return the result. PyTuple_GetItem returns a borrowed reference - meaning that the item is still "owned" by the tuple. So, does a C function which returns the result of something like PyTuple_GetItem have to INCREF the result before returning it to the interpreter?
For example:
static PyObject* my_extension_module(PyObject* tup)
{
PyObject* item = PyTuple_GetItem(tup, 1);
if (!item) { /* handle error */ }
return item; // <--- DO WE NEED TO INCREF BEFORE RETURNING HERE?
}
Python expects any function you expose to it to return a new reference, yes. If you only have a borrowed reference, you have to call Py_INCREF to give a new reference. If you return a borrowed reference, Python will proceed to call Py_DECREF (when it's done with the reference), and eventually that will cause the object to be freed while it is still in use.
(It's not uncommon for that "eventual" freeing to happen during interpreter exit, in which case it may go unnoticed, but it's still a mistake to return borrowed references.)
I have some global variables in a Python script. Some functions in that script call into C - is it possible to set one of those variables while in C and if so, how?
I appreciate that this isn't a very nice design in the first place, but I need to make a small change to existing code, I don't want to embark on major refactoring of existing scripts.
I'm not a python guru, but I found this question interesting so I googled around. This was the first hit on "python embedding API" - does it help?
If the attributes belong to the global
scope of a module, then you can use
"PyImport_AddModule" to get a handle
to the module object. For example, if
you wanted to get the value of an
integer in the main module named
"foobar", you would do the following:
PyObject *m = PyImport_AddModule("__main__");
PyObject *v = PyObject_GetAttrString(m,"foobar");
int foobar = PyInt_AsLong(v);
Py_DECREF(v);
For anyone coming here from Google, here's the direct method:
PyObject* PyEval_GetGlobals()
https://docs.python.org/2/c-api/reflection.html
https://docs.python.org/3/c-api/reflection.html
The return value is accessed as a dictionary.
I recommend using pyrex to make an extension module you can store the values in in python, and cdef a bunch of functions which can be called from C to return the values there.
Otherwise, much depends on the type of values you're trying to transmit.
Are you willing to modify the API a little bit?
You can make the C function return the new value for the global, and then call it like this:
my_global = my_c_func(...)
If you're using Robin or the Python C API directly, you can pass the globals dictionary as an additional parameter and modify it
If your global is always in the same module, Sherm's solution looks great