We are able to defeat the small integer intern in this way (a calculation allows us to avoid the caching layer):
>>> n = 674039
>>> one1 = 1
>>> one2 = (n ** 9 + 1) % (n ** 9)
>>> one1 == one2
True
>>> one1 is one2
False
How can you defeat the small string intern, i.e. to see the following result:
>>> one1 = "1"
>>> one2 = <???>
>>> type(one2) is str and one1 == one2
True
>>> one1 is one2
False
sys.intern mentions that "Interned strings are not immortal", but there's no context about how a string could kicked out of the intern, or how you can create a str instance avoiding the caching layer.
Since interning is CPython implementation detail, answers relying on undocumented implementation details are ok/expected.
Unicode consisting of only one character (with value smaller than 128 or more precisely from latin1) is the most complicated case, because those strings aren't really interned but (more similar to the integer pool or identically to the behavior for bytes) are created at the start and are stored in an array as long as the interpreter is alive:
truct _Py_unicode_state {
...
/* Single character Unicode strings in the Latin-1 range are being
shared as well. */
PyObject *latin1[256];
...
/* This dictionary holds all interned unicode strings...
*/
PyObject *interned;
...
};
So every time a length 1 unicode is created, the character value gets looked up if it is in the latin1-array. E.g. in unicode_decode_utf8:
/* ASCII is equivalent to the first 128 ordinals in Unicode. */
if (size == 1 && (unsigned char)s[0] < 128) {
if (consumed) {
*consumed = 1;
}
return get_latin1_char((unsigned char)s[0]);
}
One could even argue, if there is a way to circumvent this in the interpreter - we speak about a (performance-) bug.
A possibility is to populate the unicode-data by ourselves using C-API. I use Cython for the proof of concept, but also ctypes could be used to the same effect:
%%cython
cdef extern from *:
"""
PyObject* create_new_unicode(char *ch)
{
PyUnicodeObject *ob = (PyUnicodeObject *)PyUnicode_New(1, 127);
Py_UCS1 *data = PyUnicode_1BYTE_DATA(ob);
data[0]=ch[0]; //fill data without using the unicode_decode_utf8
return (PyObject*)ob;
}
"""
object create_new_unicode(char *ch)
def gen1():
return create_new_unicode(b"1")
Noteworthy details:
PyUnicode_New would not look up in latin1, because the characters aren't set yet.
For simplicity, the above works only for ASCII characters - thus we pass 127 as maxchar to PyUnicode_New. As result, we can interpret data via PyUnicode_1BYTE_DATA which makes it easy to manipulate it without much ado manually.
And now:
a,b=gen1(), gen1()
a is b, a == b
# yields (False, True)
as wanted.
Here is a similar idea, but implemented with ctypes:
from ctypes import POINTER, py_object, c_ssize_t, byref, pythonapi
PyUnicode_New = pythonapi.PyUnicode_New
PyUnicode_New.argtypes = (c_ssize_t, c_ssize_t)
PyUnicode_New.restype = py_object
PyUnicode_CopyCharacters = pythonapi._PyUnicode_FastCopyCharacters
PyUnicode_CopyCharacters.argtypes = (py_object, c_ssize_t, py_object, c_ssize_t, c_ssize_t)
PyUnicode_CopyCharacters.restype = c_ssize_t
def clone(orig):
cloned = PyUnicode_New(1,127)
PyUnicode_CopyCharacters(cloned, 0, orig, 0, 1)
return cloned
Noteworthy details:
It is not possible to use PyUnicode_1BYTE_DATA with ctypes, because it is a macro. An alternative would be to calculate the offset to data-member and directly access this memory (but it depends on the platform and doesn't feel very portable)
As work-around, PyUnicode_CopyCharacters is used (there are probably also other possibilities to achieve the same), which is more abstract and portable than directly calculating/accessing the memory.
Actually, _PyUnicode_FastCopyCharacters is used, because PyUnicode_CopyCharacters would check, that the target-unicode has multiple references and throw. _PyUnicode_FastCopyCharacters doesn't perform those checks and does as asked.
And now:
a="1"
b=clone(a)
a is b, a==b
# yields (False, True)
For strings longer than 1 character, it is a lot easier to avoid interning, e.g.:
a="12"
b="123"[0:2]
a is b, a == b
#yields (False, True)
How do I get the size occupied in memory by an object in Python?
Just use the sys.getsizeof function defined in the sys module.
sys.getsizeof(object[, default]):
Return the size of an object in bytes.
The object can be any type of object.
All built-in objects will return
correct results, but this does not
have to hold true for third-party
extensions as it is implementation
specific.
Only the memory consumption directly attributed to the object is
accounted for, not the memory consumption of objects it refers to.
The default argument allows to define
a value which will be returned if the
object type does not provide means to
retrieve the size and would cause a
TypeError.
getsizeof calls the object’s
__sizeof__ method and adds an additional garbage collector overhead
if the object is managed by the
garbage collector.
See recursive sizeof recipe for an example of using getsizeof() recursively to find the size of containers and all their contents.
Usage example, in python 3.0:
>>> import sys
>>> x = 2
>>> sys.getsizeof(x)
24
>>> sys.getsizeof(sys.getsizeof)
32
>>> sys.getsizeof('this')
38
>>> sys.getsizeof('this also')
48
If you are in python < 2.6 and don't have sys.getsizeof you can use this extensive module instead. Never used it though.
How do I determine the size of an object in Python?
The answer, "Just use sys.getsizeof", is not a complete answer.
That answer does work for builtin objects directly, but it does not account for what those objects may contain, specifically, what types, such as custom objects, tuples, lists, dicts, and sets contain. They can contain instances each other, as well as numbers, strings and other objects.
A More Complete Answer
Using 64-bit Python 3.6 from the Anaconda distribution, with sys.getsizeof, I have determined the minimum size of the following objects, and note that sets and dicts preallocate space so empty ones don't grow again until after a set amount (which may vary by implementation of the language):
Python 3:
Empty
Bytes type scaling notes
28 int +4 bytes about every 30 powers of 2
37 bytes +1 byte per additional byte
49 str +1-4 per additional character (depending on max width)
48 tuple +8 per additional item
64 list +8 for each additional
224 set 5th increases to 736; 21nd, 2272; 85th, 8416; 341, 32992
240 dict 6th increases to 368; 22nd, 1184; 43rd, 2280; 86th, 4704; 171st, 9320
136 func def does not include default args and other attrs
1056 class def no slots
56 class inst has a __dict__ attr, same scaling as dict above
888 class def with slots
16 __slots__ seems to store in mutable tuple-like structure
first slot grows to 48, and so on.
How do you interpret this? Well say you have a set with 10 items in it. If each item is 100 bytes each, how big is the whole data structure? The set is 736 itself because it has sized up one time to 736 bytes. Then you add the size of the items, so that's 1736 bytes in total
Some caveats for function and class definitions:
Note each class definition has a proxy __dict__ (48 bytes) structure for class attrs. Each slot has a descriptor (like a property) in the class definition.
Slotted instances start out with 48 bytes on their first element, and increase by 8 each additional. Only empty slotted objects have 16 bytes, and an instance with no data makes very little sense.
Also, each function definition has code objects, maybe docstrings, and other possible attributes, even a __dict__.
Also note that we use sys.getsizeof() because we care about the marginal space usage, which includes the garbage collection overhead for the object, from the docs:
getsizeof() calls the object’s __sizeof__ method and adds an
additional garbage collector overhead if the object is managed by the
garbage collector.
Also note that resizing lists (e.g. repetitively appending to them) causes them to preallocate space, similarly to sets and dicts. From the listobj.c source code:
/* This over-allocates proportional to the list size, making room
* for additional growth. The over-allocation is mild, but is
* enough to give linear-time amortized behavior over a long
* sequence of appends() in the presence of a poorly-performing
* system realloc().
* The growth pattern is: 0, 4, 8, 16, 25, 35, 46, 58, 72, 88, ...
* Note: new_allocated won't overflow because the largest possible value
* is PY_SSIZE_T_MAX * (9 / 8) + 6 which always fits in a size_t.
*/
new_allocated = (size_t)newsize + (newsize >> 3) + (newsize < 9 ? 3 : 6);
Historical data
Python 2.7 analysis, confirmed with guppy.hpy and sys.getsizeof:
Bytes type empty + scaling notes
24 int NA
28 long NA
37 str + 1 byte per additional character
52 unicode + 4 bytes per additional character
56 tuple + 8 bytes per additional item
72 list + 32 for first, 8 for each additional
232 set sixth item increases to 744; 22nd, 2280; 86th, 8424
280 dict sixth item increases to 1048; 22nd, 3352; 86th, 12568 *
120 func def does not include default args and other attrs
64 class inst has a __dict__ attr, same scaling as dict above
16 __slots__ class with slots has no dict, seems to store in
mutable tuple-like structure.
904 class def has a proxy __dict__ structure for class attrs
104 old class makes sense, less stuff, has real dict though.
Note that dictionaries (but not sets) got a more
compact representation in Python 3.6
I think 8 bytes per additional item to reference makes a lot of sense on a 64 bit machine. Those 8 bytes point to the place in memory the contained item is at. The 4 bytes are fixed width for unicode in Python 2, if I recall correctly, but in Python 3, str becomes a unicode of width equal to the max width of the characters.
And for more on slots, see this answer.
A More Complete Function
We want a function that searches the elements in lists, tuples, sets, dicts, obj.__dict__'s, and obj.__slots__, as well as other things we may not have yet thought of.
We want to rely on gc.get_referents to do this search because it works at the C level (making it very fast). The downside is that get_referents can return redundant members, so we need to ensure we don't double count.
Classes, modules, and functions are singletons - they exist one time in memory. We're not so interested in their size, as there's not much we can do about them - they're a part of the program. So we'll avoid counting them if they happen to be referenced.
We're going to use a blacklist of types so we don't include the entire program in our size count.
import sys
from types import ModuleType, FunctionType
from gc import get_referents
# Custom objects know their class.
# Function objects seem to know way too much, including modules.
# Exclude modules as well.
BLACKLIST = type, ModuleType, FunctionType
def getsize(obj):
"""sum size of object & members."""
if isinstance(obj, BLACKLIST):
raise TypeError('getsize() does not take argument of type: '+ str(type(obj)))
seen_ids = set()
size = 0
objects = [obj]
while objects:
need_referents = []
for obj in objects:
if not isinstance(obj, BLACKLIST) and id(obj) not in seen_ids:
seen_ids.add(id(obj))
size += sys.getsizeof(obj)
need_referents.append(obj)
objects = get_referents(*need_referents)
return size
To contrast this with the following whitelisted function, most objects know how to traverse themselves for the purposes of garbage collection (which is approximately what we're looking for when we want to know how expensive in memory certain objects are. This functionality is used by gc.get_referents.) However, this measure is going to be much more expansive in scope than we intended if we are not careful.
For example, functions know quite a lot about the modules they are created in.
Another point of contrast is that strings that are keys in dictionaries are usually interned so they are not duplicated. Checking for id(key) will also allow us to avoid counting duplicates, which we do in the next section. The blacklist solution skips counting keys that are strings altogether.
Whitelisted Types, Recursive visitor
To cover most of these types myself, instead of relying on the gc module, I wrote this recursive function to try to estimate the size of most Python objects, including most builtins, types in the collections module, and custom types (slotted and otherwise).
This sort of function gives much more fine-grained control over the types we're going to count for memory usage, but has the danger of leaving important types out:
import sys
from numbers import Number
from collections import deque
from collections.abc import Set, Mapping
ZERO_DEPTH_BASES = (str, bytes, Number, range, bytearray)
def getsize(obj_0):
"""Recursively iterate to sum size of object & members."""
_seen_ids = set()
def inner(obj):
obj_id = id(obj)
if obj_id in _seen_ids:
return 0
_seen_ids.add(obj_id)
size = sys.getsizeof(obj)
if isinstance(obj, ZERO_DEPTH_BASES):
pass # bypass remaining control flow and return
elif isinstance(obj, (tuple, list, Set, deque)):
size += sum(inner(i) for i in obj)
elif isinstance(obj, Mapping) or hasattr(obj, 'items'):
size += sum(inner(k) + inner(v) for k, v in getattr(obj, 'items')())
# Check for custom object instances - may subclass above too
if hasattr(obj, '__dict__'):
size += inner(vars(obj))
if hasattr(obj, '__slots__'): # can have __slots__ with __dict__
size += sum(inner(getattr(obj, s)) for s in obj.__slots__ if hasattr(obj, s))
return size
return inner(obj_0)
And I tested it rather casually (I should unittest it):
>>> getsize(['a', tuple('bcd'), Foo()])
344
>>> getsize(Foo())
16
>>> getsize(tuple('bcd'))
194
>>> getsize(['a', tuple('bcd'), Foo(), {'foo': 'bar', 'baz': 'bar'}])
752
>>> getsize({'foo': 'bar', 'baz': 'bar'})
400
>>> getsize({})
280
>>> getsize({'foo':'bar'})
360
>>> getsize('foo')
40
>>> class Bar():
... def baz():
... pass
>>> getsize(Bar())
352
>>> getsize(Bar().__dict__)
280
>>> sys.getsizeof(Bar())
72
>>> getsize(Bar.__dict__)
872
>>> sys.getsizeof(Bar.__dict__)
280
This implementation breaks down on class definitions and function definitions because we don't go after all of their attributes, but since they should only exist once in memory for the process, their size really doesn't matter too much.
The Pympler package's asizeof module can do this.
Use as follows:
from pympler import asizeof
asizeof.asizeof(my_object)
Unlike sys.getsizeof, it works for your self-created objects. It even works with numpy.
>>> asizeof.asizeof(tuple('bcd'))
200
>>> asizeof.asizeof({'foo': 'bar', 'baz': 'bar'})
400
>>> asizeof.asizeof({})
280
>>> asizeof.asizeof({'foo':'bar'})
360
>>> asizeof.asizeof('foo')
40
>>> asizeof.asizeof(Bar())
352
>>> asizeof.asizeof(Bar().__dict__)
280
>>> A = rand(10)
>>> B = rand(10000)
>>> asizeof.asizeof(A)
176
>>> asizeof.asizeof(B)
80096
As mentioned,
The (byte)code size of objects like classes, functions, methods, modules, etc. can be included by setting option code=True.
And if you need other view on live data, Pympler's
module muppy is used for on-line monitoring of a Python application
and module Class Tracker provides off-line analysis of the lifetime of
selected Python objects.
For numpy arrays, getsizeof doesn't work - for me it always returns 40 for some reason:
from pylab import *
from sys import getsizeof
A = rand(10)
B = rand(10000)
Then (in ipython):
In [64]: getsizeof(A)
Out[64]: 40
In [65]: getsizeof(B)
Out[65]: 40
Happily, though:
In [66]: A.nbytes
Out[66]: 80
In [67]: B.nbytes
Out[67]: 80000
You can serialize the object to derive a measure that is closely related to the size of the object:
import pickle
## let o be the object whose size you want to measure
size_estimate = len(pickle.dumps(o))
If you want to measure objects that cannot be pickled (e.g. because of lambda expressions) dill or cloudpickle can be a solution.
Use sys.getsizeof() if you DON'T want to include sizes of linked (nested) objects.
However, if you want to count sub-objects nested in lists, dicts, sets, tuples - and usually THIS is what you're looking for - use the recursive deep sizeof() function as shown below:
import sys
def sizeof(obj):
size = sys.getsizeof(obj)
if isinstance(obj, dict): return size + sum(map(sizeof, obj.keys())) + sum(map(sizeof, obj.values()))
if isinstance(obj, (list, tuple, set, frozenset)): return size + sum(map(sizeof, obj))
return size
You can also find this function in the nifty toolbox, together with many other useful one-liners:
https://github.com/mwojnars/nifty/blob/master/util.py
Python 3.8 (Q1 2019) will change some of the results of sys.getsizeof, as announced here by Raymond Hettinger:
Python containers are 8 bytes smaller on 64-bit builds.
tuple () 48 -> 40
list [] 64 ->56
set() 224 -> 216
dict {} 240 -> 232
This comes after issue 33597 and Inada Naoki (methane)'s work around Compact PyGC_Head, and PR 7043
This idea reduces PyGC_Head size to two words.
Currently, PyGC_Head takes three words; gc_prev, gc_next, and gc_refcnt.
gc_refcnt is used when collecting, for trial deletion.
gc_prev is used for tracking and untracking.
So if we can avoid tracking/untracking while trial deletion, gc_prev and gc_refcnt can share same memory space.
See commit d5c875b:
Removed one Py_ssize_t member from PyGC_Head.
All GC tracked objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes.
This can be more complicated than it looks depending on how you want to count things. For instance, if you have a list of ints, do you want the size of the list containing the references to the ints? (i.e. - list only, not what is contained in it), or do you want to include the actual data pointed to, in which case you need to deal with duplicate references, and how to prevent double-counting when two objects contain references to the same object.
You may want to take a look at one of the python memory profilers, such as pysizer to see if they meet your needs.
Having run into this problem many times myself, I wrote up a small function (inspired by #aaron-hall's answer) & tests that does what I would have expected sys.getsizeof to do:
https://github.com/bosswissam/pysize
If you're interested in the backstory, here it is
EDIT: Attaching the code below for easy reference. To see the most up-to-date code, please check the github link.
import sys
def get_size(obj, seen=None):
"""Recursively finds size of objects"""
size = sys.getsizeof(obj)
if seen is None:
seen = set()
obj_id = id(obj)
if obj_id in seen:
return 0
# Important mark as seen *before* entering recursion to gracefully handle
# self-referential objects
seen.add(obj_id)
if isinstance(obj, dict):
size += sum([get_size(v, seen) for v in obj.values()])
size += sum([get_size(k, seen) for k in obj.keys()])
elif hasattr(obj, '__dict__'):
size += get_size(obj.__dict__, seen)
elif hasattr(obj, '__iter__') and not isinstance(obj, (str, bytes, bytearray)):
size += sum([get_size(i, seen) for i in obj])
return size
Here is a quick script I wrote based on the previous answers to list sizes of all variables
for i in dir():
print (i, sys.getsizeof(eval(i)) )
Use following function to get actual size of a python object:
import sys
import gc
def actualsize(input_obj):
memory_size = 0
ids = set()
objects = [input_obj]
while objects:
new = []
for obj in objects:
if id(obj) not in ids:
ids.add(id(obj))
memory_size += sys.getsizeof(obj)
new.append(obj)
objects = gc.get_referents(*new)
return memory_size
actualsize([1, 2, [3, 4, 5, 1]])
Reference: https://towardsdatascience.com/the-strange-size-of-python-objects-in-memory-ce87bdfbb97f
If you don't need the exact size of the object but roughly to know how big it is, one quick (and dirty) way is to let the program run, sleep for an extended period of time, and check the memory usage (ex: Mac's activity monitor) by this particular python process. This would be effective when you are trying to find the size of one single large object in a python process. For example, I recently wanted to check the memory usage of a new data structure and compare it with that of Python's set data structure. First I wrote the elements (words from a large public domain book) to a set, then checked the size of the process, and then did the same thing with the other data structure. I found out the Python process with a set is taking twice as much memory as the new data structure. Again, you wouldn't be able to exactly say the memory used by the process is equal to the size of the object. As the size of the object gets large, this becomes close as the memory consumed by the rest of the process becomes negligible compared to the size of the object you are trying to monitor.
If performance is not an Issue, the easiest solution is to pickle and measure:
import pickle
data = ...
len(pickle.dumps(data))
I use this trick... May won't be accurate on small objects, but I think it's much more accurate for a complex object (like pygame surface) rather than sys.getsizeof()
import pygame as pg
import os
import psutil
import time
process = psutil.Process(os.getpid())
pg.init()
vocab = ['hello', 'me', 'you', 'she', 'he', 'they', 'we',
'should', 'why?', 'necessarily', 'do', 'that']
font = pg.font.SysFont("monospace", 100, True)
dct = {}
newMem = process.memory_info().rss # don't mind this line
Str = f'store ' + f'Nothing \tsurface use about '.expandtabs(15) + \
f'0\t bytes'.expandtabs(9) # don't mind this assignment too
usedMem = process.memory_info().rss
for word in vocab:
dct[word] = font.render(word, True, pg.Color("#000000"))
time.sleep(0.1) # wait a moment
# get total used memory of this script:
newMem = process.memory_info().rss
Str = f'store ' + f'{word}\tsurface use about '.expandtabs(15) + \
f'{newMem - usedMem}\t bytes'.expandtabs(9)
print(Str)
usedMem = newMem
On my windows 10, python 3.7.3, the output is:
store hello surface use about 225280 bytes
store me surface use about 61440 bytes
store you surface use about 94208 bytes
store she surface use about 81920 bytes
store he surface use about 53248 bytes
store they surface use about 114688 bytes
store we surface use about 57344 bytes
store should surface use about 172032 bytes
store why? surface use about 110592 bytes
store necessarily surface use about 311296 bytes
store do surface use about 57344 bytes
store that surface use about 110592 bytes
This might not be the most relevant answer, but I was interested only in object storage and retrieval. So dumping the object as pickle and checking the pickle's size was sufficient
You can make use of getSizeof() as mentioned below to determine the size of an object
import sys
str1 = "one"
int_element=5
print("Memory size of '"+str1+"' = "+str(sys.getsizeof(str1))+ " bytes")
print("Memory size of '"+ str(int_element)+"' = "+str(sys.getsizeof(int_element))+ " bytes")
From this link: How do I pass a variable by reference?, we know, Python will copy a string (an immutable type variable) when it is passed to a function as a parameter, but I think it will waste memory if the string is huge. In many cases, we need to use functions to wrap some operations for strings, so I want to know how to do it more effective?
Python does not make copies of objects (this includes strings) passed to functions:
>>> def foo(s):
... return id(s)
...
>>> x = 'blah'
>>> id(x) == foo(x)
True
If you need to "modify" a string in a function, return the new string and assign it back to the original name:
>>> def bar(s):
... return s + '!'
...
>>> x = 'blah'
>>> x = bar(x)
>>> x
'blah!'
Unfortunately, this can be very inefficient when making small changes to large strings because the large string gets copied. The pythonic way of dealing with this is to hold strings in an list and join them together once you have all the pieces.
Python does pass a string by reference. Notice that two strings with the same content are considered identical:
a = 'hello'
b = 'hello'
a is b # True
Since when b is assigned by a value, and the value already exists in memory, it uses the same reference of the string. Notice another fact, that if the string was dynamically created, meaning being created with string operations (i.e concatenation), the new variable will reference a new instance of the same string:
c = 'hello'
d = 'he'
d += 'llo'
c is d # False
That being said, creating a new string will allocate a new string in memory and returning a reference for the new string, but using a currently created string will reuse the same string instance. Therefore, passing a string as a function parameter will pass it by reference, or in other words, will pass the address in memory of the string.
And now to the point you were looking for- if you change the string inside the function, the string outside of the function will remain the same, and that stems from string immutability. Changing a string means allocating a new string in memory.
a = 'a'
b = a # b will hold a reference to string a
a += 'a'
a is b # False
Bottom line:
You cannot really change a string. The same as for maybe every other programming language (but don't quote me).
When you pass the string as an argument, you pass a reference. When you change it's value, you change the variable to point to another place in memory. But when you change a variable's reference, other variables that points to the same address will naturally keep the old value (reference) they held.
Wish the explanation was clear enough
In [7]: strs="abcd"
In [8]: id(strs)
Out[8]: 164698208
In [9]: def func(x):
print id(x)
x=x.lower() #perform some operation on string object, it returns a new object
print id(x)
...:
In [10]: func(strs)
164698208 # same as strs, i.e it actually passes the same object
164679776 # new object is returned if we perform an operation
# That's why they are called immutable
But operations on strings always return a new string object.
def modify_string( t ):
the_string = t[0]
# do stuff
modify_string( ["my very long string"] )
If you want to potentially change the value of something passed in, wrap it in a dict or a list:
This doesn't change s
def x(s):
s += 1
This does change s:
def x(s):
s[0] += 1
This is the only way to "pass by reference".
wrapping the string into a class will make it pass by reference:
class refstr:
"wrap string in object, so it is passed by reference rather than by value"
def __init__(self,s=""):
self.s=s
def __add__(self,s):
self.s+=s
return self
def __str__(self):
return self.s
def fn(s):
s+=" world"
s=refstr("hello")
fn(s) # s gets modified because objects are passed by reference
print(s) #returns 'hello world'
Just pass it in as you would any other parameter. The contents won't get copied, only the reference will.
I'm trying to figure out why this works after lots and lots of messing about with
obo.librar_version is a c function which requires char ** as the input and does a strcpy
to passed in char.
from ctypes import *
_OBO_C_DLL = 'obo.dll'
STRING = c_char_p
OBO_VERSION = _stdcall_libraries[_OBO_C_DLL].OBO_VERSION
OBO_VERSION.restype = c_int
OBO_VERSION.argtypes = [POINTER(STRING)]
def library_version():
s = create_string_buffer('\000' * 32)
t = cast(s, c_char_p)
res = obo.library_version(byref(t))
if res != 0:
raise Error("OBO error %r" % res)
return t.value, s.raw, s.value
library_version()
The above code returns
('OBO Version 1.0.1', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', '')
What I don't understand is why 's' does not have any value? Anyone have any ideas? Thx
When you cast s to c_char_p you store a new object in t, not a reference. So when you pass t to your function by reference, s doesn't get updated.
UPDATE:
You are indeed correct:
cast takes two parameters, a ctypes
object that is or can be converted to
a pointer of some kind, and a ctypes
pointer type. It returns an instance
of the second argument, which
references the same memory block as
the first argument.
In order to get a reference to your string buffer, you need to use the following for your cast:
t = cast(s, POINTER(c_char*33))
I have no idea why c_char_p doesn't create a reference where this does, but there you go.
Because library_version requires a char**, they don't want you to allocate the characters (as you're doing with create_string_buffer. Instead, they just want you to pass in a reference to a pointer so they can return the address of where to find the version string.
So all you need to do is allocate the pointer, and then pass in a reference to that pointer.
The following code should work, although I don't have obo.dll (or know of a suitable replacement) to test it.
from ctypes import *
_OBO_C_DLL = 'obo.dll'
STRING = c_char_p
_stdcall_libraries = dict()
_stdcall_libraries[_OBO_C_DLL] = WinDLL(_OBO_C_DLL)
OBO_VERSION = _stdcall_libraries[_OBO_C_DLL].OBO_VERSION
OBO_VERSION.restype = c_int
OBO_VERSION.argtypes = [POINTER(STRING)]
def library_version():
s_res = c_char_p()
res = OBO_VERSION(byref(s_res))
if res != 0:
raise Error("OBO error %r" % res)
return s_res.value
library_version()
[Edit]
I've gone a step further and written my own DLL that implements a possible implementation of OBO_VERSION that does not require an allocated character buffer, and is not subject to any memory leaks.
int OBO_VERSION(char **pp_version)
{
static char result[] = "Version 2.0";
*pp_version = result;
return 0; // success
}
As you can see, OBO_VERSION simply sets the value of *pp_version to a pointer to a null-terminated character array. This is likely how the real OBO_VERSION works. I've tested this against my originally suggested technique above, and it works as prescribed.