Develop Reference

What is the most efficient data structure to build a large word-to-index-to-word dictionary?

I would like to index a very large number of strings (mapping each string to an numeric value) but also be able to retrieve each string from its numeric index.
Using hash tables or python dict is not an option because of memory issues so I decided to use a radix trie to store the strings, I can retrieve the index of any string very quickly and handle a very large number of strings.
My problem is that I also need to retrieve the strings from their numeric index, and if I maintain a "reverse index" list [string1, string2, ..., stringn] I'll loose the memory benefit of the Trie.
I thought maybe the "reverse index" could be a list of pointers to the last node of a kind-of Trie structure but first, there are no pointers in python, and second I'm not sure I can have a "node-level" access to the Trie structure I'm currently using.
Does this kind of data-structure already exists? And if not how would you do this in python?

As per What data structure to use to have O(log n) key AND value lookup? , you need two synchronized data structures for key and value lookups, each holding references to the other's leaf nodes.
The structure for the ID lookup can be anything with sufficient efficientcy -- a balanced tree, a hash table, another trie.
To be able to extract the value from a leaf node reference, a trie needs to allow 1) leaf node references themselves (not necessarily a real Python reference, anything that its API can use); 2) walking up the trie to extract the word from that reference.
Note that a reference is effectively a unique integer so if your IDs are not larger than an integer, it makes sense to reuse something as IDs -- e.g. the trie node references themselves. Then if the trie API can validate such a reference (i.e. tell if it has a used node with such a reference) this will act as the ID lookup and you don't need the 2nd structure at all! This way, the IDs will be non-persistent though 'cuz reference values (effectively memory addresses) change between processes and runs.

I'm answering to myself because I finally end up creating my own data-structure which is perfectly suited for the word-to-index-to-word problem I had, using only python3 built-in functions.
I tried to make it clean and efficient but there's obviously room for improvement and a C binding would be better.
So the final result is a indexedtrie class that looks like a python dict (or defaultdict if you invoke it with a default_factory parameter) but can also be queried like a list because a kind of "reversed index" is automatically maintained.
The keys, which are stored in an internal radix trie, can be any subscriptable object (bytes, strings, tuples, lists) and the values you want to store anything you want inside.
Also the indextrie class is pickable, and you can benefit from the advantages of radix tries regarding "prefix search" and this kind of things!
Each key in the trie is associated with a unique integer index, you can retrieve the key with the index or the index with the key and the whole thing is fast and memory safe so I personally think that's one of the best data-structure in the world and that it should be integrated in python standard library :).
Enough talking, here is the code, feel free to adapt and use it:
"""
A Python3 indexed trie class.
An indexed trie's key can be any subscriptable object.
Keys of the indexed trie are stored using a "radix trie", a space-optimized data-structure which has many advantages (see https://en.wikipedia.org/wiki/Radix_tree).
Also, each key in the indexed trie is associated to a unique index which is build dynamically.
Indexed trie is used like a python dictionary (and even a collections.defaultdict if you want to) but its values can also be accessed or updated (but not created) like a list!
Example:
>>> t = indextrie()
>>> t["abc"] = "hello"
>>> t[0]
'hello'
>>> t["abc"]
'hello'
>>> t.index2key(0)
'abc'
>>> t.key2index("abc")
0
>>> t[:]
[0]
>>> print(t)
{(0, 'abc'): hello}
"""
__author__ = "#fbparis"
_SENTINEL = object()
class _Node(object):
"""
A single node in the trie.
"""
__slots__ = "_children", "_parent", "_index", "_key"
def __init__(self, key, parent, index=None):
self._children = set()
self._key = key
self._parent = parent
self._index = index
self._parent._children.add(self)
class IndexedtrieKey(object):
"""
A pair (index, key) acting as an indexedtrie's key
"""
__slots__ = "index", "key"
def __init__(self, index, key):
self.index = index
self.key = key
def __repr__(self):
return "(%d, %s)" % (self.index, self.key)
class indexedtrie(object):
"""
The indexed trie data-structure.
"""
__slots__ = "_children", "_indexes", "_values", "_nodescount", "_default_factory"
def __init__(self, items=None, default_factory=_SENTINEL):
"""
A list of items can be passed to initialize the indexed trie.
"""
self._children = set()
self.setdefault(default_factory)
self._indexes = []
self._values = []
self._nodescount = 0 # keeping track of nodes count is purely informational
if items is not None:
for k, v in items:
if isinstance(k, IndexedtrieKey):
self.__setitem__(k.key, v)
else:
self.__setitem__(k, v)
#classmethod
def fromkeys(cls, keys, value=_SENTINEL, default_factory=_SENTINEL):
"""
Build a new indexedtrie from a list of keys.
"""
obj = cls(default_factory=default_factory)
for key in keys:
if value is _SENTINEL:
if default_factory is not _SENTINEL:
obj[key] = obj._default_factory()
else:
obj[key] = None
else:
obj[key] = value
return obj
#classmethod
def fromsplit(cls, keys, value=_SENTINEL, default_factory=_SENTINEL):
"""
Build a new indexedtrie from a splitable object.
"""
obj = cls(default_factory=default_factory)
for key in keys.split():
if value is _SENTINEL:
if default_factory is not _SENTINEL:
obj[key] = obj._default_factory()
else:
obj[key] = None
else:
obj[key] = value
return obj
def setdefault(self, factory=_SENTINEL):
"""
"""
if factory is not _SENTINEL:
# indexed trie will act like a collections.defaultdict except in some cases because the __missing__
# method is not implemented here (on purpose).
# That means that simple lookups on a non existing key will return a default value without adding
# the key, which is the more logical way to do.
# Also means that if your default_factory is for example "list", you won't be able to create new
# items with "append" or "extend" methods which are updating the list itself.
# Instead you have to do something like trie["newkey"] += [...]
try:
_ = factory()
except TypeError:
# a default value is also accepted as default_factory, even "None"
self._default_factory = lambda: factory
else:
self._default_factory = factory
else:
self._default_factory = _SENTINEL
def copy(self):
"""
Return a pseudo-shallow copy of the indexedtrie.
Keys and nodes are deepcopied, but if you store some referenced objects in values, only the references will be copied.
"""
return self.__class__(self.items(), default_factory=self._default_factory)
def __len__(self):
return len(self._indexes)
def __repr__(self):
if self._default_factory is not _SENTINEL:
default = ", default_value=%s" % self._default_factory()
else:
default = ""
return "<%s object at %s: %d items, %d nodes%s>" % (self.__class__.__name__, hex(id(self)), len(self), self._nodescount, default)
def __str__(self):
ret = ["%s: %s" % (k, v) for k, v in self.items()]
return "{%s}" % ", ".join(ret)
def __iter__(self):
return self.keys()
def __contains__(self, key_or_index):
"""
Return True if the key or index exists in the indexed trie.
"""
if isinstance(key_or_index, IndexedtrieKey):
return key_or_index.index >= 0 and key_or_index.index < len(self)
if isinstance(key_or_index, int):
return key_or_index >= 0 and key_or_index < len(self)
if self._seems_valid_key(key_or_index):
try:
node = self._get_node(key_or_index)
except KeyError:
return False
else:
return node._index is not None
raise TypeError("invalid key type")
def __getitem__(self, key_or_index):
"""
"""
if isinstance(key_or_index, IndexedtrieKey):
return self._values[key_or_index.index]
if isinstance(key_or_index, int) or isinstance(key_or_index, slice):
return self._values[key_or_index]
if self._seems_valid_key(key_or_index):
try:
node = self._get_node(key_or_index)
except KeyError:
if self._default_factory is _SENTINEL:
raise
else:
return self._default_factory()
else:
if node._index is None:
if self._default_factory is _SENTINEL:
raise KeyError
else:
return self._default_factory()
else:
return self._values[node._index]
raise TypeError("invalid key type")
def __setitem__(self, key_or_index, value):
"""
"""
if isinstance(key_or_index, IndexedtrieKey):
self._values[key_or_index.index] = value
elif isinstance(key_or_index, int):
self._values[key_or_index] = value
elif isinstance(key_or_index, slice):
raise NotImplementedError
elif self._seems_valid_key(key_or_index):
try:
node = self._get_node(key_or_index)
except KeyError:
# create a new node
self._add_node(key_or_index, value)
else:
if node._index is None:
# if node exists but not indexed, we index it and update the value
self._add_to_index(node, value)
else:
# else we update its value
self._values[node._index] = value
else:
raise TypeError("invalid key type")
def __delitem__(self, key_or_index):
"""
"""
if isinstance(key_or_index, IndexedtrieKey):
node = self._indexes[key_or_index.index]
elif isinstance(key_or_index, int):
node = self._indexes[key_or_index]
elif isinstance(key_or_index, slice):
raise NotImplementedError
elif self._seems_valid_key(key_or_index):
node = self._get_node(key_or_index)
if node._index is None:
raise KeyError
else:
raise TypeError("invalid key type")
# switch last index with deleted index (except if deleted index is last index)
last_node, last_value = self._indexes.pop(), self._values.pop()
if node._index != last_node._index:
last_node._index = node._index
self._indexes[node._index] = last_node
self._values[node._index] = last_value
if len(node._children) > 1:
#case 1: node has more than 1 child, only turn index off
node._index = None
elif len(node._children) == 1:
# case 2: node has 1 child
child = node._children.pop()
child._key = node._key + child._key
child._parent = node._parent
node._parent._children.add(child)
node._parent._children.remove(node)
del(node)
self._nodescount -= 1
else:
# case 3: node has no child, check the parent node
parent = node._parent
parent._children.remove(node)
del(node)
self._nodescount -= 1
if hasattr(parent, "_index"):
if parent._index is None and len(parent._children) == 1:
node = parent._children.pop()
node._key = parent._key + node._key
node._parent = parent._parent
parent._parent._children.add(node)
parent._parent._children.remove(parent)
del(parent)
self._nodescount -= 1
#staticmethod
def _seems_valid_key(key):
"""
Return True if "key" can be a valid key (must be subscriptable).
"""
try:
_ = key[:0]
except TypeError:
return False
return True
def keys(self, prefix=None):
"""
Yield keys stored in the indexedtrie where key is a IndexedtrieKey object.
If prefix is given, yield only keys of items with key matching the prefix.
"""
if prefix is None:
for i, node in enumerate(self._indexes):
yield IndexedtrieKey(i, self._get_key(node))
else:
if self._seems_valid_key(prefix):
empty = prefix[:0]
children = [(empty, prefix, child) for child in self._children]
while len(children):
_children = []
for key, prefix, child in children:
if prefix == child._key[:len(prefix)]:
_key = key + child._key
_children.extend([(_key, empty, _child) for _child in child._children])
if child._index is not None:
yield IndexedtrieKey(child._index, _key)
elif prefix[:len(child._key)] == child._key:
_prefix = prefix[len(child._key):]
_key = key + prefix[:len(child._key)]
_children.extend([(_key, _prefix, _child) for _child in child._children])
children = _children
else:
raise ValueError("invalid prefix type")
def values(self, prefix=None):
"""
Yield values stored in the indexedtrie.
If prefix is given, yield only values of items with key matching the prefix.
"""
if prefix is None:
for value in self._values:
yield value
else:
for key in self.keys(prefix):
yield self._values[key.index]
def items(self, prefix=None):
"""
Yield (key, value) pairs stored in the indexedtrie where key is a IndexedtrieKey object.
If prefix is given, yield only (key, value) pairs of items with key matching the prefix.
"""
for key in self.keys(prefix):
yield key, self._values[key.index]
def show_tree(self, node=None, level=0):
"""
Pretty print the internal trie (recursive function).
"""
if node is None:
node = self
for child in node._children:
print("-" * level + "<key=%s, index=%s>" % (child._key, child._index))
if len(child._children):
self.show_tree(child, level + 1)
def _get_node(self, key):
"""
Return the node associated to key or raise a KeyError.
"""
children = self._children
while len(children):
notfound = True
for child in children:
if key == child._key:
return child
if child._key == key[:len(child._key)]:
children = child._children
key = key[len(child._key):]
notfound = False
break
if notfound:
break
raise KeyError
def _add_node(self, key, value):
"""
Add a new key in the trie and updates indexes and values.
"""
children = self._children
parent = self
moved = None
done = len(children) == 0
# we want to insert key="abc"
while not done:
done = True
for child in children:
# assert child._key != key # uncomment if you don't trust me
if child._key == key[:len(child._key)]:
# case 1: child's key is "ab", insert "c" in child's children
parent = child
children = child._children
key = key[len(child._key):]
done = len(children) == 0
break
elif key == child._key[:len(key)]:
# case 2: child's key is "abcd", we insert "abc" in place of the child
# child's parent will be the inserted node and child's key is now "d"
parent = child._parent
moved = child
parent._children.remove(moved)
moved._key = moved._key[len(key):]
break
elif type(key) is type(child._key): # don't mess it up
# find longest common prefix
prefix = key[:0]
for i, c in enumerate(key):
if child._key[i] != c:
prefix = key[:i]
break
if prefix:
# case 3: child's key is abd, we spawn a new node with key "ab"
# to replace child ; child's key is now "d" and child's parent is
# the new created node.
# the new node will also be inserted as a child of this node
# with key "c"
node = _Node(prefix, child._parent)
self._nodescount += 1
child._parent._children.remove(child)
child._key = child._key[len(prefix):]
child._parent = node
node._children.add(child)
key = key[len(prefix):]
parent = node
break
# create the new node
node = _Node(key, parent)
self._nodescount += 1
if moved is not None:
# if we have moved an existing node, update it
moved._parent = node
node._children.add(moved)
self._add_to_index(node, value)
def _get_key(self, node):
"""
Rebuild key from a terminal node.
"""
key = node._key
while node._parent is not self:
node = node._parent
key = node._key + key
return key
def _add_to_index(self, node, value):
"""
Add a new node to the index.
Also record its value.
"""
node._index = len(self)
self._indexes.append(node)
self._values.append(value)
def key2index(self, key):
"""
key -> index
"""
if self._seems_valid_key(key):
node = self._get_node(key)
if node._index is not None:
return node._index
raise KeyError
raise TypeError("invalid key type")
def index2key(self, index):
"""
index or IndexedtrieKey -> key.
"""
if isinstance(index, IndexedtrieKey):
index = index.index
elif not isinstance(index, int):
raise TypeError("index must be an int")
if index < 0 or index > len(self._indexes):
raise IndexError
return self._get_key(self._indexes[index])

Variable isn't receiving the method's return value in Python

def return_node(self, head, position):
if position == 0:
# return the node correctly
return head
else:
self.return_node(head.next_node, position - 1)
def insert_at_position(self, head, data, position):
if position == 0:
self.insert_first(head, data)
elif position == self.length:
self.insert_last(head, data)
else:
previous_node = self.return_node(head, position - 1)
# previous_node's value is None instead of the method's return value
next_node = self.return_node(head, position)
# same here
new_node = Node(data, next_node)
previous_node.next_node = new_node
self.length += 1
I'm trying to implement a method in my linked list that insert a node at a specific position. The problem is: the variables 'previous_node' and 'next_node' are not getting the values properly.
Instead of the node value they are getting None. Thank you guys!

else:
self.return_node(head.next_node, position - 1)
Will not return anything, because there is no return keyword.
return self.return_node(head.next_node, position - 1)
Will do what you are looking for.

The reason your variables are being set to None, is because that is the default value returned from a function if no value to return is provided:
def foo():
pass
>>> type(foo())
<class 'NoneType'>
Because the else clause inside of return_node() does not return a value, Python returns None. If you want to call return_node recursively and return the value returned by the subsequent call, you need to use return:
def return_node(self, head, position):
if position == 0:
# return the node correctly
return head
else:
return self.return_node(head.next_node, position - 1) # use return

How to return value from recursive function in python? [duplicate]

This question already has answers here:
Why does my recursive function return None?
(4 answers)
Closed 7 months ago.
I'm working with binary tree in python. I need to create a method which searches the tree and return the best node where a new value can be inserted. But i'm have trouble returning a value from this recursive function. I'm a total newbie in python.
def return_key(self, val, node):
if(val < node.v):
if(node.l != None):
self.return_key(val, node.l)
else:
print node.v
return node
else:
if(node.r != None):
#print node.v
self.return_key(val, node.r)
else:
print node.v
return node
Printing node.v prints the node value, but when i print the returned node :
print ((tree.return_key(6, tree.getRoot().v)))
it prints
None
as result.

You need to return the result of your recursive call. You are ignoring it here:
if(node.l != None):
self.return_key(val, node.l)
and
if(node.r != None):
self.return_key(val, node.r)
Recursive calls are no different from other function calls, you still need to handle the return value if there is one. Use a return statement:
if(node.l != None):
return self.return_key(val, node.l)
# ...
if(node.r != None):
return self.return_key(val, node.r)
Note that since None is a singleton value, you can and should use is not None here to test for the absence:
if node.l is not None:
return self.return_key(val, node.l)
# ...
if node.r is not None:
return self.return_key(val, node.r)
I suspect you are passing in the wrong arguments to the call to begin with however; if the second argument is to be a node, don't pass in the node value:
print(tree.return_key(6, tree.getRoot())) # drop the .v
Also, if all your node classes have the same method, you could recurse to that rather than using self.return_value(); on the Tree just do:
print tree.return_key(6)
where Tree.return_key() delegates to the root node:
def return_key(self, val):
root = tree.getRoot()
if root is not None:
return tree.getRoot().return_key(val)
and Node.return_key() becomes:
def return_key(self, val):
if val < self.v:
if self.l is not None:
return self.l.return_key(val)
elif val > self.v:
if self.r is not None:
return self.r.return_key(val)
# val == self.v or child node is None
return self
I updated the val testing logic here too; if val < self.v (or val < node.v in your code) is false, don't assume that val > self.v is true; val could be equal instead.

How python pass by reference?

I tried to implement delete a node in a BST.
And here is my partial code.
def delete(node,key):
#Locate that node with value k
cNode=node
target=None
while cNode:
if cNode.value==key:
target=cNode
break
elif node.value>key:
cNode=cNode.lChild
elif node.value<key:
cNode=cNode.rChild
target=None
return node
When I tried to use the above method to delete a leaf node. I failed. when the method return, it did nothing to original BST. So what's the problem of this code? I assume it should have something about how python pass arguments by reference? But I am confused now.
Many thanks in advance.

target = None only rebinds the variable target to a new value, None. Whatever target was bound to before doesn't change.
You'll have to track the parent node and set it's lChild or rChild attribute to None instead.
def delete(node,key):
cNode = node
target = parent = None
while cNode:
if cNode.value == key:
target = cNode
break
elif cNode.value > key:
parent, cNode = cNode, cNode.lChild
elif cNode.value < key:
parent, cNode = cNode, cNode.rChild
if target:
if parent:
if parent.lChild is target:
parent.lChild = None
else:
parent.rChild = None
else:
# target is top-level node; perhaps return None in that case?
return node

Binary Search Tree - Insert Function

Good evening all,
I've been tasked with designing a function in Python that will build a Binary Search tree.
When I walk through the function myself, it seems to make perfect sense and it SHOULD work. However, for whatever reason, it's only building the last 'tree' and not saving any of the prior tree information. I've included my classes, the constructors and of course the function. Any tips are appreciated! To test the function, I am using the following line:
newMap = mapInsert1('one', 1, mapInsert1('two', 2, mkEmptyMap()))
///CODE///
class EmptyMap():
__slots__ = ()
class NonEmptyMap():
__slots__ = ('left', 'key', 'value', 'right')
def mkEmptyMap():
return EmptyMap()
def mkNonEmptyMap(left, key, value, right):
nonEmptyMap = NonEmptyMap()
nonEmptyMap.left = left
nonEmptyMap.key = key
nonEmptyMap.value = value
nonEmptyMap.right = right
return nonEmptyMap
def mapInsert1(key, value, node):
if isinstance(node, EmptyMap):
node = mkNonEmptyMap(mkEmptyMap(), key, value, mkEmptyMap())
return node
else:
if key > node.key:
return mapInsert1(key, value, node.right)
elif key < node.key:
return mapInsert1(key, value, node.left)
elif key == node.key:
node.value = value
return mapInsert1(key, value, node)
else:
raise TypeError('Bad Map')

Ok, I've got your answer here. There wasn't a problem with your logic, per se, just a problem with how you were trying to implement your algorithm in Python.
And there are several problems with how you're trying to implement your algorithm. The first of these has to do with how variables are passed into functions. I would recommend reading this StackOverflow Question here which discusses how variables are passed into functions Python. The long story short is that due to the way that you are passing and updating variables in your code, you are always updating a local scope copy of the variable, which doesn't actually affect the variable that you want to update.
To see this in your code, try the following:
>>> newMap = mapInsert1('one', 1, mapInsert1('two', 2, mkEmptyMap()))
As you said, this doesn't work. But this does:
>>> newMap = mapInsert1('one', 1, mkEmptyMap())
>>> newMap.right = mapInsert1('two', 2, mkEmptyMap()))
But that's not very helpful, because you have to know what node you want to update before you try and add a new node.
In order to fix your code, what I did was clean up your class implementation. I made the following changes:
First, I started using proper constructors. Python classes use the init function as a constructor. See here for more information.
Second, I added the insert function. This is what actually solves your problem. Using this function means that you're not overwriting a locally-scoped variable, but instead mutating the outer variable. Again, take a look at the variable-passing question I linked above for more details.
Third, I made the empty map just an empty instantiation of the map class, and got rid of the isinstance() check. In Python it's usually best to avoid isinstance whenever possible. Here is more information on avoiding isinstance
Fourth, I fixed an infinite loop bug in the code. If you look at the elif key == node.key condition, you are calling mapInsert1 with the same arguments again, which gives you an infinite recursive loop.
Here's the resulting code:
class Map():
__slots__ = ('left', 'key', 'value', 'right')
def __init__(self, left, key, value, right):
self.left = left
self.key = key
self.value = value
self.right = right
def insert(self, left, key, value, right):
self.left = left
self.key = key
self.value = value
self.right = right
def isEmpty(self):
return self.left == self.right == self.key == self.value == None
def mkEmptyMap():
return Map(None, None, None, None)
def mapInsert1(key, value, node):
if node.isEmpty():
print '0'
node.insert(mkEmptyMap(), key, value, mkEmptyMap())
return node
else:
if key > node.key:
print '1'
return mapInsert1(key, value, node.right)
elif key < node.key:
print '2'
return mapInsert1(key, value, node.left)
elif key == node.key:
print '3'
node.value = value
return node
else:
raise TypeError('Bad Map')
And here's a quick test:
>>> root = mapInsert1('five', 5, mkEmptyMap())
>>> mapInsert1('four', 4, root)
>>> mapInsert1('ace', 1, root)
>>> mapInsert1('five', 'five', root)
>>> root.left.isEmpty()
Out: False
>>> root.left.key
Out: 'ace'
>>> root.left.value
Out: 1
>>> root.right.isEmpty()
Out: False
>>> root.right.key
Out: 'four'
>>> root.right.value
Out: 4
>>> root.key
Out: 'five'
>>> root.value
Out: 'five'