Develop Reference

user_items must contain 1 row for every user in userids

I want to use recommend method by imlicit library, I have made csr matrix like this
import scipy.sparse as sparse
user_items = sparse.csr_matrix((train['item_count'].astype(float),(train['client_id'], train['product_id'])))
item_users = sparse.csr_matrix((train['item_count'].astype(float),(train['product_id'], train['client_id'])))
but, when I tried to use recommend method in implicit, it showed
print('List of recommend Item for user:')
model.recommend(124, item_users)
List of recommend Item for user:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-151-100e4e122c46> in <module>
1 print('List of recommend Item for user:')
----> 2 model.recommend(124, item_users)
/usr/local/lib/python3.7/dist-packages/implicit/cpu/matrix_factorization_base.py in recommend(self, userid, user_items, N, filter_already_liked_items, filter_items, recalculate_user, items)
47 user_count = 1 if np.isscalar(userid) else len(userid)
48 if user_items.shape[0] != user_count:
---> 49 raise ValueError("user_items must contain 1 row for every user in userids")
50
51 user = self._user_factor(userid, user_items, recalculate_user)
ValueError: user_items must contain 1 row for every user in userids
I tried using the model.similar.items(), model.explain(), model.similar.user() methods, it was work perfectly, but when I tried the recoomend() methods it show error like before. Can anyone help?? thanks!

It's due to an API change: The fix is to use model.recommend(user_label, sparse_user_items[user_label]) instead of model.recommend(user_label, sparse_user_items)
See: https://github.com/benfred/implicit/issues/535

Data analysis - MD analysis Python

I am seeing this error, need help on this!
warnings.warn("Failed to guess the mass for the following atom types: {}".format(atom_type))
Traceback (most recent call last):
File "traj_residue-z.py", line 48, in
protein_z=protein.centroid()[2]
IndexError: index 2 is out of bounds for axis 0 with size 0

The problem was solved through a discussion in the mailing list thread https://groups.google.com/g/mdnalysis-discussion/c/J8oJ0M9Rjb4/m/kSD2jURODQAJ
In brief: The traj_residue-z.py script contained the line
protein=u.select_atoms('resid 1-%d' % (nprotein_res))
It turned out that the selection 'resid 1-%d' % (nprotein_res) would not select anything because the input GRO file started with resid 1327
1327LEU N 1 2.013 3.349 8.848 0.4933 -0.2510 0.2982
1327LEU H1 2 1.953 3.277 8.893 0.0174 0.1791 0.3637
1327LEU H2 3 1.960 3.377 8.762 0.6275 -0.5669 0.1094
...
and hence the selection of resids starting at 1 did not match anything. This produced an empty AtomGroup protein.
The subsequent centroid calculation
protein_z=protein.centroid()[2]
failed because for an empty AtomGroup, protein.centroid() returns an empty array and so trying to get the element at index 2 raises IndexError.
The solution (thanks to #IAlibay) was to
either change the selection string 'resid 1-%d' to accommodate start and stop resids, or
to just select the first nprotein_res residues protein = u.residues[:nprotein_res].atoms by slicing the ResiduesGroup.

Error while using sum() in Python SFrame

I'm new to python and I'm performing a basic EDA analysis on two similar SFrames. I have a dictionary as two of my columns and I'm trying to find out if the max values of each dictionary are the same or not. In the end I want to sum up the Value_Match column so that I can know how many values match but I'm getting a nasty error and I haven't been able to find the source. The weird thing is I have used the same methodology for both the SFrames and only one of them is giving me this error but not the other one.
I have tried calculating max_func in different ways as given here but the same error has persisted : getting-key-with-maximum-value-in-dictionary
I have checked for any possible NaN values in the column but didn't find any of them.
I have been stuck on this for a while and any help will be much appreciated. Thanks!
Code:
def max_func(d):
v=list(d.values())
k=list(d.keys())
return k[v.index(max(v))]
sf['Max_Dic_1'] = sf['Dic1'].apply(max_func)
sf['Max_Dic_2'] = sf['Dic2'].apply(max_func)
sf['Value_Match'] = sf['Max_Dic_1'] == sf['Max_Dic_2']
sf['Value_Match'].sum()
Error :
RuntimeError Traceback (most recent call last)
<ipython-input-70-f406eb8286b3> in <module>()
----> 1 x = sf['Value_Match'].sum()
2 y = sf.num_rows()
3
4 print x
5 print y
C:\Users\rakesh\Anaconda2\lib\site-
packages\graphlab\data_structures\sarray.pyc in sum(self)
2216 """
2217 with cython_context():
-> 2218 return self.__proxy__.sum()
2219
2220 def mean(self):
C:\Users\rakesh\Anaconda2\lib\site-packages\graphlab\cython\context.pyc in
__exit__(self, exc_type, exc_value, traceback)
47 if not self.show_cython_trace:
48 # To hide cython trace, we re-raise from here
---> 49 raise exc_type(exc_value)
50 else:
51 # To show the full trace, we do nothing and let
exception propagate
RuntimeError: Runtime Exception. Exception in python callback function
evaluation:
ValueError('max() arg is an empty sequence',):
Traceback (most recent call last):
File "graphlab\cython\cy_pylambda_workers.pyx", line 426, in
graphlab.cython.cy_pylambda_workers._eval_lambda
File "graphlab\cython\cy_pylambda_workers.pyx", line 169, in
graphlab.cython.cy_pylambda_workers.lambda_evaluator.eval_simple
File "<ipython-input-63-b4e3c0e28725>", line 4, in max_func
ValueError: max() arg is an empty sequence

In order to debug this problem, you have to look at the stack trace. On the last line we see:
File "<ipython-input-63-b4e3c0e28725>", line 4, in max_func
ValueError: max() arg is an empty sequence
Python thus says that you aim to calculate the maximum of a list with no elements. This is the case if the dictionary is empty. So in one of your dataframes there is probably an empty dictionary {}.
The question is what to do in case the dictionary is empty. You might decide to return a None into that case.
Nevertheless the code you write is too complicated. A simpler and more efficient algorithm would be:
def max_func(d):
if d:
return max(d,key=d.get)
else:
# or return something if there is no element in the dictionary
return None

py2neo rel() list indices must be integer not float

I'm trying to import nodes into Neo4j in a batch. But when I try to execute it, it throws an error: List indices must be integers, not float. I don't really understand which listitems, I do have floats, but these are cast to strings...
Partial code:
graph_db = neo4j.GraphDatabaseService("http://127.0.0.1:7474/db/data/")
batch = neo4j.WriteBatch(graph_db)
for ngram, one_grams in data.items():
ngram_rank = int(one_grams['_rank'])
ngram_prob = '%.16f' % float(one_grams['_prob'])
ngram_id = 'a'+str(n)
ngram_node = batch.create(node({"word": ngram, "rank": str(ngram_rank), "prob": str(ngram_prob)}))
for one_gram, two_grams in one_grams.items():
one_rank = int(two_grams['_rank'])
one_prob = '%.16f' % float(two_grams['_prob'])
one_node = batch.create(node({"word": one_gram, "rank": str(one_rank), "prob": one_prob}))
batch.create(rel((ngram_node, "FOLLOWED_BY", one_node))) #line 81 throwing error
results = batch.submit()
Full traceback
Traceback (most recent call last):
File "Ngram_neo4j.py", line 81, in probability_items
batch.create(rel((ngram_node, "FOLLOWED_BY", one_node))),
File "virtenv\\lib\\site-packages\\py2neo\\neo4j.py", line 2692, in create
uri = self._uri_for(entity.start_node, "relationships"),
File "virtenv\\lib\\site-packages\\py2neo\\neo4j.py", line 2537, in _uri_for
uri = "{{{0}}}".format(self.find(resource)),
File "virtenv\\lib\\site-packages\\py2neo\\neo4j.py", line 2525, in find
for i, req in pendulate(self._requests):,
File "virtenv\\lib\\site-packages\\py2neo\\util.py", line 161, in pendulate
yield index, collection[index],
TypeError: list indices must be integers, not float
running neo4j 2.0, py2neo 1.6.1, Windows 7/64bit, python 3.3/64bit
--EDIT--
Did some testing, but the error is located in the referencing to nodes.
oversimplified sample code:
for key, dict in data.items(): #string, dictionary
batch = neo4j.WriteBatch(graph_db)
three_gram_node = batch.create(node({"word": key}))
pprint(three_gram_node)
batch.add_labels(three_gram_node, "3gram") # must be int, not float
for k,v in dict.items(): #string, string
four_gram_node = batch.create(node({"word": k}))
batch.create_path(three_gram_node, "FOLLOWED_BY", four_gram_node)
# cannot cast node from BatchRequest obj
batch.submit()
When a node is created batch.create(node({props})), the pprint returns a P2Neo.neo4j. batchrequest object.
At the line add_labels(), it gives the same error as when trying to create a relation: List indices must be integers, not float.
At the batch.create_path() line it throws an error saying it can't cast a node from a P2Neo.neo4j. batchrequest object.
I'm trying the dirty-debug now to understand the indices.
--Dirty Debug Edit--
I've been meddling around with the pendulate(collection) function.
Although I don't really understand how it fits in, and how it's used, the following is happening:
Whenever it hits an uneven number, it gets cast to a float (which is weird, since count - ((i + 1) / 2), where i is an uneven number.) This float then throws the list indices error. Some prints:
count: 3
i= 0
index: 0
(int)index: 0
i= 1 # i = uneven
index: 2.0 # a float appears
(int)index: 2 # this is a safe cast
This results in the list indices error. This also happens when i=0. As this is a common case, I made an additional if() to circumvent the code (possible speedup?) Although I've not unit tested this, it seems that we can safely cast index to an int...
The pendulate function as used:
def pendulate(collection):
count = len(collection)
print("count: ", count)
for i in range(count):
print("i=", i)
if i == 0:
index = 0
elif i % 2 == 0:
index = i / 2
else:
index = count - ((i + 1) / 2)
print("index:", index)
index = int(index)
print("(int)index:", index)
yield index, collection[index]

soft debug : print ngram_node and one_node to see what they contains
dirty debug : modify File "virtenv\lib\site-packages\py2neo\util.py", line 161, add a line before :
print index
You are accessing a collection (a Python list given the traceback), so, for sure, index must be an integer :)
printing it will probably help you to understand why exception raised
(Don't forget to remove your dirty debug afterwards ;))

While it is currently possible for WriteBatch objects to be executed multiple times with edits in between, it is inadvisable to use them in this way and this will be restricted in the next version of py2neo. This is because objects created during one execution will not be available during a subsequent execution and it is not easy to detect when this is being requested.
Without looking back at the underlying code, I'm unsure why you are seeing this exact error but I would suggest refactoring your code so that each WriteBatch creation is paired with one and only one execution call (submit). You can probably achieve this by putting your batch creation within your outer loop and moving your submit call out of the inner loop into the outer loop as well.

Systematically moving through every number in a numpy array

I have a 500x500 array and I am trying to write a variable called "coordinate" that will pick out each value and then apply it to a function but I keep getting the output,
AttributeError Traceback (most recent call last)
/home/graham/<ipython-input-17-cc6ce0649eda> in <module>()
31 pass
32
---> 33 finished_array = rand_array_color(rand_array)
34
35 finished_image = Image.fromarray(finished_array)
/home/graham/<ipython-input-17-cc6ce0649eda> in rand_array_color(rand_array)
23 from PIL import Image
24 for ii in numpy.nditer(rand_array):
---> 25 coordinate = tuple(map(int, rand_image[ii,:]))
26 if ii < 128:
27 print "false"
/usr/lib/python2.7/dist-packages/PIL/Image.pyc in __getattr__(self, name)
510 new['data'] = self.tostring()
511 return new
--> 512 raise AttributeError(name)
513
514 ##
AttributeError: __getitem__
Here is my code
from PIL import Image
from numpy import random
im = Image.open('/home/graham/Desktop/Experiment/gwarner/labeled_photos/photos/003030.png')
rand_array = numpy.random.randint(255, size=(500,500)).astype('uint8')
rand_image = Image.fromarray(rand_array)
def rand_array_color(rand_array):
from PIL import Image
for ii in numpy.nditer(rand_array):
coordinate = tuple(map(int, rand_image[ii,:]))
if ii < 128:
newvalue = rand_image.putpixel(coordinate(a,ii), im.getpixel(coordinate(a,ii)))
return newvalue
else:
pass
finished_array = rand_array_color(rand_array)
I've also been fiddling around with another version of coordinate,
coordinate = tuple(int(rand_array[ii,0]),int(rand_array[ii,1])
but it just returns,
NameError: name 'ii' is not defined
Can anyone tell me how to fix one of these or recommend another one that will work?

The reason you cannot get the iith row of rand_image using rand_image[ii,:] is because rand_image is a PIL Image object, and doesn't have the same indexing access that an array has. If you want to get the iith row, you must use rand_array[ii] (I've left off the trailing colon ,: isn't required)
This appears to be similar to your
another version of coordinate
which should work because you're using the array instead of the Image. My guess is that you've not replaced it in exactly the same place as the first version, perhaps somewhere outside of the for ii in ... loop, because ii is only properly defined within that loop.
As you've defined it, coordinate is not a function, it's a tuple of length 2. But you call it as a function when you write coordinate(a, ii). Rather, you probably just want to use coordinate there (I'm not sure what a is supposed to be.
The function .putpixel does not return anything, so that when you say:
newvalue = rand_image.putpixel(...)
newvalue will be None (the 'empty' object in python). Perhaps you want to do this:
rand_image.putpixel(...)
return rand_image
without setting the newvalue and trying to return it, but instead changing the pixel in the rand_image and returning the new rand_image.
As a side note, you should not call int() on a numpy array, instead, if you want the type of each element in the array to be int, you should either use .astype as you did above, or initiate it as an int in the first place. Once it starts as an int, it should stay that way unless you cast it somehow (which you haven't done in this script).
As a final note this process could probably be done a lot faster using numpy (not using pure python loops) but I'm not sure exactly what you're trying to do. If you edit your question to explain a little bit what you're trying to do so that I can read the code with some context, I could help. For example, what is a supposed to be in the newvalue = ... line?