Develop Reference

Attritube error when importing local python file

I would like some help in figuring out an issue. The code below is attempting to import a file called game_getter.py to access it's all_games dictionary variable.
from django.db import models
from catolog import game_getter
# Create your models here.
class Game(models.Model):
url_clue = ["//images.igdb.com"]
for game in game_getter.all_games:
title = game_getter.all_games[game][0]
if url_clue in game_getter.all_games[game][1]:
cover_art = game_getter.all_games[game]
else:
pass
if game_getter.all_games[game][2] == None:
pass
else:
storyline = game_getter.all_games[game][2]
if game_getter.all_games[game][3] == None:
pass
else:
storyline = game_getter.all_games[game][3]
genre_pac = game_getter.all_games[game][4]
def __str__(self):
return self.title
class UserSearch():
user_input = str
At the bottom of this next session I used a return on the dictionary object all_games. I've even tried making it a global variable and the computer still won't see it.
# Create a search
def search_query(user_input, exp_time):
# Generate token if needed
generate_token(exp_time)
#establish client_id for wrapper
client_id = "not my client_id"
wrapper = IGDBWrapper(client_id, token)
# Query the API based on a search term.
query = wrapper.api_request(
'games', # Requesting the name, storyline, genre name, and cover art url where the user input is the search tearm
f'fields name, storyline, genres.slug, cover.url; offset 0; where name="{user_input}"*; sort first_release_date; limit: 100;',
# Also sorted by release date with a limit of 100 results
)
# Load the binary data into json
message_json = json.loads(query)
# List of all games returned
global all_games
all_games = dict()
key = 0
# Grab each value by key and separate per game
for game in message_json:
name = game.get('name')
cover_url = game.get('cover')
storyline = game.get('storyline')
summary = game.get('summary')
genre_set = game.get('genres')
# Genre posses none to many tags which needs to be sorted.
genre_list = []
if genre_set:
for type in genre_set:
for i in type:
genre_list.append(type[i])
for i in genre_list:
genre_list = [x for x in genre_list if not isinstance(x, int)]
else:
pass
# Group together by game
if game.get('cover') != None:
result = [name,cover_url.get('url'),storyline,summary,genre_list]
# Add the game to the collection of all games found
all_games[key] = result
key += 1
else:
result = [name,storyline,summary,genre_list]
# Add the game to the collection of all games found
all_games[key] = result
key += 1
return all_games
What am I missing?

How to reduce the complexity of several for loops in python

I have the following function
import requests
children_dict = {}
def get_list_of_children(base_url, username, password, folder="1"):
token = get_token()
url = f"{base_url}/unix/repo/folders/{folder}/list"
json = requests_json(url,token)
for obj in json["list"]:
if obj['name'] == 'MainFolder':
folderId = obj['id']
url_parent = f"{base_url}/unix/repo/folders/{folderId}/list"
json_parent = requests_json(url_parent,token)
for obj_child in json_parent['list']:
if obj_child['folder'] == True:
folder_grand_child_id = obj_child['id']
url_grand_child = f"{base_url}/unix/repo/folders/{folder_grand_child_id}/list"
json_grand_child = requests_json(url_grand_child,token)
for obj_grand_child in json_grand_child["list"]:
if obj_grand_child['name'] == 'SubFolder':
folder_grand_grand_child = obj_grand_child['id']
url_grand_grand_child = f"{base_url}/unix/repo/folders/{folder_grand_grand_child}/list"
json_grand_grand_child = requests_json(url_grand_grand_child,token)
for obj_grand_grand_child in json_grand_grand_child["list"]:
if obj_grand_grand_child['name'] == 'MainTasks':
folder_grand_grand_grand_child = obj_grand_grand_child['id']
url_grand_grand_grand_child = f"{base_url}/unix/repo/folders/{folder_grand_grand_grand_child}/list"
json_grand_grand_grand_child = requests_json(url_grand_grand_grand_child,token)
for obj_grand_grand_grand_child in json_grand_grand_grand_child["list"]:
children_dict[[obj_grand_grand_grand_child['id']] = obj_grand_grand_grand_child['name']
return children_dict
What i am trying to accomplish here is to make repeated api calls to traverse through http folder structure to get the list of files in the last directory
The function works as intended but sonarlint is through the below error
Refactor this function to reduce its Cognitive Complexity from 45 to the 15 allowed. [+9 locations]sonarlint(python:S3776)
is there is a better way to handle this function ?
can anyone refactor this, pointing me in the right direct will do

This isn't a complete solution, but to answer your question "how can I simplify this" more generally, you need to look for repeated patterns in your code and generalize them into a function. Perhaps it's a function you can call recursively, or in a loop. For example, in that deeply-nested statement of yours it's just the same pattern over and over again like:
url = f"{base_url}/unix/repo/folders/{folder}/list"
json = requests_json(url,token)
for obj in json["list"]:
if obj['name'] == '<some folder name>':
folderId = obj['id']
# ...repeat...
So try generalizing this into a loop, maybe, like:
url_format = "{base_url}/unix/repo/folders/{folder_id}/list"
folder_hierarchy = ['MainFolder', 'SubFolder', 'MainTasks']
folder_id = '1' # this was the argument passed to your function
for subfolder in folder_hierarchy:
url = url_format.format(base_url=base_url, folder_id=folder_id)
folder_json = requests_json(url, token)
for obj in folder_json['list']:
if obj['name'] == subfolder:
folder_id = obj['id']
break
# Now the pattern repeats for the next level of the hierarchy but
# starting with the new folder_id
This is just schematic and you may need to generalize further, but it's one idea.
If your goal is to traverse a more complicated hierarchy you might want to look into tree-traversal algorithms.

There's plenty of repeating code. Once you identify the repeating patterns, you can extract them to the classes and functions. In particular, I find it useful to isolate all the web API logic from the rest of the code:
class Client:
def __init__(self, base_url, token):
self.base_url = base_url
self.token = token
def list_folder(self, folder_id):
return request_json(
f'{self.base_url}/unix/repo/folders/{folder_id}/list', self.token
)['list']
def get_subfolders(self, parent_id=1):
return [c for c in self.list_folder(parent_id) if c['folder']]
def get_subfolder(self, parent_id, name):
children = self.list_folder(parent_id)
for child in children:
if child['name'] == name:
return child
return None
def resolve_path(self, path, root_id=1):
parent_id = root_id
for p in path:
current = self.get_subfolder(parent_id, p)
if not current:
return None
parent_id = current['id']
return current
Now you can use the class above to simplify the main code:
client = Client(base_url, token)
for folder in client.get_subfolders():
child = client.resolve_path(folder['id'], ('SubFolder', 'MainTasks'))
if child:
# do the rest of the stuff
The code above is not guaranteed to work as is, just an illustration of the idea.

I can't really test it, but I'd make it like the following in order to easily build many levels of repeating code.
class NestedProcessing:
def __init__(self, base_url):
self.base_url = base_url
self.token = get_token()
self.obj_predicates = []
def next_level_predicate(self, obj_predicate):
self.obj_predicates.append(obj_predicate)
return self
def final_action(self, obj_action):
self.obj_action = obj_action
return self
def process(self, first_folder_id):
self.process_level(0, first_folder_id)
def process_level(self, index, folder_id):
obj_is_good = self.obj_predicates[index]
url = f"{self.base_url}/unix/repo/folders/{folder_id}/list"
json = requests_json(url, self.token)
for obj in json["list"]:
if index == len(self.obj_predicates) - 1: # last level
self.obj_action(obj)
elif obj_is_good(obj):
self.process_level(index + 1, obj['id'])
def get_list_of_children(base_url, username, password, folder="1"):
children_dict = {}
NestedProcessing(base_url)
.next_level_predicate(lambda obj: obj['name'] == 'MainFolder')
.next_level_predicate(lambda obj: obj['folder'] == True)
.next_level_predicate(lambda obj: obj['name'] == 'SubFolder')
.next_level_predicate(lambda obj: obj['name'] == 'MainTasks')
.final_action(lambda obj, storage=children_dict: storage.update({obj['id']: obj['name']})
.process(folder)
return children_dict

Iteration and memory problems in Django

I need to create pairs of hashtags so people can judge whether the two tags in question refer to the same thing. The problem is that there are A LOT of hashtags, and I'm running the code on a Dreamhost VPS, so my memory is somewhat limited.
Here's my relevant models:
class Hashtag(models.Model):
text = models.CharField(max_length=140)
competitors = models.ManyToManyField('Hashtag', through='Competitors')
tweet = models.ManyToManyField('Tweet')
def __unicode__(self):
return unicode_escape(self.text)
class Competitors(models.Model):
tag1 = models.ForeignKey('Hashtag', related_name='+')
tag2 = models.ForeignKey('Hashtag', related_name='+')
yes = models.PositiveIntegerField(default=0, null=False)
no = models.PositiveIntegerField(default=0, null=False)
objects = models.Manager()
def __unicode__(self):
return u'{0} vs {1}'.format(unicode_escape(self.tag1.text), unicode_escape(self.tag2.text))
Here's the code I've developed to create the Competitors objects and save them to my DB:
class Twitterator(object):
def __init__(self, infile=None, outfile=None, verbosity=True):
...
self.competitors_i = 1
...
def __save_comps__(self,tag1, tag2):
try:
comps = Competitors(id=self.competitors_i,
tag1=tag1,
tag2=tag2,
yes=0,
no=0)
comps.save()
except IntegrityError:
self.competitors_i += 1
self.save_comps(tag1, tag2)
else:
self.competitors_i += 1
def competitors_to_db(self, start=1):
tags = Hashtag.objects.all()
i = start
while True:
try:
tag1 = tags.get(pk=i)
j = i + 1
while True:
try:
tag2 = tags.get(pk=j)
self.__save_comps__(tag1, tag2)
j += 1
except Hashtag.DoesNotExist:
break
i += 1
except Hashtag.DoesNotExist:
break
It all "works", but never manages to get that far before I run out of memory and the whole thing gets killed. I thought using .get would be less memory-intensive, but it doesn't seem to be less memory-intensive enough. I'm under the impression that Django Querysets are iterators already, so my usual 'make an iterator' trick is out. Any suggestions for further reducing my memory footprint?

I think the problem is in this function, i is not getting incremented properly and you will keep looping for same value of i.
def competitors_to_db(self, start=1):
tags = Hashtag.objects.all()
i = start
while True:
try:
tag1 = tags.get(pk=i)
j = i + 1
while True:
try:
tag2 = tags.get(pk=j)
self.__save_comps__(tag1, tag2)
j += 1
except Hashtag.DoesNotExist:
break #<------move this after i +=1 otherwise i will not increment
i += 1

Django: How to check if object with ManyToManyField contains the entire contents of other object ManyToManyField?

class Hero(models.Model):
talents = models.ManyToManyField(Talent)
(...)
class Talent(models.Model)
required_talents = models.ManyToManyField('self', symmetrical=False)
(...)
I would like to create method has_required_talents(self, talent) for Hero, which will check if this Hero has required_talents for chosen Talent.
I tried this:
def has_required_talents(self, talent)
required_list = talent.talents_required.values_list('id', flat=True)
hero_talents = self.hero.talents.values_list('id', flat=True)
if required_list in hero_talents:
return True
return False
However, it doesn't work properly when I test it, using these tests:
class HeroTalents(TestCase):
def setUp(self):
self.hero=Hero('Duck')
self.hero.save()
def test_has_required_talents(self):
self.talent1 = Talent(name = "Overpower")
self.talent1.save()
self.talent2 = Talent(name = "Overpower2")
self.talent2.save()
self.talent2.talents_required.add(self.talent1)
self.assertFalse(self.hero.has_required_talents(self.talent2), "test1")
self.hero.talents.add(self.talent1)
self.assertTrue(self.hero.has_required_talents(self.talent2), "test2")
self.talent3 = Talent(name = "Overpower3")
self.talent3.save()
self.hero.talents.add(self.talent2)
self.assertTrue(self.hero.has_required_talents(self.talent3), "test3")
self.talent1.talents_required.add(self.talent3)
self.assertFalse(self.hero.has_required_talents(self.talent1), "test4")
What needs to be done to make this work?

def has_required_talents(self, talent)
required_list = talent.talents_required.values_list('id', flat=True)
hero_talents = self.hero.talents.values_list('id', flat=True)
for item in required_list:
if not item in hero_talents:
return False
return True
That would be a list-based approach. You can also convert them to sets:
def has_required_talents(self, talent)
required_list = set(talent.talents_required.values_list('id', flat=True))
hero_talents = set(self.hero.talents.values_list('id', flat=True))
if required_list.issubset(hero_talents):
return True
return False
But (I don't know about exact internals, you could run some tests) the first approach should be quicker, because there is no conversion.

Create a Python User() class that both creates new users and modifies existing users

I'm trying to figure out the best way to create a class that can modify and create new users all in one. This is what I'm thinking:
class User(object):
def __init__(self,user_id):
if user_id == -1
self.new_user = True
else:
self.new_user = False
#fetch all records from db about user_id
self._populateUser()
def commit(self):
if self.new_user:
#Do INSERTs
else:
#Do UPDATEs
def delete(self):
if self.new_user == False:
return False
#Delete user code here
def _populate(self):
#Query self.user_id from database and
#set all instance variables, e.g.
#self.name = row['name']
def getFullName(self):
return self.name
#Create a new user
>>u = User()
>>u.name = 'Jason Martinez'
>>u.password = 'linebreak'
>>u.commit()
>>print u.getFullName()
>>Jason Martinez
#Update existing user
>>u = User(43)
>>u.name = 'New Name Here'
>>u.commit()
>>print u.getFullName()
>>New Name Here
Is this a logical and clean way to do this? Is there a better way?
Thanks.

You can do this with metaclasses. Consider this :
class MetaCity:
def __call__(cls,name):
“”“
If it’s in the database, retrieve it and return it
If it’s not there, create it and return it
““”
theCity = database.get(name) # your custom code to get the object from the db goes here
if not theCity:
# create a new one
theCity = type.__call__(cls,name)
return theCity
class City():
__metaclass__ = MetaCity
name = Field(Unicode(64))
Now you can do things like :
paris = City(name=u"Paris") # this will create the Paris City in the database and return it.
paris_again = City(name=u"Paris") # this will retrieve Paris from the database and return it.
from : http://yassinechaouche.thecoderblogs.com/2009/11/21/using-beaker-as-a-second-level-query-cache-for-sqlalchemy-in-pylons/

Off the top of my head, I would suggest the following:
1: Use a default argument None instead of -1 for user_id in the constructor:
def __init__(self, user_id=None):
if user_id is None:
...
2: Skip the getFullName method - that's just your Java talking. Instead use a normal attribute access - you can convert it into a property later if you need to.

What you are trying to achieve is called Active Record pattern. I suggest learning existing systems providing this sort of things such as Elixir.

Small change to your initializer:
def __init__(self, user_id=None):
if user_id is None: