I don't know if it's possible. Hopefully you guys knows what I try to do. I want to do the model changes in a FOR loop cause the keys of the values have always the same name as the model columns.
My current code:
sites = DataModel.objects.all()
for site in sites:
d = self.getDataBySoup(soup)
site.title = d['title']
site.text = d['text']
site.facebook = d['facebook']
site.twitter = d['twitter']
site.save()
As you can see, the keys are always the same as the django columns. So I thought its maybe possible to do it with less code.
What I tried (but it's not working):
sites = DataModel.objects.all()
for site in sites:
d = self.getDataBySoup(soup)
for key, value in d.items():
site.key = value
site.save()
I use Python 3.6
getDataBySoup Method is just returning a dict/array:
def getContentDataBySoup(self, soup):
data = {}
data['title'] = 'some text'
# etc
return data
etc.
sites = DataModel.objects.all()
for site in sites:
d = self.getDataBySoup(soup)
DataModel.objects.filter(id=site.pk).update(**d)
The code above would update each site entry with the data in the dictionary d. If you're using the update method it's important that you also specify the id, otherwise it won't know what entry to update.
Related
I've written a script in python to scrape some item names along with review texts and reviewers connected to each item name from a webpage using their api. The thing is my below script can do those things partially. I need to do those in an organized manner.
For example, in each item name there are multiple review texts and reviewer names connected to it. I wish to get them along the columns like:
Name review text reviewer review text reviewer -----
Basically, I can't get the idea how to make use of the already defined for loop in the right way within my script. Lastly, there are few item names which do not have any reviews or reviewers, so the code breaks when it doesn't find any reviews and so.
This s my approach so far:
import requests
url = "https://eatstreet.com/api/v2/restaurants/{}?yelp_site="
res = requests.get("https://eatstreet.com/api/v2/locales/madison-wi/restaurants")
for item in res.json():
itemid = item['id']
req = requests.get(url.format(itemid))
name = req.json()['name']
for texualreviews in req.json()['yelpReviews']:
reviews = texualreviews['message']
reviewer = texualreviews['reviewerName']
print(f'{name}\n{reviews}\n{reviewer}\n')
If I use print statement outside the for loop, It only gives me a single review and reviewer.
Any help to fix that will be highly appreciated.
You need to append the review and a reviewer name to an array to display as you wish.
Try the following code.
review_data = dict()
review_data['name'] = req.json()['name']
review_data['reviews'] = []
for texualreviews in req.json()['yelpReviews']:
review_sub_data = {'review': texualreviews['message'], 'reviewer': texualreviews['reviewerName']}
review_data['reviews'].append(review_sub_data)
#O/P {'name': 'xxx', 'reviews':[{'review':'xxx', 'reviewer': 'xxx'}, {'review':'xxx', 'reviewer': 'xxx'}]}
Hope this helps!
I am new to django so apologies if this is not possible or easy.
I have a view that takes a subset of a model
data = Terms.objects.filter(language = language_id)
The subset is one language. The set has a number of concepts for a language. Some languages might use the same word for multiple concepts, and I want to colour these the same in an SVG image. So I do this next:
for d in data:
if d.term is None:
d.colour = "#D3D3D3"
else:
d.colour = termColours[d.term]
Where termColours is a dictionary with keys as the unique terms and values as the hexadecimal colour I want.
I thought this would add a new colour attribute to my queryset. However, when I convert the queryset to json (in order to pass it to JS) the colour object is not there.
terms_json = serializers.serialize('json', data)
How can I add a new colour element to my queryset?
Convert your Queryset to Dict and then modify values.
Ex:
data = Terms.objects.filter(language = language_id).values()
for d in data:
if d.term is None:
d.colour = "#D3D3D3"
else:
d.colour = termColours[d.term]
If I understand correctly - you need Django ORM annotation. And it might look like that:
from django.db.models import Case, When, Value
data = Terms.objects.filter(language = language_id)
.annotate(color = Case(
When(term__isnull = True, then = "#D3D3D3"),
When(term__isnull = False, then = termColours[Value(term)]),))
Only problem here - I don't exactly know this moment - termColours[Value(term)], you need to test different combinations of that expressions to get the value of field term.
I am currently working on a script to scrape data from ClinicalTrials.gov. To do this I have written the following script:
def clinicalTrialsGov (id):
url = "https://clinicaltrials.gov/ct2/show/" + id + "?displayxml=true"
data = BeautifulSoup(requests.get(url).text, "lxml")
studyType = data.study_type.text
if studyType == 'Interventional':
allocation = data.allocation.text
interventionModel = data.intervention_model.text
primaryPurpose = data.primary_purpose.text
masking = data.masking.text
enrollment = data.enrollment.text
officialTitle = data.official_title.text
condition = data.condition.text
minAge = data.eligibility.minimum_age.text
maxAge = data.eligibility.maximum_age.text
gender = data.eligibility.gender.text
healthyVolunteers = data.eligibility.healthy_volunteers.text
armType = []
intType = []
for each in data.findAll('intervention'):
intType.append(each.intervention_type.text)
for each in data.findAll('arm_group'):
armType.append(each.arm_group_type.text)
citedPMID = tryExceptCT(data, '.results_reference.PMID')
citedPMID = data.results_reference.PMID
print(citedPMID)
return officialTitle, studyType, allocation, interventionModel, primaryPurpose, masking, enrollment, condition, minAge, maxAge, gender, healthyVolunteers, armType, intType
However, the following script won't always work as not all studies will have all items (ie. a KeyError will occur). To resolve this, I could simply wrap each statement in a try-except catch, like this:
try:
studyType = data.study_type.text
except:
studyType = ""
but it seems a bad way to implement this. What's a better/cleaner solution?
This is a good question. Before I address it, let me say that you should consider changing the second parameter to the BeautifulSoup (BS) constructor from lxml to xml. Otherwise, BS does not flag the parsed markup as XML (to verify this for yourself, access the is_xml attribute on the data variable in your code).
You can avoid generating an error when attempting to access a non-existent element by passing a list of desired element names to the find_all() method:
subset = ['results_reference','allocation','interventionModel','primaryPurpose','masking','enrollment','eligibility','official_title','arm_group','condition']
tag_matches = data.find_all(subset)
Then, if you want to get a specific element from the list of Tags without iterating through it, you can convert it to a dict using the Tag names as keys:
tag_dict = dict((tag_matches[i].name, tag_matches[i]) for i in range(0, len(tag_matches)))
I have a list of country names in different languages that I am attempting to sort by their country name. Currently, the sort is based on the index value.
Here is my truncated list of country names:
ADDRESS_COUNTRY_STYLE_TYPES = {}
for language_code in LANGUAGES.iterkeys():
ADDRESS_COUNTRY_STYLE_TYPES[language_code] = OrderedDict()
if 'af' in LANGUAGES.iterkeys():
ADDRESS_COUNTRY_STYLE_TYPES['af'][0] = " Kies 'n land of gebied" # Select a country or territory
ADDRESS_COUNTRY_STYLE_TYPES['af'][1] = "Afganistan" #Afghanistan
ADDRESS_COUNTRY_STYLE_TYPES['af'][2] = "Åland" #Aland
ADDRESS_COUNTRY_STYLE_TYPES['af'][3] = "Albanië" #Albania
....
ADDRESS_COUNTRY_STYLE_TYPES['af'][14] = "Australië" #Australia
ADDRESS_COUNTRY_STYLE_TYPES['af'][15] = "Oostenryk" #Austria
ADDRESS_COUNTRY_STYLE_TYPES['af'][16] = "Aserbeidjan" #Azerbaijan
ADDRESS_COUNTRY_STYLE_TYPES['af'][17] = "Bahamas" #Bahamas
ADDRESS_COUNTRY_STYLE_TYPES['af'][18] = "Bahrein" #Bahrain
ADDRESS_COUNTRY_STYLE_TYPES['af'][19] = "Bangladesj" #Bangladesh
ADDRESS_COUNTRY_STYLE_TYPES['af'][20] = "Barbados" #Barbados
ADDRESS_COUNTRY_STYLE_TYPES['af'][21] = "Wit-Rusland" #Belarus
ADDRESS_COUNTRY_STYLE_TYPES['af'][22] = "België" #Belgium
....
Here is my code I have in my views.py file that calls the country names:
def get_address_country_style_types(available_languages, with_country_style_zero=True):
address_country_style_types = {}
preview_labels = {}
for code, name in available_languages:
address_country_style_types[code] = ADDRESS_COUNTRY_STYLE_TYPES[code].copy()
if not with_country_style_zero:
address_country_style_types[code].pop(0)
preview_labels[code] = ADDRESS_DETAILS_LIVE_PREVIEW_LABELS[code]
# in case preview labels are not defined for the language code
# fall back to 'en', which should always be there
if len(preview_labels[code]) == 0:
preview_labels[code] = ADDRESS_DETAILS_LIVE_PREVIEW_LABELS['en']
address_country_style_types = sorted(address_country_style_types, key=lambda x:x[1])
return address_country_style_types, preview_labels
The above code only returns the index number in the html drop down list. The issue is with the following line of code (or more to the point my lack of knowledge of how to get it working):
address_country_style_types = sorted(address_country_style_types, key=lambda x:x[1])
How do I return the sorted country list ? Am I using lambda in the correct way here? Should I be using lambda here?
I have been working on this over several days, my coding skills are not very strong, and I have read many related posts to no avail, so any help is appreciated.
I have read this blog about sorting a list of multilingual country names that appear in a form HTML select drop down list - which is essentially what I am attempting to do.
EDIT
Commenting out the line of code below in the code above does return a list of country names, but the country names are sorted by the index value not the country name.
address_country_style_types = sorted(address_country_style_types, key=lambda x:x[1])
I have failed to sort the multi-language country names programatically.
Instead, I copied the list into excel and hit the sort button (based on the translated country name - the index value stays uniform), then copied the data back to the file. Works as expected - just a lot of work.
I hope that this helps someone.
Is there a way of getting all the unique keyword index i.e. Subject in Plone by querying the catalog?
I have been using this as a guide but not yet successful.
This is what I have so far
def search_content_by_keywords(self):
"""
Attempting to search the catalog
"""
catalog = self.context.portal_catalog
query = {}
query['Subject'] = 'Someval'
results = catalog.searchResults(query)
return results
Instead of passing the keyword, I want to fetch all the keywords
catalog = self.context.portal_catalog
my_keys = catalog.uniqueValuesFor('Subject')
reference: http://docs.plone.org/develop/plone/searching_and_indexing/query.html#unique-values