I have been practicing Django for a while now. Currently I am using it in a project where I'm fetching Facebook data via GET requests and then saving it to an sqlite database using Django models. I would like to know how can I improve the following code and save a list of Facebook posts and their metrics efficiently. In my current situation, I am using a for loop to iterate on a list containing several Facebook Posts and their respective metrics which is then associated to the specific Django model and finally saved.
def save_post(post_id, page_id):
facebook_post = Post(post_id=post_id,
access_token=fb_access_token)
post_db = PostsModel(page_id=page_id, post_id=post.post_id)
post_db.message = facebook_post.message
post_db.story = facebook_post.story
post_db.full_picture = facebook_post.full_picture
post_db.reactions_count = facebook_post.reactions_count
post_db.comments_count = facebook_post.comments_count
post_db.shares_count = facebook_post.shares_count
post_db.interactions_count = facebook_post.interactions_count
post_db.created_time = facebook_post.created_time
post_db.published = facebook_post.published
post_db.attachment_title = facebook_post.attachment_title
post_db.attachment_description = facebook_post.attachment_description
post_db.attachment_target_url = facebook_post.attachment_target_url
post_db.save()
post_db is a Django model object instantiated using PostsModel while Post is a normal Python Class which I wrote. The latter is simply a collection of GET requests which fetches data from Facebook's Graph API and returns JSON data whereby I associate relevant data to class attributes (message, 'shares_count`).
I read about the bulk_create function from Django's documentation but I don't know how to pass on the above. I also tried using multiprocessing and Pool but the above function does execute. Right now, I am just iterating sequentially on a list. As the list increases in length, it takes more time to save.
def create(self, request):
page_id = request.data['page_id']
page = get_object_or_404(PagesModel, pk=page_id)
post_list = get_list_or_404(PostsModel, page_id=page_id)
for post_id in post_list:
save_post(post_id=post_id, page_id=page)
The above function gets an already saved list from the database for a specific page based on the page_id. Then, the for loop iterates on each post in the list and its post_id and page instance are sent to the save_post function to fetch its data and save it.
Huge thanks if anyone can suggest a more effective way to tackle this. Thank you.
You are going in the right direction with the bulk_load. Generate a list of the PostsModel objects and then use bulk_create to upload them into the database. An important note here is that it won't work if the posts already exist in the database. For updating posts, try the bulk_update.
def save_post(post_id, page_id):
facebook_post = Post(post_id=post_id,
access_token=fb_access_token)
post_db = PostsModel(page_id=page_id, post_id=post.post_id)
post_db.message = facebook_post.message
post_db.story = facebook_post.story
post_db.full_picture = facebook_post.full_picture
post_db.reactions_count = facebook_post.reactions_count
post_db.comments_count = facebook_post.comments_count
post_db.shares_count = facebook_post.shares_count
post_db.interactions_count = facebook_post.interactions_count
post_db.created_time = facebook_post.created_time
post_db.published = facebook_post.published
post_db.attachment_title = facebook_post.attachment_title
post_db.attachment_description = facebook_post.attachment_description
post_db.attachment_target_url = facebook_post.attachment_target_url
return post_db
def create(self, request):
page_id = request.data['page_id']
page = get_object_or_404(PagesModel, pk=page_id)
post_list = get_list_or_404(PostsModel, page_id=page_id)
post_model_list = [save_post(post_id=post_id, page_id=page) for post_id in
post_list]
PostsModel.objects.bulk_create(post_model_list]
Related
I'm trying to learn Django and built a small app that contains a list of companies with some general info for each of them. The home page shows a list of all companies and then a user can click on the company name to see more info. I'm now trying figure out how APIs are consumed in Django by using Yahoo Finance to get some stock data about the company and display it on the page. (I've used the yahoo-finance package multiple times in Python and it's pretty straight forward, which is why I've stared with this). I don't need to save the data to a database (unless that's the only way), I simply want to display it.
I've pip installed the packaged and added it to the APPS in the settings.py file.
Then in my views.py I've added the yahoo-finance dependencies and tried to work in the API in the code below. Then in the template I'm trying to use {{ mkt_cap }}. Doing it this way I'm getting a YQLResponseMalformedError. I realize this may not be the correct way to go about it, but I'm having a hard time figuring it out.
from django.views import generic
from .models import Company, Articles, Transcripts, TranscriptDetails
from yahoo_finance import Share
import json
class CompanyDetails(generic.DetailView):
model = Company
template_name = 'company_details.html'
def get_context_data(self, **kwargs):
pk = self.kwargs.get('pk')
context = super(CompanyDetails, self).get_context_data(**kwargs)
context['articles'] = Articles.objects.filter(company_id=pk).order_by('-date')
context['company'] = Company.objects.get(id=pk)
context['transcripts'] = Transcripts.objects.filter(company_id=pk).order_by('-date')
# Get Yahoo API data
stock_symbol = Company.objects.filter(id=pk).values_list('stock_symbol', flat=True)
data = Share(stock_symbol)
data = json.load(data)
context['mkt_cap'] = data
return context
EDIT
Here's the final code in case someone else has a similar question. I've kept all of the API calls in the View and created a dictionary to pass them to the template.
class CompanyDetails(generic.DetailView):
model = Company
template_name = 'company_details.html'
def get_context_data(self, **kwargs):
pk = self.kwargs.get('pk')
context = super(CompanyDetails, self).get_context_data(**kwargs)
context['articles'] = Articles.objects.filter(company_id=pk).order_by('-date')
context['transcripts'] = Transcripts.objects.filter(company_id=pk).order_by('-date')
# Get Yahoo API data
stock_symbol = self.object.stock_symbol
data = Share(stock_symbol)
stock_open = data.get_open()
year_range = data.get_year_range()
fifty_day_moving_average = data.get_50day_moving_avg()
market_cap = data.get_market_cap()
yahoo_finance = dict()
yahoo_finance['stock_open'] = stock_open
yahoo_finance['year_range'] = year_range
yahoo_finance['fifty_day_moving_average'] = fifty_day_moving_average
yahoo_finance['market_cap'] = market_cap
context['yahoo_finance'] = yahoo_finance
return context
Then in the template to access these I similar use the following:
{{ yahoo_finance.stock_open }}
{{ yahoo_finance.year_range }}
{{ yahoo_finance.fifty_day_moving_average }}
{{ yahoo_finance.market_cap }}
If you want the market cap, you can just do
data = Share(stock_symbol)
market_cap = data.get_market_cap()
context['mkt_cap'] = market_cap
It looks data is a yahoo_finance.Share object that you can use directly. You can look inside with data.__dict__ but should probably just use the methods documented in the API.
Alternatively, you can likely just pass data directly into the template and use {{data.get_market_cap}}.
Hope this helps.
The values_list method returns a list. You are passing this to Share, which expects a string.
The simplest fix would be to change it to:
stock_symbol = Company.objects.filter(id=pk).values_list('stock_symbol', flat=True)[0]
data = Share(stock_symbol)
However, you don't have to fetch the company from the database at all -- the DetailView does this for you. Therefore, you can use self.object.stock_symbol to get the stock symbol
def get_context_data(self, **kwargs):
context = super(CompanyDetails, self).get_context_data(**kwargs)
context['articles'] = Articles.objects.filter(company=self.object).order_by('-date')
context['transcripts'] = Transcripts.objects.filter(company=self.object).order_by('-date')
# Get Yahoo API data
stock_symbol = self.object.stock_symbol
data = Share(stock_symbol)
data = json.load(data)
context['mkt_cap'] = data
return context
First of all, I'm very new to Django world, there could be a similar question, however i did not find a satisfactory answer.
Here is my scenario, i have few external REST endpoints, which I will hit from my Django app and get say 100-key JSON response. Now, when I'm writing my API in Django app, this response i'll have to trim and send it to outer world. Say for example,
My API is,
GET /api/profiles/1472
which will give user profile with id 1472. Now, this API will inturn call some other REST endpoint and fetch actual profile's data. So, in a way I'm writing a proxy endpoint. This proxy endpoint is supposed to trim out some fields and give it back to caller.
I've not written model classes for this.
What are best ways to achieve this in Django?
Edit 1:
Sample view will be like this,
class GetCompetitorProductsView(APIView):
"""
Get Competitor products view
"""
def post(self, request, format=None):
# I'll be having a list of fields to be trimmed from response.
# It will be separate for every API.
data = request.data
error_checks = system_errors.check_for_competitor_products_input_error(data)
if not error_checks:
response = call_to_rest(data)
return Response(response)
else :
return Response(error_checks, status = status.HTTP_412_PRECONDITION_FAILED)
And one more thing, same behavior is applied to all other APIs. So, I need more generic solution which can be easily applied to other APIs.
Basically this is how to filter in python
allowed_fields = ("first_name", "last_name", "email")
user_info = call_rest_endpoint(id=1472)
result = {key:value for key,value in user_info.items() if key in allowed_fields}
First line define what fields u want to return.
Second line call the endpoint and get the data from theird party API.
Third line consist of 3 statements.
user_info.items() convert dictionary into array key/values paris.
Build dictionary from these tuples
but only if the key was found in allowed_fields tuple
You can create function or mixin that you will put in parents of your view and then use it method for trimming. Here is example
class TrimDataMixin(object):
ALLOWED_FIELDS = None
def trim_data(self, data):
allowed_fields = self.ALLOWED_FIELDS or []
return {k: v for k, v in data.items() if k in allowed_fields}
class GetCompetitorProductsView(TrimDataMixin, APIView):
"""
Get Competitor products view
"""
ALLOWED_FIELDS = ['first_name', 'last_name']
def post(self, request, format=None):
# I'll be having a list of fields to be trimmed from response.
# It will be separate for every API.
data = request.data
error_checks = system_errors.check_for_competitor_products_input_error(data)
if not error_checks:
response = call_to_rest(data)
# trim data
response = self.trim_data(response)
return Response(response)
else:
return Response(error_checks, status = status.HTTP_412_PRECONDITION_FAILED)
I am trying to figure out the best method of accessing all the matching microseries found in the Assessment class (table) in my database. I am trying to create a link that will send a user to those matching microseries. I am new and want to better understand the nuts and bolts of front-end engineering. I also have not seen a similar question asked here on Stacks.
I am not using JSON or XML yet, so this is a static HTML process. I have a few routes that access assessments at the moment, e.g.:
config.add_route('assessments', '/assessments')
config.add_route('assessment', '/assessments/{id:\d+}')
What I would like to better understand while implementing a method of finding matching microseries in the Assessment table and sending the user to a new page with those matching series:
How routes work, especially when accessing an attribute of a class, e.g. Assessment.microseries.
The goal of View code to convey the method mentioned above.
Pyramid links and Pyramid on Routes and URL Dispatch
Using: Python 2.7, SQLAlchemy, Pyramid
Assessment table:
class Assessment(Base):
__tablename__ = 'assessments'
id = Column(Integer, primary_key=True)
name = Column(String(50), unique=True)
text = Column(String(2000))
microseries = Column(Integer)
# more code
def __init__(self, name, text, user, video, categories, microseries):
# more code
API for interacting with an assessment is based on CRUD- create, retrieve, update and delete.
View Code:
this is not doing what I need it to do as I don't have a form of link to send the user to the matching series, e.g. Link1 would send a user to a new view with GET for all subseries of Link1: 1a, 1b, 1c....
#view_config(route_name='assessments', request_method='GET', renderer='templates/unique_assessments.jinja2', permission='create')
def view_unique_microseries_group(request):
logged_in_userid = authenticated_userid(request)
if logged_in_userid is None:
raise HTTPForbidden()
all_assessments = api.retrieve_assessments() #all assessments in a list
assessments_by_microseries = {} #dictonary
for x in all_assessments:
if x.microseries in assessments_by_microseries:
print("Already seen this microseries: %s" % x.microseries)
else:
assessments_by_microseries[x.microseries] = x
unique_assessments = sorted(assessments_by_microseries.values()) #.values() method to get the, err, values of the dict.
print 'unique_assessments:', unique_assessments
#a = HTTPSeeOther(location=request.route_url('view_microseries'))
return {'logged_in': logged_in_userid, 'unique_assessments': unique_assessments} #need to send some kind of list that can be shown on the template to send a user to the appropriately matching set of microseries
Whenever the flask server is started, all the forms are initialised at that time itself. The data is interconnected between the pages, so for one form, the choices come from the database and those choices can be edited using another form on another page but after the choices are updated on that page, they remain the same for the first form. To get the new values I need to restart the server. Is there any way to refresh the values without restarting the server?
This is how the form looks like
class AddExpenses(Form):
reason = wtforms.StringField('reason', [validators.Required()])
amount = wtforms.IntegerField('amount', [validators.Required()])
allnames = []
allnames = getSalesman()
salesperson = wtforms.SelectField('salesperson', choices=[names for names in allnames])
submitfield = wtforms.SubmitField('Submit')
getSalesman() function is used to query the database and get the choices.
Why not get the choices in the view function that requires the form? For example
#app.route('/index')
def index():
form = AddExpenses()
allnames = getSalesman()
form.salesperson.choices = [names for names in allnames]
...
You may also want to use some caching on getSalesman() if the database will not change often.
I'm using the App Engine extension of WTForms to generate forms from my datastore models. This works great for adding new entities, but I would also like to be able to use the forms to edit existing entities.
Is it possible to load an existing datastore entity into a form created with model_forms from a GAE datastore model? If so, how do I do this? If not, what approach should I take to accomplish this?
This is my version of updating the google datastore.
class AdminBlogEdit(MethodView):
def __init__(self):
self.blog_form = NewBlogEntryForm(csrf_enabled=False)
def get(self,blog_key_id=None):
if blog_key_id:
self.blog_model = BlogEntryModel.get_by_id(blog_key_id)
self.blog_form = NewBlogEntryForm(obj = self.blog_model)
return render_template('admin/blog_edit.html', form=self.blog_form)
def post(self,blog_key_id=None):
if self.blog_form.validate():
self.update_post(blog_key_id)
self.blog_model.put()
return redirect(url_for(".admin"))
else:
return render_template('admin/blog_edit.html', form=self.blog_form)
return redirect(url_for(".admin"))
def update_post(self,blog_key_id):
if blog_key_id:
self.blog_model = BlogEntryModel.get_by_id(blog_key_id)
self.blog_form.populate_obj(self.blog_model)
else:
self.blog_model = BlogEntryModel(title = self.blog_form.title.data, date_created = self.blog_form.date_created.data,
entry = self.blog_form.entry.data)
The main idea is to retrieve the datastore entity and fill the form data, before displaying the GET request.
For the PUT request, retrieve the data store entity again and update it with the form data and then call the datastoremodel.put() on it