So I'm designing a website with Django that does some heavy scraping based on user input.
This can take up to 5-6 secs and while I am working on cutting that down I would like for some kind of a loader to show up while the backend is scraping.
I have put a loader as you normally would using CSS and JavaScript inside the template but that only pops up when the template is actually loading and not when the view is scraping to gather data for the template.
Tried this in Django:
def scrape(request):
render(request,'loader.html')
*do scraping*
return render(request,'results.html',scraped_data)
The method I would use to solve this is to fire an async function within the scrape view.
#shared_task
def do_the_needful():
return "hello I am doing the needful"
def scrape_result(request, scrape_id):
result = AsyncResult(scrape_id).get()
# convert result to json or some other web format
return result_as_json
def scrape(request):
scrape_request_id = do_the_needful.submit()
return render(request, "scrape.html", context={"scrape_request_id": scrape_request_id}
Then within the HTML you'll need to create some Javascript which will perform Ajax requests to the scrape_result view using the scrape_request_id in the context.
Related
I want to scrape a website using python and send as json the information that I got to my flutter application without saving it to the database.
I don't want to use the Django Restframework database. I want to send a string to the backend and trigger a function that scrapes a certain website and sends back a json to the client. Without using the database.
For simple usage, you can setup a view function, bind it to a url that will accept your string, and inside the function build the logic to scrape the third-party page and return a JsonResponse
Something like below:
in urls.py:
urlpatterns = [
path('my_scrape/<str:input_string>', views.scrape)
]
in views.py:
def scrape(request, input_string):
#scrape a website
url = 'http://google.com/{}'.format(input_string)
third_party_data = BeautifulSoup(requests.get(url).content)
my_payload = { 'data1' : third_party_data.find(text='data1') }
return JsonResponse(my_payload)
Note that this doesn't account for anything like errors and timeouts from the third-party, you will have to handle those errors in your view function, depending on what you receive.
I have to create a small web app in Flask which contains an API and also an interface and I'm facing the following problem:
This would be how i handle a GET request:
#app.route('/member/<id>', methods=['GET'])
def member_get(id):
member = cursor.execute(f"select * from members where id={id}").fetchone()
if member is not None:
return to_json(member), 200
else:
return 'Not found', 404
And I would like to create some small forms with which I could do GET,POST,PUT,DELETE operations.
This would be how I get the data from the form:
#app.route('/dashboard', methods=['POST'])
def dashboard_post():
id = request.form['get_id']
return redirect(url_for("member_get",id=id))
My question is how can I get the data from the API method without actually redirecting to that page?
More precise, can I call somehow redirect(url_for("member_get",id=id)) and get the response data directly? (if I print the return of the redirect method it only shows the request status)
I assume one solution would be using the requests module, but is there a way to do it directly in Flask?
First of all, an API should always return a response in a format that is consistent and predictable. The code for member_get returns JSON in case of success but plain text in case of failure. This is not okay. You should return JSON always, with the appropriate HTTP status code. Here you are using 404 to express Not found, this is good.
You can use the jsonify function in Flask for that. And maybe normalize the response, so that is always has the same shape, whether the member ID is found or not.
Otherwise, parsing the response from your API will be harder because it is not consistent depending on the scenario.
Second point, if I understand it right: if you want to invoke the route '/member/' from within your API, you could simply do:
return member_get(id)
You call the function that is attached to the route, not the route itself. Of course you could actually fetch the page with the requests module but this is unnecessary, since the function is available internally.
Your question is not clear, but as I understand, (1) you think you have to get your form data from one view and send it to another view to do operations. (2) you are not familiar with flask request and flask-wtf. and maybe (3) looking for a way to do this without refreshing or redirecting the page.
You don't need to separate your GET and POST methods. instead you can integrate both in one view.
#app.route('/member/<id>', methods=['GET', 'POST])
To handling data, you can use flask request.
from flask import request
and access to data in your view like this:
id = request.form.get("idField")
but you can also use Flask-WTF to simply make and handle forms.
with Flask-WTF your view would be like this:
from app.forms import SearchForm
#app.route('/your-endpoint', methods=['GET', 'POST'])
def yourView():
form = your_form()
if form.validate_on_submit():
id=form.idField.data
return render_template('test.html', form=form)
the condition form.validate_on_submit() checks if you are submitting a from or you just opened it. if you submit a form and it's data are valid based on validators defined in your form, the code runs. else just renders the template and returns the page.
To learn how to make forms with Flask-WTF I recommend reading this article:
If you don't want to refresh the page or redirect it after submitting the form, you can use AJAx on your page.
I am working on a reverse blog project where users can like and comment.
Although I successfully implement the like button to work, But it refreshes the page.
My problem is I am trying to convert it to rather use Ajax to power the button.
And to as well uses a flag to decide which button to be shown, like or unlike.
Please I need help or the logic to guide me on converting the button to work on using Ajax
Please Note: I use pagination and the like button is on the list view and detail view (CBV).
Thank you in advance.
here is my view for the button
here is my model
here is my template rendering
Set a unique id for each like button if you have multiple like buttons. Bind the click event to JS function in which you will create an ajax call. Pass the ID of your model object (to increment likes count) to your JS function which will be passed in your ajax call to your view. Specify the URL of your view at which your like increment function resides.
Below is the implementation of ajax call on button click in django
https://stackoverflow.com/a/13733855/11979793
UPDATE: Class based view changes for the ajax
In your class-based view function:
Accept the parameters:
model_id = json.loads(request.POST.get('param_name')) # model_id could be your post ID to increase the likes
instead of render return JSON response for ajax:
return JsonResponse({
'like_count': model_to_dict(total_likes), # if you want to display total likes
'success': True
})
You can use total likes in success function of ajax to display it in your template.
P.S. import json at start of the file
UPDATE: example class-based view to handle ajax POST request
class myClassView(View):
template_name='index.html'
def post(self,request): # overriding the post method
myparam = json.loads(request.POST.get('paramFromAjax'))
# perform your actions here
# make a dictionary/context say mydic and pass parameters in that
return JsonResponse({
'paramToAjax': mydic,
'success': True
})
I am trying to implement a website that uses RSS Feeds.
My use case is to be able to display one RSS feed at a time, then loop over to the next maybe after 5 seconds. These RSS feeds need to be displayed on the same placeholder.
I am new to django cms in case you need more info please let me know.
I have googled a lot but all I can see is how to add plugins from the frontend and they will all automatically show inside the placeholder.
Or modify the render method of the custom plugin class to display what you want.
But I want to display everything but one at a time in a continuous manner
#plugin_pool.register_plugin
class ExternalArticlePlugin(CMSPluginBase):
model = ExternalArticle
name = _("External article")
render_template = "external_article.html"
cache = False
def render(self, context, instance, placeholder):
context = super(ExternalArticlePlugin, self).render(
context, instance, placeholder
)
return context
I expect to display one RSS feed at a time inside my placeholder.
Those feeds are links to the actual webpage with more info.
one way would be to write a function random_rss_feed_url() in ExternalArticle Model which renders a random rss instance.
model.py
class ExternalArticle(models.Model):
def random_rss_feed_link(self):
return ExternalArticle.objects.order_by('?')[0].link
then you do in plugins external_article.html:
{{ instance.random_rss_feed_link }}
Edited:
if you want to change automatically without page reload, then you need something like this in javascript in your template:
var rss_links = ['link_1', 'link_2', 'link_3'];
setInterval(function() {
// take random rss link
var random_link = rss_links[Math.floor(Math.random()*rss_links.length)];
// change the link
document.getElementById('rss_link_element').href = random_link;
}, 5000);
I created web application in Python, Django framework. Web application gets data from MongoDB database, it gets about 10000 documents from MongoDB database and works very slow. Now looking for ways to speed up my web app. So, is it possible to store result from method render_to_response as HTML temporarily on server? It looks like this:
I have a HTML form; when the user types data in the form and clicks on submit button, web app executes a view that gets data from the Mongo database, and sends that data via variable mongo_data to home.html:
return render_to_response('home.html', {'mongo_data': mongo_data, 'request': request},
context_instance=RequestContext(request))
home.html show data that are stored in variable mongo_data. In the web application, I have a lot of identical queries, for identical queries I get identical result for home.html. So I want to store home.html to folder for example temp on server, and when user types data in the HTML form and clicks the submit button, first to check is if home.html for his data is in temp folder; if it is, then load that home.html, if not then go to view which will generate new home.html with specific mongo_data.
If this is possible , it will speed up my web app a lot ..
The Django caching framework was made exactly for this purpose; see https://docs.djangoproject.com/en/dev/topics/cache/.
In your case, you either cache the entire view for a period of time:
#cache_page(60 * 15)
def my_mongo_view(request):
return render_to_response('home.html', ...)
(From https://docs.djangoproject.com/en/dev/topics/cache/#the-per-view-cache)
or you use the Low-level Cache API (https://docs.djangoproject.com/en/dev/topics/cache/#the-low-level-cache-api):
from django.core.cache import cache
def my_mongo_view(request):
ret = cache.get('home-rendered')
if ret is None:
ret = render_to_response('home.html', ...)
cache.set('home-rendered', ret)
return ret
You'll find more options for caching (e.g. inside your templates) if you just read the doc.
P.S. you can also parametrize your cache by a variable or user ID or something else.