I want to scrape a website using python and send as json the information that I got to my flutter application without saving it to the database.
I don't want to use the Django Restframework database. I want to send a string to the backend and trigger a function that scrapes a certain website and sends back a json to the client. Without using the database.
For simple usage, you can setup a view function, bind it to a url that will accept your string, and inside the function build the logic to scrape the third-party page and return a JsonResponse
Something like below:
in urls.py:
urlpatterns = [
path('my_scrape/<str:input_string>', views.scrape)
]
in views.py:
def scrape(request, input_string):
#scrape a website
url = 'http://google.com/{}'.format(input_string)
third_party_data = BeautifulSoup(requests.get(url).content)
my_payload = { 'data1' : third_party_data.find(text='data1') }
return JsonResponse(my_payload)
Note that this doesn't account for anything like errors and timeouts from the third-party, you will have to handle those errors in your view function, depending on what you receive.
Related
I have to create a small web app in Flask which contains an API and also an interface and I'm facing the following problem:
This would be how i handle a GET request:
#app.route('/member/<id>', methods=['GET'])
def member_get(id):
member = cursor.execute(f"select * from members where id={id}").fetchone()
if member is not None:
return to_json(member), 200
else:
return 'Not found', 404
And I would like to create some small forms with which I could do GET,POST,PUT,DELETE operations.
This would be how I get the data from the form:
#app.route('/dashboard', methods=['POST'])
def dashboard_post():
id = request.form['get_id']
return redirect(url_for("member_get",id=id))
My question is how can I get the data from the API method without actually redirecting to that page?
More precise, can I call somehow redirect(url_for("member_get",id=id)) and get the response data directly? (if I print the return of the redirect method it only shows the request status)
I assume one solution would be using the requests module, but is there a way to do it directly in Flask?
First of all, an API should always return a response in a format that is consistent and predictable. The code for member_get returns JSON in case of success but plain text in case of failure. This is not okay. You should return JSON always, with the appropriate HTTP status code. Here you are using 404 to express Not found, this is good.
You can use the jsonify function in Flask for that. And maybe normalize the response, so that is always has the same shape, whether the member ID is found or not.
Otherwise, parsing the response from your API will be harder because it is not consistent depending on the scenario.
Second point, if I understand it right: if you want to invoke the route '/member/' from within your API, you could simply do:
return member_get(id)
You call the function that is attached to the route, not the route itself. Of course you could actually fetch the page with the requests module but this is unnecessary, since the function is available internally.
Your question is not clear, but as I understand, (1) you think you have to get your form data from one view and send it to another view to do operations. (2) you are not familiar with flask request and flask-wtf. and maybe (3) looking for a way to do this without refreshing or redirecting the page.
You don't need to separate your GET and POST methods. instead you can integrate both in one view.
#app.route('/member/<id>', methods=['GET', 'POST])
To handling data, you can use flask request.
from flask import request
and access to data in your view like this:
id = request.form.get("idField")
but you can also use Flask-WTF to simply make and handle forms.
with Flask-WTF your view would be like this:
from app.forms import SearchForm
#app.route('/your-endpoint', methods=['GET', 'POST'])
def yourView():
form = your_form()
if form.validate_on_submit():
id=form.idField.data
return render_template('test.html', form=form)
the condition form.validate_on_submit() checks if you are submitting a from or you just opened it. if you submit a form and it's data are valid based on validators defined in your form, the code runs. else just renders the template and returns the page.
To learn how to make forms with Flask-WTF I recommend reading this article:
If you don't want to refresh the page or redirect it after submitting the form, you can use AJAx on your page.
So I'm designing a website with Django that does some heavy scraping based on user input.
This can take up to 5-6 secs and while I am working on cutting that down I would like for some kind of a loader to show up while the backend is scraping.
I have put a loader as you normally would using CSS and JavaScript inside the template but that only pops up when the template is actually loading and not when the view is scraping to gather data for the template.
Tried this in Django:
def scrape(request):
render(request,'loader.html')
*do scraping*
return render(request,'results.html',scraped_data)
The method I would use to solve this is to fire an async function within the scrape view.
#shared_task
def do_the_needful():
return "hello I am doing the needful"
def scrape_result(request, scrape_id):
result = AsyncResult(scrape_id).get()
# convert result to json or some other web format
return result_as_json
def scrape(request):
scrape_request_id = do_the_needful.submit()
return render(request, "scrape.html", context={"scrape_request_id": scrape_request_id}
Then within the HTML you'll need to create some Javascript which will perform Ajax requests to the scrape_result view using the scrape_request_id in the context.
I am unable to send the user details along with requests module i had to hard code the user details in the data payload to identify the user.
full_url = ''.join(['http://', get_current_site(request).domain, '/am/reply'])
data = {
'agent_type':'trigger',
'input':platform,
'userid':request.user.id ####==>> had to send userid like this
}
a = requests.get(full_url,params=data)
Is there way to send all general request data using requests.?
And moreover the requests url the destination view i have implemented
def index(request):
if not request.user.is_authenticated:
return HttpResponseRedirect(reverse('login'))
And request.user.id is none when url is reached through requests module
In general how should i validate a request when using requests module
Django uses request and response objects to pass state through the system.
When a page is requested, Django creates an HttpRequest object that contains metadata about the request. Then Django loads the appropriate view, passing the HttpRequest as the first argument to the view function. Each view is responsible for returning an HttpResponse object.
Some of the middleware included in Django’s contrib apps set attributes on the request. If you don’t see the attribute on a request, be sure the appropriate middleware class like authenticationmiddleware,sessionmiddleware.
Following piece of code will give the user.id if and only if the user is authenticated.
def myview(request):
if request.user.is_authenticated:
print request.user.id
else:
... # Do something else.
https://docs.djangoproject.com/en/1.10/ref/request-response/
If I understood your question correctly, You are getting request in one view, and then making a call to other view using requests module. It that case the request object in index view will be totally different because that request was sent from your server where application works, not from user. You can only get data in index view using request.GET.get("userid") and so on. And then if you will need user info, just fetch it again from database using userid. Passing request object to other view using requests library is not possible.
I am trying to make a query system for my website, i think the best way and the most compact would be to assign search variable using url pattern.
So for example, i want to search objects of model User:
User sends HttpRequest to following url:
https://127.0.0.1/search/q="admin"
Now HttpRequest is also sent to search view, we somehow get q variable data.
def search(request):
for query in User.objects.all():
if q in query: # < We somehow need to get data of 'q'.
return HttpResponse(q)
Since i have admin in User.objects.all(), this should return HttpResponse of 'admin'.
How can this url pattern be made? So i can assign q variable from the url and then send it to system to find it?
I have problems with this URL:
https://127.0.0.1/search/q="admin"
There is no ? in the URL, so there is no query string, it's all part of the "path". Using characters like = and " in there will confuse a lot of things, if it works at all.
Either just do
https://127.0.0.1/search/admin
With an URL pattern like r'^search/(?P<querystring>.+)$', or
https://127.0.0.1/search/?q=admin
In this case the query string will be in request.GET['q']; it's also possible to use Django forms to process query parameters (e.g. for validating them).
You can capture named strings from URLs like this:
urls.py:
urlpatterns = [
url(r'^blog/page(?P<num>[0-9]+)/$', views.page),
]
views.py:
def page(request, num="1"):
tl:dr
How would a hosted django app correctly transform resource paths to match any hosted location (/ or /test or /testtest)?
Full Description
Let me try to explain what I am trying to do.
I am trying to write a somewhat re-usable django app which I intend to use from within multiple projects. This app is called systemstatus.
The systemstatus app provides a page under '$^' which provides a simple interface to query the system status.
This page makes an ajax query back to the systemstatus app to determine the actual system status and report it on the UI.
The systemstatus app provides a location '^service/$' which points to the ajax call handler.
This page has to somehow figure out the correct URI for the ajax handler depending on where this app is hosted (e.g. under / or /status or /blahblah).
I am wondering what an ideal way of doing this would be. I would say that this applies to other resources bundled inside the app too (stylesheets, images).
Right now I am using request.path to determine what the target path should be. This path is then passed down as a parameter to the template. But this approach will soon become too cumbersome to handle.
def system_status (request):
queryPath = request.path + "service/"
return render_to_response ('systemstatus.html', {'queryPath': queryPath})
My page template looks like this:
function do_ajax () {
$.getJSON ('{{ queryPath }}', function (data) {
$("#status").html (data.status);
});
}
Thanks!
You shouldn't hardcode your urls like that, but use reverse instead!
Django also has a built-in template tag to reverse urls. So you could do something like
function do_ajax () {
$.getJSON ('{% url path.to.my_ajax_view %}', function (data) {
$("#status").html (data.status);
});
}
directly in your template!
You can also send the ajax request directly to your current page's url and check if it is an ajax request or not:
def my_view(request):
if request.is_ajax():
# generate response for your ajax script
else:
# generate the response for normal request
# (render template of your page)