Sometimes I want to get from a resource the full response and sometimes I want it with pagination. Until now I was only able to use either the one or the other.
But isn't there a way to set flask-restless to use both depending on the paramters i pass on the GET request?
If I want to disable pagination for a specific resource I change the settings like this:
manager.create_api(someresource, methods=['GET'], results_per_page=None)
But now pagination is completly disabled and that's not the behaviour I wish.
And if pagination is enabled as default it returns only the first page.
Isn't there a way to tell flask-restless to get only the first page if I specifically pass the page 1 in the query string like so:
GET http://someaddress/resource?page=1
I was actually able to solve the problem using a loop but I don't think it is a nice solution because I have to use multiple requests.
I requested the resource and fetched the total_pages and then I ran a loop to total_pages and passed each iteration as an argument in the query string for another new request to fetch each page:
i = 1
while i <= response.total_pages:
page_response = requests.get("http://someurl/someresource?page=" + str(i))
...
But I don't think it is a nice way to solve that issue. If there is a possibility to change the settings on flask-restless to fetch only the first page if it is passed as an argument in the query string then I would be more than happy but if there is still another way to use both then it's also good.
You can get the behaviour you want by disabling pagination with:
manager.create_api(someresource, methods=['GET'], results_per_page=0)
And then query the API with the results_per_page parameter like so:
GET http://someaddress/resource?results_per_page=2
The results_per_page parameter has to be a positive integer and will be your new page size. The parameter is further documented here.
Getting the full response without pagination is straight forward with this configuration. Just omit the results_per_page parameter:
GET http://someaddress/resource
Related
I want to page through the results from the Shopify API using the Python wrapper. The API recently (2019-07) switched to "cursor-based pagination", so I cannot just pass a "page" query parameter to get the next set of results.
The Shopify API docs have a page dedicated to cursor-based pagination.
The API response supposedly includes a link in the response headers that includes info for making another request, but I cannot figure out how to access it. As far as I can tell, the response from the wrapper is a standard Python list that has no headers.
I think I could make this work without using the python API wrapper, but there must be an easy way to get the next set of results.
import shopify
shopify.ShopifyResource.set_site("https://example-store.myshopify.com/admin/api/2019-07")
shopify.ShopifyResource.set_user(API_KEY)
shopify.ShopifyResource.set_password(PASSWORD)
products = shopify.Product.find(limit=5)
# This works fine
for product in products:
print(product.title)
# None of these work for accessing the headers referenced in the docs
print(products.headers)
print(products.link)
print(products['headers'])
print(products['link'])
# This throws an error saying that "page" is not an acceptable parameter
products = shopify.Product.find(limit=5, page=2)
Can anyone provide an example of how to get the next page of results using the wrapper?
As mentioned by #babis21, this was a bug in the shopify python api wrapper. The library was updated in January of 2020 to fix it.
For anyone stumbling upon this, here is an easy way to page through all results. This same format also works for other API objects like Products as well.
orders = shopify.Order.find(since_id=0, status='any', limit=250)
for order in orders:
# Do something with the order
while orders.has_next_page():
orders = orders.next_page()
for order in orders:
# Do something with the remaining orders
Using since_id=0 will fetch ALL orders because order IDs are guaranteed to be greater than 0.
If you don't want to repeat the code that processes the order objects, you can wrap it all in an iterator like this:
def iter_all_orders(status='any', limit=250):
orders = shopify.Order.find(since_id=0, status=status, limit=limit)
for order in orders:
yield order
while orders.has_next_page():
orders = orders.next_page()
for order in orders:
yield order
for order in iter_all_orders():
# Do something with each order
If you are fetching a large number of orders or other objects (for offline analysis like I was), you will find that this is slow compared to your other options. The GraphQL API is faster than the REST API, but performing bulk operations with the GraphQL API was by far the most efficient.
You can find response header with below code
resp_header = shopify.ShopifyResource.connection.response.headers["link"];
then you can split(',') string of link index and then remove(<>) and can get next link url.
I am not familiar with python , but i think i will work, you can also review below links:
https://community.shopify.com/c/Shopify-APIs-SDKs/Python-API-library-for-shopify/td-p/529523
https://community.shopify.com/c/Shopify-APIs-SDKs/Trouble-with-pagination-when-fetching-products-API-python/td-p/536910
thanks
#rseabrook
I have exactly the same issue, it seems others do as well and someone has raised this: https://github.com/Shopify/shopify_python_api/issues/337
where I see there is an open PR for this: https://github.com/Shopify/shopify_python_api/pull/338
I guess it should be ready soon, so an alternative idea would be to wait a bit and use 2019-04 version (which supports the page parameter to perform pagination).
UPDATE: It seems this has been released now: https://github.com/Shopify/shopify_python_api/pull/352
I am trying to use "Filter" in request parameters while sending REST API request to AWS. Surprisingly, below request parameter just works:
request_parameters = 'Action=DescribeAvailabilityZones&Version=2016-11-15'
However, as soon as I change it to:
request_parameters = 'Action=DescribeAvailabilityZones&Filter.1.state=available&Version=2016-11-15'
I get, "The parameter state is not recognized"
I am picking up the Filter's syntax from here
Any suggestions please? Thanks in advance.
figured out the solution. The parameters list expects the filter to be passed in a key/value fashion. Below is the amendment which I found to be working:
request_parameters = 'Action=DescribeAvailabilityZones&Filter.1.Name=state&Filter.1.Value=available&Version=2016-11-15'
I also noticed that unless this option is present in the list of recognized filters, it wont work. This can be found here under specific Actions.
Also, filters tags bear relation with tags in XML response. for e.g. the filter to list state of an AvailabilityZone is "state" but in the XML response it is tagged as <zoneState>.
I want to make an HTTP GET request to ip:port/this1234 and use "1234" as a variable in Python code. The "1234" is an arbitrary int. How do I get the int as an argument to my view?
#app.route('/stop####')
def stop####():
global stopped
if stopped:
call(["./stop.sh"], shell = True)
stopped = False
return "started"
return "already started"
You'll want to take a look at the Quickstart guide.
Meanwhile, you can get POST data using
myvar = request.form["myvar"]
and GET using
myvar = request.args.get("myvar")
The guide then goes on to mention some error handling recommendations and references the more in depth request object page.
We recommend accessing URL parameters with get or by catching the KeyError because users might change the URL and presenting them a 400 bad request page in that case is not user friendly.
For a full list of methods and attributes of the request object, head
over to the request documentation.
You might also want to look at routing a bit. I'm uncertain what you're trying to accomplish with the pound sign in your routing (EDIT: I see what you mean on re-reading; see edit at bottom). Note the quote below from a comment on SitePoint.
browsers don't send the #whatever part of the URL to the server in the HTTP request when requesting the page
Ok, so if you want to pass the value in the URI, I recommend something more like: example.com/this/1234 and your routing rule would look like #app.route('/this/<myVar>') above your def my_func(myVar):
Finally, at some level, killing any process based off of an http request seems awful daring, but you know your environment best, and for all I know, it might not even be exposed to the internet. Goodluck, and don't forget to be safe about this.
I think the obvious way to do it would be:
#app.route('/kill/<int:pid>')
def kill(pid):
.....
There are lots of question discussing how to get 1 parameter, but how do you get all of the parameters, preserving their order?
There's this way: request.GET.get('q', '') to get 1 parameter.
I need to capture POST requests to my URL, then add a parameter to the URL, and send it right back to confirm it's validity and source. This is for PayPal IPN if you're wondering.
Thanks!
As #Daniel Roseman said you probably don't need to preserve the order, in which case you can just use the request.GET dict
Alternatively you can get the raw querystring:
request.META['QUERY_STRING']
https://docs.djangoproject.com/en/dev/ref/request-response/#django.http.HttpRequest.META
As Daniel Roseman mentions, order should not be significant between POST or GET request parameters; think of them like key-value pairs rather than a list.
If you want to maintain order, then perhaps pass a list as the value in your POST and grab it in Django:
myData = request.POST.get("myQuery")
Specifically, POST requests don't use the querystring* (see here). POSTs use the request body while GETs use the query string. Note that security-wise, this also means important client information isn't blatantly displayed in the URL -- which is especially important when dealing with payments.
Update: *Apparently, POSTs can use the query string, but they really shouldn't. See this SO post for more.
Yes, order is significant here. This is what I'm going to use:
newParameteres = 'cmd=_notify-validate&' + self.request.POST.urlencode()
req = urllib2.Request("http://www.paypal.com/cgi-bin/webscr", newParameteres)
I have a question about how Django's paginator module works and how to optimize it. I have a list of around 300 items from information that I get from different APIs on the internet. I am using Django's paginator module to display the list for my visitors, 10 items at a time. The pagination does not work as well as I want it to. It seems that the paginator has to get all 300 items before pulling out the ten that need to be displayed each time the page is changed. For example, if there are 30 pages, then going to page 2 requires my website to query the APIs again, put all the information in a list, and then access the ten that the visitor's browser requests. I do not want to keep querying the APIs for the same information that I already have on each page turn.
Right now, my views has a function that looks at the get request and queries the APIs for information based on the query. Then it puts all that information into a list and passes it onto the template file. So, this function always loads whenever someone turns the page, resulting in querying the APIs again.
How should I fix this?
Thank you for your help.
The paginator will in this case need the full list in order to do its job.
My advice would be to update a cache of the feeds at a regular interval, and then use that cache as the input to the paginator module. Doing an intensive or length task on each and every request is always a bad idea. If not for the page load times the user will experience, think of the vulnerability of your server to attack.
You may want to check out Django's low level cache API which would allow you to store the feed result in a globally accessible place under a key, which you can later use to retrieve the cache and paginate for each page request.
ORM's do not load data until the row is selected:
query_results = Foo(id=1) # No sql executed yet, just stored.
foo = query_results[0] # now it fires
or
for foo in query_results:
foo.bar() # sql fires
If you are using a custom data source that is loading results on initialization then the pagination will not work as expected since all feeds will be fetched at once. You may want to subclass __getitem__ or __iter__ to do the actual fetch. It will then coincide with the way Django expects the results to be loaded.
Pagination is going to need to know how many results there are to do things like has_next(). In sql it is usually inexpensive to get a count(*) with an index. So you would also, want to have know how many results there would be (or maybe just estimate if it too expensive to know exactly).