I have a tiny Flask server that is supposed to load data from a file and run a function on it. This function will return a DataFrame and I return the json version of it. Much to my surprise this all works nicely. However, how would I test this? I have included some attempts below but I don't understand Flask (nor REST) well enough yet:
#!/home/thomas/python
from flask import Flask
from flask.ext.restful import Resource, Api
app = Flask(__name__)
api = Api(app)
class UniverseAPI(Resource):
def get(self):
import pandas as pd
frame = pd.read_csv("//datasrv10//data$//AQ//test.csv", index_col=0, header=0)
return frame.to_json()
api.add_resource(UniverseAPI, '/data/universe')
I am happy to include a few of my attempts here... I appreciate any hints. I have read the official documentation.
I should specify what I mean with testing. I can run this on my linux server and can extract all the required information with the requests package. However, I want to create a unittest that comes without the need to start the server on the localhost. I think I have managed with the FLASK test-client. However, the problem now is that the requests response object and the flask response object treat the underlying json strings rather differently. So I guess my problem is more related to json string issues rather than FLASK. Thanks for all your helpful feedback though
Well, the basics of writing a REST API are essentially a set of design principles. My understanding of it is based on this article by Miguel Grinberg, http://blog.miguelgrinberg.com/post/designing-a-restful-api-with-python-and-flask .
In it, he talks about how a REST API is:
"Stateless" - All interactions with the service can happen using the information from one request.
Built upon accessing "resources" from URIs using HTTP requests like GET, PUTS, and POST. A resource could be an order in a store, a task in a web app, or whatever you like.
There's also a bunch of stuff about how the server should standardize all forms of communication between itself and the client, indicate whether it can do cacheing, and other stuff like that. From an initial design standpoint, though, this is "the point" as he put it:
"The task of designing a web service or API that adheres to the REST guidelines then becomes > an exercise in identifying the resources that will be exposed and how they will be affected > by the different request methods."
If you're looking for an interesting example of a REST API that might be suited to your interests (I know it is to mine), reddit's is open source. It's a relatable example to see how they try and structure the interactions behind requests: http://www.reddit.com/dev/api
Related
I am new to working with APIs in general and am writing code in python that needs to consume/interact with an API someone else has set up. I was wondering if there is any package out there that would build some sort of custom client class to interact with an API given a file outlining the API in some way (like a json or something where each available endpoint and http verb could be outlined in terms of stuff like allowed payload json schema for posts, general params allowed and their types, expected response json schema, the header key/value for a business verb, etc.). It would be helpful if I could have one master file outlining the endpoints available and then some package uses that to generate a client class we can use to consume the API as described.
In my googling most API packages I have found in python are much more focused on the generation of APIs but this isn't what I want.
Basically I believe you are looking for the built in requests package.
response = requests.get(f'{base_url}{endpoint}',
params={'foo': self.bar,
'foo_2':self.bar_2},
headers={'X-Api-Key': secret}
)
And from here, you can build you own class, pass it to a dataframe or whatever.
In the requests package is basically everything you need. Status handling, exception handling everything you need.
Please check the docs.
https://pypi.org/project/requests/
I am developing a DAG to be scheduled on Apache Airflow which main porpuse will be to post survey data (on json format) to an API and then getting a response (the answers to the surveys). Since this whole process is going to be automated, every part of it has to be programmed in the DAG, so I can´t use Postman or any similar app (unless there is a way to automate their usage, but I don't know if this is possible).
I was thinking of using the requests library for Python, and the function I've written for posting the json to the API looks like this:
def postFileToAPI(**context):
print('uploadFileToAPI() ------ ')
json_file = context['ti'].xcom_pull(task_ids='toJson') ## this pulls the json file from a previous task
print('--------------- Posting survey request to API')
r = requests.post('https://[request]', data = json_file)
(I haven't finished defining the http link for the request because my source data is incomplete.)
However, since this is my frst time working with APIs and the requests library, I don't know if this is enough. For example, I'm unsure if I need to provide a token from the API to perform the request.
I also don't know if there are other libraries that are better suited for this or that could be a good support.
In short: I don't know if what I'm doing will work as intended, what other information I need t provide my DAG or if there are any libraries to make my work easier.
The Python requests package that you're using is all you need, except if you're making a request that needs extra authorisation - then you should also import for example requests_jwt (then from requests_jwt import JWTAuth) if you're using JSON web tokens, or whatever relevant requests package corresponds for your authorisation style.
You make POST and GET requests and all individual requests separately.
Include the URL and data arguments as you have done and that should work!
You may also need headers and/or auth arguments to get through security,
eg for the GitLab api for a private repository you would include these extra arguments, where GITLAB_TOKEN is a GitLab web token.
```headers={'PRIVATE-TOKEN': GITLAB_TOKEN},
auth=JWTAuth(GITLAB_TOKEN)```
If you just try it it should work, if it doesn't work then test the API with curl requests directly in the Terminal, or let us know :)
I am having trouble understanding the concept of “API discovery” as used in Google products/services. Here’s some Python code that uses the said discovery service to access Google Cloud Vision:
from googleapiclient.discovery import build
from oauth2client.client import GoogleCredentials
…
API_DISCOVERY_FILE = 'https://vision.googleapis.com/$discovery/rest?version=v1'
hlh = httplib2.Http()
credentials = GoogleCredentials.get_application_default().create_scoped(
['https://www.googleapis.com/auth/cloud-platform'])
credentials.authorize(hlh)
service = build(serviceName='vision', version='v1', http=hlh, discoveryServiceUrl=API_DISCOVERY_FILE)
service_request = service.images().annotate(body={ <more JSON code here> })
Here’s another bit of Python code that also accesses Google Cloud Vision, but does not use API discovery and works just fine:
import requests
…
ENDPOINT_URL = 'https://vision.googleapis.com/v1/images:annotate'
response = requests.post(ENDPOINT_URL,
data=make_image_data(image_filenames),
params={'key': api_key},
headers={'Content-Type': 'application/json'})
What I can’t wrap my head around is this question: You need to know the details of the API that you are going to be calling so that you can tailor the call; this is obvious. So, how would API discovery help you at the time of the call, after you have already prepared the code for calling that API?
PS: I did look at the following resources prior to posting this question:
https://developers.google.com/discovery/v1/getting_started
https://developers.google.com/discovery/v1/using
I did see this answered question but would appreciate additional insight.
Note: I lack the knowledge of the Google API you are mentioning.
You need to know the details of the API that you are going to be
calling so that you can tailor the call; this is obvious.
In theory to me this is only required for the very first call, which would be the call to a starting service which would for instance list a number of resources. From there on you can have branches to underlying resources and their allowed methods (verbs if you will). So this example illustrated a tree like structure. If you provide a GUI to navigate through this discoverability in a generic way, a person would then be able to decide what to do.
In practice this is kind of hard to do once you have vast amounts of resources and all sorts of interrelationships between them.
I have a portion of my API that i am exposing using Bottle (http://bottlepy.org/docs/dev/index.html).
I am now looking to document these endpoints for the end user clients and am looking for a good solution. I am looking for something that is tightly integrated with my"routes" defined in the Bottle app so that any changes in the future keep in sync. The key areas i want to document are the HTTP method types that are accepted and the necessary query parameters.
I have included an example route below which queries whether an instance defined in the underlying API is online. As you can see the route only accepts GET requests, and the "check_valid_instance" function expects to find a query parameter. Looking at this route definition there is no indication that a query param is needed and that is what i am trying to add here! Both to the source code, and also externally to some type of help page
#app.route("/application/app_instance/is_instance_online", method="GET")
def is_instance_online():
_check_valid_instance()
function = eval("app_instance.is_instance_online")
return _process_request_for_function(function)
The above route would be called as following
http://IP:Port/applicaton/app_instance/is_instance_online?instance=instance_name
Any suggestions welcome!
Thanks!
For additional params you can create a structure similar to this:
COMMANDS = {'is_instance_online': {'mandatory_params': 'instance_name',
'description': 'command description'}}
self.bottle.route('/<command>', method='GET', commands=COMMANDS)(self.command_execute)
Then you should be able to generate JSON description of the whole API as shown below:
Automatic Generation of REST API description with json
I am new to Flask.
I have a public api, call it api.example.com.
#app.route('/api')
def api():
name = request.args.get('name')
...
return jsonify({'address':'100 Main'})
I am building an app on top of my public api (call it www.coolapp.com), so in another app I have:
#app.route('/make_request')
def index():
params = {'name':'Fred'}
r = requests.get('http://api.example.com', params=params)
return render_template('really_cool.jinja2',address=r.text)
Both api.example.com and www.coolapp.com are hosted on the same server. It seems inefficient the way I have it (hitting the http server when I could access the api directly). Is there a more efficient way for coolapp to access the api and still be able to pass in the params that api needs?
Ultimately, with an API powered system, it's best to hit the API because:
It's user testing the API (even though you're the user, it's what others still access);
You can then scale easily - put a pool of API boxes behind a load balancer if you get big.
However, if you're developing on the same box you could make a virtual server that listens on localhost on a random port (1982) and then forwards all traffic to your api code.
To make this easier I'd abstract the API_URL into a setting in your settings.py (or whatever you are loading in to Flask) and use:
r = requests.get(app.config['API_URL'], params=params)
This will allow you to make a single change if you find using this localhost method isn't for you or you have to move off one box.
Edit
Looking at your comments you are hoping to hit the Python function directly. I don't recommend doing this (for the reasons above - using the API itself is better). I can also see an issue if you did want to do this.
First of all we have to make sure the api package is in your PYTHONPATH. Easy to do, especially if you're using virtualenvs.
We from api import views and replace our code to have r = views.api() so that it calls our api() function.
Our api() function will fail for a couple of reasons:
It uses the flask.request to extract the GET arg 'name'. Because we haven't made a request with the flask WSGI we will not have a request to use.
Even if we did manage to pass the request from the front end through to the API the second problem we have is using the jsonify({'address':'100 Main'}). This returns a Response object with an application type set for JSON (not just the JSON itself).
You would have to completely rewrite your function to take into account the Response object and handle it correctly. A real pain if you do decide to go back to an API system again...
Depending on how you structure your code, your database access, and your functions, you can simply turn the other app into package, import the relevant modules and call the functions directly.
You can find more information on modules and packages here.
Please note that, as Ewan mentioned, there's some advantages to using the API. I would advise you to use requests until you actually need faster requests (this is probably premature optimization).
Another idea that might be worth considering, depending on your particular code, is creating a library that is used by both applications.