Introduction
The challenge I bring to you today is: To implement a Real Rime REST API (GET, POST, PUT, DELETE, etc) to query and update any SPARQL endpoint using the Django REST Framework for a frontend application (I am using React) to request and use the serialized data provided by the REST API.
Please note that I'm using Django because I would like to implement Web AND Mobile applications in the future, but for now I will just implement it on a React Web application.
Specifications
The REST API should be able to:
Perform (read or update) queries to a SPARQL endpoint via HTTP requests.
Serialize the response to a JSON RDF standarized table, or an RDF Graph, depending on the HTTP response.
Store the serialized response in a Python object.
Provide an endpoint with the serialized response to a frontend application such as React).
Handle incoming requests from the frontend application, "translate" and execute as a SPARQL query.
Send back the response to the frontend application's request.
ALL OF THIS while performing all queries and updates In Real Time.
What I mean with a Real Time API:
A SPARQL query is executed from the REST API to a SPARQL endpoint via an HTTP request.
The REST API reads the HTTP response generated from the request.
The REST API serializes the response to the corresponding format.
This serialized response is stored locally in a Python object for future use.
(Note: All the triples from the SPARQL endpoint in the query now exist both in the SPARQL endpoint as well as in a Python object, and are consistent both locally and remotely.)
The triples are then (hypothetically) modified or updated (Either locally or remotely).
Now the local triples are out of synch with the remote triples.
The REST API now becomes aware of this update (maybe through Listener/Observer objects?).
The REST API then automatically synchs the triples, either through an update query request (if the changes were made locally) or by updating the Python object with the response from a query request (if the update was made remotely).
Finally, both (the SPARQL endpoint and the Python object) should share the latest updated triples and, therefore, be in synch.
Previous Attempts
I have currently been able to query a SPARQL endpoint using the SPARQLWrapper package (for executing the queries), and the RDFLib and JSON packages for serializing and instantiating Python objects from the response, like this:
import json
from rdflib import RDFS, Graph
from SPARQLWrapper import GET, JSON, JSONLD, POST, TURTLE, SPARQLWrapper
class Store(object):
def __init__(self, query_endpoint, update_endpoint=None):
self.query_endpoint = query_endpoint
self.update_endpoint = update_endpoint
self.sparql = SPARQLWrapper(query_endpoint, update_endpoint)
def graph_query(self, query: str, format=JSONLD, only_conneg=True):
results = self.query(query, format, only_conneg)
results_bytes = results.serialize(format=format)
results_json = results_bytes.decode('utf8').replace("'", '"')
data = json.loads(results_json)
return data
def query(self, query: str, format=JSON, only_conneg=True):
self.sparql.resetQuery()
self.sparql.setMethod(GET)
self.sparql.setOnlyConneg(only_conneg)
self.sparql.setQuery(query)
self.sparql.setReturnFormat(format)
return self.sparql.queryAndConvert()
def update_query(self, query: str, only_conneg=True):
self.sparql.resetQuery()
self.sparql.setMethod(POST)
self.sparql.setOnlyConneg(only_conneg)
self.sparql.setQuery(query)
self.sparql.query()
store = Store('http://www.example.com/sparql/Example')
print(store.query("""SELECT ?s WHERE {?s ?p ?o} LIMIT 1"""))
print(store.graph_query("""DESCRIBE <http://www.example.com/sparql/Example/>"""))
The Challenge
The previous code solves can already:
Perform (read or update) queries to a SPARQL endpoint via HTTP requests
Serialize the response to a JSON RDF standarized table, or an RDF Graph, depending on the HTTP response
Store the serialized response in a Python object.
But still fails to implement these other aspects:
Provide an endpoint with the serialized response to a frontend application such as React).
Handle incoming requests from the frontend application, "translate" and execute as a SPARQL query.**
Send back the response to the frontend application's request.
And last, but not least, it fails completely to implement the real time aspect of this challenge.
The Questions:
How would you implement this?
Is this really the best approach?
Can the already working code be optimized?
Is there something that already does this?
Thank you so much!
Sorry but I don't know anything much about Django so can't answer here with Django specifics.
However, I can say this: SPARQL has a specification for HTTP interactions (https://www.w3.org/TR/sparql11-protocol/) and it tells you to use sparql?query=... & sparql?update... style URIs for querying a store, so why define a new way of doing things with store.query & store.graph_query etc?
Is there a Django-specific reason?
You can already pose questions to a SPARQL Endpoint using React or whatever you want right now, just as it is.
You said what is missing is to "Provide an endpoint with the serialized response" but the SPARQL responses are this! SPARQL query response formats are defined in the spec (e.g. JSON: https://www.w3.org/TR/sparql11-results-json/) and SPARQLWrapper knows how to parse them into Python objects. Other language libraries, like rdflib.js in JavaScript also know.
See YASGUI (https://triply.cc/docs/yasgui) for a stand-alone JS SPARQL client.
Related
I am new to working with APIs in general and am writing code in python that needs to consume/interact with an API someone else has set up. I was wondering if there is any package out there that would build some sort of custom client class to interact with an API given a file outlining the API in some way (like a json or something where each available endpoint and http verb could be outlined in terms of stuff like allowed payload json schema for posts, general params allowed and their types, expected response json schema, the header key/value for a business verb, etc.). It would be helpful if I could have one master file outlining the endpoints available and then some package uses that to generate a client class we can use to consume the API as described.
In my googling most API packages I have found in python are much more focused on the generation of APIs but this isn't what I want.
Basically I believe you are looking for the built in requests package.
response = requests.get(f'{base_url}{endpoint}',
params={'foo': self.bar,
'foo_2':self.bar_2},
headers={'X-Api-Key': secret}
)
And from here, you can build you own class, pass it to a dataframe or whatever.
In the requests package is basically everything you need. Status handling, exception handling everything you need.
Please check the docs.
https://pypi.org/project/requests/
I am developing a DAG to be scheduled on Apache Airflow which main porpuse will be to post survey data (on json format) to an API and then getting a response (the answers to the surveys). Since this whole process is going to be automated, every part of it has to be programmed in the DAG, so I can´t use Postman or any similar app (unless there is a way to automate their usage, but I don't know if this is possible).
I was thinking of using the requests library for Python, and the function I've written for posting the json to the API looks like this:
def postFileToAPI(**context):
print('uploadFileToAPI() ------ ')
json_file = context['ti'].xcom_pull(task_ids='toJson') ## this pulls the json file from a previous task
print('--------------- Posting survey request to API')
r = requests.post('https://[request]', data = json_file)
(I haven't finished defining the http link for the request because my source data is incomplete.)
However, since this is my frst time working with APIs and the requests library, I don't know if this is enough. For example, I'm unsure if I need to provide a token from the API to perform the request.
I also don't know if there are other libraries that are better suited for this or that could be a good support.
In short: I don't know if what I'm doing will work as intended, what other information I need t provide my DAG or if there are any libraries to make my work easier.
The Python requests package that you're using is all you need, except if you're making a request that needs extra authorisation - then you should also import for example requests_jwt (then from requests_jwt import JWTAuth) if you're using JSON web tokens, or whatever relevant requests package corresponds for your authorisation style.
You make POST and GET requests and all individual requests separately.
Include the URL and data arguments as you have done and that should work!
You may also need headers and/or auth arguments to get through security,
eg for the GitLab api for a private repository you would include these extra arguments, where GITLAB_TOKEN is a GitLab web token.
```headers={'PRIVATE-TOKEN': GITLAB_TOKEN},
auth=JWTAuth(GITLAB_TOKEN)```
If you just try it it should work, if it doesn't work then test the API with curl requests directly in the Terminal, or let us know :)
I am currently working on a bottle project and will be using Instagram api. I was hoping to use the client side authentication however I am having problems with the access token as it does not returns as a parameter.
I am making the request here:
https://api.instagram.com/oauth/authorize/?client_id=client_id&redirect_uri=redirect_uri&response_type=token&scope=basic+follower_list
The app is redirected to token page correctly and I can even see the token in the url. But when I try to parse it, it comes out empty.
#route('/oauth_callback')
def success_message():
token = request.GET.get("access_token")
print token.values()
return "success"
The token.values() returns an empty list.
ps: Keep in mind that when I try to do the same operation with server side authentication, I can successfully get the code and exchange it for a token.
Once you make a query to Instagram api you must be receiving below response?
http://your-redirect-uri#access_token=ACCESS-TOKEN
the part after # is termed as fragment and not query_string parameter and there is no way you can retrieve that information on Server side in Bottle.
To actually get fragments, bottle.request.urlparts is used
urlparts
The url string as an urlparse.SplitResult tuple. The tuple
contains (scheme, host, path, query_string and fragment), but the
fragment is always empty because it is not visible to the server.
Use the SDK and preferably Server Side operations -> https://github.com/facebookarchive/python-instagram
If you will to go with this approach, then managing a JavaScript which parses the access-token and then posts to your bottle api for your consumption which I don't recommend...
From https://instagram.com/developer/authentication/
Client-Side (Implicit) Authentication
If you are building an app that does not have a server component (a
purely javascript app, for instance), you will notice that it is
impossible to complete step three above to receive your access_token
without also having to store the secret on the client. You should
never pass or store your client_id secret onto a client. For these
situations there is the Implicit Authentication Flow.
shame to ask, but we have GraphQL Server on Java (https://github.com/graphql-java/graphql-spring-boot), where we specified type.graphqls scheme for our service.
On client side we have JS code based on Apollo Client library. And it does not need acess to this types file.
But days are come and I need to write some API tests. Most people in our team speaks Python very well, so I decided to make test workbench on python, but I cant't find any library that let me write queries schema-free or import my types.graphqls scheme.
How can I write tests on python for custom GraphQL server thogh?
Thanks!
Finally I found a gist with simple GraphQL client based on requests library:
import requests
def run_query(query, variables):
request = requests.post('https://dev.darkdata.finance:9000/graphql',
json={'query': query, 'variables': variables})
if request.status_code == 200:
return request.json()
else:
raise Exception("Query failed to run by returning code of {}. {}".format(request.status_code, query))
You can use it to test simple queries if you want.
Source: https://gist.github.com/gbaman/b3137e18c739e0cf98539bf4ec4366ad
I have a portion of my API that i am exposing using Bottle (http://bottlepy.org/docs/dev/index.html).
I am now looking to document these endpoints for the end user clients and am looking for a good solution. I am looking for something that is tightly integrated with my"routes" defined in the Bottle app so that any changes in the future keep in sync. The key areas i want to document are the HTTP method types that are accepted and the necessary query parameters.
I have included an example route below which queries whether an instance defined in the underlying API is online. As you can see the route only accepts GET requests, and the "check_valid_instance" function expects to find a query parameter. Looking at this route definition there is no indication that a query param is needed and that is what i am trying to add here! Both to the source code, and also externally to some type of help page
#app.route("/application/app_instance/is_instance_online", method="GET")
def is_instance_online():
_check_valid_instance()
function = eval("app_instance.is_instance_online")
return _process_request_for_function(function)
The above route would be called as following
http://IP:Port/applicaton/app_instance/is_instance_online?instance=instance_name
Any suggestions welcome!
Thanks!
For additional params you can create a structure similar to this:
COMMANDS = {'is_instance_online': {'mandatory_params': 'instance_name',
'description': 'command description'}}
self.bottle.route('/<command>', method='GET', commands=COMMANDS)(self.command_execute)
Then you should be able to generate JSON description of the whole API as shown below:
Automatic Generation of REST API description with json