I am trying to run a curl command using the run command in the subprocess module in python. But it is failing.
However, the curl command for my elasticsearch search query API is working fine-
curl -k XGET -H "Content-Type: application/json" -H "Authorization: ApiKey Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw==" 'https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/_search?index=.fleet-enrollment-api-keys-7&size=20&pretty' -d '{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}},"fields" : ["_id", "_index"]}'
This is the code that I am trying to run-
import json
from builtins import int
from pydantic import BaseModel, Field
from typing import List, Dict
from subprocess import PIPE, run
def elasticsearch_search_query(host: str,
port: int,
api_key: str,
path: str,
query: dict,
index: str,
size: int,
fields: List):
es_path = host + ":" + str(port) + path +"?index="+index+"&"+str(size)+"&pretty"
es_header = "Authorization: ApiKey" + " " + api_key
es_dict = {"query": query, "fields": fields}
es_json = json.dumps(es_dict)
es_replaced = " "+"'"+ es_json.replace("'", "\"")+"'"
result = run(["curl", "-k", "-XGET", "-H", "Content-Type: application/json", "-H",
es_header,
es_path,
"-d",
es_replaced]
# ,stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=False
)
print(es_path, es_header, es_replaced)
print(result.stdout)
elasticsearch_search_query(host="https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/_search", api_key="Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw==",
port=9200, size=5, query={"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, fields=["_id", "_index"], path="/_search",index=".fleet-enrollment-api-keys-7")
Here is the error-
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "request [/_search:9200/_search] contains unrecognized parameter: [5]"
}
],
"type" : "illegal_argument_exception",
"reason" : "request [/_search:9200/_search] contains unrecognized parameter: [5]"
},
"status" : 400
}
https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/_search:9200/_search?index=.fleet-enrollment-api-keys-7&5&pretty Authorization: ApiKey Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw== '{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, "fields": ["_id", "_index"]}'
None
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 350 100 350 0 0 10294 0 --:--:-- --:--:-- --:--:-- 10294
curl: (3) nested brace in URL position 12:
'{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, "fields": ["_id", "_index"]}'
UPDATE
Output of repr(result):
CompletedProcess(args=['curl', '-k', '-XGET', '-H', 'Content-Type: application/json', '-H', 'Authorization: ApiKey Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw==', 'https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200:9200/_search?index=.fleet-enrollment-api-keys-7&size=5&pretty', '-d', '{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, "fields": ["_id", "_index"]}'], returncode=3)
Since the error was
contains unrecognized parameter: [5]
The problem is not in the query but in the way you construct es_path. Instead of
&"+str(size)+
it should be
&size="+str(size)+
UPDATE:
Also from what I see your URL now looks like this
https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200:9200/_search?index=.fleet-enrollment-api-keys-7&size=5&pretty
and it's wrong for the following reasons:
the port is specified twice
the index is not at the right location
First, host should just be this (i.e. without port and path)
host="https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com"
Then you can build the URL like this instead:
es_path = host + ":" + str(port) + "/" + index + path +"?size="+str(size)+"&pretty"
i.e. it should look like this instead:
https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/.fleet-enrollment-api-keys-7/_search?size=5&pretty
Related
I have API that accepts multiple files with json data -
curl -X POST \
http://localhost:25965/v1/import \
-H 'Content-Type: application/json' \
-H 'Postman-Token: xxxxxxxx-xxxx-xxxx-xxxx-ba66a9b8d6cb' \
-H 'cache-control: no-cache' \
-H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \
-F 'files=#C:\Users\user\File1.xlsx' \
-F 'files=#C:\Users\user\File2.xlsx' \
-F 'files=#C:\Users\user\File3.xlsx' \
-F 'values=[
{
"field1": "Value11",
"field2": "Value21",
"field3": "File1.xlsx"
},
{
"field1": "Value21",
"field2": "Value22",
"field3": "File2.xlsx"
},
{
"field1": "Value31",
"field2": "Value32",
"field3": "File3.xlsx"
}
]'
I'm trying to convert this request into python.
With requests, I get error too many values to unpack (expected 2)
def PostFiles(self, apiName, values, files):
url = self.ApiUrl + apiName
params = {}
for file in files:
params.update({'files':file})
response = requests.post(url,
files=params,
data=values,
headers={
'Content-Type':'application/json'},
auth=HTTPKerberosAuth(delegate=True))
return response
I also tried requests_toolbelt but doesn't work.
def PostFiles(self, apiName, values, files):
url = self.ApiUrl + apiName
params = {}
for file in files:
params.update({'files':file})
params.update({'values':json.dumps(values)})
multipart_data = MultipartEncoder(params)
headers = { 'Content-Type':multipart_data.content_type}
response = requests.post(url,
data=multipart_data,
headers=headers,
auth=HTTPKerberosAuth(delegate=True))
return response
this is my function call -
files = [open(join(directory, filename), 'rb')]
#files = [('files',(filename, open(join(directory, filename), 'rb'),'application/vnd.ms-excel'))]
values = [{
'field1': 'value11',
'field2' : 'value21',
'field3' : 'File1.xlsx'
}]
response = PostFiles('import', values, files)
print(response)
there are posts related to this topic but I couldn't find anything with multiple files posting with json.
I modified my code as per Post JSON and file in single request and worked as expected.
def PostFiles(self, apiName, values, files):
url = self.ApiUrl + apiName
params = {
'values': (None, json.dumps(values), 'application/json')
}
for file in files:
params.update({'files':file})
response = requests.post(url,
files=params,
auth=HTTPKerberosAuth(delegate=True))
result = response.json()
if response.status_code != 200 :
raise Exception(result)
return json.dumps(result)
So the user enters the hexcode of a colour and the program uses that to get Spotifyfy playlist.
e.g user enters "red" the program will get "energetic" playlist from Spotify.
Hexcode = input("Please enter Colour: ")
Im just not sure how to set up the retrieval part from Spotify
These are the Snippets I have:
curl -X "GET" "https://api.spotify.com/v1/browse/categories/dinner/playlists?country=SE&limit=10&offset=5" -H "Accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer
if Hexcode ==:
Yellow = #FFCC00
Blue = #03025D
Red = #FF0000
Green = #339900
Purple = #580C6D
$.ajax({
url: "https://api.spotify.com/v1/users/" + $("#username").val() + "/playlists",
headers: {
Authorization: "Bearer my_OAuth_Token",
Host: "api.spotify.com"
},
accepts: "application/json",
type: "GET",
success: function (data) {
var count = data.items.length;
for (var i = 0; i < aantal; i++) {
$("#playlists").append('<option>' + data.items[i].name + '</option>');
}
},
error: function (data) {
$("#playlists").append("<option>error</option>");
}
});
They're not set up properly and are for different things. I got kind of lost and wasn't really sure what to do.
Currently within the app, requesting a UberX instantly give you a exact quote price but in the Python API, I couldn't find it. I can only find the range of the cost. Where is the exact quote at?
Try to use "POST /v1.2/requests/estimate"
Example Request
curl -X POST \
-H 'Authorization: Bearer <TOKEN>' \
-H 'Accept-Language: en_US' \
-H 'Content-Type: application/json' \
-d '{
"start_latitude": 37.7752278,
"start_longitude": -122.4197513,
"end_latitude": 37.7773228,
"end_longitude": -122.4272052
}' "https://api.uber.com/v1.2/requests/estimate"
I suggest you use "product_id" as well - to get the price for the product you need. Otherwise, if none is provided, it will default to the cheapest product for the given location.
You will get the response like:
{
"fare": {
"value": 5.73,
"fare_id": "d30e732b8bba22c9cdc10513ee86380087cb4a6f89e37ad21ba2a39f3a1ba960",
"expires_at": 1476953293,
"display": "$5.73",
"currency_code": "USD",
"breakdown": [
{
"type": "promotion",
"value": -2.00,
"name": "Promotion"
},
{
"type": "base_fare",
"notice": "Fares are slightly higher due to increased demand",
"value": 7.73,
"name": "Base Fare"
}
]
},
"trip": {
"distance_unit": "mile",
"duration_estimate": 540,
"distance_estimate": 2.39
},
"pickup_estimate": 2
}
Related to Pyton SDK - Please check: https://developer.uber.com/docs/riders/ride-requests/tutorials/api/python. You need to authenticate your user, and then get a product you want to use, and then get upfront fare (if product support this: upfront_fare_enabled field set to true). And after that you can book a ride. Code how to do it is in doc link I have sent as well:
# Get products for a location
response = client.get_products(37.77, -122.41)
products = response.json.get('products')
product_id = products[0].get('product_id')
# Get upfront fare and start/end locations
estimate = client.estimate_ride(
product_id=product_id,
start_latitude=37.77,
start_longitude=-122.41,
end_latitude=37.79,
end_longitude=-122.41,
seat_count=2
)
fare = estimate.json.get('fare')
# Request a ride with upfront fare and start/end locations
response = client.request_ride(
product_id=product_id,
start_latitude=37.77,
start_longitude=-122.41,
end_latitude=37.79,
end_longitude=-122.41,
seat_count=2,
fare_id=fare['fare_id']
)
request = response.json
request_id = request.get('request_id')
# Request ride details using `request_id`
response = client.get_ride_details(request_id)
ride = response.json
# Cancel a ride
response = client.cancel_ride(request_id)
ride = response.json
I have a working Flask API and now I want to implement search queries.
My understanding is that the filter is applied on the client and the Flask API takes care of it.
Flask==0.10.1
Flask-HTTPAuth==2.7.0
Flask-Limiter==0.9.1
Flask-Login==0.3.2
Flask-Mail==0.9.1
Flask-Principal==0.4.0
Flask-Restless==0.17.0
I have followed documentation and performed my search query, but still just retrieving same results:
http://flask-restless.readthedocs.org/en/latest/searchformat.html
No filter:
curl -u aaa:bbb -H "Content-Type: application/json" http://0.0.0.0:8080/api/1.0/job/
{
"jobs": [
{
"description": "ESXi job completed",
"reference": "07FC78BCC0",
"status": 1
},
{
"description": "Server discovery failed. Please verify HTTPS/SSH parameters",
"reference": "A6EE28F4C0",
"status": -1
}]
}
Search query based on:
http://flask-restless.readthedocs.org/en/latest/searchformat.html
curl -u aaa:bbb -G -H "Content-Type: application/json" -d '{
> "filters": [{"name": "description", "op": "like", "val": "%ESXi%"}]}' http://0.0.0.0:8080/api/1.0/job/
Or
curl -u aaa:bbb -G -H "Content-Type: application/json" -d '{"filters": [{"name": "status", "op": "eq", "val":0}]}' http://0.0.0.0:8080/api/1.0/job/
And same results are shown.
This is my Flask endpoint:
def get_jobs():
"""
:return:
"""
try:
log.info(request.remote_addr + ' ' + request.__repr__())
jobs = Model.Job.query.order_by(desc(Model.Job.job_start)).limit(settings.items_per_page).all()
# =========================================================
# Get JOBS
# =========================================================
values = ['description', 'status', 'reference']
response = [{value: getattr(d, value) for value in values} for d in jobs]
return jsonify(jobs=response)
except Exception, excpt:
log.exception(excpt.__repr__())
response = json.dumps('Internal Server Error. Please try again later')
resp = Response(response, status=500, mimetype='application/json')
return resp
My Model
class Job(db.Model, AutoSerialize, Serializer):
"""
"""
__tablename__ = 'job'
__public__ = ('status','description','reference','job_start','job_end')
id = Column(Integer, primary_key=True, server_default=text("nextval('job_id_seq'::regclass)"))
description = Column(String(200))
reference = Column(String(50))
job_start = Column(DateTime)
job_end = Column(DateTime)
fk_server = Column(ForeignKey(u'server.id'))
owner_id = Column(ForeignKey(u'auth_user.id'))
is_cluster = Column(Boolean)
host_information = Column(String(1024))
status = Column(Integer, nullable=False)
owner = relationship(u'AuthUser')
server = relationship(u'Server')
def serialize(self):
"""
:return:
"""
d = Serializer.serialize(self)
return d
Do I need to change anything?
Maybe having __public__ as a Job attribute is interfering with the way the filtering works. There's a warning in the Flask-Restless documentation about this.
I'm trying to retrieve results from the BNCF at this endpoint.
My query (with "ab" as example) is:
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT DISTINCT ?source ?label ?content
WHERE {
?source a skos:Concept;
skos:prefLabel ?label;
skos:scopeNote ?content.
FILTER regex(str(?label), "ab", "i")
}
The query is correct in fact if you try to run it works.
But when I try to get the results from my python this is the error:
SyntaxError: JSON Parse error: Unexpected EOF
This is my python code:
__3store = "http://digitale.bncf.firenze.sbn.it/openrdf-workbench/repositories/NS_03_2014/query"
sparql = SPARQLUpdateStore(queryEndpoint=__3store)
sparql.setReturnFormat(JSON)
results = sparql.query(query_rdf).convert()
print json.dumps(result, separators=(',',':'))
I tried the code above according to this answer, before my code was like this:
__3store = "http://digitale.bncf.firenze.sbn.it/openrdf-workbench/repositories/NS_03_2014/query"
sparql = SPARQLWrapper(__3store,returnFormat="json")
sparql.setQuery(query_rdf)
result = sparql.query().convert()
print json.dumps(result, separators=(',',':'))
but both throw the same error.
Does anyone know how to fix it?
Thanks
EDIT:
This is python code, hope it is enough to understand
import sys
sys.path.append ('cgi/lib')
import rdflib
from rdflib.plugins.stores.sparqlstore import SPARQLUpdateStore, SPARQLStore
import json
from SPARQLWrapper import SPARQLWrapper, JSON
#MAIN
print "Content-type: application/json"
print
prefix_SKOS = "prefix skos: <http://www.w3.org/2004/02/skos/core#>"
crlf = "\n"
query_rdf = ""
query_rdf += prefix_SKOS + crlf
query_rdf += '''
SELECT DISTINCT ?source ?title ?content
WHERE {
?source a skos:Concept;
skos:prefLabel ?title;
skos:scopeNote ?content.
FILTER regex(str(?title), "ab", "i")
}
'''
__3store = "http://digitale.bncf.firenze.sbn.it/openrdf-workbench/repositories/NS_03_2014/query"
sparql = SPARQLWrapper(__3store,returnFormat="json")
sparql.setQuery(query_rdf)
result = sparql.query().convert()
print result
Running this in Python shell returns:
Content-type: application/json
Warning (from warnings module):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/SPARQLWrapper-1.6.4-py2.7.egg/SPARQLWrapper/Wrapper.py", line 689
RuntimeWarning: Format requested was JSON, but XML (application/sparql-results+xml;charset=UTF-8) has been returned by the endpoint
<xml.dom.minidom.Document instance at 0x105add710>
So I think the result is always an XML also if I specificied Json as a return format.
There are a couple of problems playing together here:
First, you should only use SPARQLUpdateStore from rdflib if you want to access a SPARQL store via rdflib's Graph interface (e.g., you can add triples, you can iterate over them, etc.).
If you want to write a SPARQL query yourself you should use SPARQLWrapper.
Second, if you ask SPARQLWrapper to return JSON, what it does is actually ask the server for a couple of mime types that are most common and standardized for what we just call "json", as shown here and here:
_SPARQL_JSON = ["application/sparql-results+json", "text/javascript", "application/json"]
It seems as if your sever does understand application/sparql-results+json, but not a combined "give me any of these mime-types header" as rdflib compiles it for maximum interoperability (so your server essentially doesn't fully support HTTP Accept Headers):
curl -i -G -H 'Accept: application/sparql-results+json' --data-urlencode 'query=PREFIX skos:
<http://www.w3.org/2004/02/skos/core#>
SELECT DISTINCT ?source ?label ?content
WHERE {
?source a skos:Concept;
skos:prefLabel ?label;
skos:scopeNote ?content.
FILTER regex(str(?label), "ab", "i")
}' http://digitale.bncf.firenze.sbn.it/openrdf-workbench/repositories/NS_03_2014/query
will return:
HTTP/1.1 200 OK
Date: Mon, 18 May 2015 13:13:45 GMT
Server: Apache/2.2.17 (Unix) PHP/5.3.6 mod_jk/1.2.31
...
Content-Type: application/sparql-results+json;charset=UTF-8
{
"head" : {
"vars" : [ ],
"vars" : [ "source", "label", "content" ],
"link" : [ "info" ]
},
"results" : {
"bindings" : [ {
"content" : {
"type" : "literal",
"value" : "Il lasciare ingiustificatamente qualcuno o qualcosa di cui si รจ responsabili"
},
"source" : {
"type" : "uri",
"value" : "http://purl.org/bncf/tid/12445"
},
"label" : {
"xml:lang" : "it",
"type" : "literal",
"value" : "Abbandono"
}
},
...
so everything is ok, but if we ask for the combined, more interoperable mime types:
curl -i -G -H 'Accept: application/sparql-results+json,text/javascript,application/json' --data-urlencode 'query=PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT DISTINCT ?source ?label ?content
WHERE {
?source a skos:Concept;
skos:prefLabel ?label;
skos:scopeNote ?content.
FILTER regex(str(?label), "ab", "i")
}' http://digitale.bncf.firenze.sbn.it/openrdf-workbench/repositories/NS_03_2014/query
we get an xml result:
HTTP/1.1 200 OK
Server: Apache/2.2.17 (Unix) PHP/5.3.6 mod_jk/1.2.31
...
Content-Type: application/sparql-results+xml;charset=UTF-8
<?xml version='1.0' encoding='UTF-8'?>
...
So long story short: it's a bug in the server you're using. The following is a nasty workaround (it seems SPARQLWrapper doesn't just allow us to manually set the headers, but unconditionally overrides them in _createRequest), but it works:
In [1]: import SPARQLWrapper as sw
In [2]: sparql = sw.SPARQLWrapper("http://digitale.bncf.firenze.sbn.it/openrdf-workbench/repositories/NS_03_2014/query")
In [3]: sparql.setReturnFormat(sw.JSON)
In [4]: sparql.setQuery(''' PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT DISTINCT ?source ?label ?content
WHERE {
?source a skos:Concept;
skos:prefLabel ?label;
skos:scopeNote ?content.
FILTER regex(str(?label), "ab", "i")
}
''')
In [5]: request = sparql._createRequest()
In [6]: request.add_header('Accept', 'application/sparql-results+json')
In [7]: from urllib2 import urlopen
In [8]: response = urlopen(request)
In [9]: res = sw.Wrapper.QueryResult((response, sparql.returnFormat))
In [10]: result = res.convert()
In [11]: result
Out[11]:
{u'head': {u'link': [u'info'], u'vars': [u'source', u'label', u'content']},
u'results': {u'bindings': [{u'content': {u'type': u'literal',
u'value': u'Il lasciare ingiustificatamente qualcuno o qualcosa di cui si \xe8 responsabili'},
u'label': {u'type': u'literal',
u'value': u'Abbandono',
u'xml:lang': u'it'},
u'source': {u'type': u'uri', u'value': u'http://purl.org/bncf/tid/12445'}},
...