Discrepency between curl results and Python requests results - python

I am querying the New Relic API, and trying to pull CPU utilization fron the metrics that they provide. When I run the following curl command (after exporting the correct proxy settings), I see the following information (which contains the percentage value that I want) -
curl -X GET "https://api.newrelic.com/v2/applications/140456413/hosts/21044947/metrics/data.json" -H "X-Api-Key:myapikey" -i -d 'names[]=CPU/User+Time&values[]=percent&summarize=true&from=2018-07-15 16:22:30&to=2018-07-16 16:22:30'
HTTP/1.0 200 Connection Established
Proxy-agent: Apache
HTTP/1.1 200 OK
Server: openresty
Date: Tue, 17 Jul 2018 11:53:22 GMT
Content-Type: application/json
Content-Length: 290
Connection: keep-alive
Status: 200 OK
X-UA-Compatible: IE=Edge,chrome=1
ETag: "a35b451e27a49d0d5f4e16715429a17d"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: f4f8675f095aba80d5e089fbcbf1b172
X-Runtime: 0.168283
X-Rack-Cache: miss
{"metric_data":{"from":"2018-07-15T16:22:30+00:00","to":"2018-07-16T16:22:30+00:00","metrics_not_found":[],"metrics_found":["CPU/User Time"],"metrics":[{"name":"CPU/User Time","timeslices":[{"from":"2018-07-15T16:22:00+00:00","to":"2018-07-16T16:22:00+00:00","values":{"percent":9.52}}]}]}}
However, when I try to implement it inside of the Python requests module, the "Percent" value that I am interested in seeing is returning 0. This is my code to call it -
options = {"names[]": "CPU/User+Time", "values[]": "percent", "summarize": "true", "from": str(end_date), "to": str(start_date - datetime.timedelta(hours=6))}
path = "applications/140456413/hosts/" + server_id + "/metrics/data.json"
api_key = "myapikey"
headers = {'X-Api-Key': api_key}
url = "https://api.newrelic.com/v2/" + path
r = requests.get(url, headers=headers, data=options, proxies=myproxy.proxies)
This is what I get instead (notice the percent value is now 0) -
{u'metric_data': {u'metrics': [{u'timeslices': [{u'values': {u'percent': 0}, u'to': u'2018-07-17T01:35:00+00:00', u'from': u'2018-07-16T07:35:00+00:00'}], u'name': u'CPU/User+Time'}], u'to': u'2018-07-17T01:35:59+00:00', u'metrics_found': [u'CPU/User+Time'], u'from': u'2018-07-16T07:35:59+00:00', u'metrics_not_found': []}}
How can I adjust the python request to match the same output as the curl command? We were originally passing in options inside of the "options" variable using ='s instead of key/value pairs, but the requests module would not process them in this format.
These were the docs pages I referencesd -
https://docs.newrelic.com/docs/apis/rest-api-v2/requirements/specifying-time-range-v2
https://docs.newrelic.com/docs/apis/rest-api-v2/application-examples-v2/get-average-cpu-usage-host-app
https://docs.newrelic.com/docs/apis/rest-api-v2/getting-started/introduction-new-relic-rest-api-v2#examples
Thanks.

Change the options a bit:
"values[]": "percent"
to:
"values": ["percent"]

Related

PUT request is returning an error with flask

I use Flask in an API application and I have this function:
#app.route('/v1/myapi/editperson/<int:id_person>)', methods=['PUT'])
def edit_person(person_id):
req_data = request.get_json()
firstname = req_data.get('firstname')
lastname = req_data.get('lastname')
session.query(model.Persons).filter(model.Persons.id==person_id).update( firstname, lastname)
session.commit()
session.close()
return jsonify(req_data)
When I execute this curl request:
curl -i -H "Content-Type: application/json" -X PUT -d '{"firstname": "myfirstname", "lastname": "mylastname"}' http://localhost:5000/v1/myapi/editperson/38
I get this error:
HTTP/1.0 404 NOT FOUND
Content-Type: application/json
Content-Length: 27
Server: Werkzeug/0.14.1 Python/2.7.6
Date: Wed, 04 Jul 2018 07:17:09 GMT
Proxy-Connection: keep-alive
Connection: keep-alive
{
"error": "Not found"
}
I do have a row in the database with the id = 38 so I don't know why the request doesn't find it.
I use SQLAlchemy to query the database.
Any clues?
Indeed PUT/POST but also:
#app.route('/v1/myapi/editperson/<int:id_person>)', methods=['PUT'])
vs.
#app.route('/v1/myapi/editperson/<int:id_person>', methods=['PUT'])
Your annotation specifies that this API just allow PUT method. But you request this API through POST method. Try:
curl -X PUT ....
Your route passes id_person but your query uses person_id

The specifics of adding a header to an api call with a swagger codegen client in python are unclear

Point out the right document, tutorial, example, or provide one, showing how to add a specific authentication token to a specific header in the Swagger generated API client in Python?
Here's what I've tried:
My API call works just fine with the right curl command:
curl -v -H 'X-CAG-Authorization: AG_CONSUMER_TOKEN access-key=31337-70k3n' \
'https://api.company.net/api/v1/user/detail?user=1'
* Trying 10.10.1.10...
* Connected to api.company.net (10.10.1.10) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: *.company.net
* Server certificate: COMODO RSA Organization Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
> GET /api/v1/user/detail?user=1 HTTP/1.1
> Host: api.company.net
> User-Agent: curl/7.49.1
> Accept: */*
> X-CAG-Authorization: AG_CONSUMER_TOKEN access-key=31337-70k3n
>
< HTTP/1.1 200 OK
< Server: openresty
< Date: Thu, 22 Dec 2016 19:46:05 GMT
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Connection: close
< Vary: Accept-Encoding
< Vary: Accept-Encoding
< X-UA-Compatible: IE=edge
<
{"successful":true,"message":"SUCCESS","body":{"…
However, when I try the same basic request in my Python (2.7.12) client I get an authorization failure, despite confirming that the token makes it into the header about to be used. More details on the right way to use the client OR on how to get more details of the exact request and response would be appreciated.
/Users/me/VEnvs/sku-grade/bin/python /Users/me/prj/code/python_client/api_example.py
HEADERS:
{'X-CAG-Authorization': 'AG_CONSUMER_TOKEN access-key=31337-70k3n', 'User-Agent': 'Swagger-Codegen/1.0.0/python'}
Exception when calling SupplierApi->get_api_v1_user_details: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 22 Dec 2016 21:09:30 GMT', 'Content-Length': '636', 'Content-Type': 'application/json; charset=UTF-8', 'Connection': 'keep-alive', 'Server': 'nginx'})
HTTP response body: {
"code" : "PRECONDITION_FAILED",
"type" : "UnauthorizedApiDeniedException",
"message" : "Target API(/api/v1/user/details) is not available, you have to get a grant in advance.",
"messages" : {…
Here's a swagger api spec: swagger.yaml
---
swagger: "2.0"
info:
description: "API"
version: "TEMPORARY"
title: "User Details"
termsOfService: "http://wiki.company.net/tos"
contact:
name: "…"
license:
name: "…"
host: "api.company.net"
basePath: "/api/v1"
tags:
- name: "supplier"
description: "Supplier"
schemes:
- "https"
produces:
- "application/json"
paths:
/user/details:
get:
tags:
- "supplier"
summary: "userDetails"
operationId: "getApiV1UserDetails"
consumes:
- "application/json"
produces:
- "application/json;charset=utf-8"
parameters:
- name: "user"
in: "query"
description: "user id"
required: true
type: "integer"
format: "Long"
responses:
200:
description: "OK"
schema:
$ref: "#/definitions/SupplierResponseOfUserDetailsDto"
401:
description: "Unauthorized"
403:
description: "Forbidden"
404:
description: "Not Found"
definitions:
SupplierResponseOfUserDetailsDto:
type: "object"
properties:
body:
$ref: "#/definitions/UserDetailsDto"
message:
type: "string"
successful:
type: "boolean"
UserDetailsDto:
type: "object"
properties:
name:
type: "string"
The swagger-codegen was run from http://editor.swagger.io/ and I followed the api example trying to add in the extra header: api_example.py
from __future__ import print_function
import time
import swagger_client
from swagger_client import ApiClient
from swagger_client import Configuration
from swagger_client.rest import ApiException
from pprint import pprint
# Setup the authentication token header
conf = Configuration()
conf.api_key_prefix = {"teamname": "AG_CONSUMER_TOKEN"}
conf.api_key = {
"teamname": "access-key=31337-70k3n"
}
conf.api_client = ApiClient(None, "X-CAG-Authorization",
conf.get_api_key_with_prefix("teamname"))
# create an instance of the API class
api_instance = swagger_client.SupplierApi()
user = 1
try:
api_response = api_instance.get_api_v1_user_details(user)
pprint(api_response)
except ApiException as e:
print("Exception when calling "
"SupplierApi->get_api_v1_user_details: %s\n" % e)
By putting a print(self.api_client.default_headers) into the supplier_api.py I could see that the header did appear to be set.
{'X-CAG-Authorization': 'AG_CONSUMER_TOKEN access-key=31337-70k3n', 'User-Agent': 'Swagger-Codegen/1.0.0/python'}
So again what should I change in my example to get it to pass on the header and get authorized exactly the way a simple curl call does?
Update I've also tried defining it:
security:
- api_key: []
securityDefinitions:
api_key:
type: "apiKey"
name: "X-CAG-Authorization"
in: "header"
and then only setting up the key with:
swagger_client.configuration.api_key['X-CAG-Authorization'] = \
'access-key=31337-70k3n'
swagger_client.configuration.api_key_prefix['X-CAG-Authorization'] = \
'AG_CONSUMER_TOKEN'
But that didn't change much other than the header disappears from the default headers I was printing.
I've tried your code example and it looks like your headers are actually passed to server.
You can confirm this by adding print headers to swagger_client/rest.py file, just before this:
r = self.pool_manager.request(method, url,
fields=query_params,
preload_content=_preload_content,
timeout=timeout,
headers=headers)
Are you sure there is no problem on server side? Maybe some headers breaking the authentication?
Does the following curl command also work?
curl -v \
-H 'X-CAG-Authorization: AG_CONSUMER_TOKEN access-key=31337-70k3n' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json;charset=utf-8' \
-H 'User-Agent: Swagger-Codegen/1.0.0/python' \
'https://api.company.net/api/v1/user/detail?user=1'
Because this should give you the exact same answer that returned to swagger, which is 401 error. If it does, you can debug from there on server side. If it doesn't, I have no idea.
In your spec, you will need to describe the security setting (API key in your case) similar to this example defined in the securityDefinitions section of the OpenAPI/Swagger spec.
Then in the endpoint, you will need to "apply" the security definition similar to this example
After that, you can set the API key in configuration.py of the auto-generated Python API client and the HTTP request will include the API key accordingly (either in the header or URL query string as defined in the security setting of the spec)
Since the last stable release of Swagger Codegen, there has been quite a lot of enhancements to the auto-generated Python API client so I would suggest you to pull the latest master and build the JAR locally to generate API clients.

URLError doesn't return json body

I have built a REST interface. On '400 Bad Request' it returns a json body with specific information about the error.
(Pdb) error.code
400
Python correctly throws a URLError with these headers
(Pdb) print(error.headers)
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 20 Aug 2016 13:01:05 GMT
Connection: close
Content-Length: 236
There is a content of 236 char, but I cannot find a way to read the body.
I can see the extra information using DHC chrome plugin
{
"error_code": "00000001",
"error_message": "The json data is not in the correct json format.\r\nThe json data is not in the correct json format.\r\n'Execution Start Time' must not be empty.\r\n'Execution End Time' must not be empty.\r\n"
}
However, I cannot find a way in Python to read the body
Here are some of the things I have tried and what was returned.
(Pdb) len(error.read())
0
error.read().decode('utf-8', 'ignore')
''
(Pdb) error.readline()
b''
I found that this works the first time it is called, but does not work if called again.
error.read().decode('utf-8')

GoogleOAuth2 Auth Failure "unauthorized_client"/ Python GoogleAdWord API

Iam trying to accsess OAuth2 in Python (the code is the same as http://code.google.com/p/google-api-ads-python/source/browse/trunk/examples/adspygoogle/adwords/v201302/misc/use_oauth2.py?spec=svn139&r=139):
flow = OAuth2WebServerFlow(client_id='XXX',
client_secret='YYY',
scope='https://adwords.google.com/api/adwords',
user_agent='ZZZ')
authorize_url = flow.step1_get_authorize_url('urn:ietf:wg:oauth:2.0:oob')
code = raw_input('Code: ').strip()
credential = None
try:
credential = flow.step2_exchange(code) #<- error
except FlowExchangeError, e:
sys.exit('Authentication has failed: %s' % e)
This produces a "socket.error: [Errno 10054]" error at the step2_exchange and Python cuts off the excact message.
So after checking the key with OAuthplayground (to get an better errormsg) i get this Error:
HTTP/1.1 400 Bad Request
Content-length: 37
X-xss-protection: 1; mode=block
X-content-type-options: nosniff
X-google-cache-control: remote-fetch
-content-encoding: gzip
Server: GSE
Via: HTTP/1.1 GWA
Pragma: no-cache
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Date: Thu, 06 Jun 2013 13:54:29 GMT
X-frame-options: SAMEORIGIN
Content-type: application/json
Expires: Fri, 01 Jan 1990 00:00:00 GMT
{
"error" : "unauthorized_client"
}
I checked that client_id (for installed Apps) and client_secret are Identical with the one specified in the Google API Console (https://code.google.com/apis/console/).
If i do the whole proces over OAuthPlayground it will work but if i try to use a Token created by playground the App fails also.
Anyone knows how to fix it?
Fixed it. Iam behind a proxy which makes lets the step1 Auth through but apparently not the step2 auth. So a simple
h = httplib2.Http(proxy_info = httplib2.ProxyInfo PROXY DATA .....)
flow.step2_exchange(code, h)
fixed it.
There is an example of how to configure the proxy_info in httplib2 is in https://code.google.com/p/httplib2/wiki/Examples
which says:
import httplib2
import socks
httplib2.debuglevel=4
h = httplib2.Http(proxy_info = httplib2.ProxyInfo(socks.PROXY_TYPE_HTTP, 'localhost', 8000))
r,c = h.request("http://bitworking.org/news/")
However, I've found with the latest httplib2, a cleaned-up socks module comes with it, such that you really want to do something more like:
import httplib2
ht = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP, 'name_or_ip_of_the_proxy_box', proxy_port)
flow.step2_exchange(code, ht)
also, you want to be using a version of oauth2client >= 1.0beta8 which requires the version of httplib2 >= 0.7.4 which is where the support for HTTP proxy was cleaned up in both packages.

How can I view error messages with pycurl?

I have the following pycurl code:
curl = pycurl.Curl()
foo = StringIO()
curl.setopt(pycurl.WRITEFUNCTION, foo.write)
curl.setopt(pycurl.POST, 1)
curl.setopt(pycurl.URL, finalURL)
curl.setopt(pycurl.POSTFIELDS, encodedArgs)
curl.perform()
responseCode = curl.getinfo(pycurl.RESPONSE_CODE)
effectiveURL = curl.getinfo(pycurl.EFFECTIVE_URL)
curl.close()
When the command line curl command comes back I see:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/xml;charset=UTF-8
Content-Length: 216
Date: Thu, 06 Jan 2011 15:49:36 GMT
Some XML Error Here: Something you are trying to do is not permitted.
But I don't see this from pycurl.
How can I extract this alert/error message when using pycurl?
The response from the server is written using the curl option pycurl.WRITEFUNCTION.
In your case, since you are passing it a StringIO object, the response data should be in the foo variable: foo.getvalue()
Reference: http://pycurl.sourceforge.net/doc/curlobject.html

Categories

Resources