I have the following pycurl code:
curl = pycurl.Curl()
foo = StringIO()
curl.setopt(pycurl.WRITEFUNCTION, foo.write)
curl.setopt(pycurl.POST, 1)
curl.setopt(pycurl.URL, finalURL)
curl.setopt(pycurl.POSTFIELDS, encodedArgs)
curl.perform()
responseCode = curl.getinfo(pycurl.RESPONSE_CODE)
effectiveURL = curl.getinfo(pycurl.EFFECTIVE_URL)
curl.close()
When the command line curl command comes back I see:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/xml;charset=UTF-8
Content-Length: 216
Date: Thu, 06 Jan 2011 15:49:36 GMT
Some XML Error Here: Something you are trying to do is not permitted.
But I don't see this from pycurl.
How can I extract this alert/error message when using pycurl?
The response from the server is written using the curl option pycurl.WRITEFUNCTION.
In your case, since you are passing it a StringIO object, the response data should be in the foo variable: foo.getvalue()
Reference: http://pycurl.sourceforge.net/doc/curlobject.html
Related
I'm trying to use a Python script to call the API detailed in the link below:
https://developer.wmata.com/docs/services/gtfs/operations/5cdc51ea7a6be320cab064fe?
When I use the code below, it always returns a 404 error:
import requests
import json
def _url(path):
return "http://api.wmata.com" + path
def pull_data():
return requests.get(_url("/gtfs/bus-gtfsrt-tripupdates.pb"), params=params)
def jprint(obj):
# create a formatted string of the Python JSON object
text = json.dumps(obj, sort_keys=True, indent=4)
print(text)
# authenticate with your api key
params = {
"apiKey": "mykey",
}
response = pull_data()
print(response)
jprint(response.json())
I have also tried using the python code provided in the link, but it returns meaningless response content as shown below. Any attempts to decode the content have been unsuccessful.
Request-Context: appId=cid-v1:2833aead-1a1f-4ffd-874e-ef3a5ceb1de8
Cache-Control: public, must-revalidate, max-age=5
Date: Thu, 11 Feb 2021 22:05:31 GMT
ETag: 0x8D8CED90CC8419C
Content-Length: 625753
Content-MD5: fspEFl7LJ8QbZPgf677WqQ==
Content-Type: application/octet-stream
Expires: Thu, 11 Feb 2021 22:05:37 GMT
Last-Modified: Thu, 11 Feb 2021 22:04:49 GMT
1.0�Ԗ��
1818410080�
181841008020210211*52!�Ԗ�"19032("�Ԗ�"18173(#�Ԗ�"7779($�Ֆ�"18174(%�Ֆ�"7909(&�Ֆ�"7986('�Ֆ�"8039((�Ֆ�"8130()�֖�"8276(+�֖�"8313(,�ז�"8403(-�ז�"8452(.�ז�"8520(/�ؖ�"8604(1�ؖ�"8676(
7070 �Ӗ�(����������
1814174080�
181417408020210211*P129�Ԗ�"2373(:�Ԗ�"2387(;�Ԗ�"17296(=�Ֆ�"17212(>�֖�"2444(?�֖�"2493(#�֖�"2607(A�֖�"14633(B�֖�"2784(C�ז�"2832(D�ז�"2843(E�ז�"2848(F�ז�"2875(G�ؖ�"2945(H�ؖ�"2987(I�ؖ�"21946(K�ٖ�"14636(L�ٖ�"3122(M�ٖ�"3227(N�ٖ�"3308(O�ٖ�"3411(P�ٖ�"3500(Q�ٖ�"3539(R�ٖ�"14637(S�ږ�"3685(T�ږ�"15195(U�ږ�"15196(V�ۖ�"4243(W�ۖ�"4443(X�ۖ�"4517(Y�ۖ�"4631([�ܖ�"11962(
8002 �Ӗ�(/�
1825989080�
182598908020210211*7Y�Ӗ�"2158(�Ԗ�"2215(�Ԗ�"2259(�Ԗ�"2292(�Ԗ�"2299(�Ֆ�"18701(�Ֆ�"2310( �Ֆ�"2245(!�Ֆ�"2174("�Ֆ�"1987(#�֖�"1937(%�֖�"1864(
3191 �Ӗ�(��
1819988080�
Any guidance or direction would be greatly appreciated!
Change as pull_data function as follows:
def pull_data():
return requests.get(_url("/gtfs/bus-gtfsrt-tripupdates.pb"), headers=headers)
Then rename params module global variable to headers .
headers = {"apiKey": "mykey"}
WMATA looks for a apiKey in the headers, not in the query params.
Update: I noticed they use api_key for some samples, and apiKey for another ones. For example see:
https://developer.wmata.com/docs/services/gtfs/operations/5cdc51ea7a6be320cab064fe
Update 2: Notice the content type in the response headers :
print(response.headers['content-type'])
# application/octet-stream
it is not a JSON. You can get contents as follows:
print(response.content)
Worked example:
import requests
API_URL = 'https://api.wmata.com'
def _prepare_url(path):
return f'{API_URL}/{path.lstrip("/")}'
def pull_data(**options):
url = _prepare_url('/gtfs/bus-gtfsrt-tripupdates.pb')
return requests.get(url, **options)
response = pull_data(headers={'api_key': 'secret'})
print(response.content)
Try changing your URL so that it is using https instead of http. The documentation that you have linked at Bus RT Trip Updates seems to indicate that https is required.
Change this:
def _url(path):
return "http://api.wmata.com" + path
to make it this:
def _url(path):
return "https://api.wmata.com" + path
I am querying the New Relic API, and trying to pull CPU utilization fron the metrics that they provide. When I run the following curl command (after exporting the correct proxy settings), I see the following information (which contains the percentage value that I want) -
curl -X GET "https://api.newrelic.com/v2/applications/140456413/hosts/21044947/metrics/data.json" -H "X-Api-Key:myapikey" -i -d 'names[]=CPU/User+Time&values[]=percent&summarize=true&from=2018-07-15 16:22:30&to=2018-07-16 16:22:30'
HTTP/1.0 200 Connection Established
Proxy-agent: Apache
HTTP/1.1 200 OK
Server: openresty
Date: Tue, 17 Jul 2018 11:53:22 GMT
Content-Type: application/json
Content-Length: 290
Connection: keep-alive
Status: 200 OK
X-UA-Compatible: IE=Edge,chrome=1
ETag: "a35b451e27a49d0d5f4e16715429a17d"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: f4f8675f095aba80d5e089fbcbf1b172
X-Runtime: 0.168283
X-Rack-Cache: miss
{"metric_data":{"from":"2018-07-15T16:22:30+00:00","to":"2018-07-16T16:22:30+00:00","metrics_not_found":[],"metrics_found":["CPU/User Time"],"metrics":[{"name":"CPU/User Time","timeslices":[{"from":"2018-07-15T16:22:00+00:00","to":"2018-07-16T16:22:00+00:00","values":{"percent":9.52}}]}]}}
However, when I try to implement it inside of the Python requests module, the "Percent" value that I am interested in seeing is returning 0. This is my code to call it -
options = {"names[]": "CPU/User+Time", "values[]": "percent", "summarize": "true", "from": str(end_date), "to": str(start_date - datetime.timedelta(hours=6))}
path = "applications/140456413/hosts/" + server_id + "/metrics/data.json"
api_key = "myapikey"
headers = {'X-Api-Key': api_key}
url = "https://api.newrelic.com/v2/" + path
r = requests.get(url, headers=headers, data=options, proxies=myproxy.proxies)
This is what I get instead (notice the percent value is now 0) -
{u'metric_data': {u'metrics': [{u'timeslices': [{u'values': {u'percent': 0}, u'to': u'2018-07-17T01:35:00+00:00', u'from': u'2018-07-16T07:35:00+00:00'}], u'name': u'CPU/User+Time'}], u'to': u'2018-07-17T01:35:59+00:00', u'metrics_found': [u'CPU/User+Time'], u'from': u'2018-07-16T07:35:59+00:00', u'metrics_not_found': []}}
How can I adjust the python request to match the same output as the curl command? We were originally passing in options inside of the "options" variable using ='s instead of key/value pairs, but the requests module would not process them in this format.
These were the docs pages I referencesd -
https://docs.newrelic.com/docs/apis/rest-api-v2/requirements/specifying-time-range-v2
https://docs.newrelic.com/docs/apis/rest-api-v2/application-examples-v2/get-average-cpu-usage-host-app
https://docs.newrelic.com/docs/apis/rest-api-v2/getting-started/introduction-new-relic-rest-api-v2#examples
Thanks.
Change the options a bit:
"values[]": "percent"
to:
"values": ["percent"]
I have built a REST interface. On '400 Bad Request' it returns a json body with specific information about the error.
(Pdb) error.code
400
Python correctly throws a URLError with these headers
(Pdb) print(error.headers)
Cache-Control: no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sat, 20 Aug 2016 13:01:05 GMT
Connection: close
Content-Length: 236
There is a content of 236 char, but I cannot find a way to read the body.
I can see the extra information using DHC chrome plugin
{
"error_code": "00000001",
"error_message": "The json data is not in the correct json format.\r\nThe json data is not in the correct json format.\r\n'Execution Start Time' must not be empty.\r\n'Execution End Time' must not be empty.\r\n"
}
However, I cannot find a way in Python to read the body
Here are some of the things I have tried and what was returned.
(Pdb) len(error.read())
0
error.read().decode('utf-8', 'ignore')
''
(Pdb) error.readline()
b''
I found that this works the first time it is called, but does not work if called again.
error.read().decode('utf-8')
I want to specify the HTTP response charset by modifying the Content-Type header. However, it doesn't work.
Here is a short example:
#coding=utf-8
import cherrypy
class Website:
#cherrypy.expose()
def index(self):
cherrypy.response.headers['Content-Type']='text/plain; charset=gbk'
return '。。。'.encode('gbk')
cherrypy.quickstart(Website(),'/',{
'/': {
'tools.response_headers.on':True,
}
})
And when I visit that page, the Content-Type is changed mysteriously to text/plain;charset=utf-8, causing mojibake in the browser.
C:\Users\Administrator>ncat 127.0.0.1 8080 -C
GET / HTTP/1.1
Host: 127.0.0.1:8080
HTTP/1.1 200 OK
Server: CherryPy/7.1.0
Content-Length: 6
Content-Type: text/plain;charset=utf-8
Date: Mon, 22 Aug 2016 01:08:13 GMT
。。。^C
It seems that CherryPy detect content encoding and override the charset automatically. If so, how can I disable this feature?
All right. Solved this problem by tampering cherrypy.response.header_list directly.
#coding=utf-8
import cherrypy
def set_content_type():
header=(b'Content-Type',cherrypy.response._content_type.encode())
for ind,(key,_) in enumerate(cherrypy.response.header_list):
if key.lower()==b'content-type':
cherrypy.response.header_list[ind]=header
break
else:
cherrypy.response.header_list.append(header)
cherrypy.tools.set_content_type=cherrypy.Tool('on_end_resource',set_content_type)
class Website:
#cherrypy.expose()
#cherrypy.tools.set_content_type()
def index(self):
cherrypy.response._content_type='text/plain; charset=gbk'
return '。。。'.encode('gbk')
cherrypy.quickstart(Website(),'/')
I had success to set the content-type-charset by setting/manipulate the request header attribute "Accept-Charset".
cherrypy.request.headers["Accept-Charset"] = "ISO-8859-1"
cherrypy.response.headers["Content-Type"] = "text/xml"
The result:
>curl -I https://127.0.0.1/url?param=value
HTTP/1.1 200 OK
Content-Type: text/xml;charset=ISO-8859-1
Server: CherryPy/18.6.0
Date: Mon, 10 Aug 2020 11:54:49 GMT
Content-Length: 288
Set-Cookie: session_id=d28fa46a1a3d901d9502038255ce869b21add5cc; expires=Mon, 10 Aug 2020 12:54:49 GMT; Max-Age=3600; Path=/
I am trying to upload a file to the server in a simple way and I am getting the following error:
HTTP/1.1 411 Length Required
Content-Type: text/html
Date: Wed, 01 Jul 2015 03:05:33 GMT
Connection: close
Content-Length: 24
Length Required
I tried to insert length in different parts and looks like is not working, any suggestions?
import socket
import httplib
import os.path
target_host = "192.168.1.1"
target_port = 80
total_size = os.path.getsize('/root/test.html')
client = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
client.connect((target_host,target_port))
client.send("PUT /root/test.html HTTP/1.1\r\nHost:192.168.1.1\r\n\r\n" )
response = client.recv(4096)
print response
Have you tried implementing this with requests?
You could easily specify the missing content-length header that is causing your error.
Check out this answer: Is there any way to do HTTP PUT in python
Here's an example using the requests library:
payload = {'username': 'me', 'email': 'me#me.com'}
>>> r = requests.put("http://somedomain.org/endpoint", data=payload)