I would think there is not problem with that, but I am really having troubles with this piece of code and cannot seem to come up with a solution.
I have a dictionary whose keys are proper name, e.g. John Green, and I am using the API of the Sunlight Foundation to retrieve information about congress members (check here). Now I need to request using name and lastname so my code looks something like this:
for key in my_dict:
query_params2 = { 'apikey': 'xxxxxxxxxxx',
'firstname' : key.split()[0],
'lastname' : key.split()[-1]
}
endpoint2 = "http://services.sunlightlabs.com/api/legislators.get.json"
resp2 = requests.get(endpoint2, params = query_params2)
data2 = resp2.json().decode('utf-8')
print data2['response']['legislator']['bioguide_id']
Which gives some error I can not really interpret:
Traceback (most recent call last):
File "my_program.py", line 102, in <module>
data = resp.json()
File "//anaconda/lib/python2.7/site-packages/requests/models.py", line 741, in json
return json.loads(self.text, **kwargs)
File "//anaconda/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "//anaconda/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "//anaconda/lib/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
I would guess it has something to do with the encoding, but I am not sure what and how to solve it.
Needless to say, if I put by hand a name and lastname, the request works perfectly.
Anybody can help with this? Thanks a lot!
This has nothing to do with the encoding. The answer simply isn't JSON. When I try your code with 'John' and 'Green' I get a 400 Bad Request and the content of the response is 'No Such Object Exists'.
Trying John Green in the web interface also comes up with an empty answer. Also the URL in the API documentation differs from the URL in your example.
The following works for me (again no John Green):
import requests
LEGISLATORS_URL = 'https://congress.api.sunlightfoundation.com/legislators'
API_KEY = 'xxxx'
def main():
names = [('John', 'Green'), ('John', 'Kerry')]
for first_name, last_name in names:
print 'Checking', first_name, last_name
response = requests.get(
LEGISLATORS_URL,
params={
'apikey': API_KEY,
'first_name': first_name,
'last_name': last_name,
'all_legislators': 'true'
}
).json()
print response['count']
if response['count'] > 0:
print response['results'][0]['bioguide_id']
if __name__ == '__main__':
main()
Output:
Checking John Green
0
Checking John Kerry
1
K000148
Related
I am using Google Ads API SDK for Python. I want to get some Ads data and put them into a Dataframe to transform a bit. I made a call with the code below:
client = GoogleAdsClient.load_from_storage("config.yaml")
customer_id = '<customer_id>'
ga_service = client.get_service("GoogleAdsService")
query = """
SELECT
campaign.id,
campaign.name,
customer.id
FROM campaign
"""
response = ga_service.search_stream(customer_id=customer_id, query=query)
for batch in response:
for row in batch.results:
print(row)
df = pd.DataFrame(row)
print(df)
Here is the response I receive:
customer {
resource_name: "customers/<customer-id>"
id: <customer-id>
}
campaign {
resource_name: "customers/<customer-id>/campaigns/<campaign-id>"
name: "Test_campaign_1"
id: <campaign-id>
}
Traceback (most recent call last):
File "c:\Users\User\main.py", line 36, in <module>
print(dict(row))
TypeError: 'GoogleAdsRow' object is not iterable
I tried using google.protobuf.json_format to convert the result into json/dict format with the below code
from google.protobuf.json_format import MessageToJson
response = ga_service.search_stream(customer_id=customer_id, query=query)
for batch in response:
for row in batch.results:
print(row)
jsonobj = MessageToJson(row)
But I got the below error message:
File "C:\Users\User\AppData\Local\Programs\Python\Python39\lib\site-packages\proto\message.py", line 605, in __getattr__
raise AttributeError(str(ex))
AttributeError: 'DESCRIPTOR'
Could you please help me with this? Thank you.
Sorry for bothering, but I found this question and got the answer for my question.
I changed my code to below (add ._pb to the response):
response = ga_service.search(customer_id=customer_id, query=query)
dictobj = MessageToDict(response._pb)
df = pd.json_normalize(dictobj,record_path=['results'])
print(df)
and it works!
I have a function that has parameters with username and password. I want to use f strings or any sort of string formatting so that I don't have to hard code the username and password in the code and rather substitute those arguments passed to the function as well as it violates security requirements.
the payload has to be in the format as shown below unless there is another possibility to send the payload in JSON. As this needs to be send via JSON only I guess. If I try to remove the backslash it errors on me for indentation.
How can i leverage string formatting to hide username and password so that I can provide these information during runtime.
import requests
import sys, pprint, json
from getpass import getpass
from multiprocessing import Pool
import yaml
from functools import partial
http_header = {}
url_dict = {}
def getCookie(username, password, ip_addr):
url = "https://"+ip_addr+"/api/aaaLogin.json"
# payload = " {\r\n\"aaaUser\":"\
# " {\r\n\"attributes\":"\
# " {\r\n\"name\": \"admin\",\r\n" \
# " \"pwd\":\"Admin_1234!\"\r\n" \
# " }\r\n " \
# " }\r\n }\r\n"
payload = {
# 'aaaUser':'',
# 'attributes':'',
'name': username,
'pwd': password,
}
json_payload = json.dumps(payload)
headers = {
'Content-Type': "application/json",
'Cache-Control': "no-cache",
}
try:
req = requests.request("POST", url=url, data=json_payload, headers=headers, verify=False)
except:
print('Failed to obtain auth cookie: %s' % (e))
sys.exit(1)
else:
cookie=req.headers['Set-Cookie']
# print(cookie)
return cookie
def genericGetRequest(ip_addr, cookie, apiurl, verb):
url = 'https://'+ip_addr+apiurl
http_header["Cookie"]=cookie
http_header["Host"]=ip_addr
try:
req = requests.request(verb, url=url, headers=http_header, verify=False)
except:
print("There is a problem with the {} request!".format(verb))
else:
return(req)
def getResults(username, password, ip):
cookie=getCookie(username, password, ip)
if cookie:
print("User is logged in. Auth-cookie is %s\n" % cookie)
vlan_list = []
trunk_vlans_dict = {}
for i in range(1, 49):
apiurl = f"/api/mo/sys/intf/phys-[eth1/{i}]/.json"
generic = genericGetRequest(ip, cookie, apiurl, 'GET')
generic = generic.json()
imdata = generic['imdata']
vlan = imdata[0]['l1PhysIf']['attributes']
trunk_vlans_dict[f"eth1/{i}"] = vlan['trunkVlans']
vlan_list.append(trunk_vlans_dict)
print(vlan_list)
if __name__ == '__main__':
username = input("Enter username: ")
print("Enter password")
password = getpass()
if password:
deviceListFile = 'nexus_list.yaml'
with open(deviceListFile) as f:
deviceList = yaml.load(f)
num_threads = 5
print("Retreiving Configuration: ")
pool = Pool(num_threads)
partial_getResults = partial(getResults, username, password)
pool.map(partial_getResults, deviceList)
pool.close()
pool.join()
else:
print("Passwords do not match. Exiting...")
After using json dumps I am getting another error as shown below. BTW I have posted entire code for clarity sake.
ssh://vrxxx#werssefsf:22/sdfsdfsdfsdf/Python_Dev/Test1/pyVENV/bin/python -u /NetworkAutomation/Python_Dev/Test1/nxos_test5.py
Enter username: admin
admin
Enter password
Password: Admin_1234!
/NetworkAutomation/Python_Dev/Test1/nxos_test5.py:82: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
deviceList = yaml.load(f)
Retreiving Configuration:
/NetworkAutomation/Python_Dev/Test1/pyVENV/lib/python3.7/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/NetworkAutomation/Python_Dev/Test1/nxos_test5.py", line 58, in getResults
cookie=getCookie(username, password, ip)
File "/NetworkAutomation/Python_Dev/Test1/nxos_test5.py", line 42, in getCookie
cookie=req.headers['Set-Cookie']
File "/NetworkAutomation/Python_Dev/Test1/pyVENV/lib/python3.7/site-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
KeyError: 'set-cookie'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/NetworkAutomation/Python_Dev/Test1/nxos_test5.py", line 88, in <module>
pool.map(partial_getResults, deviceList)
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
KeyError: 'set-cookie'
Process finished with exit code 1
It's difficult to answer your question without a minimal example. It's also not clear what aaaUser, attributes and ip_addr is used for but I think this is what you are trying to achieve:
import json
def getCookie(username, password, ip_addr):
payload = {
'name': username,
'pwd': password
}
return json.dumps(payload)
print(getCookie(username='admin', password='Admin_1234', ip_addr="127.0.0.0"))
# {"name": "admin", "pwd": "Admin_1234"}
There is unlikely a need for all the new lines and string formatting, you just need to pass the Python dictionary through the json dumps method.
Thanks Johnny. Your suggestion worked. I just have to use aaaUser and attributes in a nested dict format as shown below.
payload = {
'aaaUser': {
'attributes': {
'name': username,
'pwd': password,
}}}
json_payload = json.dumps(payload)
First off I am total noob when it comes to writing python so a lot of what I've done thus far has been all learn as I go so with that said:
I have this bit of code here
if buycott_token != '':
print("Looking up in Buycott")
url = "https://www.buycott.com/api/v4/products/lookup"
headers = {
'Content-Type': 'application/json'
}
data={'barcode':upc,
'access_token':buycott_token
}
try:
r = requests.get(url=url, json=data, headers=headers)
j = r.json()
if r.status_code == 200:
print("Buycott found it so now we're going to gather some info here and then add it to the system")
name = j['products'][0]['product_name']
description = j['products'][0]['product_description']
#We now have what we need to add it to grocy so lets do that
#Sometimes buycott returns a success but it never actually does anything so lets just make sure that we have something
if name != '':
add_to_system(upc, name, description)
except requests.exceptions.Timeout:
print("The connection timed out")
except requests.exceptions.TooManyRedirects:
print ("Too many redirects")
except requests.exceptions.RequestException as e:
print e
98% of the time this works just fine with no issues. Then I'll scan something with my barcode scanner and I'll get
Traceback (most recent call last):
File "./barcode_reader.py", line 231, in <module>
increase_inventory(upc)
File "./barcode_reader.py", line 34, in increase_inventory
product_id_lookup(upc)
File "./barcode_reader.py", line 79, in product_id_lookup
upc_lookup(upc)
File "./barcode_reader.py", line 128, in upc_lookup
name = aj['products'][0]['product_name']
KeyError: 'products'
I am certain that it has something to do with how the json is being returned. Problem is when this is thrown it kills the script and that is that. Thank you for your assistance.
The problem is that there is no 'products' key in your response JSON. The workaround could be providing a default value if a 'products' key is not present:
default_value = [{'product_name': '', 'product_description': ''}]
j.get('products', default_value)[0]['product_name']
or you could simply check whether your response has the products key:
if 'products' not in j:
return 'Product not found'
I think this error is because of API doesn't give you proper json in response. So I think you can check from your side if key is in API response or not.
if 'products' in j:
name = j['products'][0]['product_name']
description = j['products'][0]['product_description']
else:
#Whatever you want when 'product' is not in API response
I've been trying to figure out how to load JSON objects in Python.
def do_POST(self):
length = int(self.headers['Content-Length'])
decData = str(self.rfile.read(length))
print decData, type(decData)
"{'name' : 'journal2'}" <type 'str'>
postData = json.loads(decData)
print postData, type(postData)
#{'name' : 'journal2'} <type 'unicode'>
postData = json.loads(postData)
print postData, type(postData)
# Error: Expecting property name enclosed in double quotes
Where am I going wrong?
Error Code (JScript):
var data = "{'name':'journal2'}";
var http_request = new XMLHttpRequest();
http_request.open( "post", url, true );
http_request.setRequestHeader('Content-Type', 'application/json');
http_request.send(data);
True Code (JScript):
var data = '{"name":"journal2"}';
var http_request = new XMLHttpRequest();
http_request.open( "post", url, true );
http_request.setRequestHeader('Content-Type', 'application/json');
http_request.send(JSON.stringify(data));
True Code (Python):
def do_POST(self):
length = int(self.headers['Content-Length'])
decData = self.rfile.read(length)
postData = json.loads(decData)
postData = json.loads(postData)
Your JSON data is enclosed in extra quotes making it a JSON string, and the data contained within that string is not JSON.
Print repr(decData) instead, you'll get:
'"{\'name\' : \'journal2\'}"'
and the JSON library is correctly interpreting that as one string with the literal contents {'name' : 'journal2'}. If you stripped the outer quotes, the contained characters are not valid JSON, because JSON strings must always be enclosed in double quotes.
For all the json module is concerned, decData could just as well have contained "This is not JSON" and postData would have been set to u'This is not JSON'.
>>> import json
>>> decData = '''"{'name' : 'journal2'}"'''
>>> json.loads(decData)
u"{'name' : 'journal2'}"
>>> json.loads(json.loads(decData))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 1 (char 1)
Fix whatever is producing this string, your view is fine, it's the input that is broken.
To workaround the error 'Expecting property name enclosed in double quotes' you can do:
import json
data = "{'name' : 'journal2'}"
json.loads(json.dumps(data))
You can also go with eval:
data = "{'name' : 'test'}"
eval(data)
It works and returns you a dict.
I'm trying to post a JSON object through a POST. I'm trying to do it as follows:
import json, urllib, urllib2
filename = 'test.json'
race_id = 2530
f = open(filename, 'r')
fdata = json.loads(f.read())
f.close()
prefix = 'localhost:8000'
count = 0
for points in fdata['positions'].iteritems():
print '--' + str(count) + '--------'
url = 'http://'+prefix+'/api/points'
parameters = {'point_data': json.dumps(points), 'race_id': race_id}
data = urllib.urlencode(parameters)
print data
request = urllib2.Request(url, data)
response = urllib2.urlopen(request)
count += 1
break;
print 'Finished adding points'
The data is then received on the other end (I'm using Google App Engine) with:
point_data = json.load(self.request.get('point_data'))
But I get the following error:
ERROR 2010-06-30 15:08:05,367
__init__.py:391] 'unicode' object has no attribute 'read' Traceback (most
recent call last): File
"/home/ian/workspace/google_appengine/google/appengine/ext/webapp/__init__.py",
line 513, in __call__
handler.post(*groups) File "/home/ian/workspace/codebase/track_builder/geo-api.py",
line 276, in post
point_data = json.load(self.request.get('point_data'))
File
"/home/ian/workspace/google_appengine/lib/django/django/utils/simplejson/__init__.py",
line 208, in load
return cls(encoding=encoding, **kw).decode(fp.read()) AttributeError: 'unicode' object has
no attribute 'read' INFO
2010-06-30 15:08:05,376
dev_appserver.py:3266] "POST
/api/points HTTP/1.1" 500 -
Any ideas on how to fix this?
EDIT: As requested here is an example of the points:
(u'1276859700',
{
u'24': {
u'tempc': u'-22.7',
u'gpsq': u'1',
u'altm': u'65527',
u'hd': u'112',
u'hdop': u'0.93',
u'bton': u'0',
u'maxv': u'20.15',
u'idit': u'1',
u'satc': u'10',
u'minv': u'20.15',
u'lat': u'35.271993',
u'btusr': u'0',
u'lng': u'-121.845353',
u'knots': u'7'
},
u'11': {
u'tempc': u'13.0',
u'gpsq': u'1',
u'altm': u'65535',
u'hd': u'130',
u'hdop': u'0.84',
u'bton': u'0',
u'maxv': u'15.96',
u'idit': u'1',
u'satc': u'12',
u'minv': u'15.88',
u'lat': u'34.877815',
u'btusr': u'0',
u'lng': u'-121.386116',
u'knots': u'8'
}
}
EDIT 2:
Thanks to Daniel Roseman and Nick Johnson who both caught my error. I've changed
point_data = json.loads(self.request.get('point_data'))
This has solved the error but, now I'm getting:
ERROR 2010-06-30 16:07:29,807 __init__.py:391] 'list' object has no attribute 'iteritems'
Traceback (most recent call last):
File "/home/ian/workspace/google_appengine/google/appengine/ext/webapp/__init__.py", line 513, in __call__
handler.post(*groups)
File "/home/ian/workspace/codebase/track_builder/geo-api.py", line 255, in post
for time, units in point_data.iteritems():
AttributeError: 'list' object has no attribute 'iteritems'
INFO 2010-06-30 16:07:29,816 dev_appserver.py:3266] "POST /api/points HTTP/1.1" 500 -
which relates to the following code:
class TrackPoint(webapp.RequestHandler):
def post(self):
point_data = json.loads(self.request.get('point_data'))
race_id = self.request.get('race_id')
added = []
failed = []
for time, units in point_data.iteritems():
for unit, data in units.iteritems():
...
Any ideas on this one?
It looks like self.request.get() is returning a unicode object rather than a file-like object. You could try using json.loads() instead of json.load().
json.load() expects a file object, but self.request.get returns the value of the parameter as a string.
The solution is easy: use json.loads.
Also, free tip: I presume from the name that you're bundling your own copy of the json library. App Engine actually includes a copy of simplejson that you can use - just do:
from django.utils import simplejson
first in js I recognize that json object must in STRING format (javascript file)
// using jquery, json2
var js = {"name":"nguyen","age":"1"};
$.post("/", {'data': JSON.stringify(js)}, function(ret){
alert(ret);
});
then in gae
from django.utils import simplejson as json
class PesonReq(webapp.RequestHandler):
def post(self):
t = json.loads(self.request.POST['data'])
self.response.out.write(t['name'])