Develop Reference

Only read a variable if it exists

I'm using the requests module to collect some data from a website. This application runs once every day. The amount of rows of data I get changes every time, per request I can get a maximum 250 rows of data. If there is more then 250 rows of data the API gives me a follow uplink which can be used to get the rows 251 >- 500 etc.
Now I have a problem, sometimes the amount of data is < 250 rows, this means there is no followuplink to use and that's exactly where my program gives the following error:
KeyError: #odata.nextLink
This is a piece of the application:
proxies = {'https': 'proxy.***.***.com:8080'}
headers = {"grant_type": "password",
"username": "****",
"password": "****",
"persistent": "true",
"device": '{"DeviceUniqueId":"b680c452","Name":"Chrome","DeviceVersion":"36","PlatformType":"Browser"}'}
url1 = 'https://****-***.com/odata/Results'
params_1 = (
('$filter', mod_date),
('$count', 'true'),
('$select', 'Status'),
('$expand', 'Result($select=ResultId),Specification($select=Name), SpecificationItem($select=Name,MinimumValue, MaximumValue)\n\n'),)
response_1 = requests.get(url_1, headers=headers, proxies=proxies, params=params_1)
q_1 = response_1.json()
next_link_1 = q_1['#odata.nextLink']
q_1 = [tuple(q_1.values())]
while next_link_1:
new_response_1 = requests.get(next_link_1, headers=headers, proxies=proxies)
new_data_1 = new_response_1.json()
q_1.append(tuple(new_data_1.values()))
next_link_1 = new_data_1.get('#odata.nextLink', None)
Now I actually want Python to only read the variable next_link_1 if its available otherwise it should just ignore it and collect what is available...

You only want to enter the while loop when q_1 has the key '#odata.nextLink' Inside the while loop, this is already accomplished in the line next_link_1 = new_data_1.get('#odata.nextLink', None) You could use the same approach -- setting next_link_1 to None if there is no next link -- before the while loop:
next_link_1 = q_1.get('#odata.nextLink', None)
This can be simplified to
next_link_1 = q_1.get('#odata.nextLink')
as None is already the default default value of dict.get().
NB: The question title is wrong. The variable always exists, as you are setting it. Only the existence of the key #odata.nextLink is fragile. So, what you actually want to do is check the existence of a key in a dictionary. To understand what is going on, you should familiarize yourself with the dict.get() method.
There is also some obvious refactoring possible here, getting rid of the repetition of the first iteration, and moving it into the loop:
proxies = {'https': 'proxy.***.***.com:8080'}
headers = {
'grant_type': 'password',
'username': '****',
'password': '****',
'persistent': 'true',
'device': '{"DeviceUniqueId":"b680c452","Name":"Chrome","DeviceVersion":"36","PlatformType":"Browser"}'
}
params = (
('$filter', mod_date),
('$count', 'true'),
('$select', 'Status'),
('$expand', 'Result($select=ResultId),Specification($select=Name), SpecificationItem($select=Name,MinimumValue, MaximumValue)\n\n'),
)
url = 'https://****-***.com/odata/Results'
data = []
while url:
response = requests.get(
url,
headers=headers,
proxies=proxies,
params=params,
)
response_data = response.json()
data.append(tuple(response_data.values()))
url = response_data.get('#odata.nextLink')
params = tuple()

Use get in both places. Better yet, restructure your loop so that you only need one call.
proxies = {'https': 'proxy.***.***.com:8080'}
headers = {...}
url1 = 'https://****-***.com/odata/Results'
params = (...)
qs = []
next_link = url
get_args = {'headers': headers, 'proxies': proxies, 'params': params}
while True:
response = requests.get(next_link, **get_args)
q = response.json()
qs.append(tuple(q.values())
if (next_link := q.get('#odata.nextLink', None)) is None:
break
if 'params' in get_args:
del get_args['params'] # Only needed in the first iteration
(I'm not terribly excited about how we ensure params is used only on the first iteration, but I think it's better than duplicating the process of defining next_link before the loop starts. Maybe something like this would be an improvement?
get_args = {...} # As above
new_get_args = dict(headers=..., proxies=...) # Same, but without params
while True:
...
if (next_link := ...) is None:
break
get_args = new_get_arg
Repeated assignment to get_args is probably cheaper than repeatedly testing for and deleting the params key, at the cost of having a second dict in memory. You could even drop that after the first iteration by adding a second assignment new_get_args = get_args to the end of the loop, which would result in a pair of do-nothing assignments for later iterations.)

Python Flask JSON KeyError

I'm trying to get some JSON data from another API service and update my flask app's database while user can download some PDF files. That API have 3 keys. 1st one is 'Status'. When that 'Status' key has "success" value, it also have other two keys and values too. Then app works fine without errors.
But when the 'Status' has the 'fail' value, other two keys and values won't be there. I wrote a some exception but it doesn't work and end up with a KeyError, KeyError: 'country'
Here is my code.
#app.route("/pdf/download/<int:pdf_id>", methods=['GET', 'POST'])
def downloads(pdf_id):
current_ip = someIPaddress
req = requests.request("GET", 'http://anotherwebsite.com/json/someIPaddress?fields=169')
req_two = req.json()
status = req_two['status']
country = req_two['country']
city = req_two['city']
download_file = Pdf_info.query.get(pdf_id)
if Ipaddress.query.filter(Ipaddress.ip_address == current_ip, Ipaddress.pdfid == pdf_id).first():
try:
return send_from_directory("pdfs/pdf/", filename=download_file.file_location_name, as_attachment=True)
except FileNotFoundError:
abort(404)
else:
if status == "success":
ip_adding = Ipaddress(ip_address=current_ip, pdfid=pdf_id, downtime=datetime.utcnow(), country=country, location=city)
db.session.add(ip_adding)
db.session.commit()
try:
return send_from_directory("pdfs/pdf/", filename=download_file.file_location_name, as_attachment=True)
except FileNotFoundError:
abort(404)
else:
ip_adding = Ipaddress(ip_address=current_ip, pdfid=pdf_id, downtime=datetime.utcnow())
db.session.add(ip_adding)
db.session.commit()
try:
return send_from_directory("pdfs/pdf/", filename=download_file.file_location_name, as_attachment=True)
except FileNotFoundError:
abort(404)
Can someone explain why this doesn't work or mention a solution please ?.

You are trying to fetch:
country = req_two['country']
city = req_two['city']
before you have tested the output of:
status = req_two['status']
so if status is fail then country= and city= will fail.
Use:
country = req_two.get('country')
city = req_two.get('city')
That will return None if the key is not found instead of a ``KeyError. It also allows you test the countryandcity` variables afterwards.

List in Dictionary

I can't seem to easily access a list value from within a dictionary response from an API.
data = {
'room_id': room,
'how_many': 1
}
response_url = 'https://api.clickmeeting.com/v1/conferences/'+ str(room) +'/tokens'
response1 = requests.post(response_url, headers=headers, data=data).
response1.raise_for_status()
# access JSOn content
jsonResponse = response1.json()
print(jsonResponse)
the response is:
{'access_tokens': [{'token': 'C63GJS', 'sent_to_email': None, 'first_use_date': None}]}
I'm looking to assign the token value to a variable.
Any ideas?

If the list in the access_tokens is always of length 1, you can do something like this:
token = json_response["access_token"][0]["token"]
If there's a potential for more than one item in access_tokens, then something similar:
tokens = []
access_tokens = json_response["access_token"]
tokens = [at["token"] if "token" in at for at in access_tokens]

How to properly handle error from json response

First off I am total noob when it comes to writing python so a lot of what I've done thus far has been all learn as I go so with that said:
I have this bit of code here
if buycott_token != '':
print("Looking up in Buycott")
url = "https://www.buycott.com/api/v4/products/lookup"
headers = {
'Content-Type': 'application/json'
}
data={'barcode':upc,
'access_token':buycott_token
}
try:
r = requests.get(url=url, json=data, headers=headers)
j = r.json()
if r.status_code == 200:
print("Buycott found it so now we're going to gather some info here and then add it to the system")
name = j['products'][0]['product_name']
description = j['products'][0]['product_description']
#We now have what we need to add it to grocy so lets do that
#Sometimes buycott returns a success but it never actually does anything so lets just make sure that we have something
if name != '':
add_to_system(upc, name, description)
except requests.exceptions.Timeout:
print("The connection timed out")
except requests.exceptions.TooManyRedirects:
print ("Too many redirects")
except requests.exceptions.RequestException as e:
print e
98% of the time this works just fine with no issues. Then I'll scan something with my barcode scanner and I'll get
Traceback (most recent call last):
File "./barcode_reader.py", line 231, in <module>
increase_inventory(upc)
File "./barcode_reader.py", line 34, in increase_inventory
product_id_lookup(upc)
File "./barcode_reader.py", line 79, in product_id_lookup
upc_lookup(upc)
File "./barcode_reader.py", line 128, in upc_lookup
name = aj['products'][0]['product_name']
KeyError: 'products'
I am certain that it has something to do with how the json is being returned. Problem is when this is thrown it kills the script and that is that. Thank you for your assistance.

The problem is that there is no 'products' key in your response JSON. The workaround could be providing a default value if a 'products' key is not present:
default_value = [{'product_name': '', 'product_description': ''}]
j.get('products', default_value)[0]['product_name']
or you could simply check whether your response has the products key:
if 'products' not in j:
return 'Product not found'

I think this error is because of API doesn't give you proper json in response. So I think you can check from your side if key is in API response or not.
if 'products' in j:
name = j['products'][0]['product_name']
description = j['products'][0]['product_description']
else:
#Whatever you want when 'product' is not in API response

How to prevent anonymous keys on Firebase?

I'm working on a chat project on the platform Raspberry PI 3, Openelec OS.
Trying to write to the DB and getting unwanted anonymous keys.
Unwanted key marked with yellow. Movie2 and it's keys and values are the wanted result, but I made it manually.
I only ask how can I prevent this anonymous random key to be there and how can I replace it with other key? (string, a movie name for example)
This is my code:
url = 'https://chat-example-97c62.firebaseio.com/Movie1.json'
postdata = {
'date': str(time.asctime( time.localtime(time.time()) )),
'temp': str("Hello from Kodi")
}
req = urllib2.Request(url)
req.add_header('Content-Type','application/json')
data = json.dumps(postdata)
Thanks.

When you send a POST request to Firebase, it automatically generates a Key (Anonymous Key), if you want to use your own key you need to use a PATCH request, this is an example on Python 3:
def update_entry(user, message):
my_data = dict()
my_data["user"] = user
my_data["message"] = message
json_data = json.dumps(my_data).encode()
request = urllib.requests.Request("https://<YOUR-PROJECT-ID>.firebaseio.com/movies/<REPLACE_THIS_WITH_YOUR_DESIRED_KEY>.json", data=json_data, method="PATCH")
try:
loader = urllib.request.urlopen(request)
except urllib.error.URLError as e:
message = json.loads(e.read())
print(message["error"])
else:
print(loader.read())