I'm trying to extract data from an API, and I expect to receive KeyError and IndexError, so I use the try/except function to catch them. I first create a list of items that I'll loop through to extract information from the API responses. Then I create a dataframe that stores information about items that had no errors.
l= ["a","b","c","d"]
def extract_info_from_one_response(response, idx):
try:
response = response.json()
d = {
### some codes ##
}
## ERROR MANAGEMENT
except KeyError,e:
print idx, l[idx], str(e)
return {}
except IndexError,e:
print idx, l[idx], str(e)
return {}
dat = pd.DataFrame([extract_info_from_one_response(response, idx) for idx, response in enumerate(responses)], index=l)
When errors occur, Python prints out [1]the index of the problematic item, [2] the name of the item and [3]the details on the error that took place. How do I save/capture these three outputs, save them into objects or create a dataframe with these 3 pieces of information?
Are you asking how to trace the error?
If so,the traceback module would help like this:
In [1]: import traceback
In [2]: try:
...: a = []
...: a[2] = 1
...: except Exception:
...: y = traceback.format_exc()
...:
In [4]: print y
Traceback (most recent call last):
File "<ipython-input-2-ac34fe2721d3>", line 3, in <module>
a[2] = 1
IndexError: list assignment index out of range
I am not sure what you are asking, but if you want to save what is printed:
key_err = []
idx_err = []
def extract_info_from_one_response(response, idx):
try:
# your code here
except KeyError, e:
key_err.append((idx, l[idx], '{}'.format(e),))
return {}
except IndexError, e:
idx_err.append((idx, l[idx], '{}'.format(e),))
return {}
# your normal code here
# you need to return something here
dat = pd.DataFrame([extract_info_from_one_response(response, idx) for idx, response in enumerate(responses)], index=l)
print(key_err)
print(idx_err)
Related
Is there a way to print out the contents of a dictionary after raising an exception?
Assuming that the dictionary is this,
d = {"orange":[1,2], "yellow":[5], "red":[2,6,7]}
I would like it to look something like this:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-63-719ba08e2a6c> in <module>
----> 1 raise TypeError("This is a TypeError. Printing out contents of dictionary...")
TypeError: This is a TypeError.
orange
1
2
yellow
5
red
2
6
7
Is this possible?
Append the dictionary to the exception message.
raise TypeError("This is a TypeError. Printing out contents of dictionary...\n" + str(d))
The format is not right, but this does the trick.
d = {"orange":[1,2], "yellow":[5], "red":[2,6,7]}
try:
d["melon"]
except:
raise Exception(f"{str(d)}")
or you can do:
try:
d["melon"]
except:
for k in d.keys():
print(k, d[k])
try:
# code here
except Exception as e:
# Try and replace `Exception` with the error you're getting
print(f"Error = {e}")
for name, val in zip(d.keys(), d.values()):
print(name)
for i in val:
print(i)
Just be mindful that with except Exception:, the try loop will except any error
I have a block of code where exception is happening for some records while executing an imported method (pls see below). My goal is NOT to stop execution, but perhaps print (to understand what's wrong with the data) some values when in error and continue. I tried various way to use "try...except", but no luck! Can someone pls take a look and suggest? Many thanks in advance!
code below
if student_name not in original_df_names_flat:
for org in orgs:
data = calculate(org, data)
result = imported_module.execute(data) # here's the line where exception happens
return result
else:
return data
if student_name not in original_df_names_flat:
for org in orgs:
data = calculate(org, data)
result = None # May fail to get assigned if error in imported_module
try:
result = imported_module.execute(data) # here's the line where exception happens
except Exception as e:
print(f'{e}') # probably better to log the exception
return result
else:
return data
I guess an exception can occur in 2 places. So try this. It covers your exception line and the other possibility. So your program should keep running.
Secondly I have adjusted the indentation of the second try clause so the loop processes all data in orgs. To return all results I have added a list 'results'.
if student_name not in original_df_names_flat:
results = []
for org in orgs:
try:
data = calculate(org, data)
except Exception as e:
print(str(e))
try:
result = imported_module.execute(data) # here's the line where exception happens
results.append(result)
except Exception as e:
print(str(e))
return results
else:
return data
the below solution took care of the problem:
if student_name not in original_df_names_flat:
for org in orgs:
data = calculate(org, data)
try:
result = imported_module.execute(data) # here's the line where exception happens
except exception as e:
print ("exception happened", e)
pass
return data
return result
else:
return data
the above solution took care of the problem
In my code I have a main function, which calls a function that reads some data from a file and returns this data, which is then used in diffrent ways. Obviously there is a risk that the user inputs a filename that is not to be found, resulting in an error. I want to catch this error and output a error message written by me without the traceback etc. I tried using a standard try-except statement, which works almost as intended, except now the data is not read so there are new errors as I try to calculate using empty variabels. Using sys.exit or raise SystemExit in the exception block results in errors beeig written in the console with tracebacks and the whole point of catching the first error feels redundant. I could wrap the whole program in a try-statement, but I have never seen that being done and it feels wrong. How can I either terminate the program in a clean way or hide all the subsequent errors?
def getData(fileName):
try:
file = open(fileName,"r")
data = file.readlines()
file.close()
x = []
y = []
for i in data:
noNewline = i.rstrip('\n')
x.append(float(noNewline.split("\t")[0]))
y.append(float(noNewline.split("\t")[1]))
return x,y
except FileNotFoundError:
print("Some error messages")
def main(fileName):
x,y = getData(fileName)
# diffrent calculations with x and y
Because main is a function, you could return on an error:
def main(filename):
try:
x, y = getData(filename)
except FileNotFoundError:
print("file not found")
return
# calculations here
Solution
sys.exit and SystemExit take optional arguments—0 is considered a successful termination.
Example
sys.exit(0)
raise SystemExit(0)
References
Python sys.exit: https://docs.python.org/3/library/sys.html#sys.exit
below
def getData(fileName):
file = open(fileName,"r")
data = file.readlines()
file.close()
x = []
y = []
for i in data:
noNewline = i.rstrip('\n')
x.append(float(noNewline.split("\t")[0]))
y.append(float(noNewline.split("\t")[1]))
return x,y
def main(fileName):
# if you only want to handle exception coming from 'getData'
try:
x,y = getData(fileName)
except Exception as e:
print(f'could not get data using file {filename}. Reason: {str(e)}')
return
# do something with x,y
if __name__ == "__main__":
main('get_the_file_name_from_somewhere.txt')
I am getting the following error:
ValueError Traceback (most recent call last)
<ipython-input-19-ec485c9b9711> in <module>
31 except Exception as e:
32 print(e)
---> 33 raise e
34 print(i)
35 i = i+1
<ipython-input-19-ec485c9b9711> in <module>
21 # cc = dict(x.split(':') for x in c.split(','))
22 c = '"'.join(c)
---> 23 cc = dict(x.split(':') for x in c.split(','))
24 df_temp = pd.DataFrame(cc.items())
25 df_temp = df_temp.replace('"','',regex=True)
ValueError: dictionary update sequence element #13 has length 1; 2 is required
Below is the block which is throwing the error. I checked out some of the posts here but they are code specific. Not sure is it input issue or the code.
df_final = pd.DataFrame()
i=1
for file in files:
try:
s3 = session.resource('s3')
key = file
obj = s3.Object('my-bucket',key)
n = obj.get()['Body'].read()
gzipfile = BytesIO(n)
gzipfile = gzip.GzipFile(fileobj=gzipfile)
content = gzipfile.read()
content = content.decode('utf-8')
if len(content) > 0:
content = re.findall(r"(?<=\{)(.*?)(?=\})",content)
for c in content:
c= c.split('"')
for index,val in enumerate(c):
if index%2 == 1:
c[index] = val.replace(':','_').replace(',','_')
c = '"'.join(c)
cc = dict(x.split(':') for x in c.split(','))
df_temp = pd.DataFrame(cc.items())
df_temp = df_temp.replace('"','',regex=True)
df_temp = df_temp.T
new_header = df_temp.iloc[0] #grab the first row for the header
df_temp = df_temp[1:] #take the data less the header row
df_temp.columns = new_header
df_final = pd.concat([df_final, df_temp])
except Exception as e:
print(e)
raise e
print(i)
i = i+1
Can you share what is the issue here? This used to work fine before. Do I make a change or ignore the error?
My guess is that your data is malformed. I'm guessing that at some point, x.split(':') is producing a list with only one element in it because there is no : in x, the string being split. This leads, during the creation of a dictionary from this data, to a single value being passed when a pair of values (for "key" and "value") is expected.
I would suggest that you fire up your debugger and either let the debugger stop when it hits this error, or figure out when it happens and get to a point where you're just about to have the error occur. Then look at the data that's being or about to be processed in your debugger display and see if you can find this malformed data that is causing the problem. You might have to run a prep pass on the data to fix this problem and others like it before running the line that is throwing the exception.
I'm running the following:
for server in server_list:
for item in required_fields:
print item, eval(item)
There is a possibility that some keys may not exist, but worse it's represented on a parent key not the one I'm scanning for.
So I'm scanning the json for the following key:
server['server_management']['server_total_cost_of_ownership']['description']
Which doesn't exist but it's actually the parent that is null:
server['server_management']['server_total_cost_of_ownership']
How do I write my code to account for this? It's not giving a key error. Right now I get the following traceback:
Traceback (most recent call last):
File "C:/projects/blah/scripts/test.py", line 29, in <module>
print item, eval(item)
File "<string>", line 1, in <module>
TypeError: 'NoneType' object has no attribute '__getitem__'
Full code:
import csv
import json
import os
import requests
import sys
required_fields = ["server['server_name']","server['server_info']['asset_type']['display_name']",
"server['asset_status']['display_name']", "server['record_owner']['group_name']",
"server['server_management']['server_total_cost_of_ownership']['description']",
"server['server_management']['primary_business_owner']['name']",
"server['environment']['display_name']", "server['is_virtual']",
"server['managed_by']['display_name']", "server['server_info']['billable_ibm']",
"server['server_info']['billing_sub_type']['display_name']",
"server['server_info']['serial_number']", "server['location']['display_name']",
"server['inception_date']", "server['server_info']['decommission_date']" ]
# Query API for all servers
def get_servers_info():
servers_info = requests.get('url')
return servers_info.json()
def get_server_info(sid):
server_info = requests.get('url')
return server_info.json()
server_list = get_servers_info()
for server in server_list:
for item in required_fields:
print item, eval(item)
In fact you should avoid eval. After the json load since you know the key name, you can use a list to go deeper in the tree.
server['server_management']['primary_business_owner']['name']" => ["server_management', 'primary_business_owner', 'name']
Here a snippet for a json validation against a list of required fields.
data={
"d": {
"p":{
"r":[
"test"
]
}
},
"a": 3
}
def _get_attr(dict_, attrs):
try:
src = attrs[:]
root = attrs.pop(0)
node = dict_[root]
null = object()
for i, attr in enumerate(attrs[:]):
try:
node = node.get(attr, null)
except AttributeError:
node = null
if node is null:
# i+2 pop and last element
raise ValueError("%s not present (level %s)" % (attr, '->'.join(src[: i+2])))
return node
except KeyError:
raise ValueError("%s not present" % root)
# assume list of required field
reqs = [
["d", "p", "r"],
["d"],
["k"],
["d", "p", "r", "e"],
]
for req in reqs:
try:
_get_attr(data, req)
except ValueError as E:
print(E)
# prints
# k not present
# e not present (level d->p->r->e)
Ignoring the context of the code and not understanding the use of eval here, the way to do this is to use .get() and seed it with reasonable defaults.
For example:
server['server_management']['server_total_cost_of_ownership']['description']
Can be:
server.get('server_management', {}).get('server_total_cost_of_ownership', {}).get('description', '')
Then if any of the keys do not exist you will always get back an empty description ''.
Your problem here is totally unrelated to using eval[1]. The exception you get is the same as if the code would have been there directly. What you are running (via eval) is:
a = server['server_management']
b = a['server_total_cost_of_ownership']
c = b['description']
Yet, b is None, so resolving it to c will fail. Like a KeyError, you can also catch a TypeError:
for server in server_list:
for item in required_fields:
try:
print item, eval(item)
except TypeError:
print("Guess you're lucky you didn't include a fork bomb in your own code to eval.")
You may of course alternatively pass, print the offending item, open a browser to some page or do whatever error handling is appropriate given your input data.
[1] While not bickering around, I've made a new answer that works without eval. You can use precisely the same error handling:
for server in server_list:
for item in required_fields:
value = server
for key in parse_fields(field):
try:
value = value[key]
except TypeError:
print("Remember Kiddo: Eval is Evil!")
break
else: # for: else: triggers only if no break was issued
print item, value