I'm running the following:
for server in server_list:
for item in required_fields:
print item, eval(item)
There is a possibility that some keys may not exist, but worse it's represented on a parent key not the one I'm scanning for.
So I'm scanning the json for the following key:
server['server_management']['server_total_cost_of_ownership']['description']
Which doesn't exist but it's actually the parent that is null:
server['server_management']['server_total_cost_of_ownership']
How do I write my code to account for this? It's not giving a key error. Right now I get the following traceback:
Traceback (most recent call last):
File "C:/projects/blah/scripts/test.py", line 29, in <module>
print item, eval(item)
File "<string>", line 1, in <module>
TypeError: 'NoneType' object has no attribute '__getitem__'
Full code:
import csv
import json
import os
import requests
import sys
required_fields = ["server['server_name']","server['server_info']['asset_type']['display_name']",
"server['asset_status']['display_name']", "server['record_owner']['group_name']",
"server['server_management']['server_total_cost_of_ownership']['description']",
"server['server_management']['primary_business_owner']['name']",
"server['environment']['display_name']", "server['is_virtual']",
"server['managed_by']['display_name']", "server['server_info']['billable_ibm']",
"server['server_info']['billing_sub_type']['display_name']",
"server['server_info']['serial_number']", "server['location']['display_name']",
"server['inception_date']", "server['server_info']['decommission_date']" ]
# Query API for all servers
def get_servers_info():
servers_info = requests.get('url')
return servers_info.json()
def get_server_info(sid):
server_info = requests.get('url')
return server_info.json()
server_list = get_servers_info()
for server in server_list:
for item in required_fields:
print item, eval(item)
In fact you should avoid eval. After the json load since you know the key name, you can use a list to go deeper in the tree.
server['server_management']['primary_business_owner']['name']" => ["server_management', 'primary_business_owner', 'name']
Here a snippet for a json validation against a list of required fields.
data={
"d": {
"p":{
"r":[
"test"
]
}
},
"a": 3
}
def _get_attr(dict_, attrs):
try:
src = attrs[:]
root = attrs.pop(0)
node = dict_[root]
null = object()
for i, attr in enumerate(attrs[:]):
try:
node = node.get(attr, null)
except AttributeError:
node = null
if node is null:
# i+2 pop and last element
raise ValueError("%s not present (level %s)" % (attr, '->'.join(src[: i+2])))
return node
except KeyError:
raise ValueError("%s not present" % root)
# assume list of required field
reqs = [
["d", "p", "r"],
["d"],
["k"],
["d", "p", "r", "e"],
]
for req in reqs:
try:
_get_attr(data, req)
except ValueError as E:
print(E)
# prints
# k not present
# e not present (level d->p->r->e)
Ignoring the context of the code and not understanding the use of eval here, the way to do this is to use .get() and seed it with reasonable defaults.
For example:
server['server_management']['server_total_cost_of_ownership']['description']
Can be:
server.get('server_management', {}).get('server_total_cost_of_ownership', {}).get('description', '')
Then if any of the keys do not exist you will always get back an empty description ''.
Your problem here is totally unrelated to using eval[1]. The exception you get is the same as if the code would have been there directly. What you are running (via eval) is:
a = server['server_management']
b = a['server_total_cost_of_ownership']
c = b['description']
Yet, b is None, so resolving it to c will fail. Like a KeyError, you can also catch a TypeError:
for server in server_list:
for item in required_fields:
try:
print item, eval(item)
except TypeError:
print("Guess you're lucky you didn't include a fork bomb in your own code to eval.")
You may of course alternatively pass, print the offending item, open a browser to some page or do whatever error handling is appropriate given your input data.
[1] While not bickering around, I've made a new answer that works without eval. You can use precisely the same error handling:
for server in server_list:
for item in required_fields:
value = server
for key in parse_fields(field):
try:
value = value[key]
except TypeError:
print("Remember Kiddo: Eval is Evil!")
break
else: # for: else: triggers only if no break was issued
print item, value
Related
I am trying to search within the EU parliment votes' description.
They are standard xml files. So far I noticed 2 versions of the votes' result: where the description is text and where it is an url.
Fails at this file: https://www.europarl.europa.eu/doceo/document/PV-9-2022-09-12-RCV_FR.xml
Works fine on the following file: https://www.europarl.europa.eu/doceo/document/PV-9-2022-07-06-RCV_FR.xml
My problem is not the failure itself (that's why I added the try clause) but that code execution doesn't jump to the except path, just exits here with this: string indices must be integers
Please help, why isn't the error properly handled?
for currentfile in f:
mytree = ET.parse(xmlfiles + "\\" + currentfile)
myroot = mytree.getroot()
dixml = etree_to_dict(myroot)
for votes in dixml['PV.RollCallVoteResults']['RollCallVote.Result']:
try:
title = votes['RollCallVote.Description.Text'] #fails here
titletype = type(title)
if titletype == dict:
title=title['#text']
except :
title = votes['RollCallVote.Description.Text']['a']['#text']
try:
ltitle = title.lower()
except :
print(type(title))
if stringtosearch in ltitle:
print(title,currentfile)
edit:
full trace:
Message=string indices must be integers
Source=...debug.py
StackTrace:
File "...debug.py", line 44, in <module>
title = votes['RollCallVote.Description.Text']
During handling of the above exception, another exception occurred:
File "...debug.py", line 44, in <module>
title = votes['RollCallVote.Description.Text']
File "...debug.py", line 49, in <module> (Current frame)
title = votes['RollCallVote.Description.Text']['a']['#text']
This would happen if there is a new exception raised within your catch block, I would presume specifically the votes object at some point of you accessing it, raises this exception.
Try printing out votes and then each time you access it by indexing print it out you'll see to which extent you can continue accessing it via string index ( votes['string'] ).
I have a piece of code that fetches data from the ticketmaster API using a function I've named get_event_info. The first revision of the code worked as desired, subsequently I modified the original function to make use of header based authentication instead of URL based. I also added a few lines to the function which were intended to validate the response status code. After making these changes the code began producing the following TypeError:
Traceback (most recent call last):
File "ticketmaster_only_w_headers.py", line 146, in <module>
for event in ticket_search["_embedded"]["events"].items():
TypeError: 'NoneType' object is not subscriptable
I've read quite a bit about this type of error but I'm still unable to determine why my code is producing it in this instance. I would really appreciate an explanation on why my code is producing this error and what troubleshooting methods I should have used to uncover the source error. I'm fairly comfortable with programming but certainly no expert so the simpler the language used the better.
(Function Definition)
def get_event_info(search):
if search in CACHE_DICTION:
d = CACHE_DICTION[search]
else:
api_url = '{0}events/'.format(api_url_base)
payload = {"keyword": search, "apikey": api_token,
"format": "json", "dmaId": "366", "size": 200, "radius": "2"}
response = requests.get(api_url, headers=headers, params=payload)
if response.status_code == 200:
d = json.loads(response.text)
CACHE_DICTION[search] = d
f = open(CACHE_FNAME, 'w')
f.write(json.dumps(CACHE_DICTION))
f.close()
else:
d = None
return d
(Code snippet that triggers the error)
ticket_search = get_event_info("")
for event in ticket_search["_embedded"]["events"]:
a = event["id"]
b = event["name"]
if "dateTime" in event["dates"]["start"]:
c = event["dates"]["start"]["dateTime"].replace(
"T", " ").replace("Z", "")
else:
c = "NONE"
if "end" in event["dates"] and "dateTime" in event["dates"]["end"]:
j = event["dates"]["end"]["dateTime"].replace(
"T", " ").replace("Z", "")
else:
j = "NONE"
(Code that creates, opens, and writes to the cache used in the above code)
CACHE_FNAME = "ticketmaster_cache.json"
try:
cache_file = open(CACHE_FNAME, "r")
cache_contents = cache_file.read()
CACHE_DICTION = json.loads(cache_contents)
cache_file.close()
except:
CACHE_DICTION = {}
The previous revision of the get_event_info function shown below which does not produce any TypeError:
def get_event_info(search, ticketmaster_key = ticketmaster_key):
if search in CACHE_DICTION:
d = CACHE_DICTION[search]
else:
data = requests.get("https://app.ticketmaster.com/discovery/v2/events",
params = {"keyword": search, "apikey": ticketmaster_key,
"format":"json", "dmaId": "366", "size": 200, "radius": "2"})
print(data.url)
d = json.loads(data.text)
CACHE_DICTION[search] = d
f = open(CACHE_FNAME, 'w')
f.write(json.dumps(CACHE_DICTION))
f.close()
return d
Traceback & Error message I see when I run the latest revision of the code:
Traceback (most recent call last):
File "ticketmaster_only_w_headers.py", line 146, in <module>
for event in ticket_search["_embedded"]["events"]:
TypeError: 'NoneType' object is not subscriptable
Whenever you have a function that can explicitly return None, you should always check the return value first:
def func(a):
if a == 1:
return list(range(10)) # could return a list
else:
return None # or it could return None
a = 10
f = func(a)
f[1]
# raises TypeError: NoneType is not subscriptable
# check for NoneType first
if f is not None:
print(f[1])
# otherwise, kick out different result
else:
print('Got "None" for f!')
# Got "None" for f!
Your ticket_search is returned as None, but because your for loop is trying to do a key-lookup, it's failing, because None doesn't support that operation. Your logic, following from the above, should look like:
if ticket_search is not None:
for event in ticket_search["_embedded"]["events"]:
a = event["id"]
else:
raise TypeError
# or do something else
Well, the interpreter is explicitly telling you that you are trying to evaluate something like a[i], where a is None (instead of the intended type, like a list or a dict). In your case, it is either ticket_search itself, or ticket_search["_embedded"].
In any case, if you can rerun your code at all, putting a print(ticket_search) under ticket_search = get_event_info("") should make everything clear.
The below function parses the cisco command output,stores the output in dictionary and returns the value for a given key. This function works as expected when the dictionary contains the output. However, if the command returns no output at all the length of dictionary is 0 and the function returns a key error . I have used exception KeyError: But this doesn't seem to work.
from qa.ssh import Ssh
import re
class crypto:
def __init__(self, username, ip, password, machinetype):
self.user_name = username
self.ip_address = ip
self.pass_word = password
self.machine_type = machinetype
self.router_ssh = Ssh(ip=self.ip_address,
user=self.user_name,
password=self.pass_word,
machine_type=self.machine_type
)
def session_status(self, interface):
command = 'show crypto session interface '+interface
result = self.router_ssh.cmd(command)
try:
resultDict = dict(map(str.strip, line.split(':', 1))
for line in result.split('\n') if ':' in line)
return resultDict
except KeyError:
return False
test script :
obj = crypto('uname', 'ipaddr', 'password', 'router')
out = obj.session_status('tunnel0')
status = out['Peer']
print(status)
Error
Traceback (most recent call last):
File "./test_parser.py", line 16, in <module>
status = out['Peer']
KeyError: 'Peer'
The KeyError did not happend in the function session_status,it is happend in your script at status = out['Peer'].So your try and except in session_status will not work.you should make a try and except for status = out['Peer']:
try:
status = out['Peer']
except KeyError:
print 'no Peer'
or :
status = out.get('Peer', None)
Your exception is not in the right place. As you said you just return an empty dictionary with your function. The exception is trying to lookup the key on empty dictionary object that is returned status = outertunnel['Peer']. It might be easier to check it with the dict get function. status = outertunnel.get('Peer',False) or improve the test within the function session_status, like testing the length to decide what to return False if len(resultDict) == 0
This explains the problem you're seeing.
The exception happens when you reference out['Peer'] because out is an empty dict. To see where the KeyError exception can come into play, this is how it operates on an empty dict:
out = {}
status = out['Peer']
Throws the error you're seeing. The following shows how to deal with an unfound key in out:
out = {}
try:
status = out['Peer']
except KeyError:
status = False
print('The key you asked for is not here status has been set to False')
Even if the returned object was False, out['Peer'] still fails:
>>> out = False
>>> out['Peer']
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
out['Peer']
TypeError: 'bool' object is not subscriptable
I'm not sure how you should proceed, but dealing with the result of session_status not having the values you need is the way forward, and the try: except: block inside the session_status function isn't doing anything at the moment.
I am trying to Retrieve multiple keyword term tweets by following specific group of users. Using the code below:
I have posted one more code before that regarding issues for value error:
I figure it out somehow but again I am stuck because of this traceback
import tweepy
from tweepy.error import TweepError
consumer_key=('ABC'),
consumer_secret=('ABC'),
access_key=('ABC'),
access_secret=('ABC')
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api=tweepy.API(auth)
class CustomStreamListener(tweepy.StreamListener):
def on_status(self, status):
try:
print "%s\t%s\t%s\t%s" % (status.text,
status.author.screen_name,
status.created_at,
status.source,)
except Exception, e:
print error
#def filter(self, follow=None, track=None, async=False, locations=None):
#self.parameters = {}
#self.headers['Content-type'] = "application/x-www-form-urlencoded"
#if self.running:
#raise TweepError('Stream object already connected!')
#self.url = '/%i/statuses/filter.json?delimited=length' % STREAM_VERSION
def filter(self, follow=None, track=None, async=False, locations=None):
self.parameters = {}
self.headers['Content-type'] = "application/x-www-form-urlencoded"
if self.running:
raise TweepError('Stream object already connected!')
self.url = '/%i/statuses/filter.json?delimited=length' % STREAM_VERSION
if obey:
self.parameters['follow'] = ' '.join(map(str, obey))
if track:
self.parameters['track'] = ' '.join(map(str, track))
if locations and len(locations) > 0:
assert len(locations) % 4 == 0
self.parameters['locations'] = ' '.join('%.2f' % l for l in locations)
self.body = urllib.urlencode(self.parameters)
self.parameters['delimited'] = 'length'
self._start(async)
def on_error(self, status_code):
return True
streaming_api = tweepy.streaming.Stream(auth, CustomStreamListener(), timeout=60)
list_users = ['17006157','59145948','157009365','16686144','68044757','33338729']#Some ids
list_terms = ['narendra modi','robotics']#Some terms
streaming_api.filter(follow=[list_users])
streaming_api.filter(track=[list_terms])
I am getting a traceback:
Traceback (most recent call last):
File "C:\Python27\nytimes\26052014\Multiple term search with multiple addreses.py", line 49, in <module>
streaming_api.filter(follow=[list_users])
File "build\bdist.win32\egg\tweepy\streaming.py", line 296, in filter
encoded_follow = [s.encode(encoding) for s in follow]
AttributeError: 'list' object has no attribute 'encode'
Please help me resolving the issue.
You define list_users here
list_users = ['17006157','59145948','157009365','16686144','68044757','33338729']
and then you pass it to streaming_api.filter like this
streaming_api.filter(follow=[list_users])
When the streaming_api.filter function is iterating over the value you pass as follow, it gives the error
AttributeError: 'list' object has no attribute 'encode'
The reason for this is as follows
You call streaming_api.filter like this
streaming_api.filter(follow=[list_users])
Here
streaming_api.filter(follow=[list_users])
you are trying to pass your list as value for follow, however because you put list_users in enclosing [] you are creating a list in a list. Then streaming_api.filter iterates over follow, calling .encode on each entry as we see here
[s.encode(encoding) for s in follow]
But the entry s is a list while it should be a string.
That is because you accidentally created a list in a list like you can see above.
The solution is simple. Change
streaming_api.filter(follow=[list_users])
to
streaming_api.filter(follow=list_users)
To pass a list to a function, you can just specify the name. No need to enclose it in []
Same applies to the last line. Change
streaming_api.filter(track=[list_terms])
to
streaming_api.filter(track=list_terms)
To avoid writing a long, ugly ternary a dozen times, such as p_count = root.pool_count if hasattr(root, 'pool_count') else (-1), I wrote the GetThis(el, attr, err="") function.
How do I get the function to concatenate the values of el & attr into an element, instead of attr as a literal?
test_data = ("""
<rsp stat="ok">
<group id="34427465497#N01" iconserver="1" iconfarm="1" lang="" ispoolmoderated="0" is_member="0" is_moderator="0" is_admin="0">
<name>GNEverybody</name>
<members>245</members>
<pool_count>133</pool_count>
<topic_count>106</topic_count>
<restrictions photos_ok="1" videos_ok="1" images_ok="1" screens_ok="1" art_ok="1" safe_ok="1" moderate_ok="0" restricted_ok="0" has_geo="0" />
</group>
</rsp>
""")
################
from lxml import etree
from lxml import objectify
################
def GetThis(el, attr, err=""):
if hasattr(el, attr):
return el.attr
else:
return err
################
Grp = objectify.fromstring(test_data)
root = Grp.group
gName = GetThis(root, "name", "No Name")
err_tst = GetThis(root, "not-there", "Error OK")
p_count = root.pool_count if hasattr(root, 'pool_count') else (-1)
As it is, I get the following error:
Traceback (most recent call last):
File "C:/Mirc/Python/Temp Files/test_lxml.py", line 108, in <module>
gName = GetThis(root, "name", "")
File "C:/Mirc/Python/Temp Files/test_lxml.py", line 10, in GetThis
print (el.attr)
File "lxml.objectify.pyx", line 218, in lxml.objectify.ObjectifiedElement.__getattr__ (src\lxml\lxml.objectify.c:3506)
File "lxml.objectify.pyx", line 437, in lxml.objectify._lookupChildOrRaise (src\lxml\lxml.objectify.c:5756)
AttributeError: no such child: attr
Thank you!
Your GetThis function is not needed since you can simply use getattr:
gName = getattr(root, "name", "No Name")
err_tst = getattr(root, "not-there", "Error OK")
The first argument is the object, the second is the attribute, and the third, which is optional, is a default value to return if the attribute doesn't exist (if this last part is omitted, an AttributeError is raised instead).
The two lines of code above are equivalent to these:
gName = root.name if hasattr(root, "name") else "No Name"
err_tst = root.not-there if hasattr(root, "not-there") else "Error OK"
Python has a function called getattr(foo,"bar") which return the value of the bar attr of the foo object. Just read the doc ..