Accessing dictionary elements in python - python

I working with some twitter data, I get the data by monitoring the twitter stream, then I save the results in a *.txt file
I´m trying to manipulate this txt file with python, for that I use the json.loads() instruction, with every line in the file where the twitter stream result was saved, in that way I got every file line as an json object.
for line in twitter_file
data = json.loads(line)
The json object (one json object for one file line) is loaded in a variable called "data"
Everyting is working there, when I try to access an element in the "data" json object I can do it, for example, if I can see the place element, I can do it with data["place"], so I got this:
"place":{
"id":"00a961d91f76dde8",
"url":"https:\/\/api.twitter.com\/1.1\/geo\/id\/00a961d91f76dde8.json",
"place_type":"city",
"name":"Capital - Corrientes",
"full_name":"Capital - Corrientes",
"country_code":"AR",
"country":"Argentina",
"contained_within":[]
}
I can access the "place" element, if I execute print data["place"] , I can see the text displayed above.
The problem comes when I'm trying to access an specific key in place dictionary,
I tried to do it in this way
data["place"]["place_type"]
I'm waiting to get the "city" value as result, but I can not do it, I get the following error: TypeError: "NoneType" object has no attribute '__getitem__'
I also tried other ways to display the key-value pairs with the following:
for key,value in data["data"].items()
print key
But I don't get the result
I also tried with
print data["place"].keys()[0]
To get printed the first element, but It doesn't work either. I got the following error message: AttributeError: 'NoneType' object has no attribute 'keys'
It seems that my data["place"] is not being considered as a dictionary by Python, that's what I guess, but I'm not sure, I'm pretty new at Python, so any comment will be very helpful for me.


You are looping over a file and loading each line as JSON. Some of those JSON objects have place set to a dictionary, others have it set to None.
Whenever you just loaded a line with the value associated with the place key set to None, you'll get the AttributeError exception.
You need to test for this first:
place = data['place']
if place is not None:
print place['place_type']

Related

Python web scraping nested dict key pairs - AttributeError

I'm attempting to scrape PGA stats from the API below.
url = 'https://statdata.pgatour.com/r/stats/current/02671.json?userTrackingId=exp=1594257225~acl=*~hmac=464d3dfcda2b2ccb384b77ac7241436f25b7284fb2eb0383184f48cbdff33cc4'
response = requests.get(url)
pga_stats = response.json()
I would like to only select the nested keys identified in this image. I've been able to traverse to the 'year' key with the below code, but I receive the following AttributeError for anything beyond that.
test = pga_stats.get('tours')[0].get('years')
(prints reduced dictionary)
test = pga_stats.get('tours')[0].get('years').get('stats')
'list' object has no attribute 'get'
My end goal is to write this player data to a csv file. Any suggestions would be greatly appreciated.

pga_stats.get('tours')[0].get('years') returns a list, not a dict. You actually want to use the get method on it's first element, like this:
test = pga_stats.get('tours')[0].get('years')[0].get('stats')

root.find() not working when called from dictionary- element tree

Im currently using the module element tree & urllib to access/parse and return values from xml files.
Using the root.find/root.findall() methods along with XPath syntax to locate desired info in xml. Then using (.text) to return the value of child/grandchild elements.
When assigning each root.find() to a variable then taking the variable and attaching (.text), i am able to parse and return the value with no issues.
(i.e)
x= root.find(./Cameras/Camera/Connected')
print (x.text)
==> True
However I would like to place these "root.find()" in a dictionary and call on them later in the script.
(i.e)
location= {
'Cam': "root.find('./Cameras/Camera/Connected')",
'Mic': "root.findall('./Audio/Input/Connectors/Microphone')",
'Prod_ID': "root.find('./SystemUnit/ProductPlatform')"
}
However, when indexing to dictionary by key and then attempting the add the (.text) i get the following error;
y=location['Cam']
print (y.text)
==> AttributeError: 'str' object has no attribute 'text'
So this maybe a simple issue im overlooking but do these two methods return the same value? Can the element tree module read root.find() from dictionaries?

You're setting your dictionary values to a literal string. That is, this:
'Cam': "root.find('./Cameras/Camera/Connected')",
Is setting the value of key Cam to the string value root.find('./Cameras/Camera/Connected'). You want to actually call the function and set the key to the return value, so you need to drop the quotes:
'Cam': root.find('./Cameras/Camera/Connected'),

How to unwrap json file in python?

I have a rather messy json file one of the pair values of which looks like this.
'response': '{"Results" : [{"results" : [{"id" : "2912362261001","usageLimitType":"allocation","usageLimit":"100","currentUsage":"45","remainingUsage":"55","accountValidThrough":"03-14-2020",\r"GivenName":"John","FamilyName":"Smith", "Email":[{"Address":"mizgier.agata#gmail.com","Label":"personal"}]
I would like to unwrap/tidy it up into the following
'response': "id" : "2912362261001",
"usageLimitType":"allocation",
"usageLimit":"100",
"currentUsage":"45",
"remainingUsage":"55",
"accountValidThrough":"03-14-2020",
"GivenName":"John",
"FamilyName":"Smith",
"Email":"mizgier.agata#gmail.com"}]
Not sure how to get rid of the unnecessary 'results' thing before 'id'
Not sure how to get rid of '\r' before 'Given name' as I don't know what it corresponds to
How do I remove unnecessary 'address' and 'label' from 'Email'?
I am new to json, so any help is appreciated :)

JSON is how you're storing the object, but once you've loaded it into python (using the json loads function) what you get is a dictionary that you can act on just like any other python dict.
object = json.dumps(raw_json)
object = object['Results'][0]
object['email'] = object['email']['address']
The first line converts from json to the object, the second line removes that extra "Results" parent and turns object into the subset you want, and the last line makes email just the address. You don't have to worry about the \r because it wasn't inside of a field and the json dumps removes it.

Robotframework - updating value in json dictionary gives error AttributeError: 'list' object has no attribute 'update'

I have this JSON object and i like to update a value within the object. I found a way how i should do this on stackoverflow (Json handling in ROBOT) and its failing and i don't understand why.
This is de object:
{"elementKey":"P690-C0-C3-B1","fields":[{"key":"P690-C1-C2-C1-C1-C1-F0","values":[]},{"key":"P690-C0-C2-F8","values":["1200"]},{"key":"P690-C0-C2-F9","values":["22000"]},{"key":"P690-C0-C2-F11","values":["I"]},{"key":"P690-C0-C2-F10","values":["2200"]},{"key":"P690-C0-C2-C0-C0-F0","values":["98-zsg-2"]},{"key":"P690-C1-C0-C0-F1","values":["Personenauto"]},{"key":"P690-C1-C0-C0-F2","values":["Personenauto KVP"]},{"key":"P690-C0-C2-F6","values":["B"]},{"key":"P690-C0-C2-F7","values":["75"]},{"key":"P690-C0-C2-F4","values":["2"]},{"key":"P690-C0-C2-F5","values":["5"]},{"key":"P690-C0-C2-F2","values":["model"]},{"key":"P690-C0-C2-F3","values":["2017"]},{"key":"P690-C1-C2-C2-C2-C1-F0","values":[]},{"key":"P690-C0-C2-F1","values":["merk"]}]}
In Robot frame I made this test, inspired on the given link.
${json_string}= Set Variable "see text above"
${json}= Evaluate json.loads('''${json_string}''') json
Set To Dictionary ${json["fields"]} ${new_value}
${json_string}= evaluate json.dumps(${json}) json
With ${new_value} i tried value=shizzleliz, value[0]=shizzleliz, value[1]=shizzleliz, P690-C1-C2-C1-C1-C1-F0=shizzleliz
All give the error: AttributeError: 'list' object has no attribute 'update'
When i change ${json["fields"]} to ${json} then the give value is set to the library but not in de fields section/collection.
Does anyone have a clue of what i'm doing wrong? And if you have a suggestion how i can update the value, i'd like that very much :)
target is to change: {"key":"P690-C1-C2-C1-C1-C1-F0","values":[]}
to: {"key":"P690-C1-C2-C1-C1-C1-F0","values":["shizzleliz"]}

For the first part in your question - the error AttributeError: 'list' object has no attribute 'update', you've already seen the comment - you're calling Set To Dictionary on a list object, which cannot pass.
For the second part, in order to set that value when the key is equal to something, you have to iterate over all the list members, and set it based on a condition over the key:
${json_string}= Set Variable see text above
${json1}= Evaluate json.loads('''${json_string}''') json
${target value}= Create List shizzleiz
:FOR ${element} IN #{json1["fields"]}
\ Run Keyword If "${element['key']}" == "P690-C1-C2-C1-C1-C1-F0"
... Set To Dictionary ${element} values=${target value}
${json_string}= evaluate json.dumps(${json1}) json
It looks a little cumbersome in RF (compared to python); one remark - it's never a good idea to name a local variable the same as a module - thus I've renamed it to ${json1}

I found an easier solution using Catenate where I needed to randomize two values in the json body.
${shizzleiz}= shizzleiz # or whatever you want to appear there
${json_string}= Catenate {"elementKey":"P690-C0-C3-B1","fields":[{"key":"P690-C1-C2-C1-C1-C1-F0","values":[]},{"key":"P690-C0-C2-F8","values":["1200"]},{"key":"P690-C0-C2-F9","values":["22000"]},{"key":"P690-C0-C2-F11","values":["I"]},{"key":"P690-C0-C2-F10","values":["2200"]},{"key":"P690-C0-C2-C0-C0-F0","values":["98-zsg-2"]},{"key":"P690-C1-C0-C0-F1","values":["Personenauto"]},{"key":"P690-C1-C0-C0-F2","values":["Personenauto KVP"]},{"key":"P690-C0-C2-F6","values":["B"]},{"key":"P690-C0-C2-F7","values":["75"]},{"key":"P690-C0-C2-F4","values":["2"]},{"key":"P690-C0-C2-F5","values":["5"]},{"key":"P690-C0-C2-F2","values":["model"]},{"key":"P690-C0-C2-F3","values":["2017"]},{"key":"P690-C1-C2-C2-C2-C1-F0","values": ${shizzleiz} ${the-rest-of-the-long-json-as-a-string}
then continue on with:
${json}= Evaluate json.loads('''${json_string}''') json
${json_string}= evaluate json.dumps(${json}) json
(basically do the work before reacting to the json function - obviously this requires knowing the values beforehand and could also work with more variables.)

Python: “List.append = ‘list’ object attribute ‘append’ is read-only”

I’m trying to write a response from a Solr server to a CSV file. I’m pretty new to python and have been given code to modify. Originally the code looked like this ...
for doc in response.results:
status = json.loads(doc['status'])
The script runs and prints the correct information. But it only every prints one result (last one). I think this is because the loop constantly writes over the varible 'status' until its worked through the response.
After some reading I decided to store the information in a list. That way i could print the information to seprate lines in a list. I created an empty list and changed the code below -
for doc in response.results:
list.append = json.loads(doc['status'])
I got this response back after trying to run the code -
`AttributeError: 'list' object attribute 'append' is read-only`.
Where am I going wrong? Is a list not the best approach?

>>> list.append
<method 'append' of 'list' objects>
You're trying to modify the append method of the built-in list class!
Just do
docstats = []
for doc in response.results:
docstats.append(json.loads(doc['status']))
or equivalently:
docstats = [json.loads(doc['status']) for doc in response.results]

I'm not sure what you are trying to do.
I guess you haven't created a list variable. list is a python's builtin class for lists, so if there's no variable to mask it, you'll access that. And you tried to modify one of it's propterties, which is not allowed (it's not like ruby where you can monkey-patch anything).
Is this what you want? :
l=[]
for doc in response.results:
l.append(json.loads(doc[‘status’]))

Try
list.append(json.loads(doc['status']))

Categories

Resources