Develop Reference

scrapy get data dict from json dictionary

I am trying to get all of the data stored in this json
as a dictionary that I can load and access. I am still new to writing spiders, but I believe I need something like
response.xpath().extract()
and then json.load().split() to get an element from it.
But the exact syntax I am not sure of, since there are so many elements in this file.

You can use re_first() to extract JSON from JavaScript code and next loads() it using json module:
import json
d = response.xpath('//script[contains(., "windows.PAGE_MODEL")]/text()').re_first(r'(?s)windows.PAGE_MODEL = (.+?\});')
data = json.loads(d)
property_id = data['propertyData']['id']

You're right, it pretty much works like you suggested in your question.
You can check the script tags for 'windows.PAGE_MODEL' with a simple xpath query.
Please try the following code in the callback for your request:
d = response.xpath('//script[text()[contains(., "windows.PAGE_MODEL")]]/text()').get()
from json import loads
data = loads(d)

How to pass variable to JSON, for python?

I am new in work with JSON, so sorry in advance for the stupid question.
I want to write JSON with the variable in the value field. It looks like this:
def print_json(user_name):
opened_json = open('way/to/json/file')
tmp = json.load(opened_json)
res = tmp(['path_to_folder'](user_name))
print(res)
def main(user_name):
print_json(user_name)
main('user')
It is JSON:
{"path_to_folder": "/Users/" + user_name + "/my_folder/"}
Awaiting for that output:
/Users/user/my_folder/
Please, tell me if any solution here exists.
Thanks in advance!
EDIT: My problem, that I can't add variable to JSON correctly. It marked red. Wrong syntax, when I try to concat.

What you want isn't directly possible in JSON, because it doesn't support "templating".
One solution would be to use a templating language such as Jinja to write a JSON template, then load this file without the json library and fill in the values using Jinja, and finally use json.loads to load a dictionary from your rendered string.
Your json-like file could look something like this:
{"path_to_folder": "/Users/{{ user_name }}/my_folder/"}
Your Python code:
import json
from jinja2 import Environment, FileSystemLoader
env = Environment(
FileSystemLoader("path/to/template")
)
template = env.get_template("template_filename.json")
def print_json(username):
return json.loads(
template.render(user_name=username)
)
...
In fact, if this is a simple one-time thing, it might even be better to use Python's built-in templating. I would recommend old-style formatting in the case of JSON, because otherwise you'll have to escape a lot of braces:
JSON file:
{"path_to_folder": "/Users/%(user_name)s/my_folder/"}
"Rendering":
with open("path/to/json") as f:
rendered = json.loads(f.read() % {"user_name": username})

How to get a python array from JSON

Can't solve how to convert JSON to python so that all data be in an array.
I used code to extract JSON, but the problem is that to extract strings from each new JSON data set is a new issue due to the inequality of the number of columns.
import json
data = open('acndata_sessions.json')
json.load(data)
I also tried to use https://app.quicktype.io/, but the function result is:
data_from_dict(json.loads(json_string)) doesn't work.
Data set: json

This question seems to have been asked before. Convert JSON array to Python list
They use 'json.loads' instead of 'json.load' which you use. Both are functions, but they are different.

I think your looking for something like this.
import json
with open('myfile.json','r') as jsonFile:
pythonJSON=json.load(jsonFile)
jsonFile.close()
print(pythonJSON.keys())
The json.loads() is used when you have a string type. If the example above doesn't work with just json.load() try it with json.loads().

json.load already gives you a dictionary. You just have to use it and iterate through _items
import json
data = open('acndata_sessions.json')
data_dict = json.load(data)
# Load items from dictionary
items = data_dict['_items']
# Iterate in items
for item in items:
# Print the item
print(item)
# Or if you want to further iterate on user inputs present in this item
user_inputs = item['userInputs']
# Check if there are any user inputs present in this item
if user_inputs:
for user_input in user_inputs:
print(user_input)

Try checking this question. You can parse a json file like this:
import json
# read file
with open('example.json', 'r') as myfile:
data=myfile.read()
# parse file
obj = json.loads(data)
# show values
print(str(obj['_items']))#returns a dict
print(str(obj['_meta']))

Issue with handling json api response in Python

I am using the Censys api in python to programmatically look through host and grab information about them. Censys website says it returns Json formatted data and it looks like Json formatted data but, I cant seem to figure out how to tun the api response into a json object. However, if i write the json response to a json file and load it. It works fine Any ideas?
Update: Figured out issue is with nested json that the api returns. Looking for libraries to flatten it.
Main.py
c = censys.ipv4.CensysIPv4(api_id=UID, api_secret=SECRET)
for result in c.search("autonomous_system.asn:15169 AND tags.raw:iot", max_records=1):
hostIPS.append(result["ip"]);
for host in hostIPS:
for details in c.view(host):
# test = json.dumps(details)
# test = json.load(test)
# data = json.load(details)
data = json.loads(details)
print(data)

You don't need to convert it to an object, it's already json.loaded. See the implementation here: https://github.com/censys/censys-python/blob/master/censys/base.py

JSON: Stripping file of unnecessary information in python

I'm working on a simple web app that pulls query information from a news article api. I'm looking to reduce client-side processing by stripping a json file of unnecessary information within my flask server. I want to store the edited json in a database (currently just locally in code below).
Currently my python code looks like:
def get_query(query):
response = urllib2.urlopen(link + '?q=' + query + '&fl=' + fields + '&api-key=' + key)
result = response.read()
# store json locally
with open('static/json/' + query + '.json', 'w') as stored_json:
json.dump(result, stored_json)
with open('static/json/' + query + '.json', 'r') as stored_json:
return json.load(stored_json)
My issues are:
a) I am unsure of how to properly edit the json. Currently in my javascript I am using the data on my ajax call as:
data.response.docs[i].headline.main;
where I would rather just store and return the object docs as a json. I know variable result in my python code is a string so I cannot write and return result.response.docs. I tried returning response.response.docs but I realize this is incorrect.
b) My last four lines seem redundant, I was wondering how to place my return within my first open block. I tried both 'w+' and 'r+' with no luck.

Im not sure if I am getting your question completely, but it sounds like what you want to do is:
1) receive the response
2) parse the json into a Python object
3) filter the data
4) store the filtered data locally (in a database, file, etc)
5) return the filtered data to the client
I am supposing that your json.dump / json.load combination was intended to get the json string into a format that you can manipulate easily (i.e. a Python object). If so, the json.loads (emphasis on the s) does what you need. Try something like this:
import json
def get_query(query):
response = urllib2.urlopen(...)
result = json.loads(response.read())
# result is a regular Python object holding the data from the json response
filtered = filter_the_data(result)
# filter_the_data is some function that manipulates data
with open('outfile.json', 'w') as outfile:
# here dump (no s) is used to serialize the data
# back to json and store it on the filesystem as outfile.json
json.dump(filtered, outfile)
...
At this point you have saved the data locally, and you still hold a reference to the filtered data. You can re-serialize it and send it to the client easily using Flask's jsonify function
Hope it helps