This question already has an answer here:
Python TypeError: expected string or buffer
(1 answer)
Closed 5 years ago.
I have the code below which should convert a JSON file to a CSV file
import json
import csv
infractions = open("C:\\Users\\Alan\\Downloads\\open.json","r")
infractions_parsed = json.loads(infractions)
infractions_data = infractions_parsed['infractions']
# open a file for writing
csv_data = open('Data.csv', 'w')
# create the csv writer object
csvwriter = csv.writer(csv_data)
count = 0
for inf in infractions_data:
if count == 0:
header = inf.keys()
csvwriter.writerow(header)
count += 1
csvwriter.writerow(inf.values())
employ_data.close()
However, I get this error. Any reason why this should be?
C:\Users\Alan\Desktop>python monkeytennis.py
Traceback (most recent call last):
File "monkeytennis.py", line 5, in <module>
infractions_parsed = json.loads(infractions)
File "C:\Python27\lib\json\__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
JSON is in format:
{
"count": 666,
"query": "righthere",
"infractions": [{
"status": "open",
"severity": 2.0,
"title": "Blah blah blah",
"coals": [1, 1],
"date": "2017-04-22T23:10:07",
"name": "Joe Bloggs"
},...
infractions is a file object, which can't be passed directly to json.loads(). Either read it first:
infractions_parsed = json.loads(infractions.read())
or use json.load (without the 's') which does expect a buffer.
infractions_parsed = json.load(infractions)
Related
I am currently exporting a database from firebase into a JSON and want to upload this to Bigquery. However, some of the fieldnames in the database have nested information and Bigquery does not accept it this way. How can I delete 'Peripherals' from every dataset that it is present in in my JSON. It is not present in every dataset though. I provided an example of what the JSON code looks like below. Thanks for the help!
{"AppName": "DataWorks", "foundedPeripheralCount": 1, "version": "1.6.1(8056)", "deviceType": "iPhone 6", "createdAt": "2017-04-05T07:05:30.408Z", "updatedAt": "2017-04-05T07:08:49.569Z", "Peripherals": {"1CA726ED-32B1-43B4-9071-B58BBACE20A8": "Arduino"}, "connectedPeripheralCount": 1, "iOSVersion": "10.2.1"}
{"objectId": "20H5Hg2INB", "foundedPeripheralCount": 0, "DeviceVendorID": "5B7F085E-B3B6-4270-97DC-F42903CDEAC1", "version": "1.3.5(5801)", "deviceType": "iPhone 6", "createdAt": "2015-11-10T06:16:45.459Z", "updatedAt": "2015-11-10T06:16:45.459Z", "connectedPeripheralCount": 0, "iOSVersion": "9.1"}
{"AppName": "DataWorks", "foundedPeripheralCount": 2, "version": "1.6.2(8069)", "deviceType": "iPhone 6s", "createdAt": "2017-04-12T10:05:05.937Z", "updatedAt": "2017-07-06T07:33:02.006Z", "Peripherals": {"060EBAFD-3120-4AAD-8B0A-EC14A323FA25": "28902 ", "identifierInternalSensors": "Internal Sensors", "0521A273-FAA5-462E-B9EC-FBB3D60F5E99": "28895 "}, "connectedPeripheralCount": 8, "iOSVersion": "10.2.1"}
I have tried this
import json
with open('firetobq_peripheral.json') as out_file:
out = json.load(out_file)
for element in out:
del element['Peripherals']
print(out)
but I receive this error
Traceback (most recent call last):
File "editjson.py", line 3, in <module>
out = json.load(out_file)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 290, in load
**kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 369, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 2 column 1 - line 629 column 1 (char 311 - 203056)
It looks like the data in 'firetobq_peripheral.json' is not valid json. If each row is on a new line you can use this code:
with open('firetobq_peripheral.json', 'r') as in_file:
dicts = []
for line in in_file.readlines() :
d = json.loads(line.strip())
if d.get('Peripherals'):
del d['Peripherals']
dicts += [d]
with open('firetobq_peripheral.json', 'w') as out_file:
out_file.write('[\n')
for i,v in enumerate(dicts):
out_file.write(json.dumps(v)+('\n' if i == len(dicts)-1 else ',\n'))
out_file.write(']')
Use this code for properly formatted json data:
with open('firetobq_peripheral.json', 'r') as in_file:
dicts = json.load(in_file)
for d in dicts:
if d.get('Peripherals'):
del d['Peripherals']
with open('firetobq_peripheral.json', 'w') as out_file:
out_file.write(json.dumps(dicts, indent=2))
I'm having an issue I can't understand. I sending a json data as string via Redis (as a queue) and the receiver is throwing the following error :
[ERROR JSON (in queue)] - {"ip": null, "domain": "somedomain.com", "name": "Some user name", "contact_id": 12345, "signature":
"6f496a4eaba2c1ea4e371ea2c4951ad92f41ddf45ff4949ffa761b0648a22e38"} => end is out of bounds
The code that throws the exception is the following :
try:
item = json.loads(item[1])
except ValueError as e:
sys.stderr.write("[ERROR JSON (in queue)] - {1} => {0}\n".format(str(e), str(item)))
return None
What is really odd, is that if I open a python console and do the following :
>>> import json
>>> s = '{"ip": null, "domain": "somedomain.com", "name": "Some user name", "contact_id": 12345, "signature": "6f496a4eaba2c1ea4e371ea2c4951ad92f41ddf45ff4949ffa761b0648a22e38"}'
>>> print s
I have no issue, the string (copy/pasted in the Python console) yield no errors at all, but my original code is throwing one!
Do you have any idea about what is causing the issue?
You are loading item[1], which is the second character of the string items:
>>> json.loads('"')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 381, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: end is out of bounds
You should write:
item = json.loads(item)
I am trying to access the mqlread API from Freebase but am getting a "Not Found" 404:
api_key = open("freebaseApiKey").read()
mqlread_url = 'https://www.googleapis.com/freebase/v1/mqlread'
mql_query = '[{"mid": null,"name": null, "type": "/location/statistical_region","limit": 100}]'
cursor = ""
topicService_url = 'https://www.googleapis.com/freebase/v1/topic'
params = {
'key': api_key,
'filter': '/location/statistical_region',
'limit': 0
}
for i in xrange(1000):
mql_url = mqlread_url + '?query=' + mql_query + "&cursor=" + cursor
print mql_url
statisticalRegionsResult = json.loads(urllib.urlopen(mql_url).read())
....
Obviously when I run my python file I get:
https://www.googleapis.com/freebase/v1/mqlread?query=[{"mid": null,"name": null, "type": "/location/statistical_region","limit": 100}]&cursor=
Traceback (most recent call last):
File "[Filepath]...FreeBaseDownload.py", line 37, in <module>
statisticalRegionsResult = json.loads(urllib.urlopen(mql_url).read())
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/decoder.py", line 338, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
What am I doing wrong with the API? I've read things about mqlread being deprecated, what is the parallel for my quest to get all statistical regions (the mids) in Freebase?
It was deprecated over a year ago. It was finally shut down May 2.
https://groups.google.com/forum/#!topic/freebase-discuss/WEnyO8f7xOQ
The only source for this information now is the Freebase data dump.
https://developers.google.com/freebase/#freebase-rdf-dumps
So, I've been trying to load a json file using json.load. It is the simplest, most basic code in the world but this is the error that is being thrown followed by the code and then a snippet of the .json file (in case there is a formatting issue):
Error:
Traceback (most recent call last):
File "first.py", line 8, in <module>
data = json.load(input_file)
File /usr/lib/python3.4/json/__init__.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.4/json/__init__.py", line 318, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.4/json/decoder.py", line 343, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.4/json/decoder.py", line 361, in raw_decode
raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)
Code:
import json
from pprint import pprint
input_file = open("chennai.recent.02dec2015.jsons",'r', encoding = 'UTF-8')
data = json.load(input_file)
pprint (data)
.json snippet:
{"contributors": null, "truncated": false, "text": "#Chennai Floods Leave Thousands Stranded, More Rain Expected For Coming Days #jobs #500K #: India has deployed... "is_quote_status": false, "in_reply_to_status_id": null, "id": 672203575703965696, "favorite_count": 0
I am trying to load a JSON file of mine which I have created from copying content from another JSON file. But I keep on getting the Error ValueError: Expecting property name: line 1 column 1 (char 1) when I try to read JSON data from the file in which I copied all the data, My JSON data is of the Format
{
"server": {
"ipaddress": "IP_Sample",
"name": "name_Sample",
"type": "type_Sample",
"label": "label_Sample",
"keyword": "kwd_Sample",
"uid": "uid_Sample",
"start_time": "start_Sample",
"stop_time": "stop_Sample"
}
}
And my load and write methods are
def load(self, filename):
inputfile = open(filename,'r')
self.data = json.loads(inputfile.read())
print (self.data)
inputfile.close()
return
def write(self, filename):
file = open(filename, "w")
tempObject = self.data
print type(tempObject)
#json.dump(filename, self.data)
print self.data["server"]
print >> file, self.data
file.close()
return
I cannot figure out where I am going wrong, can anybody help me with that..
To save and load JSON to and from a file, use an open file object. Your code indicates you tried to save the filename to self.data, which is not a fileobject...
The following code works:
def write(self, filename):
with open(filename, 'w') as output:
json.dump(self.data, output)
def load(self, filename):
with open(filename, 'r') as input:
self.data = json.load(input)
I use the open files as context managers, to ensure they are closed when done reading or writing.
Your other attempt, print >> file, self.data, simply prints the python representation to the file, not JSON:
>>> print example
{u'server': {u'uid': u'uid_Sample', u'keyword': u'kwd_Sample', u'ipaddress': u'IP_Sample', u'start_time': u'start_Sample', u'label': u'label_Sample', u'stop_time': u'stop_Sample', u'type': u'type_Sample', u'name': u'name_Sample'}}
which, when read back from the file would give the error message you indicated:
>>> json.loads("{u'server': {u'uid': u'uid_Sample', u'keyword': u'kwd_Sample', u'ipaddress': u'IP_Sample', u'start_time': u'start_Sample', u'label': u'label_Sample', u'stop_time': u'stop_Sample', u'type': u'type_Sample', u'name': u'name_Sample'}}")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/decoder.py", line 336, in raw_decode
obj, end = self._scanner.iterscan(s, **kw).next()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/scanner.py", line 55, in iterscan
rval, next_pos = action(m, context)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/json/decoder.py", line 171, in JSONObject
raise ValueError(errmsg("Expecting property name", s, end))
ValueError: Expecting property name: line 1 column 1 (char 1)
You'd have to print the json.dumps() output instead:
>>> print json.dumps(example)
'{"server": {"uid": "uid_Sample", "keyword": "kwd_Sample", "ipaddress": "IP_Sample", "start_time": "start_Sample", "label": "label_Sample", "stop_time": "stop_Sample", "type": "type_Sample", "name": "name_Sample"}}'