Develop Reference

ElasticSearch : TypeError: expected string or buffer?

I wrote a simple python script to put the JSON file to Elasticsearch.I want to store it based on the id field I am extracting from the JSON file.
But when I try to insert into elastic search.It raises an error TypeError: expected string or buffer
Here is the code I am working on...
#! /usr/bin/python
import requests
from elasticsearch import Elasticsearch
import json
es = Elasticsearch([{'host':'localhost','port':9200}])
r = requests.get('http://127.0.0.1:9200')
i = 1
if r.status_code == 200:
with open('d1.json') as json_data:
d = json.load(json_data)
for i in d['tc'][0]['i]['f']['h']:
if i['name'] == 'm':
m = i['value']
dope=str(m)
print dope
print type(dope)
#print type(md5)
es.index(index='lab', doc_type='report',id=dope,body=json.loads(json_data))
Error Log:
44d88612fea8a8f36de82e1278abb02f
<type 'str'>
Traceback (most recent call last):
File "elastic_insert.py", line 22, in <module>
es.index(index='labs', doc_type='report',id=dope,body=json.loads(json_data))
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
Any suggestions on how to solve this error.I even tried to convert the m to int but it gave another error.
int(m)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '44d88612fea8a8f36de82e1278abb02f'
P.S: ElasticSearch service is up and running.

The problem is not related to the id. the problem is with
"json_data". it is a file stream so you need json.load and not json.loads in your es.index

Invalid Control Character Error in Python 2.7.9

I am receiving the following error when running the below python script.
ValueError: Invalid control character at: line 7591 column 220620 (char 385678)
I did some research on this and it appeared that it would be resolved by passing 'strict=false' within json.dumps(), but I'm still receiving the same error. This is the only REST service that I have attempted to query that returns this error.
import arcgis
import json
from arcgis import ArcGIS
service = ArcGIS("http://mapping.dekalbcountyga.gov/arcgis/rest/services/LandUse/MapServer")
query = service.get(0, count_only=False)
json_query = json.dumps(query, strict=False)
f = open("dekalb_parcels.geojson", "w")
f.write(json_query)
f.close()
Any help that can be provided would be very appreciated. Thank you.
UPDATE - This is the full error that I am receiving.
Traceback (most recent call last):
File "G:\Python\Scripts\dekalb_parcel_query.py", line 8, in <module>
query = service.get(0, count_only=False)
File "C:\Python27\lib\site-packages\arcgis\arcgis.py", line 146, in get
jsobj = self.get_json(layer, where, fields, count_only, srid)
File "C:\Python27\lib\site-packages\arcgis\arcgis.py", line 90, in get_json
return response.json()
File "C:\Python27\lib\site-packages\requests\models.py", line 802, in json
return json.loads(self.text, **kwargs)
File "C:\Python27\lib\json\__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python27\lib\json\decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Invalid control character at: line 7591 column 220620 (char 385678)

I was able to fix this issue by passing strict=False within return response.json()

Python - Having trouble creating a dictionary from a JSON

I'm trying to take the JSON from a twitter get_user query and turn it into a Python object that I can extract data from (twitter handle, location, screen name, etc.)
Here is what I created. I am not sure why it doesn't work.
api = tweepy.API(auth,parser=tweepy.parsers.JSONParser())
user = api.search_users('google.com')
t_dict = json.loads(user)
pprint(t_dict)
Error:
Traceback (most recent call last):
File "Get_User_By_URL.py", line 23, in <module>
t_dict = json.loads(user)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

api.search_users is already returning a python object. It isn't a json string that needs to be parsed. According to tweetpy documentation search_users actually returns a list of users. So the following is possible:
for user in api.search_users('google.com'):
print user.screen_name

How to load unicode json in Python?

I've got some simple json formatted in unicode which I want to load using the usual python json.loads():
>>> er.rates
u"{u'sell': u'1.3477', u'buy': u'1.3588'}"
>>> json.loads(er.rates)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 381, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
So I tried using ensure_ascii=False:
>>> json.loads(er.rates, ensure_ascii=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 351, in loads
return cls(encoding=encoding, **kw).decode(s)
TypeError: __init__() got an unexpected keyword argument 'ensure_ascii'
Does anybody know how I can load this unicode json?

That is not json. It is a string representation of a python dict, which is something quite different.
You can use ast.literal_eval to load it.

Many API data providers return unicode string which is easily rendered in a browser. Unicode string (even when it looks like json) and json are not the same thing from a 'computer' perspective.
If you have a unicode (json-like) string you should be able to use json.loads(<your unicode json like string>)

Decoding JSON with Python

Why do I get
ValueError: No JSON object could be decoded
from this code:
import urllib.request,json
n = urllib.request.urlopen("http://graph.facebook.com/55")
d = json.loads(str(n.readall()))
The full error:
Traceback (most recent call last):
File "<pyshell#41>", line 1, in <module>
d= json.loads(str(n.readall()))
File "C:\Python33\lib\json\__init__.py", line 309, in loads
return _default_decoder.decode(s)
File "C:\Python33\lib\json\decoder.py", line 352, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python33\lib\json\decoder.py", line 370, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
The output of str(n.readall()):
'b\'{"id":"55","name":"Jillian Copeland","first_name":"Jillian","last_name":"Copeland","username":"JCoMD","gender":"female","locale":"en_US"}\''
Maybe the b is throwing it off?
If that is the issue, how do I convert the binary stream from the readall to a string and not have that b?
I am trying to learn a little python so please keep that in mind.
I am using Python 3.3 in Windows.

I believe that this is an exact duplicate of this question, but sadly there's no accepted answer.
On my end, this works:
import urllib.request,json
n = urllib.request.urlopen("http://graph.facebook.com/55")
d= json.loads(n.readall().decode('utf-8'))