I am doing a task in which I have to fetch the data for 20,000 records from external API. for this I am using requests module in python below is my code
def getdata():
datanotfound=[]
""" fetching the values from database to pass in the url """
values = mongo.db.collection.find()
for value in values:
value = pro.get("name")
""" adding the value parameter into the url and fetching the data """
r = requests.get('myurl'+ value)
if r!= None:
response = r.json()
info = response.get(val1)
if info != None:
val2 = info.get("val2")
val3 = info.get("val3")
val4 = info.get("val4")
val5 = info.get("val5")
""" saving the response in mongodb """
mongo.db.collection.insert_one({
'name':val1,
"address":val2,
"length":val3,
"width":val4,
"data":val5,
})
else:
""" sending the values for which response was None """
datanotfound.append(value)
return jsonify({'datanotfound':datanotfound})
This runs fine for some number of records but after some times i am gettting following errors:
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 5 (char 5)
how can i get rid of this error.
Related
The Json that its receiving in message is a byte json like so: b'{"_timestamp": 1636472787, "actual": 59.9, "target": 60.0}'
The Code is supposed to change the byte Json to String Json and load it to access the items but when I load it I get the following error:
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Code:
import json
def handle_msg(topic, message):
m = message.decode("Utf-8")
print(json.loads(m))
this is happening because you message is null value not as you expected
if you write the following it will work for you
the following running for me
message = b'{"_timestamp": 1636472787, "actual": 59.9, "target": 60.0}'
topic ="what ever"
import json
def handle_msg(topic, message):
m = message.decode("Utf-8")
print(json.loads(m))
handle_msg(topic, message)
I want to create a record, and I partially succeeded. But here's the problem, I can't record 2, I get the following error. What am I doing wrong?
Error :
File "C:\Users\bilgi\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\bilgi\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Example Code :
def loadUsers(self):
# Dosya var?
if os.path.exists('AccInformation.json'): # True ise......
with open('AccInformation.json', 'r', encoding='utf-8') as file:
users = json.load(file)
for user in users:
user = json.load(user)
newUser = Account(user_id = user['user_id'], firstName = user['first_name'], lastName = user['last_name'],
email = user['email'], username = user['username'],
password = user['password'], accountKEY = user['AccountKEY'])
self.users.append(newUser)
print(self.users)
else:
print("""'AccInformation' adlı Dosya bulunamadı.""")
So without knowing the contents of AccInformation.json it's difficult to properly answer this question, however I imagine it is a JSON with a list of dict items that represent separate users.
Based on that, the below code should work...
import os
import json
def loadUsers(self):
# Dosya var?
if os.path.exists("AccInformation.json"): # True ise......
with open("AccInformation.json", encoding="utf-8") as infile:
users = json.load(infile)
# return indent here as we've already loaded the file
for user in users:
newUser = Account(
user_id=user["user_id"],
firstName=user["first_name"],
lastName=user["last_name"],
email=user["email"],
username=user["username"],
password=user["password"],
accountKEY=user["AccountKEY"],
)
self.users.append(newUser)
print(self.users)
else:
print("""'AccInformation' adlı Dosya bulunamadı.""")
I just solved the problem. That's exactly what the problem was : " user = json.load(user)"
Ben bunu şöyle düzelttim : "user = json.loads(user)".
He's working without problems right now. I hope he doesn't cause me any trouble in the future:)
Thank you all individually for your time.
I would like fetch data from a SQL Server database and transform the result in a JSON format.
Thats not difficult, but one column is already a JSON and I would like separate it, but the result is a little bit confusing.
My code:
rows = cursor.fetchall()
print (rows)
objects_list = []
for row in rows:
d= collections.OrderedDict()
#d["Bestelldatum"]= row[0].strftime("%Y-%m-%d %H:%M")
a = json.loads(row[0])
#print(a)
#d["Adresse"] = row[0]
#d["Tor"] = a["tor"]
#d["Stiege"] = a["stg"]
#d["Stock"] = a["stk"]
#d["Tür"] = a["tür"]
#d["PLZ"] = a["plz"]
objects_list.append(d)
j = json.dumps(objects_list, ensure_ascii=False)
print (j)
My problem is row 1, that is a JSON, see the pic
Picture from database
If I run it so, I get this error messange
in loads
return _default_decoder.decode(s)
File "C:\Users\acas1\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\acas1\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I guess something is wrong with the JSON data which I get from the database
I don't understand why some data have backslashes, because at SQL they not exist.
Thats the result if I fetch just the raw data from the database:
Picture from result in python
I tried everything that they say in this post JSONDecodeError: Expecting value: line 1 column 1 (char 0)
but nothing helps.
I hope someone have an idea
I have been to various links, did hours of googling but could not find a simple way to convert the JSON data received from Tweepy StreamingListener() to python dictionary so that it can be used with pandas DataFrame.
What i did was save the data received to a json file and then read using json library. But there are various errors. I've also tried saving stream data to list and then convert to dictionary but of no use.
Here is my code:
class StreamCollector(StreamListener):
def __init__(self, api=None):
super(StreamListener, self).__init__()
self.num_tweets = 0
def on_data(self, raw_data):
try:
with open('java.json', 'a') as f:
f.write(raw_data)
self.num_tweets += 1
if self.num_tweets > 4:
return False
else:
return True
except BaseException as base_ex:
print(base_ex)
return False
def on_error(self, status_code):
print("Error Status code: --> {}".format(status_code))
return True
try:
twitterStream = Stream(auth, StreamCollector())
twitterStream.filter(track=['#Java'])
tweetDict = json.loads('java.json')
print(type(tweetDict))
print(tweetDict)
except TweepError as e:
print(e)
The above code produces following error:
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
EDIT- I checked my JSON and it appears to me that instead of one object, JSON has multiple objects which throws an error
eg:
{"name":"abc","created_at":"abc date"} //No comma
{"name":"xyz","created_at":"xyz date"}
The JSON file does not even have a root object or an array
How should i correct it?
I'm trying to parse a json but it doesn't work.
I remove the try and except in my code so you can see the Error Massege.
import sqlite3
import json
import codecs
conn = sqlite3.connect('geodata.sqlite')
cur = conn.cursor()
cur.execute('SELECT * FROM Locations')
fhand = codecs.open('where.js','w', "utf-8")
fhand.write("myData = [\n")
count = 0
for row in cur :
data = str(row[1])
print (data)
print (type(data))
#try:
js = json.loads(data)
#except: continue
if not('status' in js and js['status'] == 'OK') : continue
lat = js["results"][0]["geometry"]["location"]["lat"]
lng = js["results"][0]["geometry"]["location"]["lng"]
if lat == 0 or lng == 0 : continue
where = js['results'][0]['formatted_address']
where = where.replace("'","")
try :
print (where, lat, lng)
count = count + 1
if count > 1 : fhand.write(",\n")
output = "["+str(lat)+","+str(lng)+", '"+where+"']"
fhand.write(output)
except:
continue
fhand.write("\n];\n")
cur.close()
fhand.close()
print (count, "records written to where.js")
print ("Open where.html to view the data in a browser")
My problem is that
js = json.loads(data)
can't parse it for some reason and I get the following exception:
"raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"
I thought it becuase the data type but its doing a weird thing.
I'm asking for type(data) and I'm getting str type, but when I print data I get Byte type.
Full output for the code:
Traceback (most recent call last):
File "C:/Users/user/Desktop/Courses Online/Coursera/Using Databases with Python/geodata/geodump.py", line 17, in <module>
js = json.loads(data)
File "C:\Users\user\AppData\Local\Programs\Python\Python35-32\lib\json\__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "C:\Users\user\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\user\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
b'{\n "results" : [\n {\n "address_components" : [\n {\n ...... long json line......
<class 'str'>
I also try to use decode("utf-8") on data , but I'm getting the following Error: 'str' object has no attribute 'decode'
js = json.loads(data.decode('utf8'))
solved the same problem for me.
You are converting a bytes value to a string the wrong way here:
data = str(row[1])
You forced it to be a str() object, but for bytes objects that'll include the b prefix and quotes, because bytes objects don't have a __str__ method, only __repr__ so you get a debug representation.
Decode the row without converting to a string:
data = row[1].decode('utf8')
You really shouldn't hand-craft JSON / Javascript output in your code either. Just use json.dumps(); if you must use per-row streaming, you can still use json.dump() to create each list entry:
import sqlite3
import json
conn = sqlite3.connect('geodata.sqlite')
cur = conn.cursor()
cur.execute('SELECT * FROM Locations')
with open('where.js', 'w', encoding="utf-8") as fhand:
fhand.write("myData = [\n")
for count, row in enumerate(row):
try:
js = json.loads(row[1].decode('utf8'))
except json.JSONDecodeError:
print('Could not decode a row: ', row[1])
continue
if js.get('status') != 'OK':
continue
lat = js["results"][0]["geometry"]["location"]["lat"]
lng = js["results"][0]["geometry"]["location"]["lng"]
if not (lat and lng):
continue
where = js['results'][0]['formatted_address']
where = where.replace("'", "")
print (where, lat, lng)
if count: fhand.write(",\n")
json.dump([lat, lng, where], fhand)
fhand.write("\n];\n")
This uses plain open() (in Python 3, there is never a need to use codecs.open()), uses the file as a context manager, and adds in enumerate() to track if you have the first row processed yet.