Develop Reference

Getting keyerror while fetching JSON data from API using python

I am getting keyerror in one while printing one of the json data fetched from API using python.
Error:
Except nagios_service, I am able to print other data
Traceback (most recent call last):
File "<ipython-input-55-3a1eadbbe594>", line 1, in <module>
runfile('Y:/_Temp/MEIPE/python/20190104_Script_Jason_APIv3.py', wdir='Y:/_Temp/MEIPE/python')
File "C:\Users\MEIPE\AppData\Local\Continuum\anaconda2\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 704, in runfile
execfile(filename, namespace)
File "C:\Users\MEIPE\AppData\Local\Continuum\anaconda2\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 93, in execfile
exec(compile(scripttext, filename, 'exec'), glob, loc)
File "Y:/_Temp/MEIPE/python/20190104_Script_Jason_APIv3.py", line 68, in <module>
print data[i]["_source"]["nagios_service"]
KeyError: 'nagios_service'
My code:
url1 = "http://nagiosdatagateway.vestas.net/esq/ITE1452552/logstash-
2018.12.16/2/desc"
response = urllib.urlopen(url1)
data = json.loads(response.read())
#define db connection
cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=DKCDCVDCP42\DPA;"
"Database=VPDC;"
"Trusted_Connection=yes;")
cursor = cnxn.cursor()
sql="SELECT count(*) as count_of_rows FROM [VPDC].[pa].
[ROC_Nagios_Reporting_RawData]"
cursor.execute(sql)
for row in cursor.fetchall():
k = row.count_of_rows
i = 0
j = len(data)#find length of data set
#print j
for i in range(0,j): #loop to insert date into SQL Server
print data[i]["_source"]["nagios_service"]
print data[i]["_source"]["nagios_host"]
print data[i]["_source"]["nagios_author"]
print data[i]["_source"]["nagios_severity_label"]
print data[i]["_source"]["nagios_external_command"]
print data[i]["_source"]["#timestamp"]
cnxn.commit() #commit transaction
cursor.close()
cnxn.close()
I need help in fixing this keyerror on nagios_service. And should print all data.

We might be able to provide a better answer if you showed us the data or explained what the purpose of this was, but for now if you want to run this code without getting exceptions, you need to allow for the possibility that not all the items contain this key. One way would be to use get() calls instead of __getitem__ calls (using square brackets) - the dict.get(key, default) method returns default if key is not in the dict, or None if you don't provide default. So a basic solution would be:
for i in range(0,j): #loop to insert date into SQL Server
source_data = data[i]["_source"]
print source_data.get("nagios_service")
print source_data.get("nagios_host")
print source_data.get("nagios_author")
print source_data.get("nagios_severity_label")
print source_data.get("nagios_external_command")
print source_data.get("#timestamp")
A slightly better version that will tell you which key is missing:
for i in range(0,j): #loop to insert date into SQL Server
source_data = data[i]["_source"]
keys = ['_source', 'nagios_service', 'nagios_host', 'nagios_author',
'nagios_severity_label', 'nagios_external_command', '#timestamp']
for key in keys:
print source_data.get(key, "Missing key: '%s'" % key)

I tried using try: and except KeyError: in my code after searching SO a little more and was able to insert JSON data into SQL table with out any errors.
url1 = "http://nagiosdatagateway.vestas.net/esq/ITE1452552/logstash-" + ysday1
#print url1 #test
#url = "http://nagiosdatagateway.vestas.net/esq/ITE1452552/logstash-
2018.12.16/2/desc"
response = urllib.urlopen(url1)
data = json.loads(response.read())
#define db connection
cnxn = pyodbc.connect("Driver={SQL Server Native Client 11.0};"
"Server=DKCDCVDCP42\DPA;"
"Database=VPDC;"
"Trusted_Connection=yes;")
cursor = cnxn.cursor()
sql= "SELECT count(*) as count_of_rows FROM [VPDC].[pa].
[ROC_Nagios_Reporting_RawData]"
cursor.execute(sql)
for row in cursor.fetchall():
k = row.count_of_rows
i = 0
j = len(data)#find length of data set
#print j
#for each in data:
for i in range(0,j): #loop to insert date into SQL Server
try:
print data[i]["_source"]["nagios_author"]
print data[i]["_source"]["nagios_service"]
cursor.execute("insert into [VPDC].[pa].[ROC_Nagios_Reporting_RawData]
(Nagios_Author,Nagios_service,Nagios_host,Nagios_comment) values
(?,?,?,?)",(data[i]["_source"]["nagios_author"],data[i]["_source"]
["nagios_service"],data[i]["_source"]["nagios_host"],data[i]["_source"]
["nagios_comment"] ))
except KeyError:
pass
cnxn.commit() #commit transaction
cursor.close()
cnxn.close() #close connection

Why is this query not working from python to SQL?

I am new to coding and databases, I can not get the query to work if I write it long hand but I have a lot to carry out and want it in a function but cannot get it to work, it returns a parameters error
import mysql.connector
def connection_check_1(query, value):
mydb = mysql.connector.connect(
host="******",
user="*****",
passwd="*****",
database="****"
)
mycursor = mydb.cursor()
mycursor.execute(query, (value))
myresult = mycursor.fetchall()
mydb.close()
return myresult
value = "sheep"
query = 'select inlicence from licence where animal = %s'
myresult = connection_check_1(query, value)
print(myresult)
Here is the SQL table I have
create table licence
(
animal varchar (20) primary key,
inlicence int (1)
);
This is the error I get
Traceback (most recent call last):
File "*******************", line 20, in
myresult = connection_check_1(query, value)
File "********************", line 13, in connection_check_1
mycursor.execute(query, (value))
File "********************************************88", line 246, in execute
prepared = self._cnx.prepare_for_mysql(params)
File "/home/kev/PycharmProjects/test bed/venv/lib/python3.5/site-packages/mysql/connector/connection_cext.py", line 535, in prepare_for_mysql
raise ValueError("Could not process parameters")
ValueError: Could not process parameters
I have tried changing the way the query is written, changing it to fetchall().

Wrapping a value with () doesn't turn it in to a tuple. You probably meant to add a comma there:
mycursor.execute(query, (value,))
# Creates a one-element tuple-^

opening and closing mysql connection

I have this python function:
import MySQLdb as mdb
def sql_query(sql):
try:
con = mdb.connect('localhost', 'user', '', 'dbs')
with con:
cur = con.cursor()
cur.execute(sql)
results = cur.fetchall()
return results
except mdb.Error, e:
sys.exit(1)
finally:
if con:
con.close()
then i have loads of code that calls this function, sometime the con closes and i get this traceback:
Traceback (most recent call last):
File "users.py", line 431, in <module>
orders = get_orders(user_id, mongo_user_id, address_book_list)
File "users.py", line 343, in get_orders
order_lines = get_order_lines(order_id)
File "users.py", line 288, in get_order_lines
for item in sql_query(get_order_lines_sql):
File "users.py", line 57, in sql_query
if con:
UnboundLocalError: local variable 'con' referenced before assignment
where my get_orders code is like:
def get_orders(user_id, mongo_user_id, address_book_list):
# we just want orders that are less than 2 years old
get_orders_sql = """SELECT order_id, state, state2, stamp, reference, comments, address_id, delivery_id, user_id FROM user_orders WHERE address_id IS NOT NULL and stamp >= '2012-01-01 00:00:00' and user_id='%s' ORDER BY stamp DESC"""% (user_id)
# delivery_id - this is the shipping option the user chose.
for items in sql_query(get_orders_sql):
#print items
order_id = items[0]
there is about 50K rows in the orders
any advice much appreciated

you need to initialize con since let say there was error in mdb.connect your get error when it some how reaches final your con is still un initialized
import MySQLdb as mdb
def sql_query(sql):
con='' #this should be intiated
try:
con = mdb.connect('localhost', 'user', '', 'dbs')
with con:
cur = con.cursor()
cur.execute(sql)
results = cur.fetchall()
return results
except mdb.Error, e:
sys.exit(1)
finally:
if con:
con.close()

python mysql connector query string syntax

I am new to python mysql connector and I am trying to get results for a query that will find the user_ids with p_id = 0 will go to product table find how many products are available in that city
import mysql.connector
con = mysql.connector.connect(user='user', password = 'pass', host = 'blah.com')
cursor1 = con.cursor(buffered = True)
query = ("SELECT l.user_id, count(u.prod_id)"
"FROM db1.users as l INNER JOIN db2.product as u "
"ON l.u_city = u.p_city"
"WHERE l.p_id =0 GROUP BY l.user_id limit 10;" )
cursor1.execute(query)
the query is getting executed mysql but from python mysql connector i am getting the following error
C:\Python27\python.exe C:/Python27/Lib/site-packages/mysql/connector/update_campus_user_profile_suset.py
Traceback (most recent call last):
File "C:/Python27/Lib/site-packages/mysql/connector/update_campus_user_profile_suset.py", line 12, in <module>
cursor1.execute(camp_zip)
File "C:\Python27\lib\site-packages\mysql\connector\cursor.py", line 491, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Python27\lib\site-packages\mysql\connector\connection.py", line 683, in cmd_query
statement))
File "C:\Python27\lib\site-packages\mysql\connector\connection.py", line 601, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'l.campus_id <2 GROUP BY l.user_id' at line 1
Process finished with exit code 1

query ='''SELECT l.user_id, count(u.prod_id) FROM db1.users as l INNER JOIN db2.product as u ON l.u_city = u.p_city WHERE l.p_id =0 GROUP BY l.user_id limit 10'''
always it is better to intialize ur query to a variable with ('''stmt''') and execute
there is no need of using ;

It looks like you're just missing some spaces at the end of each line. Try:
query = ("SELECT l.user_id, count(u.prod_id) "
"FROM db1.users as l INNER JOIN db2.product as u "
"ON l.u_city = u.p_city "
"WHERE l.p_id = 0 GROUP BY l.user_id limit 10;" )
Sometimes it's more natural to use a multi-line string:
query = ("""
SELECT l.user_id, count(u.prod_id)
FROM db1.users as l INNER JOIN db2.product as u
ON l.u_city = u.p_city
WHERE l.p_id = 0 GROUP BY l.user_id limit 10
""")

like in mysql connector/python

I am using mysql-connector/python for the first time and I am facing an error.
I want to fetch the email addresses with net from a mysql table called blah.
cursor = cnx.cursor()
p = "net"
query = ("SELECT * FROM blah WHERE email LIKE %s limit 10", ("%" + p + "%",))
cursor.execute(query,(p,))
for row in cursor:
print row
cursor.close()
cnx.close()
and I am getting this error
Traceback (most recent call last):
File "<input>", line 9, in <module>
File "C:\Python27\lib\site-packages\mysql\connector\cursor.py", line 491, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "C:\Python27\lib\site-packages\mysql\connector\connection.py", line 683, in cmd_query
statement))
File "C:\Python27\lib\site-packages\mysql\connector\connection.py", line 601, in _handle_result
raise errors.get_exception(packet)
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '"SELECT * FROM blah WHERE email LIKE %s limit 10", ("%" + p +"%")' at line 1
Can anyone suggest why is the code not working? It is working on sql -
select * from blah where email LIKE "%net" limit 10

Try this:
query = "SELECT * FROM blah WHERE email LIKE CONCAT('%', %s, '%') limit 10"
cursor.execute(query,(p,))
You need to leave %s in the string so that MySQL Connector can substitute the execute parameter for it. You were setting query to a list, not a string.

the first argument to cursor.execute should be a string it looks like you have munged assigning the string and calling the function. Probably should look something like:
query = "SELECT * FROM blah WHERE email LIKE %s limit 10"
cursor.execute(query,("%" + p + "%",))