I am attempting to create a program that reads 3 sensors from a raspberry pi sense hat, and insert those values into a sql database, using python. When I execute the program, no errors are given, but when I try to select the data using python, or even going into the database itself, the data is nowhere to be found, I believe the data isn't being properly written. How would I go about correcting this?
import sqlite3
import datetime
from sense_hat import SenseHat
sense = SenseHat()
sensePressure = int(sense.get_pressure())
senseTemp = int(sense.get_temperature() * 1.8 + 32 - 18.85)
senseHumidity = int(sense.get_humidity())
currDateTime = datetime.datetime.now()
COUNT = 1
conn = sqlite3.connect('sense.db')
c = conn.cursor()
c.execute("INSERT INTO atmos (ITEM, DATETIME, TEMPERATURE, HUMIDITY, PRESSURE) VALUES (?, ?, ?, ?,
?)", (COUNT, currDateTime, senseTemp, sensePressure, senseHumidity))
conn.commit()
conn.close()
If you are writing to any SQL database, you will be better of using the pyodbc library it is quite effective and very easy to use :
import pyodbc
connection = pyodbc.connect('Driver={SQL Server};'
'Server=server_name;'
'Database=db_name;'
'Trusted_Connection=yes;')
cursor = connection.cursor()
cursor.execute("INSERT INTO atmos (ITEM, DATETIME, TEMPERATURE, HUMIDITY, PRESSURE)
VALUES (?, ?, ?, ?, ?)", COUNT, currDateTime, senseTemp,
sensePressure, senseHumidity)
Related
How can I reset the "row_factory" attribute of an sqlite3 connection object after setting it to "sqlite3.Row"? For instance:
import sqlite3
con = sqlite3.connect("db.dat", isolation_level=None)
con.row_factory = sqlite3.Row
cur = con.execute("insert into test1 (name, last) values (?, ?)", ("John","doe"))
cur = con.execute("insert into test1 (name, last) values (?, ?)", ("Liz","doe"))
cur.execute("select * from test1")
cur.fetchone() # gets {'name':'john', 'last':'doe'}
# Now I want to get a tuple when calling fetchone
con.row_factory = None
cur.fetchone() #Throw error
delattrib(con, "row_factory") # causes segmentation fault on Windows
I can't find an answer anywhere and the only solution (other than casting the dictionary) is to reopen the db connection. I read the documentation for the sqlite3 module, but it doesn't mention anything to reset the attribute back to its default state.
Thank you
I would be grateful to know what is the easiest way to diagnose this error, as it does not seem to be easy to display what SQL is being executed via pyodbc.
My stored procedure looks like this:
ALTER PROCEDURE [dbo].[s_populate_Test_sp]
#TestDateTime DATETIME,
#TestInt INT,
#TestMoney MONEY,
#TestVarChar500 VARCHAR(500),
#TestFloat FLOAT
AS
SET NOCOUNT ON
INSERT INTO [dbo].[tbl_Test_sp2] (test_datetime, test_int, test_money, test_varchar500, test_float)
VALUES (#TestDateTime, #TestInt, #TestMoney, #TestVarChar500, #TestFloat)
I can execute this stored procedure once successfully using raw text (the commented code below), but I am having difficulty with executemany:
import os
import pyodbc
import datetime
def test_sp():
# Constants
dir_path = os.path.dirname(os.path.realpath(__file__))
# Connect
server = 'xxx'
db2 = 'xxx'
conn_str = 'DRIVER={SQL Server};SERVER=' + server + \
';DATABASE=' + db2 + ';Trusted_Connection=yes'
conn=pyodbc.connect(conn_str, autocommit=False)
cursor = conn.cursor()
cursor.fast_executemany = True
for row in range(10000):
# sql = '''EXEC [dbo].[s_populate_Test_sp] #TestDateTime = '2020-01-01 13:00',
# #TestInt = 999,
# #TestMoney = '£12.34',
# #TestVarChar500 = 'Hello My Name is Jon',
# #TestFloat = 1.234567
# '''
# cursor.execute(sql)
sql = '''exec s_populate_Test_sp (#TestDateTime = ?, #TestInt = ?, #TestMoney = ?, #TestVarChar500 = ?, #TestFloat = ?)'''
values = ['2020-01-01 13:00', 999, '£12.34', 'Hello My Name is Jon', 1.234567]
cursor.executemany(sql, [values])
conn.commit()
if __name__ == '__main__':
test_sp()
Unfortunately this yields a rather cryptic error message:
ProgrammingError: ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near '#TestDateTime'. (102) (SQLExecute); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")
I can't find a way of displaying the SQL before it gets executed, so it's all a bit trial and error at the moment.
Many thanks
Per the comments, the answer was to remove the parentheses, the £ sign and use a datetime.datetime object:
sql = '''exec s_populate_Test_sp #TestDateTime = ?, #TestInt = ?, #TestMoney = ?, #TestVarChar500 = ?, #TestFloat = ?'''
values = [datetime.datetime.now(), 999, 12.34, 'Hello My Name is Jon', 1.234567]
It is frustrating that it is so slow. Over my firm's VPN it can do 400 records per minute and on a virtual machine close to the server it can do 9,000 records per minute. I suspect this is one of the limitations of pyodbc and SQL Server, and I will have to do bcp or something like that for larger datasets.
I am trying to add the contents of variables into a SQLite DB but I am getting an error of
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
My code is:-
import requests
import json
import eventlet
import os
import sqlite3
#Get the currect vuln_sets
response = requests.get('https://vulners.com/api/v3/search/stats/')
vuln_set = json.loads(response.text)
vuln_type = vuln_set['data']['type_results']
for k in vuln_type:
vuln_bulletinfamily = vuln_set['data']['type_results'][k]['bulletinFamily']
vuln_name = vuln_set['data']['type_results'][k]['displayName']
vuln_count = vuln_set['data']['type_results'][k]['count']
con = sqlite3.connect('vuln_sets.db')
with con:
cur = con.cursor()
con.row_factory = sqlite3.Row
cur.execute("REPLACE INTO vuln_sets (vulntype, displayname, bulletinfamily, count) values (?, ?, ?, ?)", (vuln_type, vuln_bulletinfamily, vuln_name, vuln_count))
con.commit()
The variables contain the JSON key pairs as I need to insert some of them into the DB for processing but a different project.
The stacktrace is:
Traceback (most recent call last):
File "test.py", line 24, in <module>
cur.execute("REPLACE INTO vuln_sets (vulntype, displayname, bulletinfamily, count) values (?, ?, ?, ?);", (vuln_type, vuln_bulletinfamily, vuln_name, vuln_count))
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
#roganjosh's comment fixed it! I needed to incude the DB transactions in the for loop as below:
import requests
import json
import eventlet
import os
import sqlite3
#Get the currect vuln_sets
response = requests.get('https://vulners.com/api/v3/search/stats/')
vuln_set = json.loads(response.text)
vuln_type = vuln_set['data']['type_results']
for k in vuln_type:
vuln_bulletinfamily = vuln_set['data']['type_results'][k]['bulletinFamily']
vuln_name = vuln_set['data']['type_results'][k]['displayName']
vuln_count = vuln_set['data']['type_results'][k]['count']
con = sqlite3.connect('vuln_sets.db')
with con:
cur = con.cursor()
con.row_factory = sqlite3.Row
cur.execute("REPLACE INTO vuln_sets (vulntype, displayname, bulletinfamily, count) values (?, ?, ?, ?)", (k, vuln_name, vuln_bulletinfamily, vuln_count))
con.commit()
I am trying to use Python to insert into MySQL database, but I have an auto-increment column (TeamID). I am using the lines below and it works like a charm. BUT I would like to not specify the TeamID in my Python script as it is an auto-increment
try:
cur.execute ("INSERT INTO teams values (%d, '%s', %d, '%s')" % (11,"Sevilla", 7, "Jorge Sampaoli"))
db.commit()
this works perfectly
How can I get rid of the first %d and 11 please? I want this value to be added automatically via the script
any help is very much appreciated
EDIT:
#!/usr/bin/python
import MySQLdb
db = MySQLdb.connect(host="localhost", # your host, usually localhost
user="username", # your username
passwd="password", # your password
db="dbname") # name of the data base
cur = db.cursor()
try:
cur.execute ("INSERT INTO teams values ('%s', %d, '%s')" % ("Sevilla", 7, "Jorge Sampaoli"))
db.commit()
except Exception as e:
print("Rolling back")
print(e)
db.rollback()
db.close()
Issue is now resolved
I did specify the column names but didn't notice I need to use %s for all columns including int values. As below:
cur.execute("INSERT INTO teams (TeamName, CountryID, TeamManager) values (%s,%s,%s)", ('Everton', 1, 'Ronald Koeman'))
Try
INSERT INTO teams (name, numb, player) VALUES ('%s', %d, '%s')
I.e.explicitly list columns.
Also PLEASE don't do it like this -- instead of doing '%s' you really need to use prepared statements,I think in Python it is something like:
cursor.execute("INSERT INTO teams (name, numb, player) VALUES (%s, %d, %s)", ['Sevilla', 7, 'John Smith'])
Read up on SQL injections.
import sqlite3
def insert_data(lVideoList, gate_id):
connection = sqlite3.connect('db.sqlite',
detect_types=sqlite3.PARSE_DECLTYPES |
sqlite3.PARSE_COLNAMES)
cursor = connection.cursor()
success = 200
# gateid = 1
default_value = 0
for gate_id in range(no_of_gate):
gate_id = i+1
for videofilename in lVideoList:
print("videofilename: ", videofilename)
insert_query = ("INSERT INTO dailyfootfall(csv_name, video_download, processed, footfall, send_status, "
"male_footfall, send_status_male, female_footfall, send_status_female, gate_id,outsiders, send_status_outsiders) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,?)")
cursor.execute(insert_query,[videofilename, success, success, default_value, success, default_value,
success, default_value, success, gate_id, default_value, success])
print("Data Inserted Successfully !")
connection.commit()
cursor.close()
connection.close()
if __name__ == "__main__":
lVideoList = getCompleteVideoList("2022_01_24", "10:00", "22:00")
no_of_gate = 3
insert_data (lVideoList, gate_id)
print("default_value inserted!!!!")
I can't show the data from database sqlite in python.
connection = sqlite3.connect('db')
connection.cursor().execute('CREATE TABLE IF NOT EXISTS users ( \
id TEXT, \
name TEXT, \
avatar TEXT \
)')
# In cycle:
query = 'INSERT INTO users VALUES ("' + str(friend.id) + '", "' + friend.name + '", "' + friend.avatar +'" )'
print query
connection.cursor().execute(query)
connection.commit()
# After cycle
print connection.cursor().fetchall()
Sample output of query variable:
INSERT INTO users VALUES ("111", "Some Name", "http://avatar/path" )
In result, fetchall returns empty tuple. Why?
UPD
Forgotten code:
connection.cursor().execute('SELECT * FROM users')
connection.cursor().fetchall()
→
[]
INSERT does not return data. To get the data back out, you'll have to issue a SELECT statement.
import sqlite3
con = sqlite3.connect("db")
con.execute("create table users(id, name, avatar)")
con.execute("insert into users(id, name, avatar) values (?, ?, ?)", (friend.id, friend.name, friend.avatar))
con.commit()
for row in con.execute("select * from users")
print row
con.close()
Because the create table string as displayed is syntactically invalid Python, as is the insert into string.
Actually, the answer to your first question is: because you use different cursors.
connection.cursor() creates a new cursor in the connection you created before. fetchall() gives you the results of the query you executed before in that same cursor. I.e. what you did was this:
# After cycle
cursor1 = connection.cursor()
cursor1.execute('SELECT * FROM users')
cursor2 = connection.cursor()
cursor2.execute("")
cursor2.fetchall()
What you should have done was this:
# After cycle
cursor = connection.cursor()
cursor.execute('SELECT * FROM users')
print cursor.fetchall()