I have some python code that copies a SQLite db across sftp. However, it is a highly active db, so many of the times I am running into a malformed db. I'm thinking of these possible options, but I don't know how to implement them because I am newer to python.
Alternate method of getting the sqlite db copied?
Maybe there is a way to query the sqlite file from the device? Not sure if that would work since sqlite is more of a local db not sure how I can query it like I could w mysql etc...
Create a loop? I could call the function again in the exception, but not sure how to retry the rest of the code.
Also, the malformed db issue can possibly occur in other sections im thinking? Maybe I need to run a pragma quick_check?
This is commonly what I am seeing.... The other catch is why am I seeing it as often as I am? Because if I load the sqlite file from my main machine, and it runs the query files?
(venv) dulanic#mediaserver:/opt/python_scripts/rpi$ cd /opt/python_scripts/rpi ; /usr/bin/env /opt/python_scripts/rpi/venv/bin/python /home/dulanic/.vscode-server/extensions/ms-python.python-2021.2.636928669/pythonFiles/lib/python/debugpy/launcher 37599 -- /opt/python_scripts/rpi/rpdb.py
An error occurred: database disk image is malformed
This is my current code:
#!/usr/bin/env python3
import psycopg2, sqlite3, sys, paramiko, sys, os, socket, time
scpuser=os.getenv('scpuser')
scppw = os.getenv('scppw')
sqdb = os.getenv('sqdb')
sqlike = os.getenv('sqlike')
pgdb = os.getenv('pgdb')
pguser = os.getenv('pguser')
pgpswd = os.getenv('pgpswd')
pghost = os.getenv('pghost')
pgport = os.getenv('pgport')
pgschema = os.getenv('pgschema')
database = r"./pihole.db"
pihole = socket.gethostbyname('pi.hole')
tabnames=[]
tabgrab = ''
def pullsqlite():
sftp.get('/etc/pihole/pihole-FTL.db','pihole.db')
sftp.close()
# SFTP pull config
ssh_client=paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect(hostname=pihole,username=scpuser,password=scppw)
sftp=ssh_client.open_sftp()
# Pull SQlite
pullsqlite()
# Load sqlite tables to list
consq=sqlite3.connect(sqdb)
cursq=consq.cursor()
cursq.execute(f"SELECT name FROM sqlite_master WHERE type='table' AND name in ({sqlike})" )
tabgrab = cursq.fetchall()
# postgres connection
conpg = psycopg2.connect(database=pgdb, user=pguser, password=pgpswd,
host=pghost, port=pgport)
#Load data to postgres from sqlite
for item in tabgrab:
tabnames.append(item[0])
start = time.perf_counter()
for table in tabnames:
curpg = conpg.cursor()
if table=='queries':
curpg.execute(f"SELECT max(id) FROM {table};")
max_id = curpg.fetchone()[0]
cursq.execute(f"SELECT * FROM {table} where id > {max_id};")
else:
cursq.execute(f"SELECT * FROM {table};")
try:
rows=cursq.fetchall()
except sqlite3.Error as e:
print("An error occurred:", e.args[0])
colcount=len(rows[0])
pholder=('%s,'*colcount)[:-1]
try:
curpg.execute(f"SET search_path TO {pgschema};" )
curpg.executemany(f"INSERT INTO {table} VALUES ({pholder}) ON CONFLICT DO NOTHING;" ,rows)
conpg.commit()
print(f'Inserted {len(rows)} rows into {table}')
except psycopg2.DatabaseError as e:
print (f'Error {e}')
sys.exit(1)
if 'start' in locals():
elapsed = time.perf_counter() - start
print(f'Time {elapsed:0.4}')
consq.close()
Related
To start off, here is an example of what I mean by the 'completion status'. This is a query run via MySQL's CLI
mysql> USE DATABASE1;
Database changed
The line,Database Changed is what I am after. The mysql.connector lib for Python only returns the result of queries that return tables of data from what I've seen so far (I will preface this by saying that I am a beginner to SQL in general, so admittedly I have not tried everything out there).
I am trying to create a MySQL gui-based front end as part of a project, so I am trying to replicate the MySQL CLI 'experience',so to speak, to as high a degree as possible. So stuff like getting out such output text is imperative
For instance,
import mysql.connector
mydb = mysql.connector.connect(host = 'localhost', user = 'root', password = key)
mycursor = mydb.cursor()
mycursor.execute('USE DATABASE1')
for i in mycursor:
print(i)
Returns... nothing (Understandable, iterating on the cursor would only provide table data after all). I want to find a way to extract Database Changed through mysql.connector. I've scoured through the docs but can't find any method that would yield me such a thing. Is there a way to eke it out?
You could do (I did after reading this example)
When you only want to print database changes when they happen, you can do something like this:
import mysql.connector
from mysql.connector import errorcode
_current_database = ''
cnx = mysql.connector.connect(host = 'localhost',
user = 'test', password = 'test', auth_plugin='mysql_native_password')
def Execute_MySQL_and_print_result(sqlstatement):
global _current_database
global cnx
try:
mycursor = cnx.cursor(buffered=True)
mycursor.execute(sqlstatement)
if (cnx.database != _current_database):
_current_database = cnx.database
print("Database changed to: ", cnx.database)
j=1
for i in mycursor:
print(j, i)
j=j+1
except mysql.connector.Error as err:
print("OOPS, something went wrong: ", err.errno)
print(err.msg)
Execute_MySQL_and_print_result('USE TEST')
Execute_MySQL_and_print_result('SHOW DATABASES')
Execute_MySQL_and_print_result('USE TEST')
Execute_MySQL_and_print_result('SHOW DATABASES')
Execute_MySQL_and_print_result('SELECT * FROM nonexistingTable')
output:
Database changed to: test
1 ('information_schema',)
2 ('sakila',)
3 ('test',)
4 ('world',)
1 ('information_schema',)
2 ('sakila',)
3 ('test',)
4 ('world',)
OOPS, something went wrong: 1146
Table 'test.nonexistingtable' doesn't exist
I added the numbers before the database names to make it clear when the second SHOW DATABASE is starting output.
Note: The second USE TEST has no output, because active database did not change.
Could someone please help me in getting the list of active databases through python code from IBM DB2
There's more than one way to do it, but here is an example that uses a monitoring query. Your userid needs appropriate authorisation to run the query, refer to the documentation for details.
#!/usr/bin/python3
#
# the shell session that runs this script needs
# the correct environment variables to be already set
# (usually by dotting in the correct db2profile for the Db2 instance)
# otherwise ibm_db will fail to import citing load failure
# on a db2 library.
#
import sys
import ibm_db
db_user=''
db_name='sample'
db_pwd=''
try:
ibm_db_conn = ibm_db.connect(db_name, db_user, db_pwd)
except:
print('Failed to connect to database' + db_name)
print(ibm_db.conn_errormsg())
sys.exit(1)
try:
action='Query snap_get_db'
active_databases_query="SELECT db_name FROM TABLE(snap_get_db(NULL))"
stmt_handle = ibm_db.exec_immediate(ibm_db_conn, active_databases_query)
if stmt_handle :
action = 'Fetching resultSet returned by query'
row = ibm_db.fetch_tuple(stmt_handle)
while (row ):
for i in row:
print(i)
row = ibm_db.fetch_tuple(stmt_handle)
except:
print('Failed on ' + action)
print(ibm_db.errormsg())
sys.exit(1)
Writing a script to convert raw data for MySQL import I worked with a temporary textfile so far which I later imported manually using the LOAD DATA INFILE... command.
Now I included the import command into the python script:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
cursor = db.cursor()
query = """
LOAD DATA INFILE 'temp.txt' INTO TABLE myDB.values
FIELDS TERMINATED BY ',' LINES TERMINATED BY ';';
"""
cursor.execute(query)
cursor.close()
db.commit()
db.close()
This works but temp.txt has to be in the database directory which isn't suitable for my needs.
Next approch is dumping the file and commiting directly:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
sql = "INSERT INTO values(`timestamp`,`id`,`value`,`status`) VALUES(%s,%s,%s,%s)"
cursor=db.cursor()
for line in lines:
mode, year, julian, time, *values = line.split(",")
del values[5]
date = datetime.strptime(year+julian, "%Y%j").strftime("%Y-%m-%d")
time = datetime.strptime(time.rjust(4, "0"), "%H%M" ).strftime("%H:%M:%S")
timestamp = "%s %s" % (date, time)
for i, value in enumerate(values[:20], 1):
args = (timestamp,str(i+28),value, mode)
cursor.execute(sql,args)
db.commit()
Works as well but takes around four times as long which is too much. (The same for construct was used in the first version to generate temp.txt)
My conclusion is that I need a file and the LOAD DATA INFILE command to be faster. To be free where the textfile is placed the LOCAL option seems useful. But with MySQL Connector (1.1.7) there is the known error:
mysql.connector.errors.ProgrammingError: 1148 (42000): The used command is not allowed with this MySQL version
So far I've seen that using MySQLdb instead of MySQL Connector can be a workaround. Activity on MySQLdb however seems low and Python 3.3 support will probably never come.
Is LOAD DATA LOCAL INFILE the way to go and if so is there a working connector for python 3.3 available?
EDIT: After development the database will run on a server, script on a client.
I may have missed something important, but can't you just specify the full filename in the first chunk of code?
LOAD DATA INFILE '/full/path/to/temp.txt'
Note the path must be a path on the server.
To use LOAD DATA INFILE with every accessible file you have to set the
LOCAL_FILES client flag while creating the connection
import mysql.connector
from mysql.connector.constants import ClientFlag
db = mysql.connector.connect(client_flags=[ClientFlag.LOCAL_FILES], <other arguments>)
I am currently connecting to a Sybase 15.7 server using sybpydb. It seems to connect fine:
import sys
sys.path.append('/dba/sybase/ase/15.7/OCS-15_0/python/python26_64r/lib')
sys.path.append('/dba/sybase/ase/15.7/OCS-15_0/lib')
import sybpydb
conn = sybpydb.connect(user='usr', password='pass', servername='serv')
is working fine. Changing any of my connection details results in a connection error.
I then select a database:
curr = conn.cursor()
curr.execute('use db_1')
however, now when I try to run queries, it always returns None
print curr.execute('select * from table_1')
I have tried running the use and select queries in the same execute, I have tried including go commands after each, I have tried using curr.connection.commit() after each, all with no success. I have confirmed, using dbartisan and isql, that the same queries I am using return entries.
Why am I not getting results from my queries in python?
EDIT:
Just some additional info. In order to get the sybpydb import to work, I had to change two environment variables. I added the lib paths (the same ones that I added to sys.path) to $LD_LIBRARY_PATH, i.e.:
setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":dba/sybase/ase/15.7/OCS-15_0/python/python26_64r/lib:/dba/sybase/ase/15.7/OCS-15_0/lib
and I had to change the SYBASE path from 12.5 to 15.7. All this was done in csh.
If I print conn.error(), after every curr.execute(), I get:
("Server message: number(5701) severity(10) state(2) line(0)\n\tChanged database context to 'master'.\n\n", 5701)
I completely understand where you might be confused by the documentation. Its doesn't seem to be on par with other db extensions (e.g. psycopg2).
When connecting with most standard db extensions you can specify a database. Then, when you want to get the data back from a SELECT query, you either use fetch (an ok way to do it) or the iterator (the more pythonic way to do it).
import sybpydb as sybase
conn = sybase.connect(user='usr', password='pass', servername='serv')
cur = conn.cursor()
cur.execute("use db_1")
cur.execute("SELECT * FROM table_1")
print "Query Returned %d row(s)" % cur.rowcount
for row in cur:
print row
# Alternate less-pythonic way to read query results
# for row in cur.fetchall():
# print row
Give that a try and let us know if it works.
Python 3.x working solution:
import sybpydb
try:
conn = sybpydb.connect(dsn="Servername=serv;Username=usr;Password=pass")
cur = conn.cursor()
cur.execute('select * from db_1..table_1')
# table header
header = tuple(col[0] for col in cur.description)
print('\t'.join(header))
print('-' * 60)
res = cur.fetchall()
for row in res:
line = '\t'.join(str(col) for col in row)
print(line)
cur.close()
conn.close()
except sybpydb.Error:
for err in cur.connection.messages:
print(f'Error {err[0]}, Value {err[1]}')
I wrote a Python script which initializes an empty database if it doesn't exist.
import os
if not os.path.exists('Database'):
os.makedirs('Database')
os.system('sqlite3 Database/testDB.db ";"')
# rest of the script...
Can I do this in a more Pythonic fashion, with a try-except, or is this kind of code acceptable?
I think you can do it like this:
import sqlite3
conn = sqlite3.connect('Database/testDB.db')
This should connect to your database and create it in case that it doesn't exist. I'm not sure this is the most pythonic way, but it does use the sqlite3 module instead of the sqlite3 command.
Making it Pythonic: create a sqlite3 database if it doesn't exist?
The most Pythonic way to do this is to use the context manager:
import sqlite3
# if we error, we rollback automatically, else commit!
with sqlite3.connect('/Temp/testDB.db') as conn:
cursor = conn.cursor()
cursor.execute('SELECT SQLITE_VERSION()')
data = cursor.fetchone()
print('SQLite version:', data)
In a python shell this echoes for me:
<sqlite3.Cursor object at 0x0CCAD4D0>
SQLite version: (u'3.5.9',)
To ensure you have a tempfile path that works across platforms, use tempfile.gettempdir:
import tempfile
with sqlite3.connect(tempfile.gettempdir() + '/testDB.db') as conn:
...
Create directory path, database file and table
Here is a recipe to create the directory path, database file and table
when necessary. If these already exist, the script will overwrite nothing and simply use what is at hand.
import os
import sqlite3
data_path = './really/deep/data/path/'
filename = 'whatever'
os.makedirs(data_path, exist_ok=True)
db = sqlite3.connect(data_path + filename + '.sqlite3')
db.execute('CREATE TABLE IF NOT EXISTS TableName (id INTEGER PRIMARY KEY, quantity INTEGER)')
db.close()
sqlite3.connect will attempt to create a database if it doesn't exist - so the only way to tell if one does exist is to try to open it and catch an IOError. Then to create a blank database, just connect using the sqlite3 module.
import sqlite3
try:
open('idonotexist')
print 'Database already exists!'
except IOError as e:
if e.args == 2: # No such file or directory
blank_db = sqlite3.connect('idontexist')
print 'Blank database created'
else: # permission denied or something else?
print e
Of course, you may still have to do something with os.makedirs depending on if the structure already exists.