Without installing additional modules, how can I use the SQLite backup API to backup an in-memory database to an on-disk database? I have managed to successfully perform a disk-to-disk backup, but passing the already-extant in-memory connection to the sqlite3_backup_init function appears to be the problem.
My toy example, adapted from https://gist.github.com/achimnol/3021995 and cut down to the minimum, is as follows:
import sqlite3
import ctypes
# Create a junk in-memory database
sourceconn = sqlite3.connect(':memory:')
cursor = sourceconn.cursor()
cursor.execute('''CREATE TABLE stocks
(date text, trans text, symbol text, qty real, price real)''')
cursor.execute("INSERT INTO stocks VALUES ('2006-01-05','BUY','RHAT',100,35.14)")
sourceconn.commit()
target = r'C:\data\sqlite\target.db'
dllpath = u'C:\\Python27\DLLs\\sqlite3.dll'
# Constants from the SQLite 3 API defining various return codes of state.
SQLITE_OK = 0
SQLITE_ERROR = 1
SQLITE_BUSY = 5
SQLITE_LOCKED = 6
SQLITE_OPEN_READONLY = 1
SQLITE_OPEN_READWRITE = 2
SQLITE_OPEN_CREATE = 4
# Tweakable variables
pagestocopy = 20
millisecondstosleep = 100
# dllpath = ctypes.util.find_library('sqlite3') # I had trouble with this on Windows
sqlitedll = ctypes.CDLL(dllpath)
sqlitedll.sqlite3_backup_init.restype = ctypes.c_void_p
# Setup some ctypes
p_src_db = ctypes.c_void_p(None)
p_dst_db = ctypes.c_void_p(None)
null_ptr = ctypes.c_void_p(None)
# Check to see if the first argument (source database) can be opened for reading.
# ret = sqlitedll.sqlite3_open_v2(sourceconn, ctypes.byref(p_src_db), SQLITE_OPEN_READONLY, null_ptr)
#assert ret == SQLITE_OK
#assert p_src_db.value is not None
# Check to see if the second argument (target database) can be opened for writing.
ret = sqlitedll.sqlite3_open_v2(target, ctypes.byref(p_dst_db), SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, null_ptr)
assert ret == SQLITE_OK
assert p_dst_db.value is not None
# Start a backup.
print 'Starting backup to SQLite database "%s" to SQLite database "%s" ...' % (sourceconn, target)
p_backup = sqlitedll.sqlite3_backup_init(p_dst_db, 'main', sourceconn, 'main')
print ' Backup handler: {0:#08x}'.format(p_backup)
assert p_backup is not None
# Step through a backup.
while True:
ret = sqlitedll.sqlite3_backup_step(p_backup, pagestocopy)
remaining = sqlitedll.sqlite3_backup_remaining(p_backup)
pagecount = sqlitedll.sqlite3_backup_pagecount(p_backup)
print ' Backup in progress: {0:.2f}%'.format((pagecount - remaining) / float(pagecount) * 100)
if remaining == 0:
break
if ret in (SQLITE_OK, SQLITE_BUSY, SQLITE_LOCKED):
sqlitedll.sqlite3_sleep(millisecondstosleep)
# Finish the bakcup
sqlitedll.sqlite3_backup_finish(p_backup)
# Close database connections
sqlitedll.sqlite3_close(p_dst_db)
sqlitedll.sqlite3_close(p_src_db)
I receive an error ctypes.ArgumentError: argument 3: <type 'exceptions.TypeError'>: Don't know how to convert parameter 3 on line 49 (p_backup = sqlitedll.sqlite3_backup_init(p_dst_db, 'main', sourceconn, 'main')). Somehow, I need to pass a reference to the in-memory database to that sqlite3_backup_init function.
I do not know enough C to grasp the specifics of the API itself.
Setup: Windows 7, ActiveState Python 2.7
It looks like as of Python 3.7, this functionality is available within the standard library. Here are some examples copied directly out of the official docs:
Example 1, copy an existing database into another:
import sqlite3
def progress(status, remaining, total):
print(f'Copied {total-remaining} of {total} pages...')
con = sqlite3.connect('existing_db.db')
bck = sqlite3.connect('backup.db')
with bck:
con.backup(bck, pages=1, progress=progress)
bck.close()
con.close()
Example 2, copy an existing database into a transient copy:
import sqlite3
source = sqlite3.connect('existing_db.db')
dest = sqlite3.connect(':memory:')
source.backup(dest)
To answer your specific question of backing up an in-memory database to disk, it looks like this works. Here's a quick script using the standard library backup method:
import sqlite3
source = sqlite3.connect(':memory:')
dest = sqlite3.connect('backup.db')
c = source.cursor()
c.execute("CREATE TABLE test(id INTEGER PRIMARY KEY, msg TEXT);")
c.execute("INSERT INTO test VALUES (?, ?);", (1, "Hello World!"))
source.commit()
source.backup(dest)
dest.close()
source.close()
And the backup.db database can be loaded into sqlite3 and inspected:
$ sqlite3 backup.db
SQLite version 3.24.0 2018-06-04 14:10:15
Enter ".help" for usage hints.
sqlite> .schema
CREATE TABLE test(id INTEGER PRIMARY KEY, msg TEXT);
sqlite> SELECT * FROM test;
1|Hello World!
An in-memory database can be accessed only through the SQLite library that created it (in this case, Python's built-in SQLite).
Python's sqlite3 module does not give access to the backup API, so it is not possible to copy an in-memory database.
You would need to install an additional module, or use an on-disk database in the first place.
While this isn't strictly a solution to your questions (as it's not using the backup API) it serves as a minimal effort approach and works well for small in memory databases.
import os
import sqlite3
database = sqlite3.connect(':memory:')
# fill the in memory db with your data here
dbfile = 'dbcopy.db'
if os.path.exists(dbfile):
os.remove(dbfile) # remove last db dump
new_db = sqlite3.connect(dbfile)
c = new_db.cursor()
c.executescript("\r\n".join(database.iterdump()))
new_db.close()
#cl is wrong. I solved this problem using ctypes to extract the underlying C pointer of a Python connection object and created a small package: https://pypi.org/project/sqlite3-backup/
Source code is at https://sissource.ethz.ch/schmittu/sqlite3_backup
Related
I am trying to read the contents/code of a stored procedure using python.
i used cx_Oracle function to establish the connection with oracle database.
here is the code
import cx_Oracle as co
import pandas as pd
dsn_tsn = co.makedsn(ip,port,SID)
db=co.connect(username,password,dsn_tsn)
cursor = db.cursor()
cursor.callproc(procedure_name,['argument']) # will be me result of the procedure.
However, i am trying to read the code of procedure itself. Is there any function to do that ?
You can call DBMS_METADATA.GET_DDL function from your code in such a way
import cx_Oracle
db = cx_Oracle.connect("<uname>/<pwd>#<host>:<port>/<service_name>")
cursor = db.cursor()
def OutputTypeHandler(cursor, name, defaultType, size, precision, scale):
if defaultType == cx_Oracle.CLOB:
return cursor.var(cx_Oracle.LONG_STRING, arraysize = cursor.arraysize)
cursor.outputtypehandler = OutputTypeHandler
cursor.execute("SELECT DBMS_METADATA.GET_DDL('PROCEDURE', :PrcName) FROM DUAL",
PrcName="MY_LITTLE_PROC")
print("returned DDL is :",cursor.fetchall())
Code of stored procedures can be accessed via user_source view. So, if you query it, you'll see what you want. Here's how:
SQL> create or replace procedure p_test is
2 begin
3 null;
4 end;
5 /
Procedure created.
SQL> desc user_source
Name Null? Type
----------------------------------------- -------- ----------------------------
NAME VARCHAR2(30)
TYPE VARCHAR2(12)
LINE NUMBER
TEXT VARCHAR2(4000)
SQL> select text from user_source where name = 'P_TEST' and type = 'PROCEDURE';
TEXT
--------------------------------------------------------------------------------
procedure p_test is
begin
null;
end;
SQL>
Though, as I don't speak Python, I can't assist in actual code you need to use there. The last select I wrote is what you need; I hope you'll manage to use it. Good luck!
import oracledb
db = oracledb.connect("<uname>/<pwd>#<host>:<port>/<service_name>")
cursor = db.cursor()
cursor.execute("SELECT DBMS_METADATA.GET_DDL('PROCEDURE', 'PROC_NAME', 'OWNER') FROM DUAL")
c = cursor.fetchall()
print( c[0][0] )
# To save on file
sample = open('samplefile.txt', 'w')
print(c[0][0], file = sample)
sample.close()
Change connection details, PROC_NAME and OWNER and run.
I have a php script that executes a python script and returns the data back and it is stored in mysql. It's working fine but when the data is stored in the database it inserts an additional blank row. My question would be, how could I make so that it stores only the actual data I recieve.
This is part of the python script
##ser.write(sync)
ser.write(counters)
a = ser.read(30)
state = binascii.hexlify(a)
asd = re.sub(rb'([0-9, a-z, A-Z])(?!$)', rb'\1,', state)
url = 'http://127.0.0.1/sastest/meters.php'
x = requests.post(url, data = asd)
print(asd)
And this is from the PHP
passthru("meters.py");
$incomingData = file_get_contents("php://input");
$qry1 = "INSERT INTO machtest(data)
values('".$incomingData."')";
mysqli_query($conn,$qry1);
From comments we discover the overall process:
When I call meters.php it activates meters.py. meters.py interrogates a devices and sends the data back to meters.php
Because PHP's passthru does not support output return but a similar function exec does as array object with each line as elements, use that instead and do not have Python post back a response. Of course, always run parameterization when interacting with databases and passing input values.
Python (meters.py)
ser.write(counters)
a = ser.read(30)
state = binascii.hexlify(a)
asd = re.sub(rb'([0-9, a-z, A-Z])(?!$)', rb'\1,', state)
print(asd)
PHP (meters.php)
// USE output ARG
exec(command = "meters.py", output = $incomingData);
// USE PARAMETERIZATION
$qry = "INSERT INTO machtest (data) VALUES (%s)";
$stmt = mysqli_prepare($qry);
mysqli_stmt_bind_param($stmt, "s", $incomingData[0]);
mysqli_stmt_execute($stmt);
See mysqli prepared statement docs
Alternatively, have Python run all processing including device and database interaction. Then, have PHP call the .py script:
Python (meters.py)
import mysql.connector # USE ANY MySQL DB-API. THIS IS AN EXAMPLE
...
### INTERROGATE DEVICE
ser.write(counters)
a = ser.read(30)
state = binascii.hexlify(a)
asd = re.sub(rb'([0-9, a-z, A-Z])(?!$)', rb'\1,', state)
### APPEND TO DATABASE
# OPEN CONNECTION AND CURSOR
conn = mysql.connector.connect(host='localhost', database='mydatabase',
user='root', password='pwd')
cur = conn.cursor()
# USE PARAMETERIZATION
qry = "INSERT INTO machtest (data) VALUES (%s)"
cur.execute(qry, (asd,))
conn.commit()
cur.close()
conn.close()
See MySQL cursor execute docs
PHP (meters.php)
// NO NEED FOR output
passthru(command = "meters.py");
I am attempting to use SQL Server 2017 filestream in python. All of the functionality i use goes through sqlalchemy, thus i am attempting to find a way of using this, since i haven't found any implementation within sqlalchemy or other libraries (may have missed something, if so please point me to a working and tested implementation).
I have decided to approach this using the dll, based on https://github.com/VisionMark/django-mssql-filestream/blob/master/sql_filestream/win32_streaming_api.py . However, my call to the OpenSqlFilestream fails and returns -1 instead of file handle. I have no idea what the issue is or how to fix it.
from ctypes import c_char, sizeof, windll
from sqlalchemy import create_engine
from sqlalchemy.orm import session_maker
import msvcrt
import os
msodbcsql = windll.LoadLibrary("C:\Windows\System32\msodbcsql17.dll")
engine = create_engine("mssql+pyodbc://user:pass#test/test?TrustedConnection=yes+driver=ODBC Driver+17+for+SQL+Server")
maker = session_maker(bind=engine)
session = session_maker()
## first query should begind transaction
path = session.execute("SELECT file_stream.PathName() FROM test_filetable").fetchall()[0][0]
## this returns str like "\\\\test\\*"
context = session.execute("SELECT GET_FILESTREAM_TRANSACTION_CONTEXT()").fetchall()[0][0]
## returns bytes
_context = (c_char*len(context)).from_buffer_copy(context)
## This call fails
handle = msodbcsql.OpenSqlFilestream(
path, # FilestreamPath
0, # DesiredAccess
0, # OpenOptions
_context, # FilestreamTransactionContext
sizeof(_context), # FilestreamTransactionContextLength
0 # AllocationSize
)
## this returns -1 instead of handle
## Never reached, but this should create usable file
desc = msvcrt.open_osfhandle(fsHandle, os.O_RDONLY)
_file = os.fdopen(desc, 'r')
All of the queries work and output (as far as i understand) correct data.
How do i obtain filestream access to a file on SQL Server 2017 from python (3.7)?
Edit: The objects i read go to the size of gigabytes and the process only needs stream access.
My guess is that your issue is related to
the fact that a SQLAlchemy Session is much more than just a raw DB API Connection, and/or
the transaction context is not appropriate for your invocation of OpenSqlFilestream
For what it's worth, the following works for me with CPython 3.7.2 and pythonnet 2.4.0:
import clr
clr.AddReference("System.Data")
from System.Data import IsolationLevel
from System.Data.SqlClient import SqlCommand, SqlConnection
from System.Data.SqlTypes import SqlFileStream
from System.IO import File, FileAccess, FileOptions
# adapted from c# code at
# https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/filestream-data
connection_string = r"Data Source=(local)\SQLEXPRESS;Initial Catalog=myDB;Integrated Security=True"
con = SqlConnection(connection_string)
con.Open()
sql = """\
SELECT Photo.PathName(), GET_FILESTREAM_TRANSACTION_CONTEXT()
FROM employees WHERE EmployeeID = 1"""
cmd = SqlCommand(sql, con)
tran = con.BeginTransaction(IsolationLevel.ReadCommitted)
cmd.Transaction = tran
rdr = cmd.ExecuteReader()
rdr.Read()
path = rdr.GetString(0)
transaction_context = rdr.GetSqlBytes(1).Buffer
rdr.Close()
allocation_size = 0
input_stream = SqlFileStream(path, transaction_context,
FileAccess.Read, FileOptions.SequentialScan, allocation_size)
output_stream = File.Create(r"C:\Users\Gord\Desktop\photo.bmp")
input_stream.CopyTo(output_stream)
output_stream.Close()
input_stream.Close()
tran.Commit()
con.Close()
I have a Pyramid / SQLAlchemy, MySQL python app.
When I execute a raw SQL INSERT query, nothing gets written to the DB.
When using ORM, however, I can write to the DB. I read the docs, I read up about the ZopeTransactionExtension, read a good deal of SO questions, all to no avail.
What hasn't worked so far:
transaction.commit() - nothing is written to the DB. I do realize this statement is necessary with ZopeTransactionExtension but it just doesn't do the magic here.
dbsession().commit - doesn't work since I'm using ZopeTransactionExtension
dbsession().close() - nothing written
dbsession().flush() - nothing written
mark_changed(session) -
File "/home/dev/.virtualenvs/sc/local/lib/python2.7/site-packages/zope/sqlalchemy/datamanager.py", line 198, in join_transaction
if session.twophase:
AttributeError: 'scoped_session' object has no attribute 'twophase'"
What has worked but is not acceptable because it doesn't use scoped_session:
engine.execute(...)
I'm looking for how to execute raw SQL with a scoped_session (dbsession() in my code)
Here is my SQLAlchemy setup (models/__init__.py)
def dbsession():
assert (_dbsession is not None)
return _dbsession
def init_engines(settings, _testing_workarounds=False):
import zope.sqlalchemy
extension = zope.sqlalchemy.ZopeTransactionExtension()
global _dbsession
_dbsession = scoped_session(
sessionmaker(
autoflush=True,
expire_on_commit=False,
extension=extension,
)
)
engine = engine_from_config(settings, 'sqlalchemy.')
_dbsession.configure(bind=engine)
Here is a python script I wrote to isolate the problem. It resembles the real-world environment of where the problem occurs. All I want is to make the below script insert the data into the DB:
# -*- coding: utf-8 -*-
import sys
import transaction
from pyramid.paster import setup_logging, get_appsettings
from sc.models import init_engines, dbsession
from sqlalchemy.sql.expression import text
def __main__():
if len(sys.argv) < 2:
raise RuntimeError()
config_uri = sys.argv[1]
setup_logging(config_uri)
aa = init_engines(get_appsettings(config_uri))
session = dbsession()
session.execute(text("""INSERT INTO
operations (description, generated_description)
VALUES ('hello2', 'world');"""))
print list(session.execute("""SELECT * from operations""").fetchall()) # prints inserted data
transaction.commit()
print list(session.execute("""SELECT * from operations""").fetchall()) # doesn't print inserted data
if __name__ == '__main__':
__main__()
What is interesting, if I do:
session = dbsession()
session.execute(text("""INSERT INTO
operations (description, generated_description)
VALUES ('hello2', 'world');"""))
op = Operation(generated_description='aa', description='oo')
session.add(op)
then the first print outputs the raw SQL inserted row ('hello2' 'world'), and the second print prints both rows, and in fact both rows are inserted into the DB.
I cannot comprehend why using an ORM insert alongside raw SQL "fixes" it.
I really need to be able to call execute() on a scoped_session to insert data into the DB using raw SQL. Any advice?
It has been a while since I mixed raw sql with sqlalchemy, but whenever you mix them, you need to be aware of what happens behind the scenes with the ORM. First, check the autocommit flag. If the zope transaction is not configured correctly, the ORM insert might be triggering a commit.
Actually, after looking at the zope docs, it seems manual execute statements need an extra step. From their readme:
By default, zope.sqlalchemy puts sessions in an 'active' state when they are
first used. ORM write operations automatically move the session into a
'changed' state. This avoids unnecessary database commits. Sometimes it
is necessary to interact with the database directly through SQL. It is not
possible to guess whether such an operation is a read or a write. Therefore we
must manually mark the session as changed when manual SQL statements write
to the DB.
>>> session = Session()
>>> conn = session.connection()
>>> users = Base.metadata.tables['test_users']
>>> conn.execute(users.update(users.c.name=='bob'), name='ben')
<sqlalchemy.engine...ResultProxy object at ...>
>>> from zope.sqlalchemy import mark_changed
>>> mark_changed(session)
>>> transaction.commit()
>>> session = Session()
>>> str(session.query(User).all()[0].name)
'ben'
>>> transaction.abort()
It seems you aren't doing that, and so the transaction.commit does nothing.
Writing a script to convert raw data for MySQL import I worked with a temporary textfile so far which I later imported manually using the LOAD DATA INFILE... command.
Now I included the import command into the python script:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
cursor = db.cursor()
query = """
LOAD DATA INFILE 'temp.txt' INTO TABLE myDB.values
FIELDS TERMINATED BY ',' LINES TERMINATED BY ';';
"""
cursor.execute(query)
cursor.close()
db.commit()
db.close()
This works but temp.txt has to be in the database directory which isn't suitable for my needs.
Next approch is dumping the file and commiting directly:
db = mysql.connector.connect(user='root', password='root',
host='localhost',
database='myDB')
sql = "INSERT INTO values(`timestamp`,`id`,`value`,`status`) VALUES(%s,%s,%s,%s)"
cursor=db.cursor()
for line in lines:
mode, year, julian, time, *values = line.split(",")
del values[5]
date = datetime.strptime(year+julian, "%Y%j").strftime("%Y-%m-%d")
time = datetime.strptime(time.rjust(4, "0"), "%H%M" ).strftime("%H:%M:%S")
timestamp = "%s %s" % (date, time)
for i, value in enumerate(values[:20], 1):
args = (timestamp,str(i+28),value, mode)
cursor.execute(sql,args)
db.commit()
Works as well but takes around four times as long which is too much. (The same for construct was used in the first version to generate temp.txt)
My conclusion is that I need a file and the LOAD DATA INFILE command to be faster. To be free where the textfile is placed the LOCAL option seems useful. But with MySQL Connector (1.1.7) there is the known error:
mysql.connector.errors.ProgrammingError: 1148 (42000): The used command is not allowed with this MySQL version
So far I've seen that using MySQLdb instead of MySQL Connector can be a workaround. Activity on MySQLdb however seems low and Python 3.3 support will probably never come.
Is LOAD DATA LOCAL INFILE the way to go and if so is there a working connector for python 3.3 available?
EDIT: After development the database will run on a server, script on a client.
I may have missed something important, but can't you just specify the full filename in the first chunk of code?
LOAD DATA INFILE '/full/path/to/temp.txt'
Note the path must be a path on the server.
To use LOAD DATA INFILE with every accessible file you have to set the
LOCAL_FILES client flag while creating the connection
import mysql.connector
from mysql.connector.constants import ClientFlag
db = mysql.connector.connect(client_flags=[ClientFlag.LOCAL_FILES], <other arguments>)