Develop Reference

Test Database is Not substituing the production db when testing in FastAPI

I was deploying my fastapi application to github using CI/CD,and it was running the job and gave an error,that a default connection was not defined (in this app I am using FastAPI to the api itself,mongoengine in most of the part of the db and pytest to tests,I only use pymongo to delete the test db content after the tests,because it is easier to do it without the errors of default connection not being set up)
Every single route has the same dependency ,of the connect function,so that when I am testing, I can replace it to the testing db (because if I was not going to test,I could just call the connect() function inside of the main.py,and it would just work fine (I tested it gentlemen)
#router.method('/route')
def route_function(db=Depends(connect):
do something
return {something}
The connect function consists of :
def connect():
try:
mongoengine.connect(host=connection_url, db=db_name,
alias=db_name, uuidRepresentation='standard')
except Exception as e:
print('Failed to connect to production db')
print(e)
finally:
disconnect(alias=db_alias)
And in every test I do a
app.dependency_overrides[connect] = test_connect
(do tests)
app.dependency_overrides.clear()
And the test_connect function is :
def test_connect():
try:
disconnect(db_alias)
mongoengine.connect(host=test_connection_url,
db=test_db_name, alias=test_db_alias, uuidRepresentation='standard')
except:
print('Failed to connect to testing db')
finally:
disconnect(alias=test_db_alias)
And now that you have read until here,PLEASE read until the end so that I can explain you all the problem
When I run the tests,it does not replace the test_connect function to the connect one,even when I use some tricks like lambda:test_connect to try to make it work,it doesnt...
And so that I can complete all my doubts,in the meta dictionary of each model,I now only use "db" keyword to say what database it is going to be stored,because when I used "db_alias",it gave an error saying that the alias that I used in the production was not defined,because the testing db alias was different of course
And when I used the same alias,it gave an error saying that I did not set an default connection (and even when I used alias of default in the connect function or even in the model,it still gave this missing default connection error or just dont still replace the prod db to the test one)
So I was tired of this problems and after a lot of research I found out that using only "db" could solve this problem
I had an idea that if I could transform a pymongo connection to mongoengine connection,I could solve this problem,but I dont know how to do this at all
So after all this (first THANK YOU SO MUCH for reading until here,second->)...any ideas please?

What is connection_record in these python sqlite pragma functions?

I am very new to sqlite3 and database. I want to configure my connection object for faster data insertion. I found these python functions in below link; they all have this connection_record argument that they don't use in the functions. Am I missing something important?
https://www.programcreek.com/python/?CodeExample=set+sqlite+pragma
Here is one of the functions:
def set_sqlite_pragma(dbapi_connection, connection_record):
isolation_level = dbapi_connection.isolation_level
dbapi_connection.isolation_level = None
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA foreign_keys=OFF")
cursor.close()
dbapi_connection.isolation_level = isolation_level
Also, am I right to say that tuning page_size and cache_size improve little on data insertion? Besides journal_mode and synchronous, are there any other major settings? Any recommended reads?
Thank you very much.

how to set autocommit = 1 in a sqlalchemy.engine.Connection

In sqlalchemy, I make the connection:
conn = engine.connect()
I found this will set autocommit = 0 in my mysqld log.
Now I want to set autocommit = 1 because I do not want to query in a transaction.
Is there a way to do this?

From The SQLAlchemy documentation: Understanding autocommit
conn = engine.connect()
conn.execute("INSERT INTO users VALUES (1, 'john')") # autocommits
The “autocommit” feature is only in effect when no Transaction has otherwise been declared. This means the feature is not generally used with the ORM, as the Session object by default always maintains an ongoing Transaction.
Full control of the “autocommit” behavior is available using the generative Connection.execution_options() method provided on Connection, Engine, Executable, using the “autocommit” flag which will turn on or off the autocommit for the selected scope. For example, a text() construct representing a stored procedure that commits might use it so that a SELECT statement will issue a COMMIT:
engine.execute(text("SELECT my_mutating_procedure()").execution_options(autocommit=True))

What is your dialect for mysql connection?
You can set the autocommit to True to solve the problem, like this mysql+mysqldb://user:password#host:port/db?charset=foo&autocommit=true

You can use this:
from sqlalchemy.sql import text
engine = create_engine(host, user, password, dbname)
engine.execute(text(sql).execution_options(autocommit=True))

In case you're configuring sqlalchemy for a python application using flask / django, you can create the engine like this:
# Configure the SqlAlchemy part of the app instance
app.config['SQLALCHEMY_DATABASE_URI'] = conn_url
session_options = {
'autocommit': True
}
# Create the SqlAlchemy db instance
db = SQLAlchemy(app, session_options=session_options)

I might be little late here, but for fox who is using sqlalchemy >= 2.0.*, above solution might not work as it did not work for me.
So, I went through the official documentation, and below solution worked for me.
from sqlalchemy import create_engine
db_engine = create_engine(database_uri, isolation_level="AUTOCOMMIT")
Above code works if you want to set autocommit engine wide.
But if you want use autocommit for a particular query then you can use below -
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as connection:
with connection.begin():
connection.execute("<statement>")
Official Documentation

This can be done using the autocommit option in the execution_option() method:
engine.execute("UPDATE table SET field1 = 'test'").execution_options(autocommit=True))
This information is available within the documentation on Autocommit

connecting sqlalchemy to MSAccess

How can I connect to MS Access with SQLAlchemy? In their website, it says connection string is access+pyodbc. Does that mean that I need to have pyodbc for the connection? Since I am a newbie, please be gentle.

In theory this would be via create_engine("access:///some_odbc_dsn"), but the Access backend hasn't been in service at all since SQLAlchemy 0.5, and it's not clear how well it was working back then either (this is why it's noted as "development" at http://docs.sqlalchemy.org/en/latest/core/engines.html#supported-databases - "development" means, "a development version of the dialect exists, but is not yet usable"). There's just not enough interest/volunteers to keep this dialect running right now. (when/if it is, you'll see it at http://docs.sqlalchemy.org/en/latest/dialects/access.html).
Your best bet for Access right now would be to export the data into a SQLite database file (or of course some other database, though SQLite is file-based in a similar way at least), then use that.
Update, September 2019:
The sqlalchemy-access dialect has been resurrected. Details here.
Usage example:
engine = create_engine("access+pyodbc://#some_odbc_dsn")

I primarily needed read access and some simple queries. The latest version of sqlalchemy has the (broken) access back end modules, but it isn't registered as an entrypoint.
It needed a few fixups, but this worked for me:
def fixup_access():
import sqlalchemy.dialects.access.base
class FixedAccessDialect(sqlalchemy.dialects.access.base.AccessDialect):
def _check_unicode_returns(self, connection):
return True
def do_execute(self, cursor, statement, params, context=None, **kwargs):
if params == {}:
params = ()
super(sqlalchemy.dialects.access.base.AccessDialect, self).do_execute(cursor, statement, params, **kwargs)
class SomeObject(object):
pass
fixed_dialect_mod = SomeObject
fixed_dialect_mod.dialect = FixedAccessDialect
sqlalchemy.dialects.access.fix = fixed_dialect_mod
fixup_access()
ENGINE = sqlalchemy.create_engine('access+fix://admin#/%s'%(db_location))

MySQL driver issues with INFORMATION_SCHEMA?

I'm trying out the Concurrence framework for Stackless Python. It includes a MySQL driver and when running some code that previously ran fine with MySQLdb it fails.
What I am doing:
Connecting to the MySQL database using dbapi with username/password/port/database.
Executing SELECT * FROM INFORMATION_SCHEMA.COLUMNS
This fails with message:
Table 'mydatabase.columns' doesn't exist
"mydatabase" is the database I specified in step 1.
When doing the same query in the MySQL console after issuing "USE mydatabase", it works perfectly.
Checking the network communication yields something like this:
>>>myusername
>>>scrambled password
>>>mydatabase
>>>CMD 3 SET AUTOCOMMIT = 0
<<<0
>>>CMD 3 SELECT * FROM INFORMATION_SCHEMA.COLUMNS
<<<255
<<<Table 'mydatabase.columns' doesn't exist
Is this a driver issue (since it works in MySQLdb)? Or am I not supposed to be able to query INFORMATION_SCHEMA this way?
If I send a specific "USE INFORMATION_SCHEMA" before trying to query it, I get the expected result. But, I do not want to have to sprinkle my code all over with "USE" queries.

It definitely looks like a driver issue. Maybe the python driver don't support the DB prefix.
Just to be sure, try the other way around: first use INFORMATION_SCHEMA and then SELECT * FROM mydatabase.sometable

I finally found the reason.
The driver just echoed the server capability flags back in the protocol handshake, with the exception of compression:
## concurrence/database/mysql/client.py ##
client_caps = server_caps
#always turn off compression
client_caps &= ~CAPS.COMPRESS
As the server has the capability...
CLIENT_NO_SCHEMA 16 /* Don't allow database.table.column */
...that was echoed back to the server, telling it not to allow that syntax.
Adding client_caps &= ~CAPS.NO_SCHEMA did the trick.