When I make an initial database connection with FlaskSQLAlchemy it is extremely slow, sometimes taking a minute or more to successfully connect and execute a query. Any subsequent database calls after the initial connection are fast. From my understanding this has something to do with "lazy loading". How can I "force" this initial connection to occur earlier, or speed up the connection altogether?
I am using MS SQL Server for my database, and pyodbc for my DB API. The application is deployed on Windows IIS server, although this problem occurs even while connecting locally - it is exacerbated while deployed on IIS.
Use latest version of PYODBC driver. First install python package.
pip install pyodbc
If you use following code, then you will get fast Response from MSSQL database for sure.
import urllib
params = urllib.parse.quote_plus(
'DRIVER={ODBC Driver 17 for SQL Server};SERVER=server-name;DATABASE=database-name;UID=sa;PWD=****;Trusted_Connection=yes;'
)
class Config:
DEBUG = True
SECRET_KEY = "E9738F40-4210ACAD94B"
SQLALCHEMY_DATABASE_URI = "mssql+pyodbc:///?odbc_connect=%s" % params
Make sure your ODBC driver is accurate in your system. If you want to get knowledge about ODBC driver, then run following query in cmd command.
import pyodbc
pyodbc.drivers()
Then put following code in your init.py file
def create_app(config_class=Config):
app = Flask(__name__)
app.config.from_object(Config)
Above things will be giving your ORM (Object Relation Mapper) which will gives you a technique of mapping object parameters to the underlying RDBMS table structure. An ORM API provides methods to perform CRUD operations without having to write raw SQL statements.
Related
I am trying to connect to a postgres instance I have in cloud sql. I have everything set up and am able to connect to it if ssl encryption is turned off. But now that I have it on I am trying to connect but running into some error.
def run():
connector = Connector()
def getconn():
conn = connector.connect(
os.getenv("CONNECTION_NAME"),
"pg8000",
user = os.getenv('DB_USERNAME'),
password = os.getenv("DB_PASSWORD"),
db=os.getenv('DB_NAME'),
ip_type= IPTypes.PRIVATE
)
return conn
pool = sqlalchemy.create_engine(
"postgresql+pg8000://",
creator=getconn,
pool.execute("CREATE TABLE........;")
All the certs are stored in secret manager as strings so I am using env variables to grab them, which is why I used cadata for example. But running into this error cadata does not contain a certificate why is this error coming up?
I'd recommend using the Cloud SQL Python Connector to connect to Cloud SQL from Python as it will generate the SSL context for you, meaning no need to manage SSL certificates! It also has additional benefits of not needing to authorize networks etc.
You can find a code sample for the Python Connector similar to the one you are using for establishing a TCP connection.
There is also an interactive getting started Colab Notebook that will walk you through using the Python Connector without you needing to change a single line of code!
It makes connecting to Cloud SQL both easy and secure.
I am trying to connect to a SQL Server instance using SQLAlchemy through Python, however I require the SQL connection to come from a specific Windows AD account that isn't the one I run VSCode with. Is there any way to modify the connection string below to explicitly feed SQL Server a Windows login that isn't the same login I am using the run VSCode?
(I am able to connect with this connection string if I "Run as a different User" in VSCode, however the AD accounts with SQL access does not have shared drive access and therefore cannot access shared files, so this won't scale long-term)
import urllib
from sqlalchemy import create_engine
params = urllib.parse.quote_plus('DRIVER={SQL Server};SERVER={server};DATABASE={database};Trusted_Connection=Yes')
engine = create_engine(f'mssql+pyodbc:///?odbc_connect={params}')
I'd like to develop a python project in client - server style.
My Python code will be deployed on a client machine and the MySQL database on the server, so all connections need to flow through a network connection.
I know how to use Python with MySQL with basics such as creating tables and queries etc, however I wanted to learn how to connect my python project to the database over a network protocol.
I searched on Google or YouTube, but only found resources that use a local database connection.
Where can I find Python resources (article, blog, sample, tutorial or video) that explains how to connect to MySQL via the network?
You have server with MySQL database running on it and a python client-application, which needs data from the database?
I think usually client software never connects directly to the database (for security reasons). Instead the client communicates with a server-application (maybe over a REST-API).
How i meant it
This is seems to be a good guide.
https://pynative.com/python-mysql-tutorial/
I tried PyMySql quickly. It works. Only problem is it is in Beta. https://pymysql.readthedocs.io/en/latest/user/examples.html
So I suggest you try the MySQL Connector Python will be slightly painful to setup initially but it comes from the vendor. so you can trust it a bit better.
or alternatively https://pypi.org/project/mysqlclient/
import pymysql.cursors
import pymysql
try:
conn = pymysql.connect(host='10.1.1.1 (remote server ip or hostname)',
user='db_user_name',
password='db_password',
db='name_of_db',
cursorclass=pymysql.cursors.DictCursor)
sql = "select * from users"
with conn.cursor() as cursor:
# Read a single record
sql = "select * from users"
cursor.execute(sql)
result = cursor.fetchall()
print(result)
except Exception as inst:
print(str(inst))
finally:
conn.close()
Our infrastructure group has asked us to "add MultiSubnetFailover=True to all application connection strings" so that we can take advantage of a new SQL Server HA setup involving Availability Groups.
I am stuck though since we have some python programs that connect (read+write) to the database via SQL Alchemy. I have been searching and I don't see anything about this MultiSubnetFailover feature being available as an option in SQL Alchemy or any other Python driver. Is it possible to connect to an HA setup utilizing the SQL Alchemy driver, or even Python, and if so how?
FYI - The link that my infrastructure guy pointed me to is here (http://msdn.microsoft.com/en-us/library/hh205662%28v=vs.110%29.aspx), and as you can see it is specifically about how .NET applications can utilize the "MultiSubnetFailover=True" setting in the connection string among other things.
http://docs.sqlalchemy.org/en/latest/dialects/mssql.html#dialect-mssql-pyodbc-connect
You could use the example towards the end of the documentation's section like this:
import urllib
from sqlalchemy import create_engine
connection_string = '127.0.0.1;Database=MyDatabase;MultiSubnetFailover=True'
engine_string = 'mssql+pyodbc:///?odbc_connect={}'.format(urllib.quote_plus(connection_string))
engine = create_engine(engine_string)
Update from comments
For newer versions of Microsoft ODBC Driver for SQL Server, you may need to use MultiSubnetFailover=Yes instead of True
I'm using SQLAlchemy in WSGI python web app to query the database. If I do two concurrent requests, the second request invariably throws an exception, with SQL Server stating
[24000] [FreeTDS][SQL Server]Invalid cursor state (0) (SQLExecDirectW)
Unfortunately it looks like I can't use caching to prevent additional requests to the database. Is there another way to resolve this issue? Ideally using native python libraries (i.e. not relying on another python module)?
The only thing I can think of is using threads to put a lock on the function making the database queries, but I'm worried this will slow down the app.
Is there anything else that can be done? Is this a configuration issue?
I'm using FreeTDS v0.91 on a Centos 5.9 server, connecting to MS SQL Server 2008.
The webapp is based on Paste.
Are your two concurrent requests using different database connections? DBAPI connections are not generally threadsafe. At the ORM level, you'd make sure you're using session per request so that each request has its own Session and therefore dedicated DBAPI connection.