Develop Reference

Connect to cloudSQL db using service account with pymysql or mysql.connector

I have a running CloudSQL instance running in another VPC and a nginx proxy to allow cross-vpc access.
I can access the db using a built-in user. But how can I access the DB using a Google Service Account?
import google.auth
import google.auth.transport.requests
import mysql.connector
from mysql.connector import Error
import os
creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
connection = mysql.connector.connect(host=HOST,
database=DB,
user=SA_USER,
password=creds.token)
if connection.is_connected():
db_Info = connection.get_server_info()
print("Connected to MySQL Server version ", db_Info)
cur = connection.cursor()
cur.execute("""SELECT now()""")
query_results = cur.fetchall()
print(query_results)
When using mysql connnector, I get this error:
DatabaseError: 2059 (HY000): Authentication plugin 'mysql_clear_password' cannot be loaded: plugin not enabled
Then I tried using pymysql
import pymysql
import google.auth
import google.auth.transport.requests
import os
creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
try:
conn = pymysql.connect(host=ENDPOINT, user=SA_USER, passwd=creds.token, port=PORT, database=DBNAME)
cur = conn.cursor()
cur.execute("""SELECT now()""")
query_results = cur.fetchall()
print(query_results)
except Exception as e:
print("Database connection failed due to {}".format(e))
Database connection failed due to (1045, "Access denied for user 'xx'#'xxx.xxx.xx.xx' (using password: YES)"
I guess these errors are all related to the token.
Anyone to suggest a proper way to get SA token to access CloudSQL DB?
PS: Using cloudsql auth proxy is not a good option for our architecture.

The error that you have mentioned in description , indicates an issue with authentication , to exactly understand what could have caused ,try these things
Verify the username and corresponding password.
Check the origin of the connection to see if it matches the URL where
the user has access privileges.
Check the user's grant privileges in the database.
As you are trying to access the DB using a Google Service Account then you should try to use the default service account credentials to include this authorization token for you. Check out the Client libraries and sample code page for more info.Alternatively, if you prefer to manually create the requests, you can use an Oauth 2.0 token. The Authorizing requests page has more information for how to create these.These access tokens are only valid for 60 minutes after which they expire - however once a token expires it does not disconnect clients but if that client connection is broken and must re-connect to the instance, and it's been more than an hour, then a new access token will need to be pulled and provided on that new connection attempt.
For your use case as you are not interested in cloud sql proxy, a service account IAM user is the better way to go.
Note that to get an appropriate access token the scope must be set to Cloud SQL Admin API.

It finally works.
I had to enforce SSL connection.
import pymysql
from google.oauth2 import service_account
import google.auth.transport.requests
scopes = ["https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/sqlservice.admin"]
credentials = service_account.Credentials.from_service_account_file('key.json', scopes=scopes)
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
config = {'user': SA_USER,
'host': ENDPOINT,
'database': DBNAME,
'password': credentials.token,
'ssl_ca': './server-ca.pem',
'ssl_cert': './client-cert.pem',
'ssl_key': './client-key.pem'}
try:
conn = pymysql.connect(**config)
with conn:
print("Connected")
cur = conn.cursor()
cur.execute("""SELECT now()""")
query_results = cur.fetchall()
print(query_results)
except Exception as e:
print("Database connection failed due to {}".format(e))

I'd recommend using the Cloud SQL Python Connector it should make your life way easier!
It manages the SSL connection for you (no need for cert files!), takes care of the credentials (uses Application Default Credentials which you can set to service account easily) and allows you to login with Automatic IAM AuthN so that you don't have to pass the credentials token as a password.
Connecting looks like this:
from google.cloud.sql.connector import Connector, IPTypes
import sqlalchemy
import pymysql
# initialize Connector object
connector = Connector(ip_type=IPTypes.PRIVATE, enable_iam_auth=True,)
# function to return the database connection
def getconn() -> pymysql.connections.Connection:
conn: pymysql.connections.Connection = connector.connect(
"project:region:instance", # your Cloud SQL instance connection name
"pymysql",
user="my-user",
db="my-db-name"
)
return conn
# create connection pool
pool = sqlalchemy.create_engine(
"mysql+pymysql://",
creator=getconn,
)
# insert statement
insert_stmt = sqlalchemy.text(
"INSERT INTO my_table (id, title) VALUES (:id, :title)",
)
# interact with Cloud SQL database using connection pool
with pool.connect() as db_conn:
# insert into database
db_conn.execute(insert_stmt, id="book1", title="Book One")
# query database
result = db_conn.execute("SELECT * from my_table").fetchall()
# Do something with the results
for row in result:
print(row)
Let me know if you run into any issues! There is also an interactive Cloud SQL Notebook that will walk your through things in more detail you can check out.

Saving psycopg2 connection within flask using SimpleConnectionPool

How do I save a psycopg2 connection on a flask server once I execute a query? I am trying to save connection from a different python file on flask. I am using this code:-
from flask import current_app
cursor = connection.cursor() # Connection with Database
cursor.execute(self.query)
connection.commit()
current_app.config['pool'].putconn(connection)
It gives me this error.
'Flask' object is not subscriptable
I use this in the flask init file to create a connection Pool:-
app.config['pool'] = psycopg2.pool.SimpleConnectionPool(
1, 10,
host = config["HOST"],
database = config["DATABASE"],
user = config["USER"],
password = config["PASSWORD"]
)
This is the get_db function. It works well.
from flask import current_app
def get_db(self):
return current_app.config['pool'].getconn()

Cannot connect to SQL Server from python with a specific user

I have to users to connect to a SQL Server. On the server, I need to read from SQL Server into python and write from python to SQL Server.
When I login with one one the users, everything goes fine with connection, whether I use Windows authentication or SQL Server authentication, with this code:
SQL Server authentication:
import sqlalchemy
engine = sqlalchemy.create_engine("mssql+pyodbc://myservername/mydatabasename/driver=SQL+Server+Native+Client+11.0?Trusted_Connection = no/UID = sa/PWD = mypassword")
conn = engine.connect()
df.to_sql(name = 'TestTable1', schema='dbo', con = conn)
Windows authentication:
import sqlalchemy
engine = sqlalchemy.create_engine("mssql+pyodbc://myservername/mydatabasename/driver=SQL+Server+Native+Client+11.0?Trusted_Connection = yes")
conn = engine.connect()
df.to_sql(name = 'TestTable1', schema='dbo', con = conn)
but with another user, I got this error:
Data source name too long State:IM010,Native:0,Origin:[Microsoft][ODBC Driver Manager]
Does this error do something with the other user?

For every login SQL Server needs credentials setup for Windows authentication or SQL Server authentication. You can test the credentials using SSMS before you use them in the Python program.

Connect to MSSQL Database using Flask-SQLAlchemy

I'm trying to connect to a local MSSQL DB through Flask-SQLAlchemy.
Here's a code excerpt from my __init__.py file:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'mssql+pyodbc://HARRISONS-THINK/LendApp'
db = SQLAlchemy(app)
SQLALCHEMY_TRACK_MODIFICATIONS = False
As you can see in SQL Server Management Studio, this information seems to match:
Here is the creation of a simple table in my models.py file:
from LendApp import db
class Transaction(db.model):
transactionID = db.Column(db.Integer, primary_key=True)
amount = db.Column(db.Integer)
sender = db.Column(db.String(80))
receiver = db.Column(db.String(80))
def __repr__(self):
return 'Transaction ID: {}'.format(self.transactionID)
I am then connecting to the database using a Python Console within Pycharm via the execution of these two lines:
>>> from LendApp import db
>>> db.create_all()
This is resulting in the following error:
DBAPIError: (pyodbc.Error) ('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)')
The only thing that I can think of is that my database connection string is incorrect. I have tried altering it to more of a standard Pyodbc connection string and including driver={SQL SERVER} but to no prevail.
If anyone could help me out with this it would be highly appreciated.
Thanks

So I just had a very similar problem and was able to solve by doing the following.
Following the SQL Alchemy documentation I found I could use the my pyodbc connection string like this:
# Python 2.x
import urllib
params = urllib.quote_plus("DRIVER={SQL Server Native Client 10.0};SERVER=dagger;DATABASE=test;UID=user;PWD=password")
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
# Python 3.x
import urllib
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 10.0};SERVER=dagger;DATABASE=test;UID=user;PWD=password")
engine = create_engine("mssql+pyodbc:///?odbc_connect=%s" % params)
# using the above logic I just did the following
params = urllib.parse.quote_plus('DRIVER={SQL Server};SERVER=HARRISONS-THINK;DATABASE=LendApp;Trusted_Connection=yes;')
app.config['SQLALCHEMY_DATABASE_URI'] = "mssql+pyodbc:///?odbc_connect=%s" % params
This then caused an additional error because I was also using Flask-Migrate and apparently it doesn't like % in the connection URI. So I did some more digging and found this post. I then changed the following line in my ./migrations/env.py file
From:
from flask import current_app
config.set_main_option('sqlalchemy.url',
current_app.config.get('SQLALCHEMY_DATABASE_URI'))
To:
from flask import current_app
db_url_escaped = current_app.config.get('SQLALCHEMY_DATABASE_URI').replace('%', '%%')
config.set_main_option('sqlalchemy.url', db_url_escaped)
After doing all this I was able to do my migrations and everything seems as if it is working correctly now.

If someone still stumbled upon this issue and trying to figure out another solution then try with pymssql instead of pyodbc;
pip install pymssql
Connection URI would be:
conn_uri = "mssql+pymssql://<username>:<password>#<servername>/<dbname>"

I just changed my connection string something like this and its worked perfectly
NOTE: you need to install pyodbc to work....
app.config["SQLALCHEMY_DATABASE_URI"] = "mssql+pyodbc://user:pwd#server/database?driver=SQL+Server"
Note:
Try to avoid '#' character in password. you will get error because connection string also has '#' character after password. This also can cause the connection error

I had the same problem, it was resolved by specifying:
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = "mssql+pyodbc://MySQLServerName/MyTestDb?driver=SQL+Server?trusted_connection=yes"
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
db.init_app(app)

using below solution i get resolve my connection issue with MSSQL server
params = urllib.parse.quote_plus('DRIVER={SQL Server};SERVER=HARRISONS-THINK;DATABASE=LendApp;Trusted_Connection=yes;')
app.config['SQLALCHEMY_DATABASE_URI'] = "mssql+pyodbc:///?odbc_connect=%s" % params
If you are getting any Login failed for User error then please go to this
http://itproguru.com/expert/2014/09/how-to-fix-login-failed-for-user-microsoft-sql-server-error-18456-step-by-step-add-sql-administrator-to-sql-management-studio/.

I believe your connection string is missing the authentication details. From Flask-SQLAlchemy documentation, the connection string should have the following format
dialect+driver://username:password#host:port/database
From your example, I believe it will look something like this
app.config['SQLALCHEMY_DATABASE_URI'] = 'mssql+pyodbc://<username>:<password>#<Host>:<Port>/LendApp'

Flask-SQLAlchemy ssl-connection with AWS RDS error

I am trying to connect flask app mysql connection with AWS RDS over ssl , It works when I am try to use mysql client like this
mysql -u user -h myrds.rds.amazonaws.com -p --ssl-ca=rds-combined-ca-bundle.pem
I am able to login but when I am try with flask app
SQLALCHEMY_DATABASE_URI = 'mysql://user:Password#myrds.rds.amazonaws.com.rds.amazonaws.com/miro_dev?ssl_cert=rds-combined-ca-bundle.pem'
it send me error
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2026, 'SSL connection error: Unable to get private key')

I was able to get this work by adding
?sslmode=verify-ca&sslrootcert=rds-combined-ca-bundle.pem
to the connection string.
This came from the postgresql docs here along with the aws docs.
You can change the sslmode to require if you do not care about verifying the rds. I downloaded the pem file from here.

I think that in your case the connection string is correct, you just need to use ssl_ca option and not ssl_cert:
SQLALCHEMY_DATABASE_URI = 'mysql://user:password#myrds.rds.amazonaws.com.rds.amazonaws.com/miro_dev?ssl_ca=rds-combined-ca-bundle.pem'

I do this:
...
ssl_args = {'ssl': {'ca': 'YOUR_SSL_CERT_PATH'}}
db_url = 'mysql://{}:{}#{}/{}'.format(username, password, server, database)
engine = create_engine(db_url, connect_args=ssl_args, echo=False)
cnx = engine.connect()
df = pd.read_sql_table('table_name', cnx)
And I'd suggest to not input a path like follows:
~/...
but:
/home/YOUR_USER/...