Develop Reference

Connect to cloudSQL db using service account with pymysql or mysql.connector

I have a running CloudSQL instance running in another VPC and a nginx proxy to allow cross-vpc access.
I can access the db using a built-in user. But how can I access the DB using a Google Service Account?
import google.auth
import google.auth.transport.requests
import mysql.connector
from mysql.connector import Error
import os
creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
connection = mysql.connector.connect(host=HOST,
database=DB,
user=SA_USER,
password=creds.token)
if connection.is_connected():
db_Info = connection.get_server_info()
print("Connected to MySQL Server version ", db_Info)
cur = connection.cursor()
cur.execute("""SELECT now()""")
query_results = cur.fetchall()
print(query_results)
When using mysql connnector, I get this error:
DatabaseError: 2059 (HY000): Authentication plugin 'mysql_clear_password' cannot be loaded: plugin not enabled
Then I tried using pymysql
import pymysql
import google.auth
import google.auth.transport.requests
import os
creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
try:
conn = pymysql.connect(host=ENDPOINT, user=SA_USER, passwd=creds.token, port=PORT, database=DBNAME)
cur = conn.cursor()
cur.execute("""SELECT now()""")
query_results = cur.fetchall()
print(query_results)
except Exception as e:
print("Database connection failed due to {}".format(e))
Database connection failed due to (1045, "Access denied for user 'xx'#'xxx.xxx.xx.xx' (using password: YES)"
I guess these errors are all related to the token.
Anyone to suggest a proper way to get SA token to access CloudSQL DB?
PS: Using cloudsql auth proxy is not a good option for our architecture.

The error that you have mentioned in description , indicates an issue with authentication , to exactly understand what could have caused ,try these things
Verify the username and corresponding password.
Check the origin of the connection to see if it matches the URL where
the user has access privileges.
Check the user's grant privileges in the database.
As you are trying to access the DB using a Google Service Account then you should try to use the default service account credentials to include this authorization token for you. Check out the Client libraries and sample code page for more info.Alternatively, if you prefer to manually create the requests, you can use an Oauth 2.0 token. The Authorizing requests page has more information for how to create these.These access tokens are only valid for 60 minutes after which they expire - however once a token expires it does not disconnect clients but if that client connection is broken and must re-connect to the instance, and it's been more than an hour, then a new access token will need to be pulled and provided on that new connection attempt.
For your use case as you are not interested in cloud sql proxy, a service account IAM user is the better way to go.
Note that to get an appropriate access token the scope must be set to Cloud SQL Admin API.

It finally works.
I had to enforce SSL connection.
import pymysql
from google.oauth2 import service_account
import google.auth.transport.requests
scopes = ["https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/sqlservice.admin"]
credentials = service_account.Credentials.from_service_account_file('key.json', scopes=scopes)
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
config = {'user': SA_USER,
'host': ENDPOINT,
'database': DBNAME,
'password': credentials.token,
'ssl_ca': './server-ca.pem',
'ssl_cert': './client-cert.pem',
'ssl_key': './client-key.pem'}
try:
conn = pymysql.connect(**config)
with conn:
print("Connected")
cur = conn.cursor()
cur.execute("""SELECT now()""")
query_results = cur.fetchall()
print(query_results)
except Exception as e:
print("Database connection failed due to {}".format(e))

I'd recommend using the Cloud SQL Python Connector it should make your life way easier!
It manages the SSL connection for you (no need for cert files!), takes care of the credentials (uses Application Default Credentials which you can set to service account easily) and allows you to login with Automatic IAM AuthN so that you don't have to pass the credentials token as a password.
Connecting looks like this:
from google.cloud.sql.connector import Connector, IPTypes
import sqlalchemy
import pymysql
# initialize Connector object
connector = Connector(ip_type=IPTypes.PRIVATE, enable_iam_auth=True,)
# function to return the database connection
def getconn() -> pymysql.connections.Connection:
conn: pymysql.connections.Connection = connector.connect(
"project:region:instance", # your Cloud SQL instance connection name
"pymysql",
user="my-user",
db="my-db-name"
)
return conn
# create connection pool
pool = sqlalchemy.create_engine(
"mysql+pymysql://",
creator=getconn,
)
# insert statement
insert_stmt = sqlalchemy.text(
"INSERT INTO my_table (id, title) VALUES (:id, :title)",
)
# interact with Cloud SQL database using connection pool
with pool.connect() as db_conn:
# insert into database
db_conn.execute(insert_stmt, id="book1", title="Book One")
# query database
result = db_conn.execute("SELECT * from my_table").fetchall()
# Do something with the results
for row in result:
print(row)
Let me know if you run into any issues! There is also an interactive Cloud SQL Notebook that will walk your through things in more detail you can check out.

ValueError: No password or public key available

I'm trying to connect to a remote MySQL database through an SSH Tunnel and deploying my code to Streamlit. When I try to do it, I get this error:
File "/home/appuser/venv/lib/python3.9/site-packages/sshtunnel.py", line 966, in __init__
(self.ssh_password, self.ssh_pkeys) = self._consolidate_auth(
File "/home/appuser/venv/lib/python3.9/site-packages/sshtunnel.py", line 1169, in _consolidate_auth
raise ValueError('No password or public key available!')
ValueError: No password or public key available!
I've tried a lot of things, from updating my SSH keys to my server and github to changing my code.
The code I have for the SSH - MySQL section looks like this:
import MySQLdb as db
from sshtunnel import SSHTunnelForwarder
def query(q):
with SSHTunnelForwarder(
ssh_address_or_host=("host_ip"),
ssh_username=("host_username"),
ssh_pkey=("path_to_private_sshkey"),
remote_bind_address=("private_host_ip", "host_port")
) as server:
conn = db.connect(
host="localhost",
port=server.local_bind_port,
user="db_username",
passwd="db_password",
db="db_database"
)
return pd.read_sql_query(q, conn)
I appreciate any help you can give me.

conn = db.connect(host="localhost"),
port=server.local_bind_port,
user=("db_username"),
passwd=("db_password"),
db=("db_database")
Because you have a closing parentheses on the first line, only the host argument is being passed to the db.connect() function. And so the function is complaining that it doesn't have a password, username, etc.
The other lines are creating plain local variables.

PyHive with Kerberos throws Authentication error after few calls

I am trying to connect to Hive using Python (PyHive Lib) to read some data and then I further connects it to hive Flask to show in Dashboard.
It all works fine for few calls to hive, however soon after that I am getting following error.
Traceback (most recent call last):
File "libs/hive.py", line 63, in <module>
cur = h.connect().cursor()
File "libs/hive.py", line 45, in connect
kerberos_service_name='hive')
File "/home1/igns/git/emsr/.venv/lib/python2.7/site-packages/pyhive/hive.py", line 94, in connect
return Connection(*args, **kwargs)
File "/home1/igns/git/emsr/.venv/lib/python2.7/site-packages/pyhive/hive.py", line 192, in __init__
self._transport.open()
File "/home1/igns/git/emsr/.venv/lib/python2.7/site-packages/thrift_sasl/__init__.py", line 79, in open
message=("Could not start SASL: %s" % self.sasl.getError()))
thrift.transport.TTransport.TTransportException: Could not start SASL: Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_cdc995595290_51CD7j))
Following is my code
from pyhive import hive
class Hive(object):
def connect(self):
return hive.connect(host='hive.hadoop-prod.abc.com',
port=10000,
database='temp',
username='gaurang.shah',
auth='KERBEROS',
kerberos_service_name='hive')
if __name__ == '__main__':
h = Hive()
cur = h.connect().cursor()
cur.execute("select * from temp.migration limit 1")
res = cur.fetchall()
print res
Calling Script
source .venv/bin/activate
for i in {1..50}
do
python get_hive_data.py
sleep 300
done
Observation
When it's working I can see hive in service principal when I do klist however, I don't when I see above error message.
Klist when it's working
Ticket cache: FILE:/tmp/krb5cc_cdc995595290_XyMnhu
Default principal: gaurang.shah#ABC.COM
Valid starting Expires Service principal
12/04/2018 14:37:28 12/05/2018 00:37:28 krbtgt/ABC.COM#ABC.COM
renew until 12/05/2018 14:37:24
12/04/2018 14:39:06 12/05/2018 00:37:28 hive/hive_server.ABC.COM#ABC.COM
renew until 12/05/2018 14:37:24
Klist when it's not working
Ticket cache: FILE:/tmp/krb5cc_cdc995595290_XyMnhu
Default principal: gaurang.shah#ABC.COM
Valid starting Expires Service principal
12/04/2018 14:37:28 12/05/2018 00:37:28 krbtgt/ABC.COM#ABC.COM
renew until 12/05/2018 14:37:24
Update:
So I don't think it's after certain call however, I think it's after certain time. ( I think one hour). I changed the sleep time to 3600 sec and just after first call I started getting error.
This is weird as, ticket for hive/hive_server.ABC.COM#ABC.COM was valid for 7 days

I know this is an old post. But if you make a new connection every time you are doing a call, you should resolve the issue.
from pyhive import hive
class Hive(object):
def connect(self):
return hive.connect(host='hive.hadoop-prod.abc.com',
port=10000,
database='temp',
username='gaurang.shah',
auth='KERBEROS',
kerberos_service_name='hive')
if __name__ == '__main__':
def newConnect(query):
h = Hive()
cur = h.connect().cursor()
cur.execute(query)
res = cur.fetchall()
return res
myConnectionAndResults = newConnect("select * from temp.migration limit 1")
print myConnectionAndResults

how can i validate username password for mongodb authentication through pymongo?

I am refering to the http://api.mongodb.org/python/current/examples/authentication.html site for authentication mechanism examples. I have created a User administrator and using its credentials I created a user for my 'reporting' database. Now i need to access the same through pymongo using the username and password. I tried the following commands in python shell. Is this the right way as my authentication is failing.
from pymongo import MongoClient
client = MongoClient('localhost')
client.reporting.authenticate('reportsUser', '123456', mechanism='MONGODB-CR')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/pymongo/database.py", line 746, in authenticate
self.connection._cache_credentials(self.name, credentials)
File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 441, in _cache_credentials
auth.authenticate(credentials, sock_info, self.__simple_command)
File "/usr/lib/python2.7/dist-packages/pymongo/auth.py", line 214, in authenticate
auth_func(credentials[1:], sock_info, cmd_func)
File "/usr/lib/python2.7/dist-packages/pymongo/auth.py", line 194, in _authenticate_mongo_cr
cmd_func(sock_info, source, query)
File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 607, in __simple_command
helpers._check_command_response(response, None, msg)
File "/usr/lib/python2.7/dist-packages/pymongo/helpers.py", line 147, in _check_command_response
raise OperationFailure(msg % errmsg, code)
pymongo.errors.OperationFailure: command SON([('authenticate', 1), ('user', u'reportsUser'), ('nonce', u'f8158a24f1c61650'), ('key', u'14cea216c54b93bae20acd2e076bb785')]) failed: auth failed

As an FYI, you can use the URI string format as well. The pseudocode looks like this:
pymongo.MongoClient('mongodb://user:password#server:port/')
Here's a simple connection code block with auth:
import pymongo
conn = pymongo.MongoClient('mongodb://root:pass#localhost:27017/')
db = conn['database']
coll = db['collection']
There are more options for the query string here: http://docs.mongodb.org/manual/reference/connection-string/
Hope that helps = looks like you already have it though. Happy coding!!

Just adding more to provided solutions.
I have been using as URI connection string and credentials being provided as f string it helps to reduce number of lines. One thing to note is about special characters in password where we convert using urllib package as shown below.
import urllib.parse
from pymongo import MongoClient
host = "localhost"
port = 27017
user_name = "myuser"
pass_word = "Pass#123"
db_name = "mydb" # database name to authenticate
# if your password has '#' then you might need to escape hence we are using "urllib.parse.quote_plus()"
client = MongoClient(f'mongodb://{user_name}:{urllib.parse.quote_plus(pass_word)}#{host}:{port}/{db_name}')

It's worked for me.
Here you can connect mongodb to python by using authentication username and password.
import pymongo
DATABASE_NAME = "your_database_name"
DATABASE_HOST = "localhost"
DATABASE_USERNAME = "database_username"
DATABASE_PASSWORD = "database_password"
try:
myclient = pymongo.MongoClient( DATABASE_HOST )
myclient.test.authenticate( DATABASE_USERNAME , DATABASE_PASSWORD )
mydb = myclient[DATABASE_NAME]
print("[+] Database connected!")
except Exception as e:
print("[+] Database connection error!")
raise e
By default Mongodb uses 27017 port

Can't connect to Meteor with Pymongo

I am trying to connect to a Meteor Mongo database through pymongo. Here's the code:
def get_mongo_url(site):
# return "mongodb://client-xxxxx:yyyyy#production-db-c1.meteor.io:27017/site"
import subprocess
p = subprocess.Popen(['meteor', 'mongo', '--url', site], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
print out
return out
from pymongo import MongoClient
client = MongoClient(get_mongo_url("mysite.com"))
And the error (the print statement yields a correct url)
>> mongodb://client-xxxxx:yyyyy#production-db-c1.meteor.io:27017/site
Traceback (most recent call last):
File "private/test.py", line 46, in <module>
client = pymongo.MongoClient(get_mongo_url(METEOR_SITE))
File "/Library/Python/2.7/site-packages/pymongo/mongo_client.py", line 369, in __init__
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: command SON([('authenticate', 1), ('user', u'client-xxxxx'), ('nonce', u'zzzzz'), ('key', u'ttttt')]) failed: auth fails
If I run meteor mongo --url mysite.com, copy the result into the return ... at the top of the function and uncomment it, the connection works. Why can't I connect programmatically?

The subprocess code appends a line feed character \n to the end of the url.
You need to strip that with .rstrip()
The right way to do that is to replace the return in your function with
return out.rstrip()
For confirmation purposes I will show what happens with the function as-is and
rstrip() applied/unapplied to the return.
murl = get_mongo_url('').rstrip()
mongodb://client-faf1d0db:746d8f43-367b-dde2-b69a-039ff8b9f7fa#production-db-a1.meteor.io:27017/_meteor_com
client = pymongo.MongoClient(murl)
Worked OK
murl = get_mongo_url('')
mongodb://client-3578a20b:d4ddeec9-6d24-713e-8ddb-c357b664948a#production-db-a1.meteor.io:27017/_meteor_com
client = pymongo.MongoClient(murl)
Traceback (most recent call last):
File "", line 1, in
File "/home/action/.local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 383, in init
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: command SON([('authenticate', 1), ('user', u'client-3578a20b'), ('nonce', u'e14e2bdb3d8484b9'), ('key', u'9
c101b78ff1a617a9c5f0def36c7e3d9')]) failed: auth fails
Failed without the rstrip.
murl = get_mongo_url('')
mongodb://client-1a193a61:4c9c572e-22e3-4b7e-44a1-dc76bfb65e86#production-db-a1.meteor.io:27017/_meteor_com
client = pymongo.MongoClient(murl)
Traceback (most recent call last):
File "", line 1, in
File "/home/action/.local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 383, in init
raise ConfigurationError(str(exc))
pymongo.errors.ConfigurationError: command SON([('authenticate', 1), ('user', u'client-1a193a61'), ('nonce', u'a2576142b1a33d8b'), ('key', u'4
419c490bcdcc65b20f2950c3b106d59')]) failed: auth fails
Failed again (no rsrtip)
murl = get_mongo_url('').rstrip()
mongodb://client-ce463608:d7dc6be0-499f-1808-43e1-fdfb8b6e8ebc#production-db-a1.meteor.io:27017/_meteor_com
client = pymongo.MongoClient(murl)
Worked (rstrip used).
The following is general info on mongodb URLs. You may know this already.
The URL that pymongo wants is not a web URL but a URL-like specifier for a mongo database connection.
For a development environment, the mongodb is usually set up on port 3001, which is not the default mongodb port for a production server.
Meteor applications can be configured to use a mongodb hosted anywhere. It does not have to be on the same server that serves the meteor content. The specification of this is done through the mongodb:// URL which is what pymongo wants. pymongo doesn't depend on the meteor website url, which can be very different from the mongodb url.
Here is some code I am using
import pymongo
MONGO_URL = r'mongodb://localhost:3001/meteor'
###
def connect():
client = pymongo.MongoClient(MONGO_URL)
return client
def findUser(c, email):
users = c.meteor.users
return users.find_one({"emails.address": email})
According to the mongodb site on Github, The MONGO_URL format is
mongodb://[username:password#]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]
so the mongodb url mongodb://localhost:3001/meteor can be interpreted like this:
* mongodb:// means this describes a mongodb connection
* localhost means connect locally
* :3001 means use non-standard port number 3001. this is how "meteor run" sets up mongo
* /meteor means connect to the database called "meteor"