I have to users to connect to a SQL Server. On the server, I need to read from SQL Server into python and write from python to SQL Server.
When I login with one one the users, everything goes fine with connection, whether I use Windows authentication or SQL Server authentication, with this code:
SQL Server authentication:
import sqlalchemy
engine = sqlalchemy.create_engine("mssql+pyodbc://myservername/mydatabasename/driver=SQL+Server+Native+Client+11.0?Trusted_Connection = no/UID = sa/PWD = mypassword")
conn = engine.connect()
df.to_sql(name = 'TestTable1', schema='dbo', con = conn)
Windows authentication:
import sqlalchemy
engine = sqlalchemy.create_engine("mssql+pyodbc://myservername/mydatabasename/driver=SQL+Server+Native+Client+11.0?Trusted_Connection = yes")
conn = engine.connect()
df.to_sql(name = 'TestTable1', schema='dbo', con = conn)
but with another user, I got this error:
Data source name too long State:IM010,Native:0,Origin:[Microsoft][ODBC Driver Manager]
Does this error do something with the other user?
For every login SQL Server needs credentials setup for Windows authentication or SQL Server authentication. You can test the credentials using SSMS before you use them in the Python program.
Related
I try to connect to Azure SQL using SQLAlchemy in python, authenticating with JWT generated for App registration. Despite checking almost any combination of parameters and granting all potentially meaningful permissions, I still get following error:
[28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user ''. (18456)
I did the following:
Created App registration 'test-app'.
Executed on the db:
CREATE USER [test-app] FROM EXTERNAL PROVIDER;
EXEC sp_addrolemember N'db_datareader', N'test-app';
GRANT SELECT ON test_vw to [test-app];
Added 'test-app' to Contributor and Reader roles for the SQL Server.
Whitelisted IP in the portal - I'm able to log in through SSMS and 'AAD Universal with MFA' using personal account.
Generated token with (tenant ID matches the one used by the organization):
POST https://login.microsoftonline.com/9ff8761-1be4-4729-b88df-e421e19d92f/oauth2/v2.0/token HTTP/1.1
Content-Type: application/x-www-form-urlencoded
client_id=[client_id]
client_secret=[client_secret]
scope=https://database.windows.net/.default
grant_type=client_credentials
Used generated JWT token in following python code:
from sqlalchemy import create_engine
import urllib
import struct
server = "test-server.database.windows.net"
database = "test-db"
driver = "{ODBC Driver 17 for SQL Server}"
jwt_token = "<jwt_token>"
SQL_COPT_SS_ACCESS_TOKEN = 1256
exptoken = b""
for i in bytes(jwt_token, "UTF-8"):
exptoken += bytes({i})
exptoken += bytes(1)
tokenstruct = struct.pack("=i", len(exptoken)) + exptoken
params = urllib.parse.quote_plus(
f"Driver={driver};Server={server};Database={database};Encrypt=yes"
)
conn_str = f"mssql+pyodbc:///?odbc_connect={params}"
engine_azure = create_engine(
conn_str,
connect_args={"attrs_before": {SQL_COPT_SS_ACCESS_TOKEN: tokenstruct}},
echo=True,
)
with engine_azure.connect() as con:
rs = con.execute("SELECT TOP 100 * FROM test_vw")
for row in rs:
print(row)
Any idea what I've missed?
I registered an app and copied ClientID, TenantId, generated client secret and copied ClientSecret value.I created azure SQL database and copied servername, databasename in azure portal.
I tried to connect azure SQL database by using token by replacing ClientID, TenantId, ClientSecret:
I installed SQLALchemy and adal packages using
pip install sqlalchemy
pip install adal
Connecting to Azure SQL database using below code:
from sqlalchemy import create_engine
import adal
server = '<servername>'
database = '<databasename>'
client_id = '<ClientId>'
tenant_id = '<TenantId>'
authority_host_uri = 'https://login.microsoftonline.com'
authority_uri = authority_host_uri + '/' + tenant_id
resource_uri = 'https://database.windows.net/'
client_secret = 'NbH8Q~x8PsGqD4C~8HUGzLmql_iRS7dxOMv2Bcjj'
context = adal.AuthenticationContext(authority_uri)
token = context.acquire_token_with_client_credentials(resource_uri, client_id, client_secret)
conn_str = (
'Driver={ODBC Driver 17 for SQL Server};'
f'Server={server};'
f'Database={database};'
'Encrypt=yes;'
'TrustServerCertificate=yes;'
f'AccessToken={token["accessToken"]};'
)
engine = create_engine('mssql+pyodbc:///?odbc_connect={}'.format(conn_str))
with engine.connect() as conn:
result = conn.execute('SELECT * FROM your-table')
for row in result:
print(row)
I got below error:
I created new group and added to registered app by following below procedure:
Goto-->Azure Active directory-->Enterprise Applications
Select the app and go to--> Group/Users-->Add User/Group select the user/group and click on assign.
Set that user/group as admin of SQL server by clicking Set admin:
I tried to connect SQL database again:
It worked successfully for me kindly check from your end.
I have a running CloudSQL instance running in another VPC and a nginx proxy to allow cross-vpc access.
I can access the db using a built-in user. But how can I access the DB using a Google Service Account?
import google.auth
import google.auth.transport.requests
import mysql.connector
from mysql.connector import Error
import os
creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
connection = mysql.connector.connect(host=HOST,
database=DB,
user=SA_USER,
password=creds.token)
if connection.is_connected():
db_Info = connection.get_server_info()
print("Connected to MySQL Server version ", db_Info)
cur = connection.cursor()
cur.execute("""SELECT now()""")
query_results = cur.fetchall()
print(query_results)
When using mysql connnector, I get this error:
DatabaseError: 2059 (HY000): Authentication plugin 'mysql_clear_password' cannot be loaded: plugin not enabled
Then I tried using pymysql
import pymysql
import google.auth
import google.auth.transport.requests
import os
creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
try:
conn = pymysql.connect(host=ENDPOINT, user=SA_USER, passwd=creds.token, port=PORT, database=DBNAME)
cur = conn.cursor()
cur.execute("""SELECT now()""")
query_results = cur.fetchall()
print(query_results)
except Exception as e:
print("Database connection failed due to {}".format(e))
Database connection failed due to (1045, "Access denied for user 'xx'#'xxx.xxx.xx.xx' (using password: YES)"
I guess these errors are all related to the token.
Anyone to suggest a proper way to get SA token to access CloudSQL DB?
PS: Using cloudsql auth proxy is not a good option for our architecture.
The error that you have mentioned in description , indicates an issue with authentication , to exactly understand what could have caused ,try these things
Verify the username and corresponding password.
Check the origin of the connection to see if it matches the URL where
the user has access privileges.
Check the user's grant privileges in the database.
As you are trying to access the DB using a Google Service Account then you should try to use the default service account credentials to include this authorization token for you. Check out the Client libraries and sample code page for more info.Alternatively, if you prefer to manually create the requests, you can use an Oauth 2.0 token. The Authorizing requests page has more information for how to create these.These access tokens are only valid for 60 minutes after which they expire - however once a token expires it does not disconnect clients but if that client connection is broken and must re-connect to the instance, and it's been more than an hour, then a new access token will need to be pulled and provided on that new connection attempt.
For your use case as you are not interested in cloud sql proxy, a service account IAM user is the better way to go.
Note that to get an appropriate access token the scope must be set to Cloud SQL Admin API.
It finally works.
I had to enforce SSL connection.
import pymysql
from google.oauth2 import service_account
import google.auth.transport.requests
scopes = ["https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/sqlservice.admin"]
credentials = service_account.Credentials.from_service_account_file('key.json', scopes=scopes)
auth_req = google.auth.transport.requests.Request()
credentials.refresh(auth_req)
config = {'user': SA_USER,
'host': ENDPOINT,
'database': DBNAME,
'password': credentials.token,
'ssl_ca': './server-ca.pem',
'ssl_cert': './client-cert.pem',
'ssl_key': './client-key.pem'}
try:
conn = pymysql.connect(**config)
with conn:
print("Connected")
cur = conn.cursor()
cur.execute("""SELECT now()""")
query_results = cur.fetchall()
print(query_results)
except Exception as e:
print("Database connection failed due to {}".format(e))
I'd recommend using the Cloud SQL Python Connector it should make your life way easier!
It manages the SSL connection for you (no need for cert files!), takes care of the credentials (uses Application Default Credentials which you can set to service account easily) and allows you to login with Automatic IAM AuthN so that you don't have to pass the credentials token as a password.
Connecting looks like this:
from google.cloud.sql.connector import Connector, IPTypes
import sqlalchemy
import pymysql
# initialize Connector object
connector = Connector(ip_type=IPTypes.PRIVATE, enable_iam_auth=True,)
# function to return the database connection
def getconn() -> pymysql.connections.Connection:
conn: pymysql.connections.Connection = connector.connect(
"project:region:instance", # your Cloud SQL instance connection name
"pymysql",
user="my-user",
db="my-db-name"
)
return conn
# create connection pool
pool = sqlalchemy.create_engine(
"mysql+pymysql://",
creator=getconn,
)
# insert statement
insert_stmt = sqlalchemy.text(
"INSERT INTO my_table (id, title) VALUES (:id, :title)",
)
# interact with Cloud SQL database using connection pool
with pool.connect() as db_conn:
# insert into database
db_conn.execute(insert_stmt, id="book1", title="Book One")
# query database
result = db_conn.execute("SELECT * from my_table").fetchall()
# Do something with the results
for row in result:
print(row)
Let me know if you run into any issues! There is also an interactive Cloud SQL Notebook that will walk your through things in more detail you can check out.
I need to connect to the ms-sql database and then create a new database there using python script.
I have the user credentials for the login. So how to create the connection to the ms-sql server using python.
If you do not have database name then use the connection string as mentioned in the code below. Create a database after connection and use the database finally.
import pyodbc
# if you have user id and password then try with this connection string
connection_string = f"DRIVER={SQL Server};SERVER={server_name};UID={user_id};PWD={password}"
# if using in the local system then use the following connection string
connection_string = f"DRIVER={SQL Server};SERVER={server_name}; Trusted_Connection=True;"
connection= pyodbc.connect(connection_string)
cursor = connection.cursor()
sql_create_database = f"CREATE DATABASE {database_name}"
cursor.execute(sql_create_database)
set_database = f"USE {database_name}"
cursor.execute(set_database)
I developed a simple python Azure function app using pyodbc to select a few rows from a public IP MS SQL server. My function app runs fine on my laptop, but it doesn't work when I publish it on Azure cloud (I used Consumption - serverless plan, linux environment). Thru the logging, I knows that it always gets stuck at the pyodbc.connect(...) command and time-out.
#...
conn_str = f'Driver={driver};Server={server},{port};Database={database};Uid={user};Pwd={password};Encrypted=yes;TrustServerCertificate=no;Connection Timeout=30'
sql_query = f'SELECT * FROM {table_name}'
try:
conn = pyodbc.connect(conn_str) # always time-out here if running on Azure cloud!!!
logging.info(f'Inventory API - connected to {server}, {port}, {user}.')
except Exception as error:
logging.info(f'Inventory API - connection error: {repr(error)}.')
else:
with conn.cursor() as cursor:
cursor.execute(sql_query)
logging.info(f'Inventory API - executed query: {sql_query}.')
data = []
for row in cursor:
data.append({'Sku' : row.Sku, 'InventoryId' : row.InventoryId, 'LocationId' : row.LocationId, 'AvailableQuantity' : row.AvailableQuantity})
#...
The logging captured:
Inventory API - connection error: OperationalError('HYT00', '[HYT00] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0) (SQLDriverConnect)').
I already include the pyodbc in the requirements.txt file. I also allows all outboundIpAddresses and possibleOutboundIpAddresses of my function app on my SQL server firewall. My function app does not have any network restriction on Azure cloud (or at least it said so on the network settings).
my config file:
driver={ODBC Driver 17 for SQL Server}
server=I tried both IP and full internet host name, both didn't work
Could someone give me a hint? Thanks.
Workarounds to fix the PYODBC connection Error
Try to use a below format
import pyodbc
server = 'tcp:myserver.database.windows.net'
database = 'mydb'
username = 'myusername'
password = 'mypassword'
conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
If you are using Domain Name add TCP with domain name server = 'tcp:myserver.database.windows.net' otherwise use the IP server = '129.0.0.1’
If you are using port use like this 'tcp:myserver.database.windows.net,1233’ or '129.0.0.1,1233'
Try to remove additional properties like Connection_Timeout, Trusted_certificate and all and check now.
Refer here
I put the following snippet into my function to check the outbound IP, and found out that Azure use a few outbound IPs that are not listed in the [outboundIpAddresses] and [possibleOutboundIpAddresses] (documented in this MS link)
import requests
#...
outbound_ip_response = requests.request('GET', 'https://checkip.amazonaws.com')
logging.info(f'Inventory API - main()- outbound ip = {outbound_ip_response.text}')
So, I followed the instructions in this link to setup a static outbound IP for my function app and allowed this IP to access my SQL server. It worked.
What I have in SQLite at the moment in the JSON configuration file:
"db": "sqlite:///\\\\F\\Evaluations\\data\\abc.db"
What I want is to connect to a database in SQL Server. How I do that?
db is used in the pipeline numerous times in different python files as follows:
...
self.db = settings['db']
...
engine = sa.create_engine( self.db, echo=False)
conn = engine.connect()
First, try to install pyodbc with the following command:
pip install pyodbc
Then, you can simply add the MS SQL URI connection to the JSON config file as follows:
"db_mssql":"mssql+pyodbc://username:password#host:port/database"
Then, in your python script:
self.db = settings['db_mssql']
engine = sa.create_engine( self.db, echo=False)
conn = engine.connect()
It looks like you are connecting through sqlalchemy. Your connection string for SQL server will need to include more information such as:
host name/server name
database name
credentials (or specify a trusted connection)
driver name
It will be good to test in a different script first.
import sqlalchemy as db
import pyodbc # you may need to install pyodbc (or use a different supported driver)
server_name = 'localhost'
database_name = 'AdventureWorks2019'
# directly specify your driver
driver = 'ODBC Driver 17 for SQL Server'
# or use pyodbc to find it
# driver = [x for x in pyodbc.drivers() if x.endswith(' for SQL Server')]
# driver = driver[0]
# specifying a username/password
engine = db.create_engine('mssql+pyodbc://USERNAME:PASSWORD#'+server_name+'/'+database_name+'?driver='+driver)
# or using a trusted connection
# sqlalchemy will add "trusted_connection=yes" if username/password isn't specified
engine = db.create_engine('mssql+pyodbc://#'+server_name+'/'+database_name+'?driver='+driver)
Then your JSON configuration will end up being something like:
"db": "mssql+pyodbc://USERNAME:PASSWORD#localhost/AdventureWorks2019?driver=ODBC Driver 17 for SQL Server"
OR
"db": "mssql+pyodbc://#localhost/AdventureWorks2019?driver=ODBC Driver 17 for SQL Server"
Side note, there is an extra parameter to speed up inserts into SQL server, if you find yourself needing it.
db.create_engine('mssql+pyodbc://USERNAME:PASSWORD#'+server_name+'/'+database_name+'?driver='+driver, fast_executemany=True)
Reference Links:
https://docs.sqlalchemy.org/en/14/core/engines.html
https://docs.sqlalchemy.org/en/14/dialects/mssql.html