Develop Reference

Oracle connect from Databricks

I am trying to connect to Oracle database in an on-premises VM from Databricks.
I tried to connect using CX_Oracle via Python through init-script by following this link. It failed with below errors
Then I tried to connect using Pyspark and it also failed with below error. Also installed OJDBC into the cluster, where I used OJDBC version compatible with Oracle DB version.
URL = "jdbc:oracle:thin:" + User_Name + "/" + Password + "#//" + IP + ":" + Port + "/" + DB_name
DbTable = DataBase_name + "." + Table_Name
empDF = spark.read.format("jdbc").option("url", URL).option("dbtable", DbTable).option("user", User_Name).option("password", Password).option("driver", "oracle.jdbc.driver.OracleDriver").load()
Using the same settings in ADF Lookup activity, using a self hosted IR I am able to establish the connection. Is there anything I am missing in the connection string from Databricks?

Select query is working but insert query is not in Python with Couchbase library

I'm trying to do some remote operations with Couchbase Python library. I have written a small code piece to test if it is working as intended.
server_name = 'xxxxxxxxxx'
bucket_name = 'vit_app_clob'
username = "Administrator"
password = "xxxxxx"
json_path = 'D:\\' + bucket_name + '_' + server_name + '.json'
cluster = Cluster('couchbase://' + server_name + ':8091', ClusterOptions(PasswordAuthenticator(username, password)))
bucket = cluster.bucket(bucket_name)
def insert(bucket):
result = cluster.query("INSERT INTO `oracle_test`(KEY, VALUE) VALUES ('key1', { 'type' : 'hotel', 'name' : 'new hotel' })")
def select():
result = cluster.query('SELECT * FROM ' + bucket_name)
myDict = []
for row in result:
name = row[bucket_name]
# print(name)
myDict.append(name)
df = pd.DataFrame(myDict)
with open(json_path, "w") as f:
json.dump(myDict, f, indent=1)
select()
# insert(bucket)
Select function works well. Query runs and there is no problem. But the other one, which I'm trying to use for inserting, doesn't work. It gives a timeout error.
couchbase.exceptions.TimeoutException: <Key='key3', RC=0xC9[LCB_ERR_TIMEOUT (201)]
What could be the reason? The query is working when I run it in the query section of couchbase interface.
Edit: There is this part in error log: 'endpoint': 'xxxxxx:11210'. Do I need to have an open connection in this port? It seems I don't have any access right now, since the telnet is not working. If this is the case, that means connecting via 8091 is not enough?
Edit2: We opened connection to port but the same problem exists

You are correct in that port 8091 is generally not enough. Check out the documentation on Couchbase port numbers.
You may not need all those ports, depending on which services you're using, but for my day-to-day work, I usually open ports 8091-8097 and 11210. (There are counterpart ports for encrypted traffic).

I don't even know why, but adding .execute() to end of the query solved the problem. SELECT query was already working without it. Also, I needed to open a connection on port 11210

ODBC connection to Azure using ActiveDirectoryInteractive

I am connecting to an Azure SQL database using pyodcb. We have multi-factor authentication enabled, so I am using ActiveDirectoryInteractive authentication. We also use single sign-on for Windows.
My code is as follows:
import pyodbc
driver = '{ODBC Driver 17 for SQL Server}'
server = 'xxxxxxxxxxxxx.database.windows.net'
database = 'xxxxxxxxxxxxx'
authentication = 'ActiveDirectoryInteractive'
username = 'xxxxxxxxxxxxx'
conn = pyodbc.connect(
'DRIVER=' + driver + ';' +
'SERVER=' + server + ';' +
'DATABASE=' + database + ';' +
'UID=' + username + ';' +
'AUTHENTICATION=' + authentication + ';')
cursor = conn.cursor()
When I run the script I get pop-ups to submit my password and MFA details. But the connection works whether I fill in these details or just close the prompts - suggesting that single sign-on is working successfully. So why am I getting the password and MFA pop-ups - and how can I suppress them?

I'm not sure if this will disable it in Office 365, but if you want to disable MFA for users in Azure ad, I think the easiest way is to disable it directly in the portal.

Connect to db2 database with Python

I'm working on an app that needs to connect to an ibm db2 database. Using DBeaver I can successfully connect to the database (I provide him the db2cc.jar and db2cc4.jar files).
It looks to me as DBeaver is using my Window's credentials to login, because I didn't need to input any login or password to connect.
Now, I've been trying to connect to the same database using python 3.7 and pypi's latest version of the ibm_db package. I didn't install anything else.
import ibm_db
# ...
connection_string = "DATABASE=" + self.params['schema'] + ";" + \
"HOSTNAME=" + self.params['host'] + ";" + \
"PORT=" + self.params['port'] + ";" + \
"PROTOCOL=TCPIP;" + \
"SECURITYMECHANISM=4;" + \
"UID=" + self.params['user'] + ";" + \
"PWD=" + self.params['password'] + ";"
try:
self.connection = ibm_db.connect(connection_string, "", "")
# ...
Using my Windows credentials in the parameters, I get the following error message:
Connection error
Bad credentials
SQLCODE=-30082
08001
From what I've seen on stack overflow connecting to a db2 database is complicated...
Does someone know how to connect ? Using the windows credentials or otherwise...
Thanks !

Db2 works fine with Python.
You need to be aware of some basics before you start such as:
what operating-system runs the target Db2-database and
what kind of client is being used (java, odbc/cli, .net etc), and
what kind of authentication/encrpytion is in place for the database
(ssl, server based authentication/+/-/encryption etc.).
is the remote database rdbms Apache DERBY or Db2.
Find out these basics before you start. You have to speak with people who run the Db2-server.
Note: in your question you mention (SecurityMechanism=4) com.ibm.db2.jcc.DB2BaseDataSource.USER_ONLY_SECURITY - this is not relevant for non-JAVA clients, it is relevant is the database manager is DERBY .
For python, the ibm_db package is not a java application.
DBeaver is a java application (hence it uses db2jcc.jar or db2jcc4.jar and a licence-file to connect to the remote database).
You can only use your Windows credentials for connecting to a Db2-database when that Db2-database run on Microsoft-Windows, and the credentials work on the hostname running the Db2-server. For any other combinations, the administrator must issue you a userid/password that is relevant for the target hostname.
The ibm_db package needs a Db2-client to be installed. The Db2-client is a separate installable. There are different kinds of Db2-client depending both on which operating-system runs your Db2-server and how much functionality you need to have in your Db2-client. If your remote Db2-server runs on Linux, Unix, Windows or Z/OS then you can use the "IBM Data Server Runtime Client" which you can either download from IBM's passport advantage website, or get from your internal IT folks. If your Db2-server runs on i-Series (AS/400) you should get its drivers from your i-Series administrator. For either Z/OS or i-Series you will additionally need a license file (which costs money) and you should get that from your administrator, unless your company uses a gateway product called Db2-connect in which case you don't need a separate license file on your workstation.

Try the following connection string, If your db2 client and server are on the same host.
Change the 'mydb' and 'DB2' (db2 instance name, you can get it with db2ilist utility) constants according your case.
ibm_db.connect('DATABASE=mydb;Instance=DB2;PROTOCOL=IPC;', '', '')

add ibm_db library and import ibm_db on top in the .py file.
def get_db_connection():
"""
This will help to get db2 connection for query execution
:return: conn
"""
dsn_driver = "{IBM DB2 ODBC DRIVER}"
dsn_database = "BLUDB"
dsn_hostname = "your_hostname"
dsn_port = "50000"
dsn_protocol = "TCPIP"
dsn_uid = "your_userid"
dsn_pwd = "your_pwd"
dsn = (
"DRIVER={0};"
"DATABASE={1};"
"HOSTNAME={2};"
"PORT={3};"
"PROTOCOL={4};"
"UID={5};"
"PWD={6};").format(dsn_driver, dsn_database, dsn_hostname, dsn_port, dsn_protocol, dsn_uid, dsn_pwd)
try:
conn = ibm_db.connect(dsn, "", "")
print("Connected!")
return conn
except Exception:
print("\nERROR: Unable to connect to the \'" + dsn_database + "\' server.")
print("error: ", ibm_db.conn_errormsg())
exit(-1)
function get_db_connection() will return the connection object. On that connection object you can perform operation like:
conn = get_db_connection()
list_results = []
select_query = 'SELECT a.STATUS, a.ID FROM "TABLE_NAME" AS a'
print(select_query)
selectStmt = ibm_db.exec_immediate(conn, select_query)
while ibm_db.fetch_row(selectStmt) != False:
list_results.append(ibm_db.result(selectStmt, 'ID'))
ibm_db.close(conn)

cx_Oracle._Error in Python script

I have attempted to search for a resolution or at least a few troubleshooting steps to understand why I see these errors when I execute a Python script. No luck so far. I SSH to Rackspace to do this.
The errors reference the Oracle db. To be more specific, it is cx_Oracle._Error and cx_Oracle.DatabaseError on separate occasions. Below is an example of a few lines from the server logs. It does not give any more information than that.
Running against Oracle server: prod.somecompanyname + prod123
Error connecting to databases: (<cx_Oracle._Error object at0x7ffff7fe9af8>,)
Error verifying existing Oracle records
My colleague is able to execute the script successfully and does not encounter the error. I compared our .bash_profile and .bashrc and nothing different stands out. The Oracle server credentials are correct in the script as well as the Oracle environment path. This may be isolated to something on my end however I cannot figure out where.
Any suggestions on where to look to fix this is appreciated.
def oraclerecords(logger, env, db1Pass, db2Pass, db3Pass, verifyRecordset):
import cx_Oracle
retval = None
try:
db1UID='somedb1name'
db2UID='somedb2name'
if env == 'p':
dbServer='prod.somecompanyname.com'
dbSID='SIDPR'
elif env == 's':
dbServer='stage.somecompanyname.com'
dbSID='SIDSTG'
elif env == 'r':
dbServer='stage.somecompanyname.com'
dbSID='SIDDEV'
db3UID = 'somedb3name'
db3SID = 'db3HUB'
logger.info('Running against Oracle server:' + dbServer + ' SID:' + dbSID)
connString = (db1UID + '/' + db1Pass + '#(DESCRIPTION=(ADDRESS_LIST=(ADDRESS='
'(PROTOCOL=TCP)(HOST=' + dbServer + ')(PORT=1234)))(CONNECT_DATA=(SID=' + dbSID + ')))')
conndb1 = cx_Oracle.connect(connString)
curdb1 = conndb1.cursor()
curdb0 = conndb1.cursor()
connString = (db2UID + '/' + db2Pass + '#(DESCRIPTION=(ADDRESS_LIST=(ADDRESS='
'(PROTOCOL=TCP)(HOST=' + dbServer + ')(PORT=1234)))(CONNECT_DATA=(SID=' + dbSID + ')))')
conndb2 = cx_Oracle.connect(connString)
curdb2 = conndb2.cursor()
connString = (db3UID + '/' + db3Pass + '#(DESCRIPTION=(ADDRESS_LIST=(ADDRESS='
'(PROTOCOL=TCP)(HOST=prod.atsomecompany.com)(PORT=1234)))(CONNECT_DATA=(SID=' + db3SID + ')))')
conndb3 = cx_Oracle.connect(connString)
curdb3 = conndb3.cursor()
except Exception as e:
logger.error('Error connecting to databases: ' + str(e.args))
return verifyRecordset, 2

The issue with your Python script is on this line:
logger.error('Error connecting to databases: ' + str(e.args))
Perhaps the simplest way to improve it is to replace it with the following:
logger.error('Error connecting to databases: ' + str(e))
I wrote the following short Python script that attempts to connect to an Oracle XE database:
import cx_Oracle
connect_string = "..."
try:
conn = cx_Oracle.connect(connect_string)
print "Got connection"
except Exception as e:
print str(e.args)
print str(e)
I knew this script would raise an exception because the database and the listener it was attempting to connect to were both down. When I ran, it I got the following output:
(<cx_Oracle._Error object at 0x02354308>,)
ORA-12541: TNS:no listener
The first line here doesn't tell me anything helpful, but the second line contains a more useful message.
Hopefully after making this change to your script you will see a more useful error message, which should help you track down what the real problem is here.