I am trying to connect to HANA in order to pull some metadata in a pandas dataframe. There are lots of mixed approaches and I couldn't find anything concrete.
All I have is:
username
password
serverip
servername
and table names.
The admin has provided all the read access to the required account for the specific tables.
What is the quickest way to get this done? I do not have the option of installing anything on SAPs site.
I have tried the below snippets but I get the error 'target machine actively refused it' and to debug at SAPs end is a lost cause. Thank you in advance.
import pyhdb
connection = pyhdb.connect(
host="123.com",
port=123,
user="user",
password="pswrd"
)
cursor = connection.cursor()
cursor.execute("SELECT * FROM Tablename")
cursor.fetchone()
connection.close()
and
from hdbcli import dbapi
conn = dbapi.connect(
address="123.com",
port=123,
user="user",
password="pswrd"
)
cursor = conn.cursor()
Given your server address and port examples, I'm not sure you got the right idea for how to connect to a HANA database.
Since you want to use pandas it is probably a good idea to have a look at the SAP HANA Machine Learning library for Python.
Check the tutorial blog post for this:
https://blogs.sap.com/2019/11/05/hands-on-tutorial-machine-learning-push-down-to-sap-hana-with-python/
To do any of this, there is no need to install or debug anything on the HANA system.
To connect to hana DB :
from hdbcli import dbapi
conn = dbapi.connect(
address="serverhost",
port=39015,
user="UserName",
password="Password",
databasename='DBNAME'
)
Make sure you enter the correct port number
To get the sql results in pandas dataframe:
query = 'select * from table'
df = pd.read_sql_query(query, conn)
df.head()
This question already has an answer here:
Postgres error after updating TimescaleDB on Ubuntu: file not found
(1 answer)
Closed 2 years ago.
I was trying to hook up the sqlalchemy with my underlying postgresql, which uses the timescaledb extension. All queries work fine when I try them from the psql terminal client. But when I try to use python & sqlalchemy to do it, it keeps throwing me an error.
Here's the very basic code snippet that I try to test it with:
engine = create_engine('postgres://usr:pwd#localhost:5432/postgres', echo=True)
engine.execute('select 1;')
And it always shows the following the error message:
File "/home/usr/.local/share/virtualenvs/redbird-lRSbFM0t/lib/python3.6/site-packages/psycopg2/extras.py", line 917, in get_oids
""" % typarray)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not access file "timescaledb-0.9.0": No such file or directory
The connection to the db is fine, otherwise it won't know the db is using timescaledb.
Any one has any insights?
UPDATE: I try to use psycopg2 directly. It basically gives the same error. DB is connected successfully, but timescaledb-0.9.0 cannot be accessed.
Here's the code snippted
conn_string = "host='localhost' dbname='db' user='usr' password='pwd'"
print("Connecting to database\n ->%s " % (conn_string))
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
print("Connected!\n")
cursor.execute("\dx")
records = cursor.fetchall()
Here's the exact same error message:
Connecting to database
Connected!
Traceback (most recent call last):
File "/home/usr/Workspace/somepath/web/model/model.py", line 21, in <module>
cursor.execute("\dx")
psycopg2.OperationalError: could not access file "timescaledb-0.9.0": No such file or directory
This seems very similar to my issue.
I guess you also updated to a new version of Timescale? The thing is: After each update of the timescale package you don't just have to make sure the library is preloaded (as the warning says on the command line) - you also have to upgrade each database that uses the extension manually via psql.
See my own answer to my issue for the steps.
--
This snipplet works for me:
#! /usr/bin/python
# -*- coding: utf-8 -*-
import psycopg2
# Connect to an existing database.
conn = psycopg2.connect(dbname='my-db-name',
user='postgres',
password='super-secret',
host='localhost',
port='5432')
# Open a cursor to perform database operations.
cur = conn.cursor()
# Query the database and obtain data as Python objects.
cur.execute('SELECT * FROM my-table-name LIMIT 100 ;')
# Print results.
results = cur.fetchall()
for result in results:
print(result)
# Close communication with the database.
cur.close()
conn.close()
Using the cursor to executue psql commands also doesn't work for me. I don't think it is supposed to. But what works reliably is doing SQL:
# Check if the database has the timescaledb extension installed.
# This is about the same as xecuting '\dx' on psql.
cur.execute('SELECT * from pg_extension;')
sqlplus sys/Oracle_1#pdborcl as sysdba;
i'm using this command to connect to Oracle 12c from Command Prompt.
How can i connect to the db using cx_Oracle. I'm new to Oracle DB.
I think this is the equivalent of the sqlplus command line that you posted:
import cx_Oracle
connect_string = "sys/Oracle_1#pdborcl"
con = cx_Oracle.connect(connect_string,mode=cx_Oracle.SYSDBA)
I tried it with a non-container database and not with a pdb so I can't verify that it would work with a pdb. You may not want to connect as sys as sysdba unless you know that you need that level of security.
Bobby
You can find the documentation here cx_Oracle docs
To query the database, use the below algorithm
import cx_Oracle
dsn = cx_Oracle.makedsn(host, port, sid)
connection = cx_Oracle.connect(dsn,mode = cx_Oracle.SYSDBA)
query = "SELECT * FROM MYTABLE"
cursor = connection.cursor()
cursor.execute(query)
resultSet=cursor.fetchall()
connection.close()
The above code works to fetch data from MYTABLE connecting to the above dsn.
Better to go through cx_Oracle docs.
This is my first question here. So, I am sorry if it is repeated or the formatting is off. I searched through other questions and the error is common but appears on multiple situations.
I have a very simple python code where I want to execute a procedure in MSSQL from pyodbc.
import pyodbc
conn = pyodbc.connect(r'DSN=myDSN')
cursor = conn.cursor()
query = r'{call myproc}'
cursor.execute(query)
I am using call instead of exec after reading that ODBC uses call for executing procedures in MSSQL.
The error I am getting is the following:
Traceback (most recent call last):
File "myscript.py", line 26, in <module>
cursor.execute(query)
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]The current transaction has aborted, and any pending changes have been rolled back. Cause: A transaction in a rollback-only state was not explicitly rolled back before a DDL, DML or SELECT statement. (111233) (SQLExecDirectW)')
Thanks for the help
In case someone is having the same issue. I was able to find out what the problems was. When you open a connection with DSN, the autocommit is set to False. For some reason, this should be True for the code to work (this depends largely on what I was doing on MSSQL).
import pyodbc
conn = pyodbc.connect(r'DSN=myDSN', autocommit=True)
cursor = conn.cursor()
query = r'{call myproc}'
cursor.execute(query)
This runs well!
Here are two examples on how you can execute a stored proc in MS SQL Server through pyodbc:
Passing a NULL, and the VARCHAR 'tallen' as positional parameter variables:
cursor.execute('EXEC usp_get_user_data ?, ?', None, 'flipperpa')
Passing two VARCHAR values as named parameter variables:
cursor.execute('EXEC usp_get_user_data #user_full_name = ?, #user_username = ?', 'flip', 'flipperpa')
Then to loop through the returned data:
rows = cursor.fetchall()
for row in rows:
# Do stuff
print(row.user_id)
Good luck!
I had issue with executing the Stored procedure using the SQL server 2008. The first step I did is to go to control panel > Administrative tools > Data Sources (ODBC) > add the driver
The only change I made in my python code is use "{call procdev_2017}". I got an error when I tried using exec instead of call
import pyodbc
conn = pyodbc.connect(driver='{SQL Server Native Client 11.0}', server='XXXXXXX', database='XXX',
trusted_connection='yes', autocommit=True)
bepcur = conn.cursor()
ipcur.execute("{call procdev_2017}")
How can I access Oracle from Python? I have downloaded a cx_Oracle msi installer, but Python can't import the library.
I get the following error:
import cx_Oracle
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
import cx_Oracle
ImportError: DLL load failed: The specified module could not be found.
I will be grateful for any help.
Here's what worked for me. My Python and Oracle versions are slightly different from yours, but the same approach should apply. Just make sure the cx_Oracle binary installer version matches your Oracle client and Python versions.
My versions:
Python 2.7
Oracle Instant Client 11G R2
cx_Oracle 5.0.4 (Unicode, Python 2.7, Oracle 11G)
Windows XP SP3
Steps:
Download the Oracle Instant Client package. I used instantclient-basic-win32-11.2.0.1.0.zip. Unzip it to C:\your\path\to\instantclient_11_2
Download and run the cx_Oracle binary installer. I used cx_Oracle-5.0.4-11g-unicode.win32-py2.7.msi. I installed it for all users and pointed it to the Python 2.7 location it found in the registry.
Set the ORACLE_HOME and PATH environment variables via a batch script or whatever mechanism makes sense in your app context, so that they point to the Oracle Instant Client directory. See oracle_python.bat source below. I'm sure there must be a more elegant solution for this, but I wanted to limit my system-wide changes as much as possible. Make sure you put the targeted Oracle Instant Client directory at the beginning of the PATH (or at least ahead of any other Oracle client directories). Right now, I'm only doing command-line stuff so I just run oracle_python.bat in the shell before running any programs that require cx_Oracle.
Run regedit and check to see if there's an NLS_LANG key set at \HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE. If so, rename the key (I changed it to NLS_LANG_OLD) or unset it. This key should only be used as the default NLS_LANG value for Oracle 7 client, so it's safe to remove it unless you happen to be using Oracle 7 client somewhere else. As always, be sure to backup your registry before making changes.
Now, you should be able to import cx_Oracle in your Python program. See the oracle_test.py source below. Note that I had to set the connection and SQL strings to Unicode for my version of cx_Oracle.
Source: oracle_python.bat
#echo off
set ORACLE_HOME=C:\your\path\to\instantclient_11_2
set PATH=%ORACLE_HOME%;%PATH%
Source: oracle_test.py
import cx_Oracle
conn_str = u'user/password#host:port/service'
conn = cx_Oracle.connect(conn_str)
c = conn.cursor()
c.execute(u'select your_col_1, your_col_2 from your_table')
for row in c:
print row[0], "-", row[1]
conn.close()
Possible Issues:
"ORA-12705: Cannot access NLS data files or invalid environment specified" - I ran into this before I made the NLS_LANG registry change.
"TypeError: argument 1 must be unicode, not str" - if you need to set the connection string to Unicode.
"TypeError: expecting None or a string" - if you need to set the SQL string to Unicode.
"ImportError: DLL load failed: The specified procedure could not be found." - may indicate that cx_Oracle can't find the appropriate Oracle client DLL.
You can use any of the following way based on Service Name or SID whatever you have.
With SID:
import cx_Oracle
dsn_tns = cx_Oracle.makedsn('server', 'port', 'sid')
conn = cx_Oracle.connect(user='username', password='password', dsn=dsn_tns)
c = conn.cursor()
c.execute('select count(*) from TABLE_NAME')
for row in c:
print(row)
conn.close()
OR
With Service Name:
import cx_Oracle
dsn_tns = cx_Oracle.makedsn('server', 'port', service_name='service_name')
conn = cx_Oracle.connect(user='username', password='password', dsn=dsn_tns)
c = conn.cursor()
c.execute('select count(*) from TABLE_NAME')
for row in c:
print(row)
conn.close()
Here is how my code looks like. It also shows an example of how to use query parameters using a dictionary. It works on using Python 3.6:
import cx_Oracle
CONN_INFO = {
'host': 'xxx.xx.xxx.x',
'port': 12345,
'user': 'SOME_SCHEMA',
'psw': 'SECRETE',
'service': 'service.server.com'
}
CONN_STR = '{user}/{psw}#{host}:{port}/{service}'.format(**CONN_INFO)
QUERY = '''
SELECT
*
FROM
USER
WHERE
NAME = :name
'''
class DB:
def __init__(self):
self.conn = cx_Oracle.connect(CONN_STR)
def query(self, query, params=None):
cursor = self.conn.cursor()
result = cursor.execute(query, params).fetchall()
cursor.close()
return result
db = DB()
result = db.query(QUERY, {'name': 'happy'})
import cx_Oracle
dsn_tns = cx_Oracle.makedsn('host', 'port', service_name='give service name')
conn = cx_Oracle.connect(user='username', password='password', dsn=dsn_tns)
c = conn.cursor()
c.execute('select count(*) from schema.table_name')
for row in c:
print row
conn.close()
Note :
In (dsn_tns) if needed, place an 'r' before any parameter in order to address any special character such as '\'.
In (conn) if needed, place an 'r' before any parameter in order to address any special character such as '\'. For example, if your user name contains '\', you'll need to place 'r' before the user name: user=r'User Name' or password=r'password'
use triple quotes if you want to spread your query across multiple lines.
Note if you are using pandas you can access it in following way:
import pandas as pd
import cx_Oracle
conn= cx_Oracle.connect('username/pwd#host:port/service_name')
try:
query = '''
SELECT * from dual
'''
df = pd.read_sql(con = conn, sql = query)
finally:
conn.close()
df.head()
In addition to the Oracle instant client, you may also need to install the Oracle ODAC components and put the path to them into your system path. cx_Oracle seems to need access to the oci.dll file that is installed with them.
Also check that you get the correct version (32bit or 64bit) of them that matches your: python, cx_Oracle, and instant client versions.
In addition to cx_Oracle, you need to have the Oracle client library installed and the paths set correctly in order for cx_Oracle to find it - try opening the cx_Oracle DLL in "Dependency Walker" (http://www.dependencywalker.com/) to see what the missing DLL is.
Ensure these two and it should work:-
Python, Oracle instantclient and cx_Oracle are 32 bit.
Set the environment variables.
Fixes this issue on windows like a charm.
If you are using virtualenv, it is not as trivial to get the driver using the installer. What you can do then: install it as described by Devon. Then copy over cx_Oracle.pyd and the cx_Oracle-XXX.egg-info folder from Python\Lib\site-packages
into the Lib\site-packages from your virtual env. Of course, also here, architecture and version are important.
import cx_Oracle
from sshtunnel import SSHTunnelForwarder
# remote server variables
remote_ip_address = "<PUBLIC_IP_ADDRESS_OF_DB_SERVER>"
remote_os_username = "<OS_USERNAME>"
ssh_private_key = "<PATH_TO_PRIVATE_KEY>"
# Oracle database variables
database_username = "<DATABASE_USER>"
database_password = "<DATABASE_USER_PASSWORD>"
database_server_sid = "<ORACLE_SID>"
def server_connection():
server = SSHTunnelForwarder(
remote_ip_address,
ssh_username=remote_os_username,
ssh_password=ssh_private_key,
remote_bind_address=('localhost', 1521) # default Oracle DB port
)
return server
def database_connection():
data_source_name = cx_Oracle.makedsn("localhost",
server.local_bind_port,
service_name=database_server_sid)
connection = cx_Oracle.connect(database_username,
database_password,
data_source_name,
mode=cx_Oracle.SYSDBA) # If logging in with SYSDBA privs,
# leave out if not.
return connection
def database_execute():
connection = database_connection()
cursor = connection.cursor()
for row in cursor.execute("SELECT * FROM HELLO_WORLD_TABLE"):
print(row)
if __name__ == '__main__':
server = server_connection()
server.start() # start remote server connection
connection = database_connection() # create Oracle database connection
database_execute() # execute query
connection.close() # close Oracle database connection
server.stop() # close remote server connection
If you're accessing the Oracle database via a bastion tunnel, you just need to modify this piece of code:
def server_connection():
server = SSHTunnelForwarder(
remote_ip_address, # public IP of bastion server
ssh_username=remote_os_username,
ssh_password=ssh_private_key,
remote_bind_address=('localhost', 1521),
local_bind_address=('0.0.0.0', 3333) # Suppose local bind is '3333'
)
return server