Being a newbie to python, trying to write a python code to connect to the oracle database without using any Instant client. i'm using jaydebeapi and jpype as suggested in some other threads in this forum. After lot of hurdles, i now got stuck at this error. here is the code.
import jaydebeapi
import jpype
try:
con = jaydebeapi.connect('oracle.jdbc.driver.OracleDriver', ['windb19.ams.com', 'AA3112D1OS', 'advantage', 'C:\Tools\ojdbc8.jar'])
cur = con.cursor()
cur.execute('select * from r_sc_user_info')
except Exception as e:
print e
and the error i'm receiving is as below
C:\Python27\python.exe C:/Project/Robot_Framework/SampleProject/CustomLibraries/DBLibrary.py
java.lang.Exception: Class oracle.jdbc.driver.OracleDriver not found
Process finished with exit code 0
As I couldn't modify anything in the Environment variables, as per the policy, I had to modify the code as below to make it work. I had to keep the ojdbc8.jar in the same path as that of this python file and add following lines of code.
jar=os.getcwd()+'\ojdbc8.jar'
args = '-Djava.class.path=%s' % jar
jvm_path = jpype.getDefaultJVMPath()
jpype.startJVM(jvm_path, args)
try:
con = jaydebeapi.connect("oracle.jdbc.driver.OracleDriver", "jdbc:oracle:thin:#HOSTNAME",["USERID", "PASSWORD"], jar)
Related
When I'm trying to read data from sqlalchemy df=pd.read_sql_table(table, con, schema) getting runtime error :
runtime/cgo: could not obtain pthread_keys
tried 0x115 0x116 0x117 0x118 0x119 0x11a 0x11b 0x11c 0x11d 0x11e 0x11f 0x120 0x121 0x122 0x123 0x124 0x125 0x126 0x127 0x128 0x129 0x12a 0x12b 0x12c 0x12d 0x12e 0x12f 0x130 0x131 0x132 0x133 0x134 0x135 0x136 0x137 0x138 0x139 0x13a 0x13b 0x13c 0x13d 0x13e 0x13f 0x140 0x141 0x142 0x143 0x144 0x145 0x146 0x147 0x148 0x149 0x14a 0x14b 0x14c 0x14d 0x14e 0x14f 0x150 0x151 0x152 0x153 0x154 0x155 0x156 0x157 0x158 0x159 0x15a 0x15b 0x15c 0x15d 0x15e 0x15f 0x160 0x161 0x162 0x163 0x164 0x165 0x166 0x167 0x168 0x169 0x16a 0x16b 0x16c 0x16d 0x16e 0x16f 0x170 0x171 0x172 0x173 0x174 0x175 0x176 0x177 0x178 0x179 0x17a 0x17b 0x17c 0x17d 0x17e 0x17f 0x180 0x181 0x182 0x183 0x184 0x185 0x186 0x187 0x188 0x189 0x18a 0x18b 0x18c 0x18d 0x18e 0x18f 0x190 0x191 0x192 0x193 0x194
Below is the code:
class TeradataWriter:
def __init__(self):
print("in init")
def read_data_from_teradata(self):
try:
print('Create main')
import pdb;pdb.set_trace()
eng = self.create_connection_engine()
df = pd.read_sql_table("table_name", eng, schema="schema")
print(df)
except Exception as ex:
print('Exception: %s', ex.with_traceback())
def create_connection_engine(self):
try:
return create_engine('teradatasql://' + constants.TERADATA_HOST + '/?user='+ constants.TERADATA_USER_NAME + '&password=' + constants.TERADATA_PWD, echo=False)
except Exception as ex:
LOGGER.error('Exception: %s', ex)
raise Exception(message_constants.ERROR_WHILE_CREATING_CONNECTION_WITH_TERADATA)
if __name__ == "__main__":
p = TeradataWriter()
p.write_dataframe_to_teradata()
Edit: This is fixed. I was finally able to get their support and engineering team to reproduce the issue. They now build the driver with a newer version of go. Upgrade to >= 17.0.3, and you shouldn't see anymore segfaults.
I think I finally figured out why this happens. According to this Go issue, it happens if "If the host process spawns threads prior to loading the shared library, the offset will have changed."
In my case, I was importing matplotlib.pyplot in IPython before calling code that loads the shared library. This starts an event loop and causes the conditions that lead to the segfault.
I changed my code to import matplotlib.pyplot after configuring the teradata driver, and it went away.
According to the Go issue, they just need to recompile the library with a newer version of Go, which I've asked them to do. We'll see what they say.
I have run in to same issue -
So to fix the problem, I moved connect statement to main and it kind of fixed. Its worth trying in your case.
I have run into a peculiar issue while using the teradatasql package (installed from pypi). I use the following code (let's call it pytera.py) to query a database:
from dotenv import load_dotenv
import pandas as pd
import teradatasql
# Load the database credentials from .env file
_ = load_dotenv()
db_host = os.getenv('db_host')
db_username = os.getenv('db_username')
db_password = os.getenv('db_password')
def run_query(query):
"""Run query string on teradata and return DataFrame."""
if query.strip()[-1] != ';':
query += ';'
with teradatasql.connect(host=db_host, user=db_username,
password=db_password) as connect:
df = pd.read_sql(query, connect)
return df
When I import this function in the IPython/Python interpreter or in Jupyter Notebook, I can run queries just fine like so:
import pytera as pt
pt.run_query('select top 5 * from table_name;')
However, if I save the above code in a .py file and try to run it, I get an error message most of the time (not all the time). The error message is below.
E teradatasql.OperationalError: [Version 16.20.0.49] [Session 0] [Teradata SQL Driver] Hostname lookup failed for None
E at gosqldriver/teradatasql.(*teradataConnection).makeDriverError TeradataConnection.go:1046
E at gosqldriver/teradatasql.(*Lookup).getAddresses CopDiscovery.go:65
E at gosqldriver/teradatasql.discoverCops CopDiscovery.go:137
E at gosqldriver/teradatasql.newTeradataConnection TeradataConnection.go:133
E at gosqldriver/teradatasql.(*teradataDriver).Open TeradataDriver.go:32
E at database/sql.dsnConnector.Connect sql.go:600
E at database/sql.(*DB).conn sql.go:1103
E at database/sql.(*DB).Conn sql.go:1619
E at main.goCreateConnection goside.go:229
E at main._cgoexpwrap_e6e101e164fa_goCreateConnection _cgo_gotypes.go:214
E at runtime.call64 asm_amd64.s:574
E at runtime.cgocallbackg1 cgocall.go:316
E at runtime.cgocallbackg cgocall.go:194
E at runtime.cgocallback_gofunc asm_amd64.s:826
E at runtime.goexit asm_amd64.s:2361
E Caused by lookup None on <ip address redacted>: server misbehaving
I am using Python 3.7.3 and teradatasql 16.20.0.49 on Ubuntu (WSL) 18.04.
Perhaps not coincidentally, I run into a similar issue when trying a similar workflow on Windows (using the teradata package and the Teradata Python drivers installed). Works when I connect inside the interpreter or in Jupyter, but not in a script. In the Windows case, the error is:
E teradata.api.DatabaseError: (10380, '[08001] [Teradata][ODBC] (10380) Unable to establish connection with data source. Missing settings: {[DBCName]}')
I have a feeling that there's something basic that I'm missing, but I can't find a solution to this anywhere.
Thanks ravioli for the fresh eyes. Turns out the issue was loading in the environment variables using dotenv. My module is in a Python package (separate folder), and my script and .env files are in the working directory.
dotenv successfully reads in the environment variables (.env in my working directory) when I run the code in my original post, line by line, in the interpreter or in Jupyter. However, when I run the same code in a script, it does not find in the .env file in my working directory. That will be a separate question I'll have to find the answer to.
import teradatasql
import pandas as pd
def run_query(query, db_host, db_username, db_password):
"""Run query string on teradata and return DataFrame."""
if query.strip()[-1] != ';':
query += ';'
with teradatasql.connect(host=db_host, user=db_username,
password=db_password) as connect:
df = pd.read_sql(query, connect)
return df
The code below runs fine in a script now:
import pytera as pt
from dotenv import load_dotenv()
_ = load_dotenv()
db_host = os.getenv('db_host')
db_username = os.getenv('db_username')
db_password = os.getenv('db_password')
data = pt.run_query('select top 5 * from table_name;', db_host, db_username, db_password)
It looks like your client can't find the Teradata server, which is why you see that DBCName missing error. This should be the "system name" of your Teradata server (i.e. TDServProdA).
A couple things to try:
If you are trying to connect directly with a hostname, try disabling COP discovery in your connection with this flag: cop = false. More info
Try updating your hosts file on your local system. From the documentation:
Modifying the hosts File
If your site does not use DNS, you must define the IP address and the
Teradata Database name to use in the system hosts file on the
computer.
Locate the hosts file on the computer. This file is typically located in the following folder: %SystemRoot%\system32\drivers\etc
Open the file with a text editor, such as Notepad.
Add the following entry to the file: xxx.xx.xxx.xxx sssCOP1 where xxx.xx.xxx.xxx is the IP address and where sss is the Teradata
Database name.
Save the hosts file.
Link 1
Link 2
I try to connect to a JDBC:hive client via knox and if I run the following code:
import jaydebeapi;
import jpype;
conn = jaydebeapi.connect("org.apache.hive.jdbc.HiveDriver","<hive_jdbc_url>",["<username>", "<password>"],"/path/to/hive-jdbc-uber-<version>.jar")
curs = conn.cursor();
curs.execute("SELECT * FROM Database WHERE day = 20181114 Limit 10");
curs.fetchall();
curs.close();
Python.exe stops working, and if I run the code in pycharm, I get the following error:
Process finished with exit code -1073741819 (0xC0000005)
I think it could be a problem with the environment variable, but I don't know what to chance to fix this issue.
I am very new to python and I just can't seem to find an answer to this error. When I run the code below I get the error
AttributeError: module 'odbc' has no attribute 'connect'
However, the error only shows in eclipse. There's no problem if I run it via command line. I am running python 3.5. What am I doing wrong?
try:
import pyodbc
except ImportError:
import odbc as pyodbc
# Specifying the ODBC driver, server name, database, etc. directly
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=PXLstr,17;DATABASE=Dept_MR;UID=guest;PWD=password')
The suggestion to remove the try...except block did not work for me. Now the actual import is throwing the error as below:
Traceback (most recent call last):
File "C:\Users\a\workspace\TestPyProject\src\helloworld.py", line 2, in <module>
import pyodbc
File "C:\Users\a\AppData\Local\Continuum\Anaconda3\Lib\site-packages\sqlalchemy\dialects\mssql\pyodbc.py", line 105, in <module>
from .base import MSExecutionContext, MSDialect, VARBINARY
I do have pyodbc installed and the import and connect works fine with the command line on windows.
thank you
The problem here is that the pyodbc module is not importing in your try / except block. I would highly recommend not putting import statements in try blocks. First, you would want to make sure you have pyodbc installed (pip install pyodbc), preferably in a virtualenv, then you can do something like this:
import pyodbc
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=PXLstr,17;DATABASE=Dept_MR;UID=guest;PWD=password')
cursor = cnxn.cursor()
cursor.execute('SELECT 1')
for row in cursor.fetchall():
print(row)
If you're running on Windows (it appears so, given the DRIVER= parameter), take a look at virtualenvwrapper-win for managing Windows Python virtual environments: https://pypi.python.org/pypi/virtualenvwrapper-win
Good luck!
Flipper's answer helped to establish that the problem was with referencing an incorrect library in External Libraries list in eclipse. After fixing it, the issue was resolved.
What is the name of your python file? If you inadvertently name it as 'pyodbc.py', you got that error. Because it tries to import itself instead of the intended pyodbc module.
here is the solution!
simply install and use 'pypyodbc' instead of 'pyodbc'!
I have my tested example as below. change your data for SERVER_NAME and DATA_NAME and DRIVER. also put your own records.good luck!
import sys
import pypyodbc as odbc
records = [
['x', 'Movie', '2020-01-09', 2020],
['y', 'TV Show', None, 2019]
]
DRIVER = 'ODBC Driver 11 for SQL Server'
SERVER_NAME = '(LocalDB)\MSSQLLocalDB'
DATABASE_NAME = 'D:\ASPNET\SHOJA.IR\SHOJA.IR\APP_DATA\DATABASE3.MDF'
conn_string = f"""
Driver={{{DRIVER}}};
Server={SERVER_NAME};
Database={DATABASE_NAME};
Trust_Connection=yes;
"""
try:
conn = odbc.connect(conn_string)
except Exception as e:
print(e)
print('task is terminated')
sys.exit()
else:
cursor = conn.cursor()
insert_statement = """
INSERT INTO NetflixMovies
VALUES (?, ?, ?, ?)
"""
try:
for record in records:
print(record)
cursor.execute(insert_statement, record)
except Exception as e:
cursor.rollback()
print(e.value)
print('transaction rolled back')
else:
print('records inserted successfully')
cursor.commit()
cursor.close()
finally:
if conn.connected == 1:
print('connection closed')
conn.close()
This might be an obvious error but I'm trying to create a database within python from a script I've already created.
conn = sqlite3.connect('testDB')
c = conn.cursor()
c.execute('.read create.sql')
This gives an error "sqlite3.OperationalError: near ".": syntax error"
If I do the same thing at the sqlite3 cmd line it works fine
[me#myPC ~]$ sqlite3 testDB
SQLite version 3.3.6
Enter ".help" for instructions
sqlite> .read create.sql
sqlite>
It seems that any commands that start with a . give me problems.
just pass the content of the file to the .execute method:
conn = sqlite3.connect('testDB')
c = conn.cursor()
SQL = open('create.sql').read()
c.executescript(SQL)
I would suppose that commands starting with . are for the CLI client itself, not for the backend.
So you have no chance to do so and would have to do file reading and executing the queries by yourself, i.e. in Python.