Develop Reference

Using Python to connect to Impala database (thriftpy error)

What I'm trying to do is very basic: connect to an Impala db using Python:
from impala.dbapi import connect
conn = connect(host='impala', port=21050, auth_mechanism='PLAIN')
I'm using Impyla package to do so. I got this error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/thriftpy/transport/socket.py", line 96, in open
self.sock.connect(addr)
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/alaaeddine/PycharmProjects/test/data_test.py", line 3, in <module>
conn = connect(host='impala', port=21050, auth_mechanism='PLAIN')
File "/usr/local/lib/python3.6/dist-packages/impala/dbapi.py", line 147, in connect
auth_mechanism=auth_mechanism)
File "/usr/local/lib/python3.6/dist-packages/impala/hiveserver2.py", line 758, in connect
transport.open()
File "/usr/local/lib/python3.6/dist-packages/thrift_sasl/__init__.py", line 61, in open
self._trans.open()
File "/usr/local/lib/python3.6/dist-packages/thriftpy/transport/socket.py", line 104, in open
message="Could not connect to %s" % str(addr))
thriftpy.transport.TTransportException: TTransportException(type=1, message="Could not connect to ('impala', 21050)")
Tried also the Ibis package but failed with the same thriftpy related error.
In Windows using Dbeaver, I could connect to the database using the official Cloudera JDBC connector. My questions are:
Should pass my JDBC connector as parameter in my connect code? I have made some search I could not find something pointing at this direction.
Should I try something else than Ibis and Impyla packages? I had experienced a lot of version related issues and dependencies when using them. If yes, what would you recommend as alternatives?
Thanks!

Solved:
I used pyhive package instead of Ibis/Impyla. Here's an example:
#import hive from pyhive
from pyhive import hive
#establish the connection to the db
conn = hive.Connection(host='host_IP_addr', port='conn_port', auth='auth_type', database='my_db')
#prepare the cursor for the queries
cursor = conn.cursor()
#execute a query
cursor.execute("SHOW TABLES")
#navigate and display the results
for table in cursor.fetchall():
print(table)

Your impala domain name must not be resolving. Are you able to do nslookup impala in command prompt? If you're using Docker, you need to have the docker service name in docker-compose as "impala" or have "extra_hosts" option. Or you can always add it to /etc/hosts (Windows/Drivers/etc/hosts) as impala 127.0.0.1
Also try 'NOSASL' instead of PLAIN sometimes that works better with security turned off.

This is the simple method, connecting impala through impala shell using python.
import commands
import re
query1 = "select * from table_name limit 10"
impalad = str('hostname')
port = str('21000')
database = str('database_name')
result_string = 'impala-shell -i "'+ impalad+':'+port +'" -k -B --delimited -q "'+query1+'"'
status, output = commands.getstatusoutput(result_string)
print output
if status == 0:
print output
else:
print "Error encountered while executing HiveQL queries."

Why does pyodbc generate garbled error messages on Fedora 27 ppc64?

I'm using pyodbc (3.0.10-9) with Python 3 (3.6.6-1) on Fedora 27 ppc64 (on an IBM Power 720). I have a very simple script that connects to a MS SQL Server (2K8R2 on W2K8R2) database table as follows:
import time
import pyodbc
# import pyximport; pyximport.install()
def main():
cnxn = pyodbc.connect('DSN=something;UID=uu;PWD=pppp;', ansi=True)
cursor = cnxn.cursor()
cursor.execute('SELECT * FROM dbo.tb_Table;')
for row in cursor:
print(row)
if __name__ == '__main__':
time.sleep(3)
main()
input()
If the UID and PWD are correct, everything runs fine.
So I stopped the SQL Server service on the Windows server, in order to see the error message on the Power7 machine.
However, I am getting this garbled error on a Mac OSX (10.11.6) terminal and on the Fedora 27 gnome-terminal:
Traceback (most recent call last):
File "podbc.py", line 16, in <module>
main()
File "podbc.py", line 6, in main
cnxn = pyodbc.connect('DSN=something;UID=uu;PWD=pppp;', ansi=True)
pyodbc.OperationalError: ('08S01', '[08S01] 嬀䘀爀攀攀吀䐀匀崀嬀匀儀䰀\u2000匀攀爀瘀攀爀崀唀渀愀戀氀攀\u2000琀漀\u2000挀漀渀渀攀挀琀㨀\u2000䄀搀愀瀀琀椀瘀攀\u2000匀攀爀瘀攀爀\u2000椀猀\u2000甀渀愀瘀愀椀氀愀戀氀攀\u2000漀爀\u2000搀漀攀猀\u2000渀漀琀\u2000攀砀椀猀琀 (20009) (SQLDriverConnect)')
Why? How can I "translate" this into a readable (english) form?
Any help or pointers appreciated.

pyodbc 3.0.10 is very old. Upgrade to the most recent version (currently 4.0.24). It includes the fix for this issue.
Update:
You have a classic case of mojibake. ppc64 defaults to big-endian byte ordering but Microsoft SQL Server (and hence FreeTDS) use little-endian byte ordering so the UTF_16-encoded message returned from the server is being mangled:
>>> foo = '嬀䘀爀攀攀吀䐀匀崀嬀匀儀䰀\u2000匀攀爀瘀攀爀崀唀渀愀戀氀攀\u2000琀漀\u2000挀漀渀渀攀挀琀㨀\u2000䄀搀愀瀀琀椀瘀攀\u2000匀攀爀瘀攀爀\u2000椀猀\u2000甀渀愀瘀愀椀氀愀戀氀攀\u2000漀爀\u2000搀漀攀猀\u2000渀漀琀\u2000攀砀椀猀琀'
>>> foo.encode('utf-16be').decode('utf-16le')
'[FreeTDS][SQL Server]Unable to connect: Adaptive Server is unavailable or does not exist'
Normally one would use pyodbc's Connection.setdecoding method (described here) to work around the mismatch in endian-ness, but your error is happening before the connection is established. You may want to raise an issue on GitHub regarding your particular circumstances.

Sqlalchemy setup for postgresql with timescaledb extension [duplicate]

This question already has an answer here:
Postgres error after updating TimescaleDB on Ubuntu: file not found
(1 answer)
Closed 2 years ago.
I was trying to hook up the sqlalchemy with my underlying postgresql, which uses the timescaledb extension. All queries work fine when I try them from the psql terminal client. But when I try to use python & sqlalchemy to do it, it keeps throwing me an error.
Here's the very basic code snippet that I try to test it with:
engine = create_engine('postgres://usr:pwd#localhost:5432/postgres', echo=True)
engine.execute('select 1;')
And it always shows the following the error message:
File "/home/usr/.local/share/virtualenvs/redbird-lRSbFM0t/lib/python3.6/site-packages/psycopg2/extras.py", line 917, in get_oids
""" % typarray)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not access file "timescaledb-0.9.0": No such file or directory
The connection to the db is fine, otherwise it won't know the db is using timescaledb.
Any one has any insights?
UPDATE: I try to use psycopg2 directly. It basically gives the same error. DB is connected successfully, but timescaledb-0.9.0 cannot be accessed.
Here's the code snippted
conn_string = "host='localhost' dbname='db' user='usr' password='pwd'"
print("Connecting to database\n ->%s " % (conn_string))
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
print("Connected!\n")
cursor.execute("\dx")
records = cursor.fetchall()
Here's the exact same error message:
Connecting to database
Connected!
Traceback (most recent call last):
File "/home/usr/Workspace/somepath/web/model/model.py", line 21, in <module>
cursor.execute("\dx")
psycopg2.OperationalError: could not access file "timescaledb-0.9.0": No such file or directory

This seems very similar to my issue.
I guess you also updated to a new version of Timescale? The thing is: After each update of the timescale package you don't just have to make sure the library is preloaded (as the warning says on the command line) - you also have to upgrade each database that uses the extension manually via psql.
See my own answer to my issue for the steps.
--
This snipplet works for me:
#! /usr/bin/python
# -*- coding: utf-8 -*-
import psycopg2
# Connect to an existing database.
conn = psycopg2.connect(dbname='my-db-name',
user='postgres',
password='super-secret',
host='localhost',
port='5432')
# Open a cursor to perform database operations.
cur = conn.cursor()
# Query the database and obtain data as Python objects.
cur.execute('SELECT * FROM my-table-name LIMIT 100 ;')
# Print results.
results = cur.fetchall()
for result in results:
print(result)
# Close communication with the database.
cur.close()
conn.close()
Using the cursor to executue psql commands also doesn't work for me. I don't think it is supposed to. But what works reliably is doing SQL:
# Check if the database has the timescaledb extension installed.
# This is about the same as xecuting '\dx' on psql.
cur.execute('SELECT * from pg_extension;')

How do I display the hosts inside the Google Chrome sqlite3 "cookie" database using Python

I'm using Python to access the "cookie" chrome sqlite3 db to retrieve the host keys, but getting error below
import sqlite3
conn = sqlite3.connect(r"C:\Users\tikka\AppData\Local\Google\Chrome\User Data\Default\Cookies")
cursor = conn.cursor()
cursor.execute("select host_key from cookies")
results = cursor.fetchall()
print results
conn.close()
Error
Traceback (most recent call last):
File "C:\Python27\cookies.py", line 4, in <module>
cursor.execute("select host_key from cookies")
DatabaseError: malformed database schema (is_transient) - near "where": syntax error
>>>

thanks to link provided by alecxe was able to fix it by upgrading sqlite3 version from 3.6.21 to 3.9.2. I upgraded by downloading new version from this site and placing the dll in C:\Python27\DLLs

Connect to SQL Server using python from Raspberry pi

I am trying to connect to a SQL Server database using python.I have followed,
http://blog.tryolabs.com/2012/06/25/connecting-sql-server-database-python-under-ubuntu/
I have used following Python code to connect with the Microsoft SQL Server Management Studio 2014 with above setting.
import pyodbc
user='sa'
password='PC#1234'
database='climate'
port='1433'
TDS_Version='8.0'
server='192.168.1.146'
con_string= 'UID=%s;PWD=%s;DATABASE=%s;PORT=%s;TDS=%s;SERVER=%s;' %
(user,password, database,port,TDS_Version,server)
cnxn=pyodbc.connect(con_string)
cursor=cnxn.cursor()
cursor.execute("select * from mytable")
row=cursor.fetchone()
print row
I got following error,
Traceback (most recent call last):
File "sql.py", line 15, in <module>
cnxn=pyodbc.connect(con_string)
pyodbc.Error: ('IM002', '[IM002] [unixODBC][Driver Manager]Data source
name not found, and no default driver specified (0) (SQLDriverConnect)')
I also have installed pymssql and tried to connect to SQL Server. For this I have used following python code,
import pymssql
connection=pymssql.connect(user='sa',password='PC#1234',
host='192.168.1.146',database='climate',as_dict=True)
cursor=connection.cursor()
cursor.execute('select * from mytable;')
rows=cursor.fetchall()
I have got following error,
connection=pymssql.connect(user='sa',password='PC#1234',
host='192.168.1.146',database='climate',as_dict=True)
File "/usr/lib/pymodules/python2.7/pymssql.py", line 607, in connect
raise OperationalError, e[0]
pymssql.OperationalError: DB-Lib error message 20009, severity 9:
Unable to connect: Adaptive Server is unavailable or does not exist
Net-Lib error during Operation now in progress Error 115
- Operation now in progress
what is the reason for showing data source name not found and adaptive server is not available?