Develop Reference

Python alternative to R RJDBC connection to SQL

I have a R code which connect to the Vertica database using RJDBC driver. The code is following:
library(RJDBC)
#for vertica save
user = "myuser"
pwd = "mypwd"
driver<- JDBC(driverClass="com.vertica.jdbc.Driver", classPath=Pathto thedriver")
connection<- dbConnect(driver, databasewithport, user, pwd)
sql_code = paste("SELECT .....")
mydata= dbGetQuery(connection, sql_code )
I am searching for a solution that helps do the same thing but using Python. I found the following link, but do not understand which example to use and what else to do. As I understood here no need to connect to the RJDBC driver. Could you help to find the solution which gives the same output as R version.
The code below works well, however, data is retrieved as one value, to get another I need to change ....cur.fetchone()[ANYNUMBER]). How can I get a data frame of the SQL code?
import vertica_python
conn_info = {'host': '127.0.0.1',
'port': 5433,
'user': 'some_user',
'password': 'some_password',
'database': 'vdb',
'connection_load_balance': True}
# Server enables load balancing
with vertica_python.connect(**conn_info) as conn:
cur = conn.cursor()
cur.execute("SELECT NODE_NAME FROM V_MONITOR.CURRENT_SESSION")
print("Client connects to primary node:", cur.fetchone()[0])
cur.execute("SELECT SET_LOAD_BALANCE_POLICY('ROUNDROBIN')")

First of all you will need to install the vertica-python package:
pip install vertica-python
Next, you need to create a connection and perform the query. When retrieving the query results, you can (1) load them all or (2) process them one by one.
import vertica_python
conn_info = {'host': '127.0.0.1',
'port': 5433,
'user': 'myuser',
'password': 'mypass',
'database': 'vdb',
'connection_load_balance': True}
with vertica_python.connect(**conn_info) as connection:
cur = conn.cursor()
cur.execute("SELECT NODE_NAME FROM V_MONITOR.CURRENT_SESSION")
# (1) If you want to load all the results in-memory
data = cur.fetchall()
print(data)
# (2) If you want to process one by one
for row in cur.iterate():
print(row)

How to get key(column name)-value pair from a Select query using default connection in Django

I have 4 applications in one Django Project. I am using the default Postgres database connection which I have included it in my setting.py file.
The object django.db.connection represents the default database connection. To use the database connection, I call connection.cursor() to get a cursor object. Then, I call cursor.execute(sql, [params]) to execute the raw Postgres queries and cursor.fetchone() or cursor.fetchall() to return the resulting rows.
Now, in one application I want to use the (connection.cursor(cursor_factory = psycopg2.extras.RealDictCursor)) to get the records in a (key(column name), value) pair, which is provided by psycopg2.connect but the default connection that I have is not compatible with cursor_factory = psycopg2.extras.RealDictCursor.
How do I get a (key(column name), value) pair from the database by using the default connection?
In setting.py
`DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'OPTIONS' : {
'options': '-c search_path=django,public'
},
'NAME': 'postgres',
'USER': 'abc',
'PASSWORD': 'password!',
'HOST': '',
'PORT': '',
}
}
`
In python file .py
from django.db import connection
cur = connection.cursor(cursor_factory = psycopg2.extras.RealDictCursor)
cur.execute("SELECT * FROM sometable")
data= cur.fetchall()
ERROR: cursor() got an unexpected keyword argument 'cursor_factory'

You have to get an underlying postgres connection, ensure it is established and then you can specify custom cursor_factory.
from django.db import connections
import psycopg2
def scan_tables(app):
conn = connections['default']
conn.ensure_connection()
with conn.connection.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as cursor:
cursor.execute("SELECT table_name, column_name "
"FROM information_schema.columns AS c "
"WHERE table_name LIKE '{}_%'".format(app))
print(cursor.fetchall())
scan_tables('django')
This is an adapted answer from https://stackoverflow.com/a/48844401/803174 #kert just putting different cursor type.

Python - Returning the output as a dictionary - MySQL

I have coded a class to handle mysql connecting using the external modules but when I get data I want it to be returned like:
{ 'id': '1', 'username': 'Test'}
it's currently returned:
(1, u'Test')
The code which I use is:
def executeQuery(self, query):
if(self.mysqlSuccess):
cursor = self.cnx.cursor()
cursor.execute(query)
for row in cursor:
print row
executeQuery('SELECT * FROM `users`')

I found an answer to my problem when your defining the cursor you do it like this
cursor = db.cursor(dictionary=True)
and the mysql package I was using is mysql.connector

You can use a DictCursor when initializing your connection to MySQL.
import MySQLdb.cursors
cnx = MySQLdb.connect(user='joe', passwd='password', db='dbname',
cursorclass=MySQLdb.cursors.DictCursor)
If you are using a diff mysql package then just check the docs for DictCursor.

Oracle connection string with at sign # in pasword

I have a code that connect to oracle using connection string:
conn = cx_Oracle.connect('username/password#server:port/services')
But the problem is my password contain # character so it may become
conn = cx_Oracle.connect('username/p#ssword#server:port/services')
it return
DatabaseError: ORA-12154: TNS:could not resolve the connect identifier
specified
I use Django with Oracle with this settings
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.oracle',
'NAME': 'Services',
'USER': 'user',
'PASSWORD': 'p#ssword',
'HOST': 'ip',
'PORT': 'port',
}
}
I cant change password :( Does anyone know this problem?

I haven't tried cx_Oracle, but you might be able to connect by specifying the individual parameters -
conn = cx_Oracle.connect(user='username', password='p#ssword', dsn='server:port/services')
OR
dsn_tns = cx_Oracle.makedsn('server', 'port', 'services')
conn = cx_Oracle.connect(user='username', password='p#ssword', dsn=dsn_tns)

You can use any of the following way based on Service Name or SID whatever you have.
With SID:
dsn_tns = cx_Oracle.makedsn('server', 'port', 'sid')
conn = cx_Oracle.connect(user='username', password='p#ssword', dsn=dsn_tns)
OR
With Service Name:
dsn_tns = cx_Oracle.makedsn('server', 'port', service_name='service_name')
conn = cx_Oracle.connect(user='username', password='p#ssword', dsn=dsn_tns)

Does this work?
conn = cx_Oracle.connect('username/"p#ssword"#server:port/services')

FYI: This was a long-standing bug in Django. The first stable version containing the fix is v2.1

All the answers mentioned here did not work for SQLAlchemy.
Parsing the characters to create a valid URL worked:
from sqlalchemy.engine import create_engine
import urllib.parse
url_password = urllib.parse.quote_plus(password)
connection_string = f"oracle+cx_oracle://{username}:{url_password}#{server}:{port}/?service_name={service}"
connection = create_engine(connection_string)
Should probably do the same with the other components of the connection string.

Python: decryption failed or bad record mac when calling from Thread

I'm getting a "decryption failed or bad record mac" error in this code-fragment:
conn = psycopg2.connect(...)
cursor = conn.cursor()
cursor.execute("SELECT id, ip FROM schema.table;")
rows = cursor.fetchall()
cursor.close()
conn.commit()
conn.close()
This is called in the run() method of a Thread, several times in a while(True) loop.
I'm just opening a connection to my PostgreSQL database using the psycopg2 driver.
Any idea of how safe is opening db connections into Threads in Python?
I don't know what is raising this error.

Ok, looks like I've fixed the problem. I was creating too many connections and seems I was running out of memory or something.
I gathered all the queries and do cursor.execute(...) once with a huge query, instead performing hundreds of small queries/connections.
conn = psycopg2.connect(...)
cursor = conn.cursor()
cursor.execute("SELECT id, ip FROM schema.table;")
rows = cursor.fetchall()
cursor.close()
conn.commit()
conn.close()
conn = None

The cause of this issue could be, there were too many processes(multi) were trying to access PostGres, it was not able to handle that. I was using Django & PostGres in BeanStalk.
Adding 'OPTIONS': {'sslmode': 'disable'} in the database config helped.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': ...
'USER': ....
'PASSWORD': ...
'HOST': ....
'PORT': '5432',
'OPTIONS': {
'sslmode': 'disable',
}
}
}