Develop Reference

How to set a query timeout in sqlalchemy using Oracle database?

I want to create a query timeout in sqlalchemy. I have an oracle database.
I have tried following code:
import sqlalchemy
engine = sqlalchemy.create_engine('oracle://db', connect_args={'querytimeout': 10})
I got following error:
TypeError: 'querytimeout' is an invalid keyword argument for this function
I would like a solution looking like:
connection.execute('query').set_timeout(10)
Maybe it is possible to set timeout in sql query? I found how to do it in pl/sql, but i need just sql.
How could i set a query timeout?

The only way how you can set connection timeout for the Oracle engine from the Sqlalchemy is create and configure the sqlnet.ora
Linux
Create file sqlnet.ora in folder
/opt/oracle/instantclient_19_9/network/admin
Windows
For windows please create such folder as \network\admin
C:\oracle\instantclient_19_9\network\admin
Example sqlnet.ora file
SQLNET.INBOUND.CONNECT_TIMEOUT = 120
SQLNET.SEND_TIMEOUT = 120
SQLNET.RECV_TIMEOUT = 120
More parameters you can find here https://docs.oracle.com/cd/E11882_01/network.112/e10835/sqlnet.htm

The way to do it in Oracle is via resource manager. Have a look here

timeout decorator
Get your session handle as you normally would. (Notice that the session has not actually connected yet.) Then, test the session in a function that is decorated with wrapt_timeout_decorator.timeout.
#!/usr/bin/env python3
from time import time
from cx_Oracle import makedsn
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.sql import text
from wrapt_timeout_decorator import timeout
class ConnectionTimedOut(Exception):
pass
class Blog:
def __init__(self):
self.port = None
def connect(self, connection_timeout):
#timeout(connection_timeout, timeout_exception=ConnectionTimedOut)
def test_session(session):
session.execute(text('select dummy from dual'))
session = sessionmaker(bind=self.engine())()
test_session(session)
return session
def engine(self):
return create_engine(
self.connection_string(),
max_identifier_length=128
)
def connection_string(self):
driver = 'oracle'
username = 'USR'
password = 'solarwinds123'
return '%s://%s:%s#%s' % (
driver,
username,
password,
self.dsn()
)
def dsn(self):
host = 'hn.com'
dbname = 'ORCL'
print('port: %s expected: %s' % (
self.port,
'success' if self.port == 1530 else 'timeout'
))
return makedsn(host, self.port, dbname)
def run(self):
self.port = 1530
session = self.connect(connection_timeout=4)
for r in session.execute(text('select status from v$instance')):
print(r.status)
self.port = 1520
session = self.connect(connection_timeout=4)
for r in session.execute(text('select status from v$instance')):
print(r.status)
if __name__ == '__main__':
Blog().run()
In this example, the network is firewalled with port 1530 open. Port 1520 is blocked and leads to a TCP connection timeout. Output:
port: 1530 expected: success
OPEN
port: 1520 expected: timeout
Traceback (most recent call last):
File "./blog.py", line 68, in <module>
Blog().run()
File "./blog.py", line 62, in run
session = self.connect(connection_timeout=4)
File "./blog.py", line 27, in connect
test_session(session)
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrapt_timeout_decorator.py", line 123, in wrapper
return wrapped_with_timeout(wrap_helper)
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrapt_timeout_decorator.py", line 131, in wrapped_with_timeout
return wrapped_with_timeout_process(wrap_helper)
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrapt_timeout_decorator.py", line 145, in wrapped_with_timeout_process
return timeout_wrapper()
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrap_function_multiprocess.py", line 43, in __call__
self.cancel()
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrap_function_multiprocess.py", line 51, in cancel
raise_exception(self.wrap_helper.timeout_exception, self.wrap_helper.exception_message)
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrap_helper.py", line 178, in raise_exception
raise exception(exception_message)
__main__.ConnectionTimedOut: Function test_session timed out after 4.0 seconds
Caution
Do not decorate the function that calls sessionmaker, or you will get:
_pickle.PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session
SCAN
This implementation is a "connection timeout" without regard to underlying cause. The client could time out before trying all available SCAN listeners.

Unable to connect to hive using python using impyla/dbapi.py

I am trying to connect to hive[with default derby db] using python:
from impala.dbapi import connect
conn = connect( host='localhost', port=10000)
cursor = conn.cursor()
cursor.execute('SELECT * FROM employee')
print cursor.description # prints the result set's schema
results = cursor.fetchall()
but I am getting error:
Traceback (most recent call last):
File "hivetest_b.py", line 2, in <module>
conn = connect( host='localhost', port=10000)
File "/home/ubuntu/.local/lib/python2.7/site-packages/impala/dbapi.py", line 147, in connect
auth_mechanism=auth_mechanism)
File "/home/ubuntu/.local/lib/python2.7/site-packages/impala/hiveserver2.py", line 758, in connect
transport.open()
File "/home/ubuntu/.local/lib/python2.7/site-packages/thrift/transport/TTransport.py", line 149, in open
return self.__trans.open()
File "/home/ubuntu/.local/lib/python2.7/site-packages/thrift/transport/TSocket.py", line 101, in open
message=message)
thrift.transport.TTransport.TTransportException: Could not connect to localhost:10000
entry in my /etc/hosts is:
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
I am using default hive-site.xml and defult derby database for running my hive. When I run hive through shell it shows me that table:
hive> show databases;
OK
default
test
test_db
Time taken: 0.937 seconds, Fetched: 3 row(s)
hive> show tables;
OK
employee
Time taken: 0.054 seconds, Fetched: 1 row(s)
hive> describe employee;
OK
empname string
age int
gender string
income float
department string
dept string
# Partition Information
# col_name data_type comment
dept string
Time taken: 0.451 seconds, Fetched: 11 row(s)
I am not sure what exactly am I missing here. Any quick references/pointers would be appreciated.
Regards,
Bhupesh

You can check and validate the port with:
hive> set hive.server2.thrift.port;
And try 0.0.0.0 and 127.0.0.1 instead of localhost as your host for the connection.

SSH Tunnel to MariaDB

I've looked at several very similar examples, but I'm doing something wrong...probably because I'm mixing up local or remote binding addresses or well, not sure. I've yet to find a document that can describe what each is supposed to be to a newb like me.
I have a raspberry pi in a robot which has MariaDB installed. I can connect to the server from my PC with SQL Workbench.
I have a second Pi that needs a python script that can send data to the first pi...
IP Addresses, names and passwords have been changed to protect the innocent, but the whole thing is a closed network anyways.
import mysql.connector
import sshtunnel
_host = Robot's IP Address
_ssh_port = 22
_username = Robot user login
_password = Robot Password
_remote_bind_address = Robot's IP Address
_remote_mysql_port = 3308
_local_bind_address = Second Pi's IP Address
_local_mysql_port = 3308
_db_user = Database User Name
_db_password = Database Password
_db_name = "joycap"
with sshtunnel.SSHTunnelForwarder(
(_host, _ssh_port),
ssh_username=_username,
ssh_password=_password,
remote_bind_address=(_remote_bind_address, _remote_mysql_port),
local_bind_address=(_local_bind_address,_local_mysql_port)
) as tunnel:
connection = mysql.connector.connect(
user=_db_user,
passwd=_db_password,
host=_local_bind_address,
database=_db_name,
port=_local_mysql_port)
Here's the error I'm current getting...
pi#raspberrypi:~ $ python dbtest1.py
2018-04-05 06:11:42,262| ERROR | Secsh channel 0 open FAILED: Connection refused: Connect failed
2018-04-05 06:11:42,277| ERROR | Could not establish connection from ('Second Pi's IP Address', 3308) to remote side of the tunnel
Traceback (most recent call last):
File "dbtest1.py", line 29, in <module>
port=_local_mysql_port)
File "/usr/local/lib/python2.7/dist-packages/mysql/connector/__init__.py", line 184, in connect
return MySQLConnection(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/mysql/connector/connection.py", line 100, in __init__
self.connect(**kwargs)
File "/usr/local/lib/python2.7/dist-packages/mysql/connector/abstracts.py", line 733, in connect
self._open_connection()
File "/usr/local/lib/python2.7/dist-packages/mysql/connector/connection.py", line 241, in _open_connection
self._do_handshake()
File "/usr/local/lib/python2.7/dist-packages/mysql/connector/connection.py", line 108, in _do_handshake
packet = self._socket.recv()
File "/usr/local/lib/python2.7/dist-packages/mysql/connector/network.py", line 248, in recv_plain
raise errors.InterfaceError(errno=2013)
mysql.connector.errors.InterfaceError: 2013: Lost connection to MySQL server during query
Would appreciate any advice! Would think someone would have made a video on doing this by now :)
Regards,
Matt

Finally worked it out through trial and error.
_host = "192.168.8.1" #Server where database is hosted
_ssh_port = 22 #ssh port
_username = "pi" #user name to host server
_password = "raspberry" #password to host server
_remote_bind_address = "127.0.0.1" #local connection IP to db on host(should probably not change)
_remote_mysql_port = 3306
_local_bind_address = "0.0.0.0" #local bind address(should probably not change)
_local_mysql_port = 3306
_db_user = "databaseuser"
_db_password = "databasepass"
_db_name = "databasename"

Python connecting to mysql

I installed Python 2.7 to try to connect to MySQL online. Basically, MySQL and phpMyAdmin is on a server and I can access it via localhost:8888/phpmyadmin via putty on my windows desktop. I cant seem to connect to it even with the putty on. Any idea? I face the same issue with Python 3.3 using CyMySQL.
import MySQLdb
db = MySQLdb.connect(host="127.0.0.1", # your host, usually 127.0.0.1
user="megamonster", # your username
passwd="", # your password
db="extractor") # name of the data base
# you must create a Cursor object. It will let
# you execute all the query you need
cur = db.cursor()
# Use all the SQL you like
cur.execute("SELECT * FROM abc")
# print all the first cell of all the rows
for row in cur.fetchall() :
print row[0]
Error:
Traceback (most recent call last):
File "C:\Users\Jonathan\Desktop\testSQL.py", line 6, in <module>
db="extractor") # name of the data base
File "C:\Python27\lib\site-packages\MySQLdb\__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "C:\Python27\lib\site-packages\MySQLdb\connections.py", line 187, in __init__
super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (2003, "Can't connect to MySQL server on '127.0.0.1' (10061)")
Update
i added port(3306) and got this.
OperationalError: (2013, "Lost connection to MySQL server at 'waiting for initial communication packet', system error: 0")
Currently looking at
MySQL error: 2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0"
Hmm cant work still...

I used sqlplus it worked
sqlplus User_name/password#Host_ip:Port/Service_Name#$SQLFILENAME
Just specify the SQLFILENAME if you want to utilize a file. Otherwise, you can ignore this parameter and can directly run sql statements

It could be a number of things, but as far as MySQL is concerned, permissions are set independently for localhost and for 127.0.0.1. Make sure you can connect with the exact host and credentials. Possibly related
For example, check this when connected with your PUTTY connection.
mysql> use mysql;
Database changed
mysql> SELECT host,user,select_priv FROM user;
+-------------------------+------+-------------+
| host | user | select_priv |
+-------------------------+------+-------------+
| localhost | root | Y |
| 127.0.0.1 | root | Y |
+-------------------------+------+-------------+
Also check who you are connected as (on PUTTY) and use that same info in the script:
mysql> SELECT USER(),CURRENT_USER();
+----------------+----------------+
| USER() | CURRENT_USER() |
+----------------+----------------+
| root#localhost | root#localhost |
+----------------+----------------+

Thrift : TypeError: getaddrinfo() argument 1 must be string or None

Hi I am trying to write a simple thrift server in python (named PythonServer.py) with a single method that returns a string for learning purposes. The server code is below. I am having the following errors in the Thrift's python libraries when I run the server. Has anyone experienced this problem and suggest a workaround?
The execution output:
Starting server
Traceback (most recent call last):
File "/home/dae/workspace/BasicTestEnvironmentV1.0/src/PythonServer.py", line 38, in <module>
server.serve()
File "usr/lib/python2.6/site-packages/thrift/server/TServer.py", line 101, in serve
File "usr/lib/python2.6/site-packages/thrift/transport/TSocket.py", line 136, in listen
File "usr/lib/python2.6/site-packages/thrift/transport/TSocket.py", line 31, in _resolveAddr
TypeError: getaddrinfo() argument 1 must be string or None
PythonServer.java
port = 9090
import MyService as myserv
#from ttypes import *
# Thrift files
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer
# Server implementation
class MyHandler:
# return server message
def sendMessage(self, text):
print text
return 'In the garage!!!'
# set handler to our implementation
handler = MyHandler()
processor = myserv.Processor(handler)
transport = TSocket.TServerSocket(port)
tfactory = TTransport.TBufferedTransportFactory()
pfactory = TBinaryProtocol.TBinaryProtocolFactory()
# set server
server = TServer.TThreadedServer(processor, transport, tfactory, pfactory)
print 'Starting server'
server.serve() ##### LINE 38 GOES HERE ##########

Your problem is the line:
transport = TSocket.TServerSocket(port)
When calling TSocket.TServerSocket which a single argument, the value is treated as a host identifier, hence the error with getaddrinfo().
To fix that, change the line to:
transport = TSocket.TServerSocket(port=port)

I had this problem while running PythonServer.py ...
I changed this line
transport = TSocket.TServerSocket(9090)
to
transport = TSocket.TServerSocket('9090')
and my server started running.

I had a similar problem. My fix is
TSocket.TServerSocket('your server ip',port)