I know questions with the same title have been asked but I cannot figure out the problem based on the solutions given. I am using argparser to connect to database and fetch data. Here
import os
import logging
import argparse
import mysql.connector
def main():
parser = create_parser()
args = parser.parse_args()
query_result = get_name_and_id(args.user, args.password, args.host, args.database)
def create_parser():
parser = argparse.ArgumentParser(description="Require database credentials")
parser.add_argument("--host", required="True", metavar="[host]", dest='host', help="Database host")
parser.add_argument("--database-name", required="True", metavar="[database]", dest='database', help="Name of the database to connect to.")
parser.add_argument("--user-name", metavar="[user]", dest='user', required="True")
parser.add_argument("--password", required="True", metavar="[password]", dest='password')
return parser
def get_name_and_id(user,password,host,database):
con = mysql.connector.connect(user, password, host, database)
cursor = con.cursor()
query = ("SELECT id, name FROM some_table")
cursor.execute(query)
name_id_list = dict()
for id, name in cursor:
name_id_list[id] = name
return name_id_list
if __name__ == "__main__":
main()
to run this program, I use the command
python test_database.py --host somehost --database somedb --user-name myuser-name --password my password
ANd this is the stacktrace :
File "test_arg_parsing.py", line 59, in
main() File "test_arg_parsing.py", line 14, in main
query_result = get_name_and_id(args.user, args.password, args.host, args.database) File "test_arg_parsing.py", line 32, in
get_enqueuer_data
con = mysql.connector.connect(user, password, host, database) File "/Library/Python/2.7/site-packages/mysql/connector/init.py",
line 179, in connect
return MySQLConnection(*args, **kwargs) File "/Library/Python/2.7/site-packages/mysql/connector/connection.py",
line 57, in init
super(MySQLConnection, self).init(*args, **kwargs) TypeError: init() takes exactly 1 argument (5 given)
The thing I do not understand here is whether it is complaining about the MySQLConnection init function? Or am I not running the right command? I have successfully connected to this same database using mysql connector. But I want to pass the credentials as arguments now. What is going wrong here?
These need to be keyword arguments. Try:
con = mysql.connector.connect(user=user, password=password, host=host, database=database)
First of all, the get_name_and_id function should accept the connection arguments:
def get_name_and_id(user, password, host, database):
Then, according to the connect() function definition, you need to specify keyword arguments:
def get_name_and_id(user, password, host, database):
con = mysql.connector.connect(user=user, password=password, host=host, database=database)
Or, you can just pass around the arbitrary keyword arguments:
def get_name_and_id(**kwargs):
con = mysql.connector.connect(**kwargs)
# ...
Usage:
def main():
parser = create_parser()
args = parser.parse_args()
query_result = get_name_and_id(user=args.user,
password=args.password,
host=args.host,
database=args.database)
Related
I'm getting following error
Traceback (most recent call last):
File "/databricks/spark/python/pyspark/serializers.py", line 473, in dumps
return cloudpickle.dumps(obj, pickle_protocol)
File "/databricks/spark/python/pyspark/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/databricks/spark/python/pyspark/cloudpickle/cloudpickle_fast.py", line 563, in dump
return Pickler.dump(self, obj)
TypeError: cannot pickle 'psycopg2.extensions.cursor' object
PicklingError: Could not serialize object: TypeError: cannot pickle 'psycopg2.extensions.cursor' object
while running the below script
def get_connection():
conn_props = brConnect.value
print(conn_props)
#extract value from broadcast variables
database = conn_props.get("database")
user = conn_props.get("user")
pwd = conn_props.get("password")
host = conn_props.get("host")
db_conn = psycopg2.connect(
host = host,
user = user,
password = pwd,
database = database,
port = 5432
)
return db_conn
def process_partition_up(partition, db_cur):
updated_rows = 0
try:
for row in partition:
process_row(row, myq, db_cur)
except Exception as e:
print("Not connected")
return updated_rows
def update_final(df, db_cur):
df.rdd.coalesce(2).foreachPartition(lambda x: process_partition_up(x, db_cur))
def etl_process():
for id in ['003']:
conn = get_connection()
for t in ['email_table']:
query = f'''(select * from public.{t} where id= '{id}') as tab'''
df_updated = load_data(query)
if df_updated.count() > 0:
q1 = insert_ops(df_updated, t) #assume this function returns a insert query
query_props = q1
sc = spark.sparkContext
brConnectQ = sc.broadcast(query_props)
db_conn = get_connection()
db_cur = db_conn.cursor()
update_final(df_updated, db_cur)
conn.commit()
conn.close()
Explanation:
Here etl_process() internally calling get_connection() which returns a psycopg2 connection object. After that it's calling a update_final() which takes dataframe and psycopg2 cursor object as an arguments.
Now update_final() is calling process_partition_up() on each partition(df.rdd.coalesce(2).foreachPartition) which takes dataframe and psycopg2 cursor object as an arguments.
Here after passing psycopg2 cursor object to the process_partition_up(), I'm not getting cursor object rather I'm getting above error.
Can anyone help me out to resolve this error?
Thank you.
I think that you don't understand what's happening here.
You are creating a database connection in your driver(etl_process), and then trying to ship that live connection from the driver, across your network to executor to do the work.(your lambda in foreachPartitions is executed on the executor.)
That is what spark is telling you "cannot pickle 'psycopg2.extensions.cursor'". (It can't serialize your live connection to the database to ship it to an executor.)
You need to call conn = get_connection() from inside process_partition_up this will initialize the connection to the database from inside the executor.(And any other book keeping you need to do.)
FYI: The worst part that I want to call out is that this code will work on your local machine. This is because it's both the executor and the driver.
I want to create a query timeout in sqlalchemy. I have an oracle database.
I have tried following code:
import sqlalchemy
engine = sqlalchemy.create_engine('oracle://db', connect_args={'querytimeout': 10})
I got following error:
TypeError: 'querytimeout' is an invalid keyword argument for this function
I would like a solution looking like:
connection.execute('query').set_timeout(10)
Maybe it is possible to set timeout in sql query? I found how to do it in pl/sql, but i need just sql.
How could i set a query timeout?
The only way how you can set connection timeout for the Oracle engine from the Sqlalchemy is create and configure the sqlnet.ora
Linux
Create file sqlnet.ora in folder
/opt/oracle/instantclient_19_9/network/admin
Windows
For windows please create such folder as \network\admin
C:\oracle\instantclient_19_9\network\admin
Example sqlnet.ora file
SQLNET.INBOUND.CONNECT_TIMEOUT = 120
SQLNET.SEND_TIMEOUT = 120
SQLNET.RECV_TIMEOUT = 120
More parameters you can find here https://docs.oracle.com/cd/E11882_01/network.112/e10835/sqlnet.htm
The way to do it in Oracle is via resource manager. Have a look here
timeout decorator
Get your session handle as you normally would. (Notice that the session has not actually connected yet.) Then, test the session in a function that is decorated with wrapt_timeout_decorator.timeout.
#!/usr/bin/env python3
from time import time
from cx_Oracle import makedsn
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.sql import text
from wrapt_timeout_decorator import timeout
class ConnectionTimedOut(Exception):
pass
class Blog:
def __init__(self):
self.port = None
def connect(self, connection_timeout):
#timeout(connection_timeout, timeout_exception=ConnectionTimedOut)
def test_session(session):
session.execute(text('select dummy from dual'))
session = sessionmaker(bind=self.engine())()
test_session(session)
return session
def engine(self):
return create_engine(
self.connection_string(),
max_identifier_length=128
)
def connection_string(self):
driver = 'oracle'
username = 'USR'
password = 'solarwinds123'
return '%s://%s:%s#%s' % (
driver,
username,
password,
self.dsn()
)
def dsn(self):
host = 'hn.com'
dbname = 'ORCL'
print('port: %s expected: %s' % (
self.port,
'success' if self.port == 1530 else 'timeout'
))
return makedsn(host, self.port, dbname)
def run(self):
self.port = 1530
session = self.connect(connection_timeout=4)
for r in session.execute(text('select status from v$instance')):
print(r.status)
self.port = 1520
session = self.connect(connection_timeout=4)
for r in session.execute(text('select status from v$instance')):
print(r.status)
if __name__ == '__main__':
Blog().run()
In this example, the network is firewalled with port 1530 open. Port 1520 is blocked and leads to a TCP connection timeout. Output:
port: 1530 expected: success
OPEN
port: 1520 expected: timeout
Traceback (most recent call last):
File "./blog.py", line 68, in <module>
Blog().run()
File "./blog.py", line 62, in run
session = self.connect(connection_timeout=4)
File "./blog.py", line 27, in connect
test_session(session)
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrapt_timeout_decorator.py", line 123, in wrapper
return wrapped_with_timeout(wrap_helper)
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrapt_timeout_decorator.py", line 131, in wrapped_with_timeout
return wrapped_with_timeout_process(wrap_helper)
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrapt_timeout_decorator.py", line 145, in wrapped_with_timeout_process
return timeout_wrapper()
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrap_function_multiprocess.py", line 43, in __call__
self.cancel()
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrap_function_multiprocess.py", line 51, in cancel
raise_exception(self.wrap_helper.timeout_exception, self.wrap_helper.exception_message)
File "/home/exagriddba/lib/python3.8/site-packages/wrapt_timeout_decorator/wrap_helper.py", line 178, in raise_exception
raise exception(exception_message)
__main__.ConnectionTimedOut: Function test_session timed out after 4.0 seconds
Caution
Do not decorate the function that calls sessionmaker, or you will get:
_pickle.PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session
SCAN
This implementation is a "connection timeout" without regard to underlying cause. The client could time out before trying all available SCAN listeners.
I'm trying to run the below code to connect to a database using python 3.6, I'm not sure I'm using Python correctly, I read in the config file and create and instance of the Dao class and pass the config details back to the parent class of Dao which is Db.
When I then go an try and open and close a connection on the Dao object it says that the dbhost isn't set.
Any help greatly appreciated.
Error
Traceback (most recent call last):
File "parse.py", line 12, in <module>
sources = daoObj.getSourceUrls()
File "E:\classes\Dao.py", line 14, in getSourceUrls
conn = super().open()
File "E:\classes\Db.py", line 22, in open
conn = pymysql.connect(dbhost, dbuser, dbpass, dbname);
NameError: name 'dbhost' is not defined
parse.py
import configparser
from classes.Dao import Dao
# Load configuration settings
config = configparser.ConfigParser()
config.read("./config.ini")
# Create instance of database class
daoObj = Dao(config)
# Test database connection
daoObj.test()
Dao.py
from classes.Db import Db
class Dao(Db):
"""Contains all SQL queries used for database interaction"""
node = None
def __init__(self, config):
"""Default constructor"""
super().__init__(config)
def getSourceUrls(self):
conn = super().open()
super().close(conn)
Db.py
import pymysql
class Db:
"""Database connection class, handles all opening and closing of MySQL database connections."""
dbuser = None
dbpass = None
dbhost = None
dbname = None
def __init__(self, config):
"""Default constructor"""
# Assign the database login credentials
dbuser = config["DB"]["USER"]
dbpass = config["DB"]["PASS"]
dbhost = config["DB"]["HOST"]
dbname = config["DB"]["DATABASE"]
def open(self):
"""Open database connection."""
conn = None
try:
conn = pymysql.connect(dbhost, dbuser, dbpass, dbname);
except pymysql.err.InternalError as e:
print("Error connecting to database.")
return conn
def close(self, conn):
"""Close passed in database connection"""
conn.close()
config.ini
[DB]
USER=username
PASS=password
HOST=127.0.0.l
DATABASE=database
I have config.ini:
[mysql]
host=localhost
port=3306
user=root
passwd=abcdefgh
db=testdb
unix_socket=/opt/lampp/var/mysql/mysql.sock
I have this class:
#!/usr/bin/python
import MySQLdb,ConfigParser
config = ConfigParser.ConfigParser()
config.read("config.ini")
class MySQL( object ):
def __init__( self ):
self.host = config.get("mysql","host")
self.port = config.get("mysql","port")
self.user = config.get("mysql","user")
self.passwd = config.get("mysql","passwd")
self.db = config.get("mysql","db")
self.unix_socket = config.get("mysql","unix_socket")
self.conn = MySQLdb.Connect(self.host,
self.port,
self.user,
self.passwd,
self.db,
self.unix_socket)
self.cursor = self.conn.cursor ( MySQLdb.cursors.DictCursor )
def __del__( self ):
self.cursor.close()
self.conn.close()
and this:
#!/usr/bin/env python
from mysql import MySQL
class Incident( MySQL ):
def getIncidents( self ):
self.cursor.execute("""*VALID QUERY*""")
return self.cursor.fetchall()
and finally this:
import subprocess, os, alarm
from Queue import Queue
from incident_model import Incident
fileQueue = Queue()
def enumerateFilesPath():
global fileQueue
incident = Incident()
incidents = incident.getIncidents()
for i in incidents:
fileQueue.put("MD5")
def main():
global fileQueue
enumerateFilesPath()
Output:
Traceback (most recent call last):
File "./mwmonitor.py", line 202, in
main() File "./mwmonitor.py", line 184, in main
enumerateFilesPath() File "./mwmonitor.py", line 86, in
enumerateFilesPath
incident = Incident() File "/usr/share/mwanalysis/core/mysql.py",
line 23, in init
self.unix_socket) File "/usr/lib/pymodules/python2.6/MySQLdb/init.py",
line 81, in Connect
return Connection(*args, **kwargs) File
"/usr/lib/pymodules/python2.6/MySQLdb/connections.py",
line 170, in init
super(Connection, self).init(*args, **kwargs2)
TypeError: an integer is required
Exception AttributeError: "'Incident'
object has no attribute 'cursor'" in
0xa03d46c>> ignored
If someone can help detect and correct the error would greatly appreciate.
Thanks in advance.
Your __del__ method is causing confusion. Specifically, it refers to self.cursor and self.conn which may never get created if, for example, MySQLdb.Connect raises an exception (which is what seems to happen).
I suggest you modify your class as follows:
class MySQL( object ):
def __init__( self ):
self.conn = None
self.cursor = None
self.host = config.get("mysql","host")
self.port = config.get("mysql","port")
self.user = config.get("mysql","user")
self.passwd = config.get("mysql","passwd")
self.db = config.get("mysql","db")
self.unix_socket = config.get("mysql","unix_socket")
self.conn = MySQLdb.Connect(self.host,
self.port,
self.user,
self.passwd,
self.db,
self.unix_socket)
self.cursor = self.conn.cursor ( MySQLdb.cursors.DictCursor )
def __del__( self ):
if self.cursor is not None:
self.cursor.close()
if self.conn is not None:
self.conn.close()
This won't solve the problem, but should give better diagnostic.
Now to the actual problem that you're experiencing. I strongly suspect that you're supplying the arguments to Connect in the wrong order, or the types aren't quite right, or something along those lines. To quote the docstring for Connection.__init__:
Create a connection to the database. It is strongly recommended
that you only use keyword parameters. Consult the MySQL C API
documentation for more information.
host
string, host to connect
user
string, user to connect as
passwd
string, password to use
db
string, database to use
port
integer, TCP/IP port to connect to
unix_socket
string, location of unix_socket to use
...
"It is strongly that you only use keyword parameters." I recommend that you do just that when you call MySQLdb.Connect. Also, make sure that port is an int and not a string.
I suspect it's expecting port to be an integer rather than a string. Try:
self.port = int(config.get("mysql","port"))
I am not sure if this is a connectivity error. Have you checked the type of the incident_model ?
TypeError: an integer is required
Exception AttributeError: "'Incident'object has no attribute 'cursor'" in
I'm using sqlobject in Python. I connect to the database with
conn = connectionForURI(connStr)
conn.makeConnection()
This succeeds, and I can do queries on the connection:
g_conn = conn.getConnection()
cur = g_conn.cursor()
cur.execute(query)
res = cur.fetchall()
This works as intended. However, I also defined some classes, e.g:
class User(SQLObject):
class sqlmeta:
table = "gui_user"
username = StringCol(length=16, alternateID=True)
password = StringCol(length=16)
balance = FloatCol(default=0)
When I try to do a query using the class:
User.selectBy(username="foo")
I get an exception:
...
File "c:\python25\lib\site-packages\SQLObject-0.12.4-py2.5.egg\sqlobject\main.py", line 1371, in selectBy
conn = connection or cls._connection
File "c:\python25\lib\site-packages\SQLObject-0.12.4-py2.5.egg\sqlobject\dbconnection.py", line 837, in __get__
return self.getConnection()
File "c:\python25\lib\site-packages\SQLObject-0.12.4-py2.5.egg\sqlobject\dbconnection.py", line 850, in getConnection
"No connection has been defined for this thread "
AttributeError: No connection has been defined for this thread or process
How do I define a connection for a thread? I just realized I can pass in a connection keyword which I can give conn to to make it work, but how do I get it to work if I weren't to do that?
Do:
from sqlobject import sqlhub, connectionForURI
sqlhub.processConnection = connectionForURI(connStr)