Develop Reference

Error: 'Authentication plugin 'caching_sha2_password' is not supported when running executable of Python Module using pyinstaller

I'm attempting to run an executable, "main.exe", that was built from three python modules (with "main.py" being the main script) using the pyinstaller module. The command that was used to build an executable from the scripts is,
pyinstaller --onefile main.py
This script invokes functions from the "tictactoe_office_release.py" script which establishes connection to a MySQL 8.0.31 server database for performing CRUD operations. When running the executable from the command line, I receive the following string of errors:
Error: 'Authentication plugin 'caching_sha2_password' is not supported'
Traceback (most recent call last):
File "main.py", line 24, in <module>
File "tictactoe_office_release.py", line 42, in __init__
File "mysql_python_lib.py", line 124, in __init__
File "mysql_python_lib.py", line 96, in read_query
AttributeError: 'NoneType' object has no attribute 'cursor'
[25743] Failed to execute script 'main' due to unhandled exception!
It is important to note, however, that my main.py script executes without errors when run outside of the executable. Now, I have troubleshooted the errors using numerous comments from Authentication Plugin 'caching_sha2_password' is not supportedincluding the following
uninstalling 'mysql-connector' and installing 'mysql-connector-python'
2)Setting the 'auth_plugin' parameter to 'mysql_native_password' in the 'mysql.connector.connect()' function calls
3)Modifying the mysql encryption by running
ALTER USER 'root'#'localhost' IDENTIFIED WITH caching_sha2_password BY 'Panther021698';
but am receiving the same error after I re-build and run the executable.
The relevant code in my "tictactoe_office_release.py" module that depicts the function definitions for enabling communication between the Python interpreter and the MySQL server, and database is provided below:
from distutils.util import execute
import mysql.connector
from mysql.connector import Error
from mysql.connector.locales.eng import client_error
class mysql_python_connection:
''' Provide class definition for creating connection to MySQL server,
initializing database, and executing queries '''
def __init__(self):
self.host_name = "localhost"
self.user_name = "root"
self.passwd = "Panther021698"
def create_server_connection(self):
''' This function establishes a connection between Python
interpreter and the MySQL Community Server that we are attempting
to connect to '''
self.connection = None # Close any existing connections
try:
self.connection = mysql.connector.connect(
host = self.host_name,
user = self.user_name,
passwd = self.passwd
)
print("MySQL connection successful")
except Error as err:
print(f"Error: '{err}'")
def create_database(self, query):
''' This function initializes a new database
on the connected MySQL server'''
cursor = self.connection.cursor()
try:
cursor.execute(query)
print("Database created successfully")
except Error as err:
print(f"Error: '{err}'")
def create_db_connection(self, db_name):
''' This function establishes a connection between Python
the MySQL Community Server and a database that we
are initializing on the server '''
self.connection = None # Close any existing connections
self.db_name = db_name
try:
self.connection = mysql.connector.connect(
host = self.host_name,
user = self.user_name,
passwd = self.passwd,
database = self.db_name
)
print("MySQL Database connection successful")
except Error as err:
print(f"Error: '{err}'")
def execute_query(self, query):
''' This function takes SQL queries stored
in Python as strings and passes them
to the "cursor.execute()" method to
execute them on the server'''
cursor = self.connection.cursor()
try:
cursor.execute(query)
self.connection.commit() # Implements commands detailed in SQL queries
print(query + "Query successful")
except Error as err:
print(f"Error: '{err}'")
def read_query(self,query):
''' This function reads and returns data from
a MySQL database using the specified query '''
cursor = self.connection.cursor()
print("cursor datatype is ")
print(type(cursor))
#result = None
try:
cursor.execute(query)
result = cursor.fetchall()
return result
except Error as err:
print(f"Error: '{err}'")
Additionally, my MySQL environment variables are provided in the image below.

How to pass psycopg2 cursor object to foreachPartition() in pyspark?

I'm getting following error
Traceback (most recent call last):
File "/databricks/spark/python/pyspark/serializers.py", line 473, in dumps
return cloudpickle.dumps(obj, pickle_protocol)
File "/databricks/spark/python/pyspark/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/databricks/spark/python/pyspark/cloudpickle/cloudpickle_fast.py", line 563, in dump
return Pickler.dump(self, obj)
TypeError: cannot pickle 'psycopg2.extensions.cursor' object
PicklingError: Could not serialize object: TypeError: cannot pickle 'psycopg2.extensions.cursor' object
while running the below script
def get_connection():
conn_props = brConnect.value
print(conn_props)
#extract value from broadcast variables
database = conn_props.get("database")
user = conn_props.get("user")
pwd = conn_props.get("password")
host = conn_props.get("host")
db_conn = psycopg2.connect(
host = host,
user = user,
password = pwd,
database = database,
port = 5432
)
return db_conn
def process_partition_up(partition, db_cur):
updated_rows = 0
try:
for row in partition:
process_row(row, myq, db_cur)
except Exception as e:
print("Not connected")
return updated_rows
def update_final(df, db_cur):
df.rdd.coalesce(2).foreachPartition(lambda x: process_partition_up(x, db_cur))
def etl_process():
for id in ['003']:
conn = get_connection()
for t in ['email_table']:
query = f'''(select * from public.{t} where id= '{id}') as tab'''
df_updated = load_data(query)
if df_updated.count() > 0:
q1 = insert_ops(df_updated, t) #assume this function returns a insert query
query_props = q1
sc = spark.sparkContext
brConnectQ = sc.broadcast(query_props)
db_conn = get_connection()
db_cur = db_conn.cursor()
update_final(df_updated, db_cur)
conn.commit()
conn.close()
Explanation:
Here etl_process() internally calling get_connection() which returns a psycopg2 connection object. After that it's calling a update_final() which takes dataframe and psycopg2 cursor object as an arguments.
Now update_final() is calling process_partition_up() on each partition(df.rdd.coalesce(2).foreachPartition) which takes dataframe and psycopg2 cursor object as an arguments.
Here after passing psycopg2 cursor object to the process_partition_up(), I'm not getting cursor object rather I'm getting above error.
Can anyone help me out to resolve this error?
Thank you.

I think that you don't understand what's happening here.
You are creating a database connection in your driver(etl_process), and then trying to ship that live connection from the driver, across your network to executor to do the work.(your lambda in foreachPartitions is executed on the executor.)
That is what spark is telling you "cannot pickle 'psycopg2.extensions.cursor'". (It can't serialize your live connection to the database to ship it to an executor.)
You need to call conn = get_connection() from inside process_partition_up this will initialize the connection to the database from inside the executor.(And any other book keeping you need to do.)
FYI: The worst part that I want to call out is that this code will work on your local machine. This is because it's both the executor and the driver.

how can i validate username password for mongodb authentication through pymongo?

I am refering to the http://api.mongodb.org/python/current/examples/authentication.html site for authentication mechanism examples. I have created a User administrator and using its credentials I created a user for my 'reporting' database. Now i need to access the same through pymongo using the username and password. I tried the following commands in python shell. Is this the right way as my authentication is failing.
from pymongo import MongoClient
client = MongoClient('localhost')
client.reporting.authenticate('reportsUser', '123456', mechanism='MONGODB-CR')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/pymongo/database.py", line 746, in authenticate
self.connection._cache_credentials(self.name, credentials)
File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 441, in _cache_credentials
auth.authenticate(credentials, sock_info, self.__simple_command)
File "/usr/lib/python2.7/dist-packages/pymongo/auth.py", line 214, in authenticate
auth_func(credentials[1:], sock_info, cmd_func)
File "/usr/lib/python2.7/dist-packages/pymongo/auth.py", line 194, in _authenticate_mongo_cr
cmd_func(sock_info, source, query)
File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 607, in __simple_command
helpers._check_command_response(response, None, msg)
File "/usr/lib/python2.7/dist-packages/pymongo/helpers.py", line 147, in _check_command_response
raise OperationFailure(msg % errmsg, code)
pymongo.errors.OperationFailure: command SON([('authenticate', 1), ('user', u'reportsUser'), ('nonce', u'f8158a24f1c61650'), ('key', u'14cea216c54b93bae20acd2e076bb785')]) failed: auth failed

As an FYI, you can use the URI string format as well. The pseudocode looks like this:
pymongo.MongoClient('mongodb://user:password#server:port/')
Here's a simple connection code block with auth:
import pymongo
conn = pymongo.MongoClient('mongodb://root:pass#localhost:27017/')
db = conn['database']
coll = db['collection']
There are more options for the query string here: http://docs.mongodb.org/manual/reference/connection-string/
Hope that helps = looks like you already have it though. Happy coding!!

Just adding more to provided solutions.
I have been using as URI connection string and credentials being provided as f string it helps to reduce number of lines. One thing to note is about special characters in password where we convert using urllib package as shown below.
import urllib.parse
from pymongo import MongoClient
host = "localhost"
port = 27017
user_name = "myuser"
pass_word = "Pass#123"
db_name = "mydb" # database name to authenticate
# if your password has '#' then you might need to escape hence we are using "urllib.parse.quote_plus()"
client = MongoClient(f'mongodb://{user_name}:{urllib.parse.quote_plus(pass_word)}#{host}:{port}/{db_name}')

It's worked for me.
Here you can connect mongodb to python by using authentication username and password.
import pymongo
DATABASE_NAME = "your_database_name"
DATABASE_HOST = "localhost"
DATABASE_USERNAME = "database_username"
DATABASE_PASSWORD = "database_password"
try:
myclient = pymongo.MongoClient( DATABASE_HOST )
myclient.test.authenticate( DATABASE_USERNAME , DATABASE_PASSWORD )
mydb = myclient[DATABASE_NAME]
print("[+] Database connected!")
except Exception as e:
print("[+] Database connection error!")
raise e
By default Mongodb uses 27017 port

Python and MySQL connection problems (mysqldb api)

I have config.ini:
[mysql]
host=localhost
port=3306
user=root
passwd=abcdefgh
db=testdb
unix_socket=/opt/lampp/var/mysql/mysql.sock
I have this class:
#!/usr/bin/python
import MySQLdb,ConfigParser
config = ConfigParser.ConfigParser()
config.read("config.ini")
class MySQL( object ):
def __init__( self ):
self.host = config.get("mysql","host")
self.port = config.get("mysql","port")
self.user = config.get("mysql","user")
self.passwd = config.get("mysql","passwd")
self.db = config.get("mysql","db")
self.unix_socket = config.get("mysql","unix_socket")
self.conn = MySQLdb.Connect(self.host,
self.port,
self.user,
self.passwd,
self.db,
self.unix_socket)
self.cursor = self.conn.cursor ( MySQLdb.cursors.DictCursor )
def __del__( self ):
self.cursor.close()
self.conn.close()
and this:
#!/usr/bin/env python
from mysql import MySQL
class Incident( MySQL ):
def getIncidents( self ):
self.cursor.execute("""*VALID QUERY*""")
return self.cursor.fetchall()
and finally this:
import subprocess, os, alarm
from Queue import Queue
from incident_model import Incident
fileQueue = Queue()
def enumerateFilesPath():
global fileQueue
incident = Incident()
incidents = incident.getIncidents()
for i in incidents:
fileQueue.put("MD5")
def main():
global fileQueue
enumerateFilesPath()
Output:
Traceback (most recent call last):
File "./mwmonitor.py", line 202, in
main() File "./mwmonitor.py", line 184, in main
enumerateFilesPath() File "./mwmonitor.py", line 86, in
enumerateFilesPath
incident = Incident() File "/usr/share/mwanalysis/core/mysql.py",
line 23, in init
self.unix_socket) File "/usr/lib/pymodules/python2.6/MySQLdb/init.py",
line 81, in Connect
return Connection(*args, **kwargs) File
"/usr/lib/pymodules/python2.6/MySQLdb/connections.py",
line 170, in init
super(Connection, self).init(*args, **kwargs2)
TypeError: an integer is required
Exception AttributeError: "'Incident'
object has no attribute 'cursor'" in
0xa03d46c>> ignored
If someone can help detect and correct the error would greatly appreciate.
Thanks in advance.

Your __del__ method is causing confusion. Specifically, it refers to self.cursor and self.conn which may never get created if, for example, MySQLdb.Connect raises an exception (which is what seems to happen).
I suggest you modify your class as follows:
class MySQL( object ):
def __init__( self ):
self.conn = None
self.cursor = None
self.host = config.get("mysql","host")
self.port = config.get("mysql","port")
self.user = config.get("mysql","user")
self.passwd = config.get("mysql","passwd")
self.db = config.get("mysql","db")
self.unix_socket = config.get("mysql","unix_socket")
self.conn = MySQLdb.Connect(self.host,
self.port,
self.user,
self.passwd,
self.db,
self.unix_socket)
self.cursor = self.conn.cursor ( MySQLdb.cursors.DictCursor )
def __del__( self ):
if self.cursor is not None:
self.cursor.close()
if self.conn is not None:
self.conn.close()
This won't solve the problem, but should give better diagnostic.
Now to the actual problem that you're experiencing. I strongly suspect that you're supplying the arguments to Connect in the wrong order, or the types aren't quite right, or something along those lines. To quote the docstring for Connection.__init__:
Create a connection to the database. It is strongly recommended
that you only use keyword parameters. Consult the MySQL C API
documentation for more information.
host
string, host to connect
user
string, user to connect as
passwd
string, password to use
db
string, database to use
port
integer, TCP/IP port to connect to
unix_socket
string, location of unix_socket to use
...
"It is strongly that you only use keyword parameters." I recommend that you do just that when you call MySQLdb.Connect. Also, make sure that port is an int and not a string.

I suspect it's expecting port to be an integer rather than a string. Try:
self.port = int(config.get("mysql","port"))

I am not sure if this is a connectivity error. Have you checked the type of the incident_model ?
TypeError: an integer is required
Exception AttributeError: "'Incident'object has no attribute 'cursor'" in

sqlobject: No connection has been defined for this thread or process

I'm using sqlobject in Python. I connect to the database with
conn = connectionForURI(connStr)
conn.makeConnection()
This succeeds, and I can do queries on the connection:
g_conn = conn.getConnection()
cur = g_conn.cursor()
cur.execute(query)
res = cur.fetchall()
This works as intended. However, I also defined some classes, e.g:
class User(SQLObject):
class sqlmeta:
table = "gui_user"
username = StringCol(length=16, alternateID=True)
password = StringCol(length=16)
balance = FloatCol(default=0)
When I try to do a query using the class:
User.selectBy(username="foo")
I get an exception:
...
File "c:\python25\lib\site-packages\SQLObject-0.12.4-py2.5.egg\sqlobject\main.py", line 1371, in selectBy
conn = connection or cls._connection
File "c:\python25\lib\site-packages\SQLObject-0.12.4-py2.5.egg\sqlobject\dbconnection.py", line 837, in __get__
return self.getConnection()
File "c:\python25\lib\site-packages\SQLObject-0.12.4-py2.5.egg\sqlobject\dbconnection.py", line 850, in getConnection
"No connection has been defined for this thread "
AttributeError: No connection has been defined for this thread or process
How do I define a connection for a thread? I just realized I can pass in a connection keyword which I can give conn to to make it work, but how do I get it to work if I weren't to do that?

Do:
from sqlobject import sqlhub, connectionForURI
sqlhub.processConnection = connectionForURI(connStr)