I have a sql file generated during database backup process and I want to load all database content from that sql file to a different MySQL database (secondary database).
I have created a python function to load the whole database in that sql file but when I execute the function, I get an error
'str' object is not callable
Below is python script
def load_database_dump_to_secondary_mysql(file_path='db_backup_file.sql'):
query = f'source {file_path}'
try:
connection = mysql_hook.get_conn() # connection to secondary db
cursor = connection.cursor(query)
print('LOAD TO MYSQL COMPLETE')
except Exception as xerror:
print("LOAD ERROR: ", xerror)
NB: mysql_hook is an airflow connector that contains MySQL DB connection info such as Host, user/passwd, Database name. Also, I don't have connection to the primary database, I'm only receiving sql dump file.
What I'm I missing?
source is a client builtin command: https://dev.mysql.com/doc/refman/8.0/en/mysql-commands.html
It's not an SQL query that MySQL's SQL parser understands.
So you can't execute source using cursor.execute(), because that goes directly to the dynamic SQL interface.
You must run it using the MySQL command-line client as a subprocess:
subprocess.run(['mysql', '-e', f'source {file_path}'])
You might need other options to the mysql client, such as user, password, host, etc.
try this
import mysql.connector as m
# database which you want to backup
db = 'geeksforgeeks'
connection = m.connect(host='localhost', user='root',
password='123', database=db)
cursor = connection.cursor()
# Getting all the table names
cursor.execute('SHOW TABLES;')
table_names = []
for record in cursor.fetchall():
table_names.append(record[0])
backup_dbname = db + '_backup'
try:
cursor.execute(f'CREATE DATABASE {backup_dbname}')
except:
pass
cursor.execute(f'USE {backup_dbname}')
for table_name in table_names:
cursor.execute(
f'CREATE TABLE {table_name} SELECT * FROM {db}.{table_name}')
I'm writing an application to upload data for my customer's database. When I tested it on my PC, it works ok but when connecting to the client's database, it has some errors. I'm using this code to connect to MySQL:
def connect_sql(self):
mydb = mysql.connector.connect(
host=host,
passwd=pw,
port=port,
database=db,
user=user,
)
return mydb
When tried to connect to the database on the client's PC, it shows this error:
1115 (42000): Unknown character set: 'utf8mb4'
I've noticed that this error maybe because different database server version between me and my client. While he uses 5.0.67-community-nt (which does not support 'utf8mb4') and I use 10.0.21 MariaDB-log.
Is there any way to fix this error on my site (maybe export .sql file to set the default character set to prevent this error, etc..), or do I need to ask my customer to update his MySQL database's version?
Thanks for any help!!!
Try it as following:
def connect_sql(self):
mydb = mysql.connector.connect(
host=host,
passwd=pw,
port=port,
database=db,
user=user,
charset='utf8',
use_unicode=True
)
return mydb
Here's documentation.
I am beginning with sqlite3 in Python.
I know how to connect to local database. See simple example below:
import sqlite3
connection = sqlite3.connect('mydb.db')
cursor = connection.cursor()
But now I would like to send data into a database that is running on a server. I have following config information:
[db-my_db]
user=my_user
password=my_pass
host=db.my_db.com
database=my_db.com
How can I connect to this database? Thanks
I have a LAMP server and then I installed MySQLdb for my Python scripts. Now I can't access the MySQL (from LAMP) from Python scripts because it isn't connecting to the MySQLdb, and also I can't access the MySQLdb with phpMyAdmin with (root root). I got "#2002 Cannot log in to the MySQL server" error. Is it possible to connect to one db with Python and phpMyAdmin?
Here is my Python code, which can't connect to the LAMP MySQL, but can connect to the MySQLdb:
db = MySQLdb.connect(host="localhost", port=3303, user="root", passwd="rootroot", db="test")
cursor = db.cursor()
sql = "CREATE TABLE TT(ID int NOT NULL AUTO_INCREMENT PRIMARY KEY)"
cursor.execute(sql)
db.commit()
db.close()
If you're getting #2002 Cannot log in to the MySQL server when logging in to phpmyadmin, then edit phpmyadmin/config.inc.php file and change:
$cfg['Servers'][$i]['host'] = 'localhost';
to:
$cfg['Servers'][$i]['host'] = '127.0.0.1';
You might want to visit this link:
http://blog.ryantremaine.com/2011/03/2002-cannot-log-in-to-mysql-server.html
UPDATE:
you wud not have configured your php.ini well and so that it cannot connect to mysql server.
Wrong path for the mysql.sock
mysql.sock is the instance of the mysql, so first you have to find where does it place at.
You may find it at "/tmp/mysql.sock" or "/var/mysql/mysql.sock".
Go to your php.ini and make sure the value for "pdo_mysql.default_socket", "mysql.default_socket", "mysqli.default_socket" is the right path.
Then restart your web server and try again.
ELSE
Try this:
Go to config.inc.php and check for the following line:
$cfg['Servers'][$i]['user'] = 'YOUR USER NAME IS HERE';
$cfg['Servers'][$i]['password'] = 'AND YOU PASSWORD IS HERE';
Check whether the user name and password that you gave is present or not
How can I access Oracle from Python? I have downloaded a cx_Oracle msi installer, but Python can't import the library.
I get the following error:
import cx_Oracle
Traceback (most recent call last):
File "<pyshell#1>", line 1, in <module>
import cx_Oracle
ImportError: DLL load failed: The specified module could not be found.
I will be grateful for any help.
Here's what worked for me. My Python and Oracle versions are slightly different from yours, but the same approach should apply. Just make sure the cx_Oracle binary installer version matches your Oracle client and Python versions.
My versions:
Python 2.7
Oracle Instant Client 11G R2
cx_Oracle 5.0.4 (Unicode, Python 2.7, Oracle 11G)
Windows XP SP3
Steps:
Download the Oracle Instant Client package. I used instantclient-basic-win32-11.2.0.1.0.zip. Unzip it to C:\your\path\to\instantclient_11_2
Download and run the cx_Oracle binary installer. I used cx_Oracle-5.0.4-11g-unicode.win32-py2.7.msi. I installed it for all users and pointed it to the Python 2.7 location it found in the registry.
Set the ORACLE_HOME and PATH environment variables via a batch script or whatever mechanism makes sense in your app context, so that they point to the Oracle Instant Client directory. See oracle_python.bat source below. I'm sure there must be a more elegant solution for this, but I wanted to limit my system-wide changes as much as possible. Make sure you put the targeted Oracle Instant Client directory at the beginning of the PATH (or at least ahead of any other Oracle client directories). Right now, I'm only doing command-line stuff so I just run oracle_python.bat in the shell before running any programs that require cx_Oracle.
Run regedit and check to see if there's an NLS_LANG key set at \HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE. If so, rename the key (I changed it to NLS_LANG_OLD) or unset it. This key should only be used as the default NLS_LANG value for Oracle 7 client, so it's safe to remove it unless you happen to be using Oracle 7 client somewhere else. As always, be sure to backup your registry before making changes.
Now, you should be able to import cx_Oracle in your Python program. See the oracle_test.py source below. Note that I had to set the connection and SQL strings to Unicode for my version of cx_Oracle.
Source: oracle_python.bat
#echo off
set ORACLE_HOME=C:\your\path\to\instantclient_11_2
set PATH=%ORACLE_HOME%;%PATH%
Source: oracle_test.py
import cx_Oracle
conn_str = u'user/password#host:port/service'
conn = cx_Oracle.connect(conn_str)
c = conn.cursor()
c.execute(u'select your_col_1, your_col_2 from your_table')
for row in c:
print row[0], "-", row[1]
conn.close()
Possible Issues:
"ORA-12705: Cannot access NLS data files or invalid environment specified" - I ran into this before I made the NLS_LANG registry change.
"TypeError: argument 1 must be unicode, not str" - if you need to set the connection string to Unicode.
"TypeError: expecting None or a string" - if you need to set the SQL string to Unicode.
"ImportError: DLL load failed: The specified procedure could not be found." - may indicate that cx_Oracle can't find the appropriate Oracle client DLL.
You can use any of the following way based on Service Name or SID whatever you have.
With SID:
import cx_Oracle
dsn_tns = cx_Oracle.makedsn('server', 'port', 'sid')
conn = cx_Oracle.connect(user='username', password='password', dsn=dsn_tns)
c = conn.cursor()
c.execute('select count(*) from TABLE_NAME')
for row in c:
print(row)
conn.close()
OR
With Service Name:
import cx_Oracle
dsn_tns = cx_Oracle.makedsn('server', 'port', service_name='service_name')
conn = cx_Oracle.connect(user='username', password='password', dsn=dsn_tns)
c = conn.cursor()
c.execute('select count(*) from TABLE_NAME')
for row in c:
print(row)
conn.close()
Here is how my code looks like. It also shows an example of how to use query parameters using a dictionary. It works on using Python 3.6:
import cx_Oracle
CONN_INFO = {
'host': 'xxx.xx.xxx.x',
'port': 12345,
'user': 'SOME_SCHEMA',
'psw': 'SECRETE',
'service': 'service.server.com'
}
CONN_STR = '{user}/{psw}#{host}:{port}/{service}'.format(**CONN_INFO)
QUERY = '''
SELECT
*
FROM
USER
WHERE
NAME = :name
'''
class DB:
def __init__(self):
self.conn = cx_Oracle.connect(CONN_STR)
def query(self, query, params=None):
cursor = self.conn.cursor()
result = cursor.execute(query, params).fetchall()
cursor.close()
return result
db = DB()
result = db.query(QUERY, {'name': 'happy'})
import cx_Oracle
dsn_tns = cx_Oracle.makedsn('host', 'port', service_name='give service name')
conn = cx_Oracle.connect(user='username', password='password', dsn=dsn_tns)
c = conn.cursor()
c.execute('select count(*) from schema.table_name')
for row in c:
print row
conn.close()
Note :
In (dsn_tns) if needed, place an 'r' before any parameter in order to address any special character such as '\'.
In (conn) if needed, place an 'r' before any parameter in order to address any special character such as '\'. For example, if your user name contains '\', you'll need to place 'r' before the user name: user=r'User Name' or password=r'password'
use triple quotes if you want to spread your query across multiple lines.
Note if you are using pandas you can access it in following way:
import pandas as pd
import cx_Oracle
conn= cx_Oracle.connect('username/pwd#host:port/service_name')
try:
query = '''
SELECT * from dual
'''
df = pd.read_sql(con = conn, sql = query)
finally:
conn.close()
df.head()
In addition to the Oracle instant client, you may also need to install the Oracle ODAC components and put the path to them into your system path. cx_Oracle seems to need access to the oci.dll file that is installed with them.
Also check that you get the correct version (32bit or 64bit) of them that matches your: python, cx_Oracle, and instant client versions.
In addition to cx_Oracle, you need to have the Oracle client library installed and the paths set correctly in order for cx_Oracle to find it - try opening the cx_Oracle DLL in "Dependency Walker" (http://www.dependencywalker.com/) to see what the missing DLL is.
Ensure these two and it should work:-
Python, Oracle instantclient and cx_Oracle are 32 bit.
Set the environment variables.
Fixes this issue on windows like a charm.
If you are using virtualenv, it is not as trivial to get the driver using the installer. What you can do then: install it as described by Devon. Then copy over cx_Oracle.pyd and the cx_Oracle-XXX.egg-info folder from Python\Lib\site-packages
into the Lib\site-packages from your virtual env. Of course, also here, architecture and version are important.
import cx_Oracle
from sshtunnel import SSHTunnelForwarder
# remote server variables
remote_ip_address = "<PUBLIC_IP_ADDRESS_OF_DB_SERVER>"
remote_os_username = "<OS_USERNAME>"
ssh_private_key = "<PATH_TO_PRIVATE_KEY>"
# Oracle database variables
database_username = "<DATABASE_USER>"
database_password = "<DATABASE_USER_PASSWORD>"
database_server_sid = "<ORACLE_SID>"
def server_connection():
server = SSHTunnelForwarder(
remote_ip_address,
ssh_username=remote_os_username,
ssh_password=ssh_private_key,
remote_bind_address=('localhost', 1521) # default Oracle DB port
)
return server
def database_connection():
data_source_name = cx_Oracle.makedsn("localhost",
server.local_bind_port,
service_name=database_server_sid)
connection = cx_Oracle.connect(database_username,
database_password,
data_source_name,
mode=cx_Oracle.SYSDBA) # If logging in with SYSDBA privs,
# leave out if not.
return connection
def database_execute():
connection = database_connection()
cursor = connection.cursor()
for row in cursor.execute("SELECT * FROM HELLO_WORLD_TABLE"):
print(row)
if __name__ == '__main__':
server = server_connection()
server.start() # start remote server connection
connection = database_connection() # create Oracle database connection
database_execute() # execute query
connection.close() # close Oracle database connection
server.stop() # close remote server connection
If you're accessing the Oracle database via a bastion tunnel, you just need to modify this piece of code:
def server_connection():
server = SSHTunnelForwarder(
remote_ip_address, # public IP of bastion server
ssh_username=remote_os_username,
ssh_password=ssh_private_key,
remote_bind_address=('localhost', 1521),
local_bind_address=('0.0.0.0', 3333) # Suppose local bind is '3333'
)
return server