I'd like to create new database and new tables to this database using sqlalchemy. and I have to write create_engine twice, is there any easier writing method to do these things?
my code is here:
from sqlalchemy import create_engine
import pymysql
user = 'admin'
password = ''
host = 'database-1.czswegfdjhpn.us-east-1.rds.amazonaws.com'
port = 3306
engine = create_engine(url="mysql+pymysql://{0}:{1}#{2}:{3}".format(
user, password, host, port))
conn = engine.connect()
print('connect successfull')
conn.execute("commit")
conn.execute('create database covid_19')
database = 'covid_19'
engine1 = create_engine(url="mysql+pymysql://{0}:{1}#{2}:{3}/{4}".format(
user, password, host, port, database))
conn1 = engine.connect()
print('connect successfull')
df.to_sql(name="covid_19_world_cases_deaths_testing",con=engine1, if_exists='append', index=False, chunksize=200)
Haven’t tested this, however could you simply call
conn.execute(“USE covid_19”)
to change the database of the existing connection?
Related
I need to read from AWS- Aurora table and write the content to Oracle table.
My code is-
import pandas as pd
import psycopg2
from sqlalchemy import types, create_engine
import cx_Oracle
import sys
**# Connect to Aurora**
host = sys.argv[1]
username = sys.argv[2]
password = sys.argv[3]
database = sys.argv[4]
db_conn = psycopg2.connect(host=host, database=database, user=username, password=password)
sql = "SELECT * FROM Table_Name;"
data_df = pd.io.sql.read_sql(sql, db_conn)
print(data_df.head(2))
db_conn.close()
# Connect to Oracle and write data_df dataframe
dsn = cx_Oracle.makedsn('10.z.y.xx', '1521', service_name='abcd')
u_name = sys.argv[5]
pwd = sys.argv[6]
conn = cx_Oracle.connect(user=u_name, password=pwd, dsn=dsn)
ora_engine = create_engine(f'oracle+cx_oracle://{u_name}:{pwd}#{dsn}', echo=True)
ora_engine.connect()
data_df.to_sql(name='oracle_table_name', con=conn)
conn.close()
Connect to Aurora is working but I'm unable to create engine in Oracle and write the dataframe!
The code is correct, due to high volume of data and low RAM being configured, it was failing.
Thanks.
I have my report dashboard that I'm developing using Django and I want to connect it to my MySQL database without using models.py or migrations.
Basically, here's what I've done so far.
views.py
import configparser
config = configparser.ConfigParser()
config.read('db.ini')
def getConnection():
host = config['ord']['host']
port = config['ord']['port']
database = config['ord']['database']
password = config['ord']['password']
user = config['ord']['user']
connection = pymssql.connect(host=host, user=user, password=password, database=database)
cursor = connection.cursor()
print("Connection Succesfull")
return cursor
def fill_weight(request):
try:
sql = u """
SELECT * FROM fill_weight;
"""
sql =sql.encode('utf-8')
cursor.execute(sql)
cursor.fetchall()
db.ini
[oof_ord]
host = localhost
port = 3306
database = xxxxxxxxxxx
user = xxxxxxxxxxx
password = xxxxxxxxxxx
default-character-set = utf8
The reason why I want to do this is because I don't have the official database that I'm going to use for this system and I want to easily access it by putting it's database credentials to my db.ini and access it to show the data without doing any migrations.
Is there anyone that can help me ?
I'm new to hadoop and impala. I managed to connect to impala by installing impyla and executing the following code. This is connection by LDAP:
from impala.dbapi import connect
from impala.util import as_pandas
conn = connect(host="server.lrd.com",port=21050, database='tcad',auth_mechanism='PLAIN', user="alexcj", use_ssl=True,timeout=20, password="secret1pass")
I'm then able to grab a cursor and execute queries as:
cursor = conn.cursor()
cursor.execute('SELECT * FROM tab_2014_m LIMIT 10')
df = as_pandas(cursor)
I'd like to be able use sqlalchemy to connect to impala and be able to use some nice sqlalchemy functions. I found a test file in imyla source code that illustrates how to create an sqlalchemy engine with impala driver like:
engine = create_engine('impala://localhost')
I'd like to be able to do that but I'm not able to because my call to the connect function above has a lot more parameters; and I do not know how to pass those to sqlalchemy's create_engine to get a successful connection. Has anyone done this? Thanks.
As explained at https://github.com/cloudera/impyla/issues/214
import sqlalchemy
def conn():
return connect(host='some_host',
port=21050,
database='default',
timeout=20,
use_ssl=True,
ca_cert='some_pem',
user=user, password=pwd,
auth_mechanism='PLAIN')
engine = sqlalchemy.create_engine('impala://', creator=conn)
If your Impala is secured by Kerberos below script works (due to some reason I need to use hive:// instead of impala://)
import sqlalchemy
from sqlalchemy.engine import create_engine
connect_args={'auth': 'KERBEROS', 'kerberos_service_name': 'impala'}
engine = create_engine('hive://impalad-host:21050', connect_args=connect_args)
conn = engine.connect()
ResultProxy = conn.execute("SELECT * FROM db1.table1 LIMIT 5")
print(ResultProxy.fetchall())
import time
from sqlalchemy import create_engine, MetaData, Table, select, and_
ENGINE = create_engine(
'impala://{host}:{port}/{database}'.format(
host=host, # your host
port=port,
database=database,
)
)
METADATA = MetaData(ENGINE)
TABLES = {
'table': Table('table_name', METADATA, autoload=True),
}
Ok so here is my use case. I've to make connection to different types of DB(MSSQL, oracl, MYSQL, etc.). I've .sql files for each of these database. As it seems sqlalchemy can't run .sql file so we need to open and execute the statements one by one from the .sql files over the connection.
So guys, I'm having this information and I wanted to connect using SQL Alchemy.
<db type="MSSQL" version="2005" patch_level="SP2" port="1433" id="MSSQLSERVER"/>
here MSSQLServer is the instance. No DB information is provide. so do I need DB name to connect to DB?
this is my command
engine = create_engine('mssql+pyodbc://sa:pass#172.21.153.227/MSSQLSERVER', echo=True)
this is my complete code
from sqlalchemy.engine import create_engine
engine = create_engine('mssql+pyodbc://sa:pass#172.21.153.227', echo=False)
connection = engine.connect()
connection.execute(
"""
select ##version
"""
)
connection.close()
you don't need a db name, you can use this function i wrote:
(it works for me on mySQL)
def ConnectToDB(self, server, uid, password):
"""
this method if for connecting to a db
#param server: server name
#param uid: username
#param password: password
"""
connection_string = 'mysql://{}:{}#{}'.format(uid, password, server)
try:
self.engine = create_engine(connection_string)
self.connection = self.engine.connect()
except exc.SQLAlchemyError, e:
self.engine = None
return False, e
return True, None
in your SQL statements you will say the DB and table some thing like this:
INSERT INTO `dbName`.`dbTable`.........
I have a .ini (configuration file) where I have mentioned the server name, Database Name, UserName and Password with which I can connect my app to the MSSQL
self.db = pyodbc.connect(
'driver={SQL Server};server=homeserver;database=testdb;uid=home;pwd=1234')`
corresponding data mentioned above connect statement is now in config.ini
self.configwrite = ConfigParser.RawConfigParser()
configread = SafeConfigParser()
configread.read('config.ini')
driver = configread.get('DataBase Settings','Driver')
server = str(configread.get('DataBase Settings','Server'))
db = str(configread.get('DataBase Settings','Database'))
user = str(configread.get('DataBase Settings','Username'))
password = str(configread.get('DataBase Settings','Password'))'
How can I pass these variables in the pyodbc connect statement?
I tried this:
self.db = pyodbc.connect('driver={Driver};server=server;database=db;uid=user;pwd=password')
But I am getting an error.
Other options for the connect function:
# using keywords for SQL Server authentication
self.db = pyodbc.connect(driver=driver, server=server, database=db,
user=user, password=password)
# using keywords for Windows authentication
self.db = pyodbc.connect(driver=driver, server=server, database=db,
trusted_connection='yes')
self.db = pyodbc.connect('driver={%s};server=%s;database=%s;uid=%s;pwd=%s' % ( driver, server, db, user, password ) )
%s is used to include variables into the string
the variables are placed into the string according to their order after the %
Mixing strings and input variable in sql connection string using pyodbc library - Python
inspired of this answer
`conn=pyodbc.connect('Driver={SQL Server};'
'Server='+servername+';'
'Database=master;'
'UID=sa;'
'PWD='+pasword+';'
)`