I have two separate database files each with tables with matching primary keys, but different data. I want to pull out rows from one table based on values in the other. In the CLI for sqlite3, I would do this like:
.open data.db
.open details.db
attach 'data.db' as data;
attach 'details.db' as details;
select details.A.colA from data.A join details.A using ('key') where data.A.colB = 0 and data.A.colC = 1;
How can I recreate such a cross-database join using pysqlite?
You can attach additional databases with ATTACH DATABASE:
conn = sqlite3.connect('data.db')
conn.execute('ATTACH DATABASE details.db AS details')
For query purposes, the first database is known as main:
cursor = conn.cursor()
cursor.execute('''
select details.A.colA
from main.A
join details.A using ('key')
where main.A.colB = 0 and main.A.colC = 1
''')
Related
Here's the goal raw SQL:
INSERT INTO "db2"."schema2"."table" ("col1", "col2")
SELECT "db1"."schema1"."tablea"."column1",
"db1"."schema1"."tableb"."column1"
FROM "db1"."schema1"."tablea" JOIN "db1"."schema1"."tableb"
ON
"db1"."schema1"."tablea"."join_col" = "db1"."schema1"."tableb"."join_col";
I'm using SQLAlchemy Core to reflect the tables. This is simple enough but the issue is I'm unable to reflect them in a fully qualified way.
from sqlalchemy import create_engine, MetaData, Table, text
from snowflake.sqlalchemy import URL
from sqlalchemy.sql import select, insert
engine = create_engine(URL(
account="my_account",
user="my_username",
password="my_password",
warehouse="my_warehouse",
role="my_role",
)
)
metadata = MetaData()
with engine.begin() as cnxn:
cnxn.execute(text("USE DATABASE db1;"))
tablea = Table("tablea", metadata, autoload_with=engine, schema="schema1")
tableb = Table("tableb", metadata, autoload_with=engine, schema="schema1")
with engine.begin() as cnxn:
cnxn.execute(text("USE DATABASE db2;"))
table = Table("table", metadata, autoload_with=engine, schema="schema2")
sql = select(tablea.c.column1, tableb.c.column1)
sql = sql.select_from(tablea.join(tableb, tablea.c.join_col == tableb.c.join_col))
insert_stmnt = insert(table).from_select([c.name for c in table.c]), sql)
with engine.begin() as cnxn:
cnxn.execute(insert_stmnt)
The problem is the insert statement fails because I have three tables and two different databases. I think the easy answer is when reflecting the tables they need to be fully qualified. Is that possible? If so, how? I've been trying all different combinations of table_name and schema in the reflection trying to include the database but have been completely striking out.
The only thing I've been able to do is print the insert statement and then manually add the databases to each table in the resulting string. Then, run the string. This completely defeats the purpose of SQLAlchemy IMO.
python_full_version = "3.9.6"
sqlalchemy = "1.4.45"
snowflake-sqlalchemy = "1.4.4"
Should be done using single function
Shouldn't use Pandas or merge function or any other inbuilt database libraries
You can use native driver like psycopg2 for postgres https://www.psycopg.org/docs/usage.html
import psycopg2
# Connect to an existing database
conn = psycopg2.connect("dbname=test user=postgres")
cur = conn.cursor()
# Query the database and obtain data as Python objects
cur.execute("""
SELECT * FROM test t
left join test1 t1 on (t.t1_id = t1.id);
""")
fetched_rows = cur.fetchall()
# Make the changes to the database persistent
conn.commit()
# Close communication with the database
cur.close()
conn.close()
Creating a list in python and inserting into oracle table , but no records found in oracle table.
Created a list in python.
Created a Oracle table using Python code.
Using executeMany inserted the list.
Run the count(*) query in python and obtained the number of rows and printed in python.
Output : Table has been created in Oracle using python code succesfully , but cannot find the records inserted using python
import cx_Oracle
con = cx_Oracle.connect('username/password#127.0.0.1/orcl')
cursor = con.cursor()
create_table = """CREATE TABLE python_modules ( module_name VARCHAR2(1000) NOT NULL, file_path VARCHAR2(1000) NOT NULL )"""
cursor.execute(create_table)
M = []
M.append(('Module1', 'c:/1'))
M.append(('Module2', 'c:/2'))
M.append(('Module3', 'c:3'))
cursor.prepare("INSERT INTO python_modules(module_name, file_path) VALUES (:1, :2)")
cursor.executemany(None,M)
con.commit
cursor.execute("SELECT COUNT(*) FROM python_modules")
print(cursor.fetchone() [0])
Executing below query "select * from python_modules " should have the 3 records in Oracle SQL Developer tool
Change your commit to con.commit().
I am trying to work out why the schema of a dropped table returns when I attempt to create a table using a different set of column names?
After dropping the table, I can confirm in an SQLite explorer that the table has disappeared. Once trying to load the new file via ODO it then returns an error "Column names of incoming data don't match column names of existing SQL table names in SQL table". Then I can see the same table is re-created in the database, using the previously dropped schema! I attempted a VACUUM statement after dropping the table but still same issue.
I can create the table fine using a different table name, however totally confused as to why I can't use the previously dropped table name I want to use?
import sqlite3
import pandas as pd
from odo import odo, discover, resource, dshape
conn = sqlite3.connect(dbfile)
c = conn.cursor()
c.execute("DROP TABLE <table1>")
c.execute("VACUUM")
importfile = pd.read_csv(csvfile)
odo(importfile,'sqlite:///<db_path>::<table1'>)
ValueError: Column names of incoming data don't match column names of existing SQL table Names in SQL table:
import sqlite3
import pandas as pd
from odo import odo, discover, resource, dshape
conn = sqlite3.connect('test.db')
cursor = conn.cursor();
table = """ CREATE TABLE IF NOT EXISTS TABLE1 (
id integer PRIMARY KEY,
name text NOT NULL
); """;
cursor.execute(table);
conn.commit(); # Save table into database.
cursor.execute(''' DROP TABLE TABLE1 ''');
conn.commit(); # Save that table has been dropped.
cursor.execute(table);
conn.commit(); # Save that table has been created.
conn.close();
I'm having troubles with creating a database and tables. The database needs to be created within a Python script.
#connect method has 4 parameters:
#localhost (where mysql db is located),
#database user name,
#account password,
#database name
db1 = MS.connect(host="localhost",user="root",passwd="****",db="test")
returns
_mysql_exceptions.OperationalError: (1049, "Unknown database 'test'")
So clearly, the db1 needs to be created first, but how? I've tried CREATE before the connect() statement but get errors.
Once the database is created, how do I create tables?
Thanks,
Tom
Here is the syntax, this works, at least the first time around. The second time naturally returns that the db already exists. Now to figure out how to use the drop command properly.
db = MS.connect(host="localhost",user="root",passwd="****")
db1 = db.cursor()
db1.execute('CREATE DATABASE test1')
So this works great the first time through. The second time through provides a warning "db already exists". How to deal with this? The following is how I think it should work, but doesn't. OR should it be an if statement, looking for if it already exists, do not populate?
import warnings
warnings.filterwarnings("ignore", "test1")
Use CREATE DATABASE to create the database:
db1 = MS.connect(host="localhost",user="root",passwd="****")
cursor = db1.cursor()
sql = 'CREATE DATABASE mydata'
cursor.execute(sql)
Use CREATE TABLE to create the table:
sql = '''CREATE TABLE foo (
bar VARCHAR(50) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1
'''
cursor.execute(sql)
There are a lot of options when creating a table. If you are not sure what the right SQL should be, it may help to use a graphical tool like phpmyadmin to create a table, and then use SHOW CREATE TABLE to discover what SQL is needed to create it:
mysql> show create table foo \G
*************************** 1. row ***************************
Table: foo
Create Table: CREATE TABLE `foo` (
`bar` varchar(50) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
phpmyadmin can also show you what SQL it used to perform all sorts of operations. This can be a convenient way to learn some basic SQL.
Once you've experimented with this, then you can write the SQL in Python.
I think the solution is a lot easier, use "if not":
sql = "CREATE DATABASE IF NOT EXISTS test1"
db1.execute(sql)
import MySQLdb
# Open database connection ( If database is not created don't give dbname)
db = MySQLdb.connect("localhost","yourusername","yourpassword","yourdbname" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# For creating create db
# Below line is hide your warning
cursor.execute("SET sql_notes = 0; ")
# create db here....
cursor.execute("create database IF NOT EXISTS yourdbname")
# create table
cursor.execute("SET sql_notes = 0; ")
cursor.execute("create table IF NOT EXISTS test (email varchar(70),pwd varchar(20));")
cursor.execute("SET sql_notes = 1; ")
#insert data
cursor.execute("insert into test (email,pwd) values('test#gmail.com','test')")
# Commit your changes in the database
db.commit()
# disconnect from server
db.close()
#OUTPUT
mysql> select * from test;
+-----------------+--------+
| email | pwd |
+-----------------+--------+
| test#gmail.com | test |
+-----------------+--------+