Should be done using single function
Shouldn't use Pandas or merge function or any other inbuilt database libraries
You can use native driver like psycopg2 for postgres https://www.psycopg.org/docs/usage.html
import psycopg2
# Connect to an existing database
conn = psycopg2.connect("dbname=test user=postgres")
cur = conn.cursor()
# Query the database and obtain data as Python objects
cur.execute("""
SELECT * FROM test t
left join test1 t1 on (t.t1_id = t1.id);
""")
fetched_rows = cur.fetchall()
# Make the changes to the database persistent
conn.commit()
# Close communication with the database
cur.close()
conn.close()
I have a sqlite db in my home dir.
stephen#stephen-AO725:~$ pwd
/home/stephen
stephen#stephen-AO725:~$ sqlite db1
SQLite version 2.8.17
Enter ".help" for instructions
sqlite> select * from test
...> ;
3|4
5|6
sqlite> .quit
when I try to connect from a jupiter notebook with sqlalchemy and pandas, sth does not work.
db=sqla.create_engine('sqlite:////home/stephen/db1')
pd.read_sql('select * from db1.test',db)
~/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py in do_execute(self, cursor, statement, parameters, context)
578
579 def do_execute(self, cursor, statement, parameters, context=None):
--> 580 cursor.execute(statement, parameters)
581
582 def do_execute_no_params(self, cursor, statement, context=None):
DatabaseError: (sqlite3.DatabaseError) file is not a database
[SQL: select * from db1.test]
(Background on this error at: http://sqlalche.me/e/4xp6)
I also tried:
db=sqla.create_engine('sqlite:///~/db1')
same result
Personally, just to complete the code of #Stephen with the modules required:
# 1.-Load module
import sqlalchemy
import pandas as pd
#2.-Turn on database engine
dbEngine=sqlalchemy.create_engine('sqlite:////home/stephen/db1.db') # ensure this is the correct path for the sqlite file.
#3.- Read data with pandas
pd.read_sql('select * from test',dbEngine)
#4.- I also want to add a new table from a dataframe in sqlite (a small one)
df_todb.to_sql(name = 'newTable',con= dbEngine, index=False, if_exists='replace')
Another way to read is using sqlite3 library, which may be more straighforward:
#1. - Load libraries
import sqlite3
import pandas as pd
# 2.- Create your connection.
cnx = sqlite3.connect('sqlite:////home/stephen/db1.db')
cursor = cnx.cursor()
# 3.- Query and print all the tables in the database engine
cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
print(cursor.fetchall())
# 4.- READ TABLE OF SQLITE CALLED test
dfN_check = pd.read_sql_query("SELECT * FROM test", cnx) # we need real name of table
# 5.- Now I want to delete all rows of this table
cnx.execute("DELETE FROM test;")
# 6. -COMMIT CHANGES! (mandatory if you want to save these changes in the database)
cnx.commit()
# 7.- Close the connection with the database
cnx.close()
Please let me know if this helps!
import sqlalchemy
engine=sqlalchemy.create_engine(f'sqlite:///db1.db')
Note: that you need three slashes in sqlite:/// in order to use a relative path for the DB. If you want an absolute path, use four slashes: sqlite:////
Source: Link
The issue is no backward compatibility as noted by Everila. anaconda installs its own sqlite, which is sqlite3.x and that sqlite cannot load databases created by sqlite 2.x
after creating a db with sqlite 3 the code works fine
db=sqla.create_engine('sqlite:////home/stephen/db1')
pd.read_sql('select * from test',db)
which confirms the 4 slashes are needed.
None of the sqlalchemy solutions worked for me with python 3.10.6 and sqlalchemy 2.0.0b4, it could be a beta issue or version 2.0.0 changed things. #corina-roca's solution was close, but not right as you need to pass a connection object, not an engine object. That's what the documentation says, but it didn't actually work. After a bit of experimentation, I discovered that engine.raw_connect() works, although you get a warning on the CLI. Here are my working examples
The sqlite one works out of the box - but it's not ideal if you are thinking of changing databases later
import sqlite3
conn = sqlite3.connect("sqlite:////home/stephen/db1")
df = pd.read_sql_query('SELECT * FROM test', conn)
df.head()
# works, no problem
sqlalchemy lets you abstract your db away
from sqlalchemy import create_engine, text
engine = create_engine("sqlite:////home/stephen/db1")
conn = engine.connect() # <- this is also what you are supposed to
# pass to pandas... it doesn't work
result = conn.execute(text("select * from test"))
for row in result:
print(row) # outside pands, this works - proving that
# connection is established
conn = engine.raw_connection() # with this workaround, it works; but you
# get a warning UserWarning: pandas only
# supports SQLAlchemy connectable ...
df = pd.read_sql_query(sql='SELECT * FROM test', con=conn)
df.head()
Creating a list in python and inserting into oracle table , but no records found in oracle table.
Created a list in python.
Created a Oracle table using Python code.
Using executeMany inserted the list.
Run the count(*) query in python and obtained the number of rows and printed in python.
Output : Table has been created in Oracle using python code succesfully , but cannot find the records inserted using python
import cx_Oracle
con = cx_Oracle.connect('username/password#127.0.0.1/orcl')
cursor = con.cursor()
create_table = """CREATE TABLE python_modules ( module_name VARCHAR2(1000) NOT NULL, file_path VARCHAR2(1000) NOT NULL )"""
cursor.execute(create_table)
M = []
M.append(('Module1', 'c:/1'))
M.append(('Module2', 'c:/2'))
M.append(('Module3', 'c:3'))
cursor.prepare("INSERT INTO python_modules(module_name, file_path) VALUES (:1, :2)")
cursor.executemany(None,M)
con.commit
cursor.execute("SELECT COUNT(*) FROM python_modules")
print(cursor.fetchone() [0])
Executing below query "select * from python_modules " should have the 3 records in Oracle SQL Developer tool
Change your commit to con.commit().
I'm trying to read the Table names from a database into a list using Pandas.read_sql.
I have tried different SQL queries found online:
cnxn = pyodbc.connect('DRIVER={Microsoft Access Driver (*.mdb)};DBQ=' + str(self.file_selected)+';Uid=Admin;Pwd=; ')
# sql = "SELECT * FROM SYS.TABLES" # tried this - also an error
sql = "SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA='database_name.MDB'"
self.TableNames = pd.io.sql.read_sql(sql, cnxn)
cnxn.close()
but I get an error that it can not find the file database_name.INFORMATION_SCHEMA.TABLES
what should I use for the sql query?
In MS Access, you can retrieve metadata on a database using its system table, MSysObjects. Below is a DML call to retrieve all table names:
SELECT MSysObjects.Name
FROM MsysObjects
WHERE ((MSysObjects.Type)=1)
ORDER BY MSysObjects.Name;
However, by default this will not work with external ODBC calls such as you do in Python as permission is not allowed. To resolve, consider two routes:
Grant Permission (for Admin user)
Inside the MSAccess.exe GUI, open database and run VBA subroutine (in standalone module) which only needs to be run once:
Public Sub GrantMSysPermission()
Dim strSQL As String
strSQL = "GRANT SELECT ON MSysObjects TO Admin;"
CurrentProject.Connection.Execute strSQL
End Sub
Once done run above query in pandas read_sql call.
Saved Table
Inside MS Access.exe GUI program, run below make-table query:
SELECT MSysObjects.Name
INTO DBTables
FROM MsysObjects
WHERE ((MSysObjects.Type)=1)
ORDER BY MSysObjects.Name;
Then in Python pandas, refer to new table:
cnxn = pyodbc.connect('DRIVER={{Microsoft Access Driver (*.mdb)}};DBQ=' + \
'{};Uid=Admin;Pwd=;'.format(str(self.file_selected)))
sql = "SELECT * DBTables"
self.TableNames = pd.io.sql.read_sql(sql, cnxn)
cnxn.close()
I have a SQL database which I wish to run a query on through python. I have the following code:
sql='select * from mf where frequency=220258.0;'
cur.execute(sql)
Where I use the same select command in sqlite3 directly it works, but through Python no database entries are outputted.
What am I doing wrong?
Consider this SQLite database.
$ sqlite3 so.sqlite3 .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE t (f1 integer, f2 text);
INSERT INTO "t" VALUES(1,'foo');
INSERT INTO "t" VALUES(2,'bar');
INSERT INTO "t" VALUES(3,'baz');
COMMIT;
Python connects and queries this database like this.
import sqlite3
con = sqlite3.connect("so.sqlite3")
cur = con.cursor()
cur.execute("select * from t where f1 = 2")
print(cur.fetchone())
Output:
(2, 'bar')
You have to use one of cur.fetchone(), cur.fetchall(), or cur.fetchmany() to get rows from the cursor. Just doing cur.execute() does not return the rows.