Develop Reference

Is it possible to view a postgresql table using Pandas?

def connectxmlDb(dbparams):
conn_string = "host='{}' dbname='{}' user='{}' password='{}' port='{}'"\
.format(dbparams['HOST'], dbparams['DB'], dbparams['USERNAME'], dbparams['PASSWORD'], dbparams['PORT'])
try:
conn = psycopg2.connect(conn_string)
pass
except Exception as err:
print('Connection to Database Failed : ERR : {}'.format(err))
return False
print('Connection to Database Success')
return conn
dbconn = connectxmlDb(params['DATABASE'])
The above mentioned is the code I use to connect to postgresql database
sql_statement = """ select a.slno, a.clientid, a.filename, a.user1_id, b.username, a.user2_id, c.username as username2, a.uploaded_ts, a.status_id
from masterdb.xmlform_joblist a
left outer join masterdb.auth_user b
on a.user1_id = b.id
left outer join masterdb.auth_user c
on a.user2_id = c.id
"""
cursor.execute(sql_statement)
result = cursor.fetchall()
This is my code to extract data from a postgres table using python.
I want to know if it is possible to view this data using Pandas
This is the code I used:
df = pd.read_sql_query(sql_statement, dbconn)
print(df.head(10))
but it is showing error.
C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-
packages\pandas\io\sql.py:762: UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy
warnings.warn(
Traceback (most recent call last):
File "C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 2023, in execute
cur.execute(*args, **kwargs)
psycopg2.errors.SyntaxError: syntax error at or near ";"
LINE 1: ...ELECT name FROM sqlite_master WHERE type='table' AND name=?;
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\Users\Lukmana\Desktop\user_count\usercount.py", line 55, in <module>
df = pd.read_sql_table(result, dbconn)
File "C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 286, in read_sql_table
if not pandas_sql.has_table(table_name):
File "C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 2200, in has_table
return len(self.execute(query, [name]).fetchall()) > 0
File "C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 2035, in execute
raise ex from exc
pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': syntax error at or near ";"
LINE 1: ...ELECT name FROM sqlite_master WHERE type='table' AND name=?;

How do I rectify the error in mysql-connector where I try to add multiple records at the same time into a MySQL Database?

I am trying to add multiple records into a database using mysql-connector-python. I was able to add one record initially but for multiple records, this error has been persistent.
class DataBase:
def __init__(self):
try:
self.connection = mysql.connector.connect(host='xx.xxx.xx.xx',
database='XXX',
user='XXXX',
password='xxxx')
except Error as error:
print("Failed to connect: {}".format(error))
def store_into_table(self, df):
mySql_insert_query = """INSERT INTO data ('data_id', 'a', 'b', 'c') VALUES ('%S','%S', '%S', '%S') """
records_to_insert = df
cursor = self.connection.cursor()
cursor.executemany(mySql_insert_query, records_to_insert)
self.connection.commit()
print(cursor.rowcount, " Records inserted successfully into data table")
The variable that I am passing to an instance of DataBase is:
data = [('101', 'name_1', '3', 'sample'), ('102', 'name_2', '5', 'sample_1')]
Exception being thrown:
Traceback (most recent call last):
File "/Users/dev/anaconda3/envs/sql/lib/python3.7/site-packages/mysql/connector/cursor_cext.py", line 317, in _batch_insert
"Not all parameters were used in the SQL statement")
ProgrammingError: Not all parameters were used in the SQL statement
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<ipython-input-123-44fdf194cd10>", line 1, in <module>
db.store_into_table(data)
File "/Users/dev/Detector/database.py", line 30, in store_into_table
cursor.executemany(mySql_insert_query, records_to_insert)
File "/Users/dev/anaconda3/envs/sql/lib/python3.7/site-packages/mysql/connector/cursor_cext.py", line 350, in executemany
stmt = self._batch_insert(operation, seq_params)
File "/Users/dev/anaconda3/envs/sql/lib/python3.7/site-packages/mysql/connector/cursor_cext.py", line 329, in _batch_insert
"Failed executing the operation; %s" % err)
InterfaceError: Failed executing the operation; Not all parameters were used in the SQL statement
Any clues as to why this isn't working?

Try this:
mySql_insert_query = """INSERT INTO data (data_id, a, b, c) VALUES (%s, %s, %s, %s) """

Python "cx_Oracle.InterfaceError: not a query" for Oracle Function return type cursor

Please help me to understand what I am doing wrong because with same value if I execute procedure which gives multiple lines of response but in python which is failing and giving exception in cursor call to function.
-----Oracle Function----------
FUNCTION get_details (id IN VARCHAR2, actNo IN VARCHAR2, code IN VARCHAR2)
RETURN SYS_REFCURSOR
----Oracle Function Logic----------
OPEN cur_msg_dtls FOR SELECT NAME,VALUE,CODE FROM TABLE
returns cur_msg_dtls
----Python Code--------
import cx_Oracle
if __name__ == '__main__':
con = cx_Oracle.connect('username/pass#host.com:1521/schema')
cur_var = con.cursor()
actNo = 'actNo';
code ='Code';
Id ='Id';
try:
cur_var.callfunc('SCHEMA.PKG.GET_DETAILS',cx_Oracle.CURSOR, (Id,actNo, actCd))
print ("hello")
print(cur_var)
print(type(cur_var))
for row in cur_var:
print(row)
except cx_Oracle.DatabaseError as e:
print (e)
finally:
cur_var.close()
con.close()
pass
----Error--------------
hello
<cx_Oracle.Cursor on <cx_Oracle.Connection to username/pass#host.com:1521/schema>>
<class 'cx_Oracle.Cursor'>
Traceback (most recent call last):
File "C:\Users\TestDBPRoc.py", line 26, in <module>
for row in cur_var:
cx_Oracle.InterfaceError: not a query

Issue in Model-View-Controller Pattern implementation

I am running an example for learning The Model-View-Controller Pattern in python, but the code is giving an error. I tried to debug the code, but I couldn't find the main root/cause. Removing close connection system works, but what is the issue of the code? Can you advise me what is wrong?
# Filename: mvc.py
import sqlite3
import types
class DefectModel:
def getDefectList(self, component):
query = '''select ID from defects where Component = '%s' ''' %component
defectlist = self._dbselect(query)
list = []
for row in defectlist:
list.append(row[0])
return list
def getSummary(self, id):
query = '''select summary from defects where ID = '%d' ''' % id
summary = self._dbselect(query)
for row in summary:
return row[0]
def _dbselect(self, query):
connection = sqlite3.connect('example.db')
cursorObj = connection.cursor()
results = cursorObj.execute(query)
connection.commit()
cursorObj.close()
return results
class DefectView:
def summary(self, summary, defectid):
print("#### Defect Summary for defect# %d ####\n %s" % (defectid,summary) )
def defectList(self, list, category):
print("#### Defect List for %s ####\n" % category )
for defect in list:
print(defect )
class Controller:
def __init__(self): pass
def getDefectSummary(self, defectid):
model = DefectModel()
view = DefectView()
summary_data = model.getSummary(defectid)
return view.summary(summary_data, defectid)
def getDefectList(self, component):
model = DefectModel()
view = DefectView()
defectlist_data = model.getDefectList(component)
return view.defectList(defectlist_data, component)
This is related run.py.
#run.py
import mvc
controller = mvc.Controller()
# Displaying Summary for defect id # 2
print(controller.getDefectSummary(2))
# Displaying defect list for 'ABC' Component print
controller.getDefectList('ABC')
If you need to create the database, it is available here:
# Filename: datbase.py
import sqlite3
import types
# Create a database in RAM
db = sqlite3.connect('example.db')
# Get a cursor object
cursor = db.cursor()
cursor.execute("drop table defects")
cursor.execute("CREATE TABLE defects(id INTEGER PRIMARY KEY, Component TEXT, Summary TEXT)")
cursor.execute("INSERT INTO defects VALUES (1,'XYZ','File doesn‘t get deleted')")
cursor.execute("INSERT INTO defects VALUES (2,'XYZ','Registry doesn‘t get created')")
cursor.execute("INSERT INTO defects VALUES (3,'ABC','Wrong title gets displayed')")
# Save (commit) the changes
db.commit()
# We can also close the connection if we are done with it.
# Just be sure any changes have been committed or they will be lost.
db.close()
My error is as below:
> Windows PowerShell Copyright (C) Microsoft Corporation. All rights
> reserved.
>
> PS E:\Projects\test> & python e:/Projects/test/mvc.py
> Traceback (most recent call last): File
> "e:/Projects/test/mvc.py", line 56, in <module>
> import mvc File "e:\Projects\test\mvc.py", line 65, in <module>
> cursor.execute("drop table defects") sqlite3.OperationalError: no such table: defects PS E:\Projects\test> & python
> e:/Projects/ramin/mvc.py Traceback (most recent call last):
> File "e:/Projects/test/mvc.py", line 56, in <module>
> import mvc File "e:\Projects\test\mvc.py", line 80, in <module>
> print(controller.getDefectSummary(2)) File "e:\Projects\test\mvc.py", line 44, in getDefectSummary
> summary_data = model.getSummary(defectid) File "e:\Projects\test\mvc.py", line 18, in getSummary
> for row in summary: sqlite3.ProgrammingError: Cannot operate on a closed cursor. PS E:\Projects\test>

I suspect that the problem is this line: cursor.execute("drop table defects")
Maybe you dropped that table in a previous run, and since it's no longer there, sqlite3 raises an OperationalError exception.
In your code there is a comment that says that you are using an in-memory sqlite database, but you are not. This is how you create an in-memory db:
db = sqlite3.connect(:memory:)
If you use an in-memory db you don't need to drop anything, since you are creating the db on the fly when you run your script.
Note: last year I wanted to understand the MVC better, so I wrote a series of articles about it. Here is the one where I use SQLite as a storage backend for my Model.

Problem Using Variable in Sqlite3 Statement

I have a problem with a Python 2.7 project.
I'm trying to set a variable to a value retrieved from an sqlite3 database, but I'm having trouble. Here is my code thus far, and the error I'm receiving. Yes, the connection opens just fine, and the table, columns, and indicated row are there as they should be.
import sqlite3 import Trailcrest
conn = sqlite3.connect('roster.paw')
c = conn.cursor()
def Lantern(AI):
"""Pulls all of the data for the selected user."""
Trailcrest.FireAutoHelp = c.execute("""select fireautohelp
from roster
where index = ?;""", (AI,) ).fetchall()
The error is:
> Traceback (most recent call last):
> File "<pyshell#4>", line 1, in <module> Lantern(1)
> File "C:\Users\user\MousePaw Games\Word4Word\PYM\Glyph.py", line 20,
> in Lantern
> Trailcrest.FireAutoHelp = c.execute("""select fireautohelp from roster where index = ?;""", (AI,)).fetchall()
> OperationalError: near "index": syntax error

As Thomas K mentions in a comment, index is a SQL keyword.
You can either rename that column, or enclose in backticks:
Trailcrest.FireAutoHelp = c.execute("""select fireautohelp
from roster
where `index` = ?;""", (AI,) ).fetchall()