Develop Reference

Is it possible to view a postgresql table using Pandas?

def connectxmlDb(dbparams):
conn_string = "host='{}' dbname='{}' user='{}' password='{}' port='{}'"\
.format(dbparams['HOST'], dbparams['DB'], dbparams['USERNAME'], dbparams['PASSWORD'], dbparams['PORT'])
try:
conn = psycopg2.connect(conn_string)
pass
except Exception as err:
print('Connection to Database Failed : ERR : {}'.format(err))
return False
print('Connection to Database Success')
return conn
dbconn = connectxmlDb(params['DATABASE'])
The above mentioned is the code I use to connect to postgresql database
sql_statement = """ select a.slno, a.clientid, a.filename, a.user1_id, b.username, a.user2_id, c.username as username2, a.uploaded_ts, a.status_id
from masterdb.xmlform_joblist a
left outer join masterdb.auth_user b
on a.user1_id = b.id
left outer join masterdb.auth_user c
on a.user2_id = c.id
"""
cursor.execute(sql_statement)
result = cursor.fetchall()
This is my code to extract data from a postgres table using python.
I want to know if it is possible to view this data using Pandas
This is the code I used:
df = pd.read_sql_query(sql_statement, dbconn)
print(df.head(10))
but it is showing error.
C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-
packages\pandas\io\sql.py:762: UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy
warnings.warn(
Traceback (most recent call last):
File "C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 2023, in execute
cur.execute(*args, **kwargs)
psycopg2.errors.SyntaxError: syntax error at or near ";"
LINE 1: ...ELECT name FROM sqlite_master WHERE type='table' AND name=?;
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\Users\Lukmana\Desktop\user_count\usercount.py", line 55, in <module>
df = pd.read_sql_table(result, dbconn)
File "C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 286, in read_sql_table
if not pandas_sql.has_table(table_name):
File "C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 2200, in has_table
return len(self.execute(query, [name]).fetchall()) > 0
File "C:\Users\Lukmana\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 2035, in execute
raise ex from exc
pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': syntax error at or near ";"
LINE 1: ...ELECT name FROM sqlite_master WHERE type='table' AND name=?;

Executing python function in postgres

I'm trying to run a python function on the cursor.execute parameter but it just throws me this error.
I'm using psycopg2
Traceback (most recent call last):
File "cliente.py", line 55, in <module>
cursorDB.execute(get_datos_animal('falsa'))
psycopg2.errors.UndefinedColumn: column "falsa" does not exist
LINE 1: ...e, clasificacion FROM animales WHERE animales.hierro = falsa
and my python function is this one
def get_datos_animal(hierro_v):
return "SELECT hierro, registro, nombre, fecha_nacimiento, raza, sexo, hierro_madre, hierro_padre, clasificacion FROM animales WHERE animales.hierro = " + str(hierro_v)
any idea what i´m doing wrong?
Have several functions like this with same errors.

Use the automatic parameter quoting provided by your connection to ensure that values in queries are always quoted correctly, and to avoid SQL injection attacks.
stmt = """SELECT hierro, registro, nombre, fecha_nacimiento, raza, sexo, hierro_madre, hierro_padre, clasificacion
FROM animales
WHERE animales.hierro = %s"""
cursor.execute(stmt, (hierro_v,))

In postgres if you pass value without quotes it will treat that as column name.
Try this:
def get_datos_animal(hierro_v):
return "SELECT hierro, registro, nombre, fecha_nacimiento, raza, sexo, hierro_madre, hierro_padre, clasificacion FROM animales WHERE animales.hierro = '"+str(hierro_v)+"'"

Using Python and WMI Queries to get a list of running services

I am attempting to get a list of services that are running on a windows machine with python.
My Code:
import wmi
c = wmi.WMI()
wql = "SELECT * FROM Win32_Service WHERE State = ""Running"""
for x in c.query(wql):
print(x)
I am getting an error and I do not understand why. I have a few other wql statements in my script and they seem to be working fine.
Error:
Traceback (most recent call last):
File "C:/Users/i861470/Desktop/Scripts/test.py", line 79, in <module>
for x in c.query(wql):
File "C:\Users\i861470\Desktop\Scripts\venv\lib\site-packages\wmi.py", line 1009, in query
return [ _wmi_object (obj, instance_of, fields) for obj in self._raw_query(wql) ]
File "C:\Users\i861470\Desktop\Scripts\venv\lib\site-packages\wmi.py", line 1009, in <listcomp>
return [ _wmi_object (obj, instance_of, fields) for obj in self._raw_query(wql) ]
File "C:\Users\i861470\Desktop\Scripts\venv\lib\site- packages\win32\com\client\dynamic.py", line 280, in __getitem__
return self._get_good_object_(self._enum_.__getitem__(index))
File "C:\Users\i861470\Desktop\Scripts\venv\lib\site-packages\win32\com\client\util.py", line 41, in __getitem__
return self.__GetIndex(index)
File "C:\Users\i861470\Desktop\Scripts\venv\lib\site-packages\win32\com\client\util.py", line 62, in __GetIndex
result = self._oleobj_.Next(1)
win32.types.com_error: (-2147217385, 'OLE error 0x80041017', None, None)

wql = "SELECT * FROM Win32_Service WHERE State = ""Running"""
results to an invalid WQL query (checked using print(wql))
SELECT * FROM Win32_Service WHERE State = Running
You need
wql = 'SELECT * FROM Win32_Service WHERE State = "Running"'
which results to a valid WQL query (read WHERE Clause docs)
SELECT * FROM Win32_Service WHERE State = "Running"
BTW, you may use string literals, such as "Running" or 'Running', in a WHERE clause. Hence, the following WQL query works as well:
wql = "SELECT * FROM Win32_Service WHERE State = 'Running'"

Peewee raw SELECT sql with multiple programmatic filter parameters

I am facing a problem returning the users from this get_users() function. Here is the code:
I am using Peewee, Pymysql and MySQL
def get_users(self, filter_columns=None, parameters=[], operator=None, ):
#Define the operator to be used in the WHERE statement
if operator:
operator = operator
else:
operator = '&'
#Contruct the sql_where statement
where = ''
if filter_columns:
for field in filter_columns:
where = where + field + '=%s, '
if len(where)>2:
where = where[:-2]
#Build Parameter list
param_list = ''
for param in parameters:
param_list = param_list + param + ", "
if len(param_list)>2:
param_list = param_list[:-2]
#Select the users and return.
sql = "SELECT * FROM user WHERE " + where
user = U.raw(sql, param_list)
return user
When i call the function like this:
users = user.get_users(filter_columns=['first_name', 'status'], parameters=['awa', 'active'], operator='|')
print(users)
for u in users:
print(u.first_name, u.last_name)
This is what i get as result:
Traceback (most recent call last):
File "D:/projects/micro bank/tests/smanager/randomtest.py", line 10, in <module>
for u in users:
File "C:\Python34\lib\site-packages\peewee.py", line 2963, in __iter__
return iter(self.execute())
File "C:\Python34\lib\site-packages\peewee.py", line 2959, in execute
self._qr = QRW(self.model_class, self._execute(), None)
File "C:\Python34\lib\site-packages\peewee.py", line 2902, in _execute
return self.database.execute_sql(sql, params, self.require_commit)
File "C:\Python34\lib\site-packages\peewee.py", line 3758, in execute_sql
cursor.execute(sql, params or ())
File "C:\Python34\lib\site-packages\pymysql\cursors.py", line 164, in execute
query = self.mogrify(query, args)
File "C:\Python34\lib\site-packages\pymysql\cursors.py", line 143, in mogrify
query = query % self._escape_args(args, conn)
TypeError: not enough arguments for format string
When i print out the user returned, i get this:
<class 'common.models.User'> SELECT * FROM user WHERE first_name=%s, status=%s ['awa, active']
From observation, the problem comes from this last area ['awa, active'] which is supposed to be ['awa', 'active']
The problem now is establishing a parameter_list that when i use it, it should print out like this ['awa', 'active']
Thanks for assistance.

I assume you need param_list to be ['awa', 'active']
Try this:
param_list = []
for param in parameters:
param_list.append(param)
# Since `param_list` in needed in the format `'"awa", "active"'`
param_list = ', '.join('"{0}"'.format(w) for w in param_list)
instead of
param_list = ''
for param in parameters:
param_list = param_list + param + ", "

Impossible to insert data with PyMySQL when I use parameter

I'm currently working on a basic query which insert data depending on the input parameters, and I'm unable to perfom it.
cur.execute("INSERT INTO foo (bar1, bar2) values (?, ?)", (foo1, foo2))
I have this error message:
Exception in Tkinter callback Traceback (most recent call last):
File "/usr/lib/python3.2/tkinter/init.py", line 1426, in call
return self.func(*args) File "test.py", line 9, in register
cur.execute("INSERT INTO foo (bar1, bar2) values (?,?)", (foo1, foo2)) File
"/usr/local/lib/python3.2/dist-packages/pymysql/cursors.py", line 108,
in execute
query = query % escaped_args TypeError: unsupported operand type(s) for %: 'bytes' and 'tuple'
foo1 and foo2 are both string type. I tried with %s, same error.

It seems like a bug in cursors.py. As suggested here and here you should replace this line in cursors.py:
query = query % conn.escape(args)
With this:
query = query.decode(charset) % conn.escape(args)
In case it didn't work try this one instead:
query = query.decode(charset) % escaped_args