Develop Reference

Postgres: invalid input syntax for type date

I have created a database and I am trying to fetch data from it. I have a class Query and inside the class I have a function that calls a table called forecasts. The function is as follows:
def forecast(self, provider: str, zone: str='Mainland',):
self.date_start = date_start)
self.date_end = (date_end)
self.df_forecasts = pd.DataFrame()
fquery = """
SELECT dp.name AS provider_name, lf.datetime_from AS date, fr.name AS run_name, lf.value AS value
FROM load_forecasts lf
INNER JOIN bidding_zones bz ON lf.zone_id = bz.zone_id
INNER JOIN data_providers dp ON lf.provider_id = dp.provider_id
INNER JOIN forecast_runs fr ON lf.run_id = fr.run_id
WHERE bz.name = '{zone}'
AND dp.name = '{provider}'
AND date(lf.datetime_from) BETWEEN '{self.date_start}' AND '{self.date_end}'
"""
df_forecasts = pd.read_sql_query(fquery, self.connection)
return df_forecasts
In the scripts that I run I am calling the Query class giving it my inputs
query = Query(date_start, date_end)
And the function
forecast_df = query.forecast(provider='Meteologica')
I run my script in the command line in the classic way
python myscript.py '2022-11-10' '2022-11-18'
My script shows the error
sqlalchemy.exc.DataError: (psycopg2.errors.InvalidDatetimeFormat) invalid input syntax for type date: "{self.date_start}"
LINE 9: AND date(lf.datetime_from) BETWEEN '{self.date_start...
when I use this syntax, but when I manually input the string for date_start and date_end it works.

I cannot find a way to solve the problem with sqlalchemy, so I opened a cursor with psycopg2.
# Returns the datetime, value and provider name and issue date of the forecasts in the load_forecasts table
# The dates range is specified by the user when the class is called
def forecast(self, provider: str, zone: str='Mainland',):
# Opens a cursor to get the data
cursor = self.connection.cursor()
# Query to run
query = """
SELECT dp.name, lf.datetime_from, fr.name, lf.value, lf.issue_date
FROM load_forecasts lf
INNER JOIN bidding_zones bz ON lf.zone_id = bz.zone_id
INNER JOIN data_providers dp ON lf.provider_id = dp.provider_id
INNER JOIN forecast_runs fr ON lf.run_id = fr.run_id
WHERE bz.name = %s
AND dp.name = %s
AND date(lf.datetime_from) BETWEEN %s AND %s
"""
# Execute the query, bring the data and close the cursor
cursor.execute(query, (zone, provider, self.date_start, self.date_end))
self.df_forecasts = cursor.fetchall()
cursor.close()
return self.df_forecasts
If anyone finds the answer with sqlalchemy, I would love to see it!

mock psycopg2 fetchone and fetchall to return different dataset doesn't work [duplicate]

I have this code segment in Python2:
def super_cool_method():
con = psycopg2.connect(**connection_stuff)
cur = con.cursor(cursor_factory=DictCursor)
cur.execute("Super duper SQL query")
rows = cur.fetchall()
for row in rows:
# do some data manipulation on row
return rows
that I'd like to write some unittests for. I'm wondering how to use mock.patch in order to patch out the cursor and connection variables so that they return a fake set of data? I've tried the following segment of code for my unittests but to no avail:
#mock.patch("psycopg2.connect")
#mock.patch("psycopg2.extensions.cursor.fetchall")
def test_super_awesome_stuff(self, a, b):
testing = super_cool_method()
But I seem to get the following error:
TypeError: can't set attributes of built-in/extension type 'psycopg2.extensions.cursor'

You have a series of chained calls, each returning a new object. If you mock just the psycopg2.connect() call, you can follow that chain of calls (each producing mock objects) via .return_value attributes, which reference the returned mock for such calls:
#mock.patch("psycopg2.connect")
def test_super_awesome_stuff(self, mock_connect):
expected = [['fake', 'row', 1], ['fake', 'row', 2]]
mock_con = mock_connect.return_value # result of psycopg2.connect(**connection_stuff)
mock_cur = mock_con.cursor.return_value # result of con.cursor(cursor_factory=DictCursor)
mock_cur.fetchall.return_value = expected # return this when calling cur.fetchall()
result = super_cool_method()
self.assertEqual(result, expected)
Because you hold onto references for the mock connect function, as well as the mock connection and cursor objects you can then also assert if they were called correctly:
mock_connect.assert_called_with(**connection_stuff)
mock_con.cursor.asset_called_with(cursor_factory=DictCursor)
mock_cur.execute.assert_called_with("Super duper SQL query")
If you don't need to test these, you could just chain up the return_value references to go straight to the result of cursor() call on the connection object:
#mock.patch("psycopg2.connect")
def test_super_awesome_stuff(self, mock_connect):
expected = [['fake', 'row', 1], ['fake', 'row' 2]]
mock_connect.return_value.cursor.return_value.fetchall.return_value = expected
result = super_cool_method()
self.assertEqual(result, expected)
Note that if you are using the connection as a context manager to automatically commit the transaction and you use as to bind the object returned by __enter__() to a new name (so with psycopg2.connect(...) as conn: # ...) then you'll need to inject an additional __enter__.return_value in the call chain:
mock_con_cm = mock_connect.return_value # result of psycopg2.connect(**connection_stuff)
mock_con = mock_con_cm.__enter__.return_value # object assigned to con in with ... as con
mock_cur = mock_con.cursor.return_value # result of con.cursor(cursor_factory=DictCursor)
mock_cur.fetchall.return_value = expected # return this when calling cur.fetchall()
The same applies to the result of with conn.cursor() as cursor:, the conn.cursor.return_value.__enter__.return_value object is assigned to the as target.

Since the cursor is the return value of con.cursor, you only need to mock the connection, then configure it properly. For example,
query_result = [("field1a", "field2a"), ("field1b", "field2b")]
with mock.patch('psycopg2.connect') as mock_connect:
mock_connect.cursor.return_value.fetchall.return_value = query_result
super_cool_method()

The following answer is the variation of above answers.
I was using django.db.connections cursor object.
So following code worked for me
#patch('django.db.connections')
def test_supercool_method(self, mock_connections):
query_result = [("field1a", "field2a"), ("field1b", "field2b")]
mock_connections.__getitem__.return_value.cursor.return_value.__enter__.return_value.fetchall.return_value = query_result
result = supercool_method()
self.assertIsInstance(result, list)

#patch("psycopg2.connect")
async def test_update_task_after_launch(fake_connection):
"""
"""
fake_update_count =4
fake_connection.return_value = Mock(cursor=lambda : Mock(execute=lambda x,y :"",
fetch_all=lambda:['some','fake','rows'],rowcount=fake_update_count,close=lambda:""))

Using Neo4j Transactions with Python Bolt Driver

I have some code that successfully creates a new node using the Python Bolt Neo4j driver. However, I cannot create new relationships in the same transaction.
I am using Python 2.7 with the Neo4j Bolt drive 1.7.2.
with conn.session() as session:
uuid = getNewUUID()
tx = None
try:
tx = session.begin_transaction()
stmt = "CREATE (a:{type} {{{uuid_attrib}: $uuid, {name_attrib}: $name, {desc_attrib}: $desc, {has_phi_attrib}: $has_phi}}) RETURN a.{uuid_attrib}".format(
type=ENTITY_NODE_NAME, uuid_attrib=UUID_ATTRIBUTE,
name_attrib=NAME_ATTRIBUTE, desc_attrib=DESCRIPTION_ATTRIBUTE,
has_phi_attrib=HAS_PHI_ATTRIBUTE)
#print "EXECUTING: " + stmt
tx.run(stmt, uuid=uuid, name=name, desc=description, has_phi=hasPHI)
create_relationship(tx, uuid, DERIVED_FROM_REL, parentUUID)
create_relationship(tx, uuid, LAB_CREATED_AT_REL, labCreatedUUID)
create_relationship(tx, uuid, CREATED_BY_REL, createdByUUID)
tx.commit()
return uuid
here is the create_relationship method:
def create_relationship(tx, startuuid, rel_label, enduuid):
try:
stmt = "MATCH (a),(b) WHERE a.uuid = '$startuuid' AND b.uuid = '$enduuid' CREATE (a)-[r:{rel_label}]->(b) RETURN type(r)".format(
rel_label=rel_label)
temp_stmt = stmt
temp_stmt = temp_stmt.replace("$startuuid", startuuid)
temp_stmt = temp_stmt.replace("$enduuid", enduuid)
print "EXECUTING: " + temp_stmt
result = tx.run(stmt,startuuid=startuuid, enduuid=enduuid)
The code successfully creates the node in Neo4j. However, the relationships are never created. I expected the relationships to be added to the node. If I copy and paste the relationship CREATE commands into the bolt web interface, the CREATE command works.

I am not sure if this is the exact issue but it looks like transaction tx is passed as value and create_relationship function creates its own local copy of tx so original tx is not modified by the function create_relationship.
When you commit the tx, transactions from create_relationship function are not committed as these are not part of the tx.
You should consider running these transactions in the calling function itself instead of create_relationship, use create_relationship or similar function to create and return the statement and run these statement in the calling function.
Function to get statement:
def get_relationship_statement(startuuid, rel_label, enduuid):
stmt = "MATCH (a),(b) WHERE a.uuid = '$startuuid' AND b.uuid = '$enduuid' CREATE (a)-[r:{rel_label}]->(b) RETURN type(r)".format(
rel_label=rel_label)
temp_stmt = stmt
temp_stmt = temp_stmt.replace("$startuuid", startuuid)
temp_stmt = temp_stmt.replace("$enduuid", enduuid)
print "Statement: " + temp_stmt
return stmt
Replace
create_relationship(tx, uuid, DERIVED_FROM_REL, parentUUID)
with
tx.run(get_relationship_statement(uuid, DERIVED_FROM_REL, parentUUID),startuuid=uuid, enduuid=parentUUID)

Issues returning python parameter to main function

problem: Im trying to extract values out of a mysql search to use later within the code.
I have setup a mysql function (connector_mysql) that i use to connect/run the mysql commands.
The problem i'm having is returning these values back out of the mysql function to the rest of the code.
There are two scenarios that i've tried which i think should work but dont when run.
full code sample below...
1. In the mysql function result acts like a dictionary if - using just..
result = cursor.fetchone(variable1, variable2, variable3)
return result
AND in the main function
result1 = connector_mysql(subSerialNum, ldev, today_date)
print(result1)
This appears to work and when printing result1 i get a dictionary looking output:
{'ldev_cap': '0938376656', 'ldev_usedcap': '90937763873'}
HOWEVER...
I cant then use dictionary methods to get or separate the values out.. eg
used_capacity = result1['ldev_cap']
which i would have expected that now 'used_capacity' to represent or equal 0938376656.
Instead i get an error about the object is not able to be subscripted...?
Error below:
File "/Users/graham/PycharmProjects/VmExtrat/VmExtract.py", line 160, in openRead
used_capacity = result1['ldev_cap']
TypeError: 'NoneType' object is not subscriptable
In the mysql function the result acts like a dictionary if I manipulate it and try and return multiple values with the tuple concept - using..
cursor.execute(sql_query, {'serial': subSerialNum, 'lun': ldev, 'todayD': today_date})
result = cursor.fetchone()
while result:
ldev_cap = result['ldev_cap']
ldev_usdcap = result['ldev_usdcap']
return ldev_cap, ldev_usdcap
Here, result acts like a dictionary and i'm able to assign a parameter to the key like:
ldev_cap = result['ldev_cap']
and if you print ldev_cap you get the correct figure...
If I return one figure, the main function line of:
result1 = connector_mysql(subSerialNum, ldev, today_date)
it Works....
HOWEVER...
When then trying to return multiple parameters from the mysql function by doing
return ldev_cap, ldev_usdcap
and in the main function:
capacity, usd_capacity = connector_mysql(subSerialNum, ldev, today_date)
I get errors again
File "/Users/graham/PycharmProjects/VmExtrat/VmExtract.py", line 156, in openRead
capacity, usd_capacity = connector_mysql(subSerialNum, ldev, today_date)
TypeError: 'NoneType' object is not iterable
I think i'm doing the right thing with the dictionary and the tuple but i'm obviously missing something or not doing it correctly, as i need to do this with 4-5 paramaters per sql query,
I didn't want to do multiple querys for the same thing to get the individual paramaters out.
Any help or suggestions would be greatly welcomed...
full code below.
main code:
capacity, usd_capacity = connector_mysql(subSerialNum, ldev, today_date)
print(capacity)
print(usd_capacity)
def connector_mysql(subSerialNum, ldev, today_date):
import pymysql.cursors
db_server = 'localhost'
db_name = 'CBDB'
db_pass = 'secure_password'
db_user = 'user1'
sql_query = (
"SELECT ldev_cap, ldev_usdcap FROM Ldevs WHERE sub_serial=%(serial)s "
"and ldev_id=%(lun)s and data_date=%(todayD)s")
connection = pymysql.connect(host=db_server,
user=db_user,
password=db_pass,
db=db_name,
cursorclass=pymysql.cursors.DictCursor)
try:
with connection.cursor() as cursor:
cursor.execute(sql_query, {'serial': subSerialNum, 'lun': ldev, 'todayD': today_date})
result = cursor.fetchone()
#return result #used for returning dict
while result:
ldev_cap = result['ldev_cap']
ldev_usdcap = result['ldev_usdcap']
print(result)
return ldev_cap, ldev_usdcap
finally:
connection.close()

cx_Oracle: How can I receive each row as a dictionary?

By default, cx_Oracle returns each row as a tuple.
>>> import cx_Oracle
>>> conn=cx_Oracle.connect('scott/tiger')
>>> curs=conn.cursor()
>>> curs.execute("select * from foo");
>>> curs.fetchone()
(33, 'blue')
How can I return each row as a dictionary?

You can override the cursor's rowfactory method. You will need to do this each time you perform the query.
Here's the results of the standard query, a tuple.
curs.execute('select * from foo')
curs.fetchone()
(33, 'blue')
Returning a named tuple:
def makeNamedTupleFactory(cursor):
columnNames = [d[0].lower() for d in cursor.description]
import collections
Row = collections.namedtuple('Row', columnNames)
return Row
curs.rowfactory = makeNamedTupleFactory(curs)
curs.fetchone()
Row(x=33, y='blue')
Returning a dictionary:
def makeDictFactory(cursor):
columnNames = [d[0] for d in cursor.description]
def createRow(*args):
return dict(zip(columnNames, args))
return createRow
curs.rowfactory = makeDictFactory(curs)
curs.fetchone()
{'Y': 'brown', 'X': 1}
Credit to Amaury Forgeot d'Arc:
http://sourceforge.net/p/cx-oracle/mailman/message/27145597

A very short version:
curs.rowfactory = lambda *args: dict(zip([d[0] for d in curs.description], args))
Tested on Python 3.7.0 & cx_Oracle 7.1.2

Old question but adding some helpful links with a Python recipe
According to cx_Oracle documentation:
Cursor.rowfactory
This read-write attribute specifies a method to call for each row that
is retrieved from the database. Ordinarily a tuple is returned for
each row but if this attribute is set, the method is called with the
tuple that would normally be returned, and the result of the method is
returned instead.
The cx_Oracle - Python Interface for Oracle Database Also points to GitHub repository for lots of helpful sample examples. Please check GenericRowFactory.py.
Googled: This PPT can be further helpful: [PDF]CON6543 Python and Oracle Database - RainFocus
Recipe
Django database backend for Oracle under the hood uses cx_Oracle. In earlier versions ( Django 1.11- ) they have written _rowfactory(cursor, row) That also cast cx_Oracle's numeric data types into relevant Python data and strings into unicode.
If you have installed Django Please check base.py as follows:
$ DJANGO_DIR="$(python -c 'import django, os; print(os.path.dirname(django.__file__))')"
$ vim $DJANGO_DIR/db/backends/oracle/base.py
One can borrow _rowfactory() from $DJANGO_DIR/db/backends/oracle/base.py and can apply below decorator naming to make it return namedtuple instead of simple tuple.
mybase.py
import functools
from itertools import izip, imap
from operator import itemgetter
from collections import namedtuple
import cx_Oracle as Database
import decimal
def naming(rename=False, case=None):
def decorator(rowfactory):
#functools.wraps(rowfactory)
def decorated_rowfactory(cursor, row, typename="GenericRow"):
field_names = imap(case, imap(itemgetter(0), cursor.description))
return namedtuple(typename, field_names)._make(rowfactory(cursor, row))
return decorated_rowfactory
return decorator
use it as:
#naming(rename=False, case=str.lower)
def rowfactory(cursor, row):
casted = []
....
....
return tuple(casted)
oracle.py
import cx_Oracle as Database
from cx_Oracle import *
import mybase
class Cursor(Database.Cursor):
def execute(self, statement, args=None):
prepareNested = (statement is not None and self.statement != statement)
result = super(self.__class__, self).execute(statement, args or [])
if prepareNested:
if self.description:
self.rowfactory = lambda *row: mybase.rowfactory(self, row)
return result
def close(self):
try:
super(self.__class__, self).close()
except Database.InterfaceError:
"already closed"
class Connection(Database.Connection):
def cursor(self):
Cursor(self)
connect = Connection
Now, instead of import cx_oracle import oracle in user script as:
user.py
import oracle
dsn = oracle.makedsn('HOSTNAME', 1521, service_name='dev_server')
db = connect('username', 'password', dsn)
cursor = db.cursor()
cursor.execute("""
SELECT 'Grijesh' as FirstName,
'Chauhan' as LastName,
CAST('10560.254' AS NUMBER(10, 2)) as Salary
FROM DUAL
""")
row = cursor.fetchone()
print ("First Name is %s" % row.firstname) # => Grijesh
print ("Last Name is %s" % row.lastname) # => Chauhan
print ("Salary is %r" % row.salary) # => Decimal('10560.25')
Give it a Try!!

Building up on answer by #maelcum73 :
curs.rowfactory = lambda *args: dict(zip([d[0] for d in curs.description], args))
The issue with this solution is that you need to re-set this after every execution.
Going one step further, you can create a shell around the cursor object like so:
class dictcur(object):
# need to monkeypatch the built-in execute function to always return a dict
def __init__(self, cursor):
self._original_cursor = cursor
def execute(self, *args, **kwargs):
# rowfactory needs to be set AFTER EACH execution!
self._original_cursor.execute(*args, **kwargs)
self._original_cursor.rowfactory = lambda *a: dict(
zip([d[0] for d in self._original_cursor.description], a)
)
# cx_Oracle's cursor's execute method returns a cursor object
# -> return the correct cursor in the monkeypatched version as well!
return self._original_cursor
def __getattr__(self, attr):
# anything other than the execute method: just go straight to the cursor
return getattr(self._original_cursor, attr)
dict_cursor = dictcur(cursor=conn.cursor())
Using this dict_cursor, every subsequent dict_cursor.execute() call will return a dictionary. Note: I tried monkeypatching the execute method directly, however that was not possible because it is a built-in method.