Develop Reference

Postgres: invalid input syntax for type date

I have created a database and I am trying to fetch data from it. I have a class Query and inside the class I have a function that calls a table called forecasts. The function is as follows:
def forecast(self, provider: str, zone: str='Mainland',):
self.date_start = date_start)
self.date_end = (date_end)
self.df_forecasts = pd.DataFrame()
fquery = """
SELECT dp.name AS provider_name, lf.datetime_from AS date, fr.name AS run_name, lf.value AS value
FROM load_forecasts lf
INNER JOIN bidding_zones bz ON lf.zone_id = bz.zone_id
INNER JOIN data_providers dp ON lf.provider_id = dp.provider_id
INNER JOIN forecast_runs fr ON lf.run_id = fr.run_id
WHERE bz.name = '{zone}'
AND dp.name = '{provider}'
AND date(lf.datetime_from) BETWEEN '{self.date_start}' AND '{self.date_end}'
"""
df_forecasts = pd.read_sql_query(fquery, self.connection)
return df_forecasts
In the scripts that I run I am calling the Query class giving it my inputs
query = Query(date_start, date_end)
And the function
forecast_df = query.forecast(provider='Meteologica')
I run my script in the command line in the classic way
python myscript.py '2022-11-10' '2022-11-18'
My script shows the error
sqlalchemy.exc.DataError: (psycopg2.errors.InvalidDatetimeFormat) invalid input syntax for type date: "{self.date_start}"
LINE 9: AND date(lf.datetime_from) BETWEEN '{self.date_start...
when I use this syntax, but when I manually input the string for date_start and date_end it works.

I cannot find a way to solve the problem with sqlalchemy, so I opened a cursor with psycopg2.
# Returns the datetime, value and provider name and issue date of the forecasts in the load_forecasts table
# The dates range is specified by the user when the class is called
def forecast(self, provider: str, zone: str='Mainland',):
# Opens a cursor to get the data
cursor = self.connection.cursor()
# Query to run
query = """
SELECT dp.name, lf.datetime_from, fr.name, lf.value, lf.issue_date
FROM load_forecasts lf
INNER JOIN bidding_zones bz ON lf.zone_id = bz.zone_id
INNER JOIN data_providers dp ON lf.provider_id = dp.provider_id
INNER JOIN forecast_runs fr ON lf.run_id = fr.run_id
WHERE bz.name = %s
AND dp.name = %s
AND date(lf.datetime_from) BETWEEN %s AND %s
"""
# Execute the query, bring the data and close the cursor
cursor.execute(query, (zone, provider, self.date_start, self.date_end))
self.df_forecasts = cursor.fetchall()
cursor.close()
return self.df_forecasts
If anyone finds the answer with sqlalchemy, I would love to see it!

Why is Twisted's adbapi failing to recover data from within unittests?

Overview
Context
I am writing unit tests for some higher-order logic that depends on writing to an SQLite3 database. For this I am using twisted.trial.unittest and twisted.enterprise.adbapi.ConnectionPool.
Problem statement
I am able to create a persistent sqlite3 database and store data therein. Using sqlitebrowser, I am able to verify that the data has been persisted as expected.
The issue is that calls to t.e.a.ConnectionPool.run* (e.g.: runQuery) return an empty set of results, but only when called from within a TestCase.
Notes and significant details
The problem I am experiencing occurs only within Twisted's trial framework. My first attempt at debugging was to pull the database code out of the unit test and place it into an independent test/debug script. Said script works as expected while the unit test code does not (see examples below).
Case 1: misbehaving unit test
init.sql
This is the script used to initialize the database. There are no (apparent) errors stemming from this file.
CREATE TABLE ajxp_changes ( seq INTEGER PRIMARY KEY AUTOINCREMENT, node_id NUMERIC, type TEXT, source TEXT, target TEXT, deleted_md5 TEXT );
CREATE TABLE ajxp_index ( node_id INTEGER PRIMARY KEY AUTOINCREMENT, node_path TEXT, bytesize NUMERIC, md5 TEXT, mtime NUMERIC, stat_result BLOB);
CREATE TABLE ajxp_last_buffer ( id INTEGER PRIMARY KEY AUTOINCREMENT, type TEXT, location TEXT, source TEXT, target TEXT );
CREATE TABLE ajxp_node_status ("node_id" INTEGER PRIMARY KEY NOT NULL , "status" TEXT NOT NULL DEFAULT 'NEW', "detail" TEXT);
CREATE TABLE events (id INTEGER PRIMARY KEY AUTOINCREMENT, type text, message text, source text, target text, action text, status text, date text);
CREATE TRIGGER LOG_DELETE AFTER DELETE ON ajxp_index BEGIN INSERT INTO ajxp_changes (node_id,source,target,type,deleted_md5) VALUES (old.node_id, old.node_path, "NULL", "delete", old.md5); END;
CREATE TRIGGER LOG_INSERT AFTER INSERT ON ajxp_index BEGIN INSERT INTO ajxp_changes (node_id,source,target,type) VALUES (new.node_id, "NULL", new.node_path, "create"); END;
CREATE TRIGGER LOG_UPDATE_CONTENT AFTER UPDATE ON "ajxp_index" FOR EACH ROW BEGIN INSERT INTO "ajxp_changes" (node_id,source,target,type) VALUES (new.node_id, old.node_path, new.node_path, CASE WHEN old.node_path = new.node_path THEN "content" ELSE "path" END);END;
CREATE TRIGGER STATUS_DELETE AFTER DELETE ON "ajxp_index" BEGIN DELETE FROM ajxp_node_status WHERE node_id=old.node_id; END;
CREATE TRIGGER STATUS_INSERT AFTER INSERT ON "ajxp_index" BEGIN INSERT INTO ajxp_node_status (node_id) VALUES (new.node_id); END;
CREATE INDEX changes_node_id ON ajxp_changes( node_id );
CREATE INDEX changes_type ON ajxp_changes( type );
CREATE INDEX changes_node_source ON ajxp_changes( source );
CREATE INDEX index_node_id ON ajxp_index( node_id );
CREATE INDEX index_node_path ON ajxp_index( node_path );
CREATE INDEX index_bytesize ON ajxp_index( bytesize );
CREATE INDEX index_md5 ON ajxp_index( md5 );
CREATE INDEX node_status_status ON ajxp_node_status( status );
test_sqlite.py
This is the unit test class that fails unexpectedly. TestStateManagement.test_db_clean passes, indicated that the tables were properly created. TestStateManagement.test_inode_create fails, reporitng that zero results were retrieved.
import os.path as osp
from twisted.internet import defer
from twisted.enterprise import adbapi
import sqlengine # see below
class TestStateManagement(TestCase):
def setUp(self):
self.meta = mkdtemp()
self.db = adbapi.ConnectionPool(
"sqlite3", osp.join(self.meta, "db.sqlite"), check_same_thread=False,
)
self.stateman = sqlengine.StateManager(self.db)
with open("init.sql") as f:
script = f.read()
self.d = self.db.runInteraction(lambda c, s: c.executescript(s), script)
def tearDown(self):
self.db.close()
del self.db
del self.stateman
del self.d
rmtree(self.meta)
#defer.inlineCallbacks
def test_db_clean(self):
"""Canary test to ensure that the db is initialized in a blank state"""
yield self.d # wait for db to be initialized
q = "SELECT name FROM sqlite_master WHERE type='table' AND name=?;"
for table in ("ajxp_index", "ajxp_changes"):
res = yield self.db.runQuery(q, (table,))
self.assertTrue(
len(res) == 1,
"table {0} does not exist".format(table)
)
#defer.inlineCallbacks
def test_inode_create_file(self):
yield self.d
path = osp.join(self.ws, "test.txt")
with open(path, "wt") as f:
pass
inode = mk_dummy_inode(path)
yield self.stateman.create(inode, directory=False)
entry = yield self.db.runQuery("SELECT * FROM ajxp_index")
emsg = "got {0} results, expected 1. Are canary tests failing?"
lentry = len(entry)
self.assertTrue(lentry == 1, emsg.format(lentry))
sqlengine.py
These are the artefacts being tested by the above unit tests.
def values_as_tuple(d, *param):
"""Return the values for each key in `param` as a tuple"""
return tuple(map(d.get, param))
class StateManager:
"""Manages the SQLite database's state, ensuring that it reflects the state
of the filesystem.
"""
log = Logger()
def __init__(self, db):
self._db = db
def create(self, inode, directory=False):
params = values_as_tuple(
inode, "node_path", "bytesize", "md5", "mtime", "stat_result"
)
directive = (
"INSERT INTO ajxp_index (node_path,bytesize,md5,mtime,stat_result) "
"VALUES (?,?,?,?,?);"
)
return self._db.runOperation(directive, params)
Case 2: bug disappears outside of twisted.trial
#! /usr/bin/env python
import os.path as osp
from tempfile import mkdtemp
from twisted.enterprise import adbapi
from twisted.internet.task import react
from twisted.internet.defer import inlineCallbacks
INIT_FILE = "example.sql"
def values_as_tuple(d, *param):
"""Return the values for each key in `param` as a tuple"""
return tuple(map(d.get, param))
def create(db, inode):
params = values_as_tuple(
inode, "node_path", "bytesize", "md5", "mtime", "stat_result"
)
directive = (
"INSERT INTO ajxp_index (node_path,bytesize,md5,mtime,stat_result) "
"VALUES (?,?,?,?,?);"
)
return db.runOperation(directive, params)
def init_database(db):
with open(INIT_FILE) as f:
script = f.read()
return db.runInteraction(lambda c, s: c.executescript(s), script)
#react
#inlineCallbacks
def main(reactor):
meta = mkdtemp()
db = adbapi.ConnectionPool(
"sqlite3", osp.join(meta, "db.sqlite"), check_same_thread=False,
)
yield init_database(db)
# Let's make sure the tables were created as expected and that we're
# starting from a blank slate
res = yield db.runQuery("SELECT * FROM ajxp_index LIMIT 1")
assert not res, "database is not empty [ajxp_index]"
res = yield db.runQuery("SELECT * FROM ajxp_changes LIMIT 1")
assert not res, "database is not empty [ajxp_changes]"
# The details of this are not important. Suffice to say they (should)
# conform to the DB schema for ajxp_index.
test_data = {
"node_path": "/this/is/some/arbitrary/path.ext",
"bytesize": 0,
"mtime": 179273.0,
"stat_result": b"this simulates a blob of raw binary data",
"md5": "d41d8cd98f00b204e9800998ecf8427e", # arbitrary
}
# store the test data in the ajxp_index table
yield create(db, test_data)
# test if the entry exists in the db
entry = yield db.runQuery("SELECT * FROM ajxp_index")
assert len(entry) == 1, "got {0} results, expected 1".format(len(entry))
print("OK")
Closing remarks
Again, upon checking with sqlitebrowser, it seems as though the data is being written to db.sqlite, so this looks like a retrieval problem. From here, I'm sort of stumped... any ideas?
EDIT
This code will produce an inode that that can be used for testing.
def mk_dummy_inode(path, isdir=False):
return {
"node_path": path,
"bytesize": osp.getsize(path),
"mtime": osp.getmtime(path),
"stat_result": dumps(stat(path), protocol=4),
"md5": "directory" if isdir else "d41d8cd98f00b204e9800998ecf8427e",
}

Okay, it turns out that this is a bit of a tricky one. Running the tests in isolation (as was posted to this question) makes it such that the bug only rarely occurs. However, when running in the context of an entire test suite, it fails almost 100% of the time.
I added yield task.deferLater(reactor, .00001, lambda: None) after writing to the db and before reading from the db, and this solves the issue.
From there, I suspected this might be a race condition stemming from the connection pool and sqlite's limited concurrency-tolerance. I tried setting the cb_min and cb_max parameters to ConnectionPool to 1, and this also solved the issue.
In short: it seems as though sqlite doesn't play very nicely with multiple connections, and that the appropriate fix is to avoid concurrency to the extent possible.

If you take a look at your setUp function, you're returning self.db.runInteraction(...), which returns a deferred. As you've noted, you assume that it waits for the deferred to finish. However this is not the case and it's a trap that most fall victim to (myself included). I'll be honest with you, for situations like this, especially for unit tests, I just execute the synchronous code outside the TestCase class to initialize the database. For example:
def init_db():
import sqlite3
conn = sqlite3.connect('db.sqlite')
c = conn.cursor()
with open("init.sql") as f:
c.executescript(f.read())
init_db() # call outside test case
class TestStateManagement(TestCase):
"""
My test cases
"""
Alternatively, you could decorate the setup and yield runOperation(...) but something tells me that it wouldn't work... In any case, it's surprising that no errors were raised.
PS
I've been eyeballing this question for a while and it's been in the back of my head for days now. A potential reason for this finally dawned on me at nearly 1am. However, I'm too tired/lazy to actually test this out :D but it's a pretty damn good hunch. I'd like to commend you on your level of detail in this question.

MySQL Python connector does not commit

I have this function that it's supposed should receive json and store the values on a RDS MySQL db.
def saveMetric(metrics):
cnx = RDS_Connect()
cursor = cnx.cursor()
jsonMetrics = json.loads(metrics)
#print type(jsonMetrics['Metrics'])
# Every 2000 registries, the script will start overriding values
persistance = 2000
save_metrics_query = (
"REPLACE INTO metrics "
"SET metric_seq = (SELECT COALESCE(MAX(row_id), 0) %% %(persistance)d + 1 FROM metrics AS m), "
"instance_id = \'%(instance_id)s\', "
"service = \'%(service)s\' , "
"metric_name = \'%(metric_name)s\', "
"metric_value = %(metric_value)f"
)
for metric in jsonMetrics['Metrics']:
formatData = {}
formatData['persistance'] = persistance
formatData['instance_id'] = arguments.dimensionValue
formatData['service'] = jsonMetrics['Service']
formatData['metric_name'] = metric
formatData['metric_value'] = jsonMetrics['Metrics'][metric]
print save_metrics_query % formatData
try:
cursor.execute(save_metrics_query, formatData, multi=True)
logger('info','Metrics were saved successfully!')
cnx.commit()
except mysql.connector.Error as err:
logger('error', "Something went wrong: %s" % err)
cursor.close()
cnx.close()
RDS_Connect() was already tested and it works just fine. The problem is that after running the function, the data is not saved to the DB. I think there is a problem with the commit but I don't see any errors or warning message. If I run the query manually, the data gets stored.
Here is the query that runs after parsing the json:
REPLACE INTO metrics SET metric_seq = (SELECT COALESCE(MAX(row_id), 0) % 2000 + 1 FROM metrics AS m), instance_id = 'i-03932937bd67622c4', service = 'AWS/EC2' , metric_name = 'CPUUtilization', metric_value = 0.670000
If it helps, this is the json that the function receives:
{
"Metrics": {
"CPUUtilization": 1.33,
"NetworkIn": 46428.0,
"NetworkOut": 38772.0
},
"Id": "i-03932937bd67622c4",
"Service": "AWS/EC2"
}
I'd appreciate some help.
Regards!
UPDATE:
I found that the problem was related to the formatting codes on the query template.
I re wrote the function like this:
def saveMetric(metrics):
cnx = RDS_Connect()
jsonMetrics = json.loads(metrics)
print json.dumps(jsonMetrics,indent=4)
persistance = 2000
row_id_query_template = "SELECT COALESCE(MAX(row_id), 0) % {} + 1 FROM metrics AS m"
row_id_query = row_id_query_template.format(persistance)
save_metrics_query = (
"REPLACE INTO metrics "
"SET metric_seq = (" + row_id_query + "),"
"instance_id = %(instance_id)s,"
"service = %(service)s,"
"metric_name = %(metric_name)s,"
"metric_value = %(metric_value)s"
)
for metric in jsonMetrics['Metrics']:
formatData = {}
formatData['instance_id'] = arguments.dimensionValue
formatData['service'] = jsonMetrics['Service']
formatData['metric_name'] = metric
formatData['metric_value'] = jsonMetrics['Metrics'][metric]
if arguments.verbose == True:
print "Data: ",formatData
print "Query Template: ",save_metrics_query.format(**formatData)
try:
cursor = cnx.cursor()
cursor.execute(save_metrics_query, formatData)
logger('info','Metrics were saved successfully!')
cnx.commit()
cursor.close()
except mysql.connector.Error as err:
logger('error', "Something went wrong: %s" % err)
cnx.close()
As you can see, I format the SELECT outside. I believe the whole problem was due to this line:
"metric_value = %(metric_value)f"
I changed to:
"metric_value = %(metric_value)s"
and now it works. I think the formatting was wrong, given a syntax error (tho I don't know how the exception was never thrown).
Thanks to everyone who took time to help me!

I haven't actually used MySQL, but the docs seem to indicate that calling cursor.execute with multi=True just returns an iterator. If that is true, then it wouldn't actually insert anything - you'd need to call .next() on the iterator (or just iterate over it) to actually insert the record.
It also goes on to advise against using parameters with multi=True:
If multi is set to True, execute() is able to execute multiple statements specified in the operation string. It returns an iterator that enables processing the result of each statement. However, using parameters does not work well in this case, and it is usually a good idea to execute each statement on its own.
tl;dr: remove that parameter, as the default is False.

This was the solution. I changed:
"metric_value = %(metric_value)f"
To:
"metric_value = %(metric_value)s"
Doing some troubleshooting I found a syntax error on the SQL. Somehow the exception didn't show.

Arcpy.python ArcServer sde permissions

I have this python script that basically selects a point by it's ID, then selects all points within a distance and returns only a subset of those that match the type field. i.e. find all hospitals within 3 miles of this location..
My python script works, does what it's supposed to. So I create a GP service from it. Add the service back into the map and run it. Now I notice no matter what distance I use, the data doesn't change. If I delete the created Featureclass to see if it's really working. It does not create a new featureclass but it says it completed successfully.
Now the weird part, if I hit the GP service at the rest endpoint with the parameters it says it works but returns no records. I've been careful to avoid schema locks when using the Rest endpoint ArcMap and ArcCatalog are closed.
It's like the GP service doesn't have permission to write to the sde database, However my sde connection works fine on my PC.
Any ideas?
import arcpy, os, string
from arcpy import env
db = r"Connection to racdev1.sde"
theworkspace = r"Connection to racdev1.sde"
arcpy.env.workspace = theworkspace
arcpy.env.overwriteOutput = True
#facilityID = '1249'
facilityID = arcpy.GetParameterAsText(0)
#facilityIDType= 'PFI'
facilityIDType = arcpy.GetParameterAsText(1)
thedistance = arcpy.GetParameterAsText(2)
#thedistance = '3 miles'
#withindistance = "3 Miles"
withindistance = thedistance + ' Miles'
sql_query = "\"%s\" = '%s'" % ("ID", facilityID)
sql_query2 = "\"%s\" = '%s'" % ("IDTYPE", facilityIDType)
# Local variables:
Facilities = "DOHGIS.NYSDOH_CI_DATA"
featLayer = "thePts"
arcpy.MakeFeatureLayer_management(Facilities, featLayer)
# Process: Select Layer By Attribute
arcpy.SelectLayerByAttribute_management(featLayer, "NEW_SELECTION", sql_query)
# Process: Select Layer By Location 311
arcpy.SelectLayerByLocation_management(featLayer, "WITHIN_A_DISTANCE",featLayer, withindistance, "NEW_SELECTION")
#print " now for the subset"
arcpy.SelectLayerByAttribute_management("thePts", "SUBSET_SELECTION", sql_query2 )
# creaate the new featureclss..
arcpy.CopyFeatures_management("thePts",'DOHGIS.NYSDOH_FacilitiesQuery')
#print "Done"

I'm getting a 'ghost' ROLLBACK from sqlalchemy but not when use psql with postgres

I have a weird behaviour with postgres + sqlalchemy.
I call a function that insert into a table, but when called from sqlalchemy it roolback at the end, and when called from psql it succeed:
Logs when called by sqlalchemy (as reported by the logs):
Jan 21 13:17:28 intersec.local postgres[3466]: [18-9] STATEMENT: SELECT name, suffix
Jan 21 13:17:28 intersec.local postgres[3466]: [18-10] FROM doc_codes('195536d95bd155b9ea412154b3e920761495681a')
Jan 21 13:17:28 intersec.local postgres[3466]: [19-9] STATEMENT: ROLLBACK
Jan 21 13:17:28 intersec.local postgres[3465]: [13-9] STATEMENT: COMMIT
If using psql:
Jan 21 13:28:47 intersec.local postgres[3561]: [20-9] STATEMENT: SELECT name, suffix FROM doc_codes('195536d95bd155b9ea412154b3e920761495681a');
Note not transaction stuff at all.
This is my python code:
def getCon(self):
conStr = "postgresql+psycopg2://%(USER)s:%(PASSWORD)s#%(HOST)s/%(NAME)s"
config = settings.DATABASES['default']
#print conStr % config
con = sq.create_engine(
conStr % config,
echo=ECHO
)
event.listen(con, 'checkout', self.set_path)
self.con = con
self.meta.bind = con
return con
def getDocPrefixes(self, deviceId):
f = sq.sql.func.doc_codes(deviceId, type_=types.String)
columns = [
sq.Column('name', types.String),
sq.Column('suffix', types.String)
]
return [dict(x.items()) for x in self.con.execute
(
select(columns).
select_from(f)
).fetchall()]
sync = dbSync('malab')
for k in sync.getDocPrefixes('195536d95bd155b9ea412154b3e920761495681a'):
print k['name'], '=', k['suffix']
What could trigger the ROLLBACK?
P.D: My DB functions:
CREATE OR REPLACE FUNCTION next_letter (
table_name TEXT,
OUT RETURNS TEXT
)
AS
$$
DECLARE
result TEXT = 'A';
nextLetter TEXT;
num INTEGER;
BEGIN
SELECT INTO num nextval('letters');
nextLetter := chr(num);
result := nextLetter;
WHILE true LOOP
--RAISE NOTICE '%', result;
IF EXISTS(SELECT 1 FROM DocPrefix WHERE Name=result AND TableName=table_name) THEN
SELECT max(SUBSTRING(name FROM '\d+'))
FROM DocPrefix WHERE Name=result AND TableName=table_name
INTO num;
result := nextLetter || (coalesce(num,0) + 1);
ELSE
EXIT;
END IF;
END LOOP;
RETURNS = result;
END;
$$
LANGUAGE 'plpgsql';
-- Retorna el prefijo unico para la tabla/dispositivo.
CREATE OR REPLACE FUNCTION prefix_fordevice (
table_name TEXT,
device_id TEXT,
OUT RETURNS TEXT
)
AS
$$
DECLARE
result TEXT = NULL;
row RECORD;
BEGIN
IF NOT(EXISTS(SELECT 1 FROM DocPrefix WHERE MachineId=device_id AND TableName=table_name)) THEN
INSERT INTO DocPrefix
(Name, MachineId, TableName)
VALUES
(next_letter(table_name), device_id, table_name);
END IF;
SELECT name FROM DocPrefix WHERE
MachineId=device_id AND TableName=table_name
INTO result;
RETURNS = result;
END;
$$
LANGUAGE 'plpgsql';
--Retornar los prefijos exclusivos para el ID de dispositvo
CREATE OR REPLACE FUNCTION doc_codes(device_id TEXT) RETURNS TABLE("name" TEXT, "suffix" TEXT) AS $$
SELECT name, prefix_fordevice(name, device_id) AS suffix FROM doccode;
$$ LANGUAGE SQL;

the antipattern here is that you're confusing a SQLAlchemy Engine for a connection, when you do something like this:
con = sq.create_engine(<url>)
result = con.execute(statement)
the Engine is associated with a connection pool as a source of connections. When you call the execute() method on Engine, it checks out a connection from the pool, runs the statement, and returns the results; when the result set is exhausted, it returns the connection to the pool. At that stage, the pool will either close the connection fully, or it will re-pool it. Storing the connection in the pool means that any remaining transactional state must be cleared (note that DBAPI connections are always in a transaction when they are used), so it emits a rollback.
Your program should create a single Engine per URL at the module level, and when it needs a connection, should call upon engine.connect().
the document Working with Engines and Connections explains all of this.

I finally found the answer here:
Make SQLAlchemy COMMIT instead of ROLLBACK after a SELECT query
def getDocPrefixes(self, deviceId):
f = sq.sql.func.doc_codes(deviceId, type_=types.String)
columns = [
sq.Column('name', types.String),
sq.Column('sufix', types.String)
]
with self.con.begin():
return [dict(x.items()) for x in self.con.execute
(
select(columns).
select_from(f)
).fetchall()]
The thing is, the function can insert data + also return a SELECT, so, sqlalchemy think this is a normal SELECT when in fact the function also change data and need commit.