I am attempting to call a sproc that wraps T-SQL's sp_send_dbmail procedure. It simply creates the query, and then passes that and a hardcoded subject and recipient to the Microsoft supplied sproc. When running from sql management studio, the sproc executes as expected and I receive an email. From pyodbc, it doesn't send an email. The contents of my sproc look similar to:
declare #qry varchar(MAX)
set #qry = 'select * from table'
EXEC msdb.dbo.sp_send_dbmail
#recipients = 'email#email.com',
#subject = 'my email',
#query = #qry
select * from table
where 1=0
I have also tried toggling the #exclude_query_output, flag but that had no effect. I am invoking that sproc via the following method:
def execute_sproc(query, cnxn):
cursor = cnxn.cursor();
rows = cursor.execute(query)
columns = [column[0] for column in cursor.description]
return pd.DataFrame.from_records(rows, columns=columns)
where the query is simply executing my wrapper sproc. As I mentioned before, the wrapper sproc works when run from the management studio, but no email is sent when called here. I am using the same credentials to access my database in both places. I have also used this function to successfully call other sprocs, but none of them have contained exec statements, nor have they done many of the other things that im sure sp_send_dbmail is doing.
Please let me know if you have any ideas.
Thanks,
Max Goldman
So i think this comes down to a misunderstanding of either the sendmail sproc, pyodbc, or both. I have another method for calling sprocs that edit the database:
def execute_commit_sproc(query, cnxn):
cursor = cnxn.cursor();
cursor.execute(query)
cnxn.commit()
The only difference being that the former expected a result set, whereas the latter makes changes to the db, and uses pyodbc's commit mechanism to save the results. I did not realize the sp_send_dbmail required this to be called in order for the email to be sent. I am still unsure of why (what is getting written to, what pyodbc::commit() is doing under the hood, etc.)
Related
Background
I maintain a Python application that automatically applies SQL schema migrations (adding/removing tables and columns, adjusting the data, etc) to our database (SQL2016). Each migration is executed via PyODBC within a transaction so that it can be rolled back if something goes wrong. Sometimes a migration requires one or more batch statements (GO) to execute correctly. Since GO is not actually a T-SQL command but rather a special keyword in SSMS, I've been splitting each SQL migration on GO and executing each SQL fragment separately within the same transaction.
import pyodbc
import re
conn_args = {
'driver': '{ODBC Driver 17 for SQL Server}',
'hostname': 'MyServer',
'port': 1298,
'server': r'MyServer\MyInstance',
'database': 'MyDatabase',
'user': 'MyUser',
'password': '********',
'autocommit': False,
}
connection = pyodbc.connect(**conn_args)
cursor = connection.cursor()
sql = '''
ALTER TABLE MyTable ADD NewForeignKeyID INT NULL FOREIGN KEY REFERENCES MyParentTable(ID)
GO
UPDATE MyTable
SET NewForeignKeyID = 1
'''
sql_fragments = re.split(r'^\s*GO;?\s*$', sql, flags=re.IGNORECASE|re.MULTILINE)
for sql_frag in sql_fragments:
cursor.execute(sql_frag)
# Wait for the command to complete. This is necessary for some database system commands
# (backup, restore, etc). Probably not necessary for schema migrations, but included
# for completeness.
while cursor.nextset():
pass
connection.commit()
Problem
SQL statement batches aren't being executed like I expected. When the above schema migration is executed in SSMS, it succeeds. When executed in Python, the first batch (adding the foreign key) executes just fine, but the second batch (setting the foreign key value) fails because it isn't aware of the new foreign key.
('42S22', "[42S22] [FreeTDS][SQL Server]Invalid column name 'NewForeignKeyID'. (207) (SQLExecDirectW)")
Goal
Execute a hierarchy of SQL statement batches (i.e. where each statement batch depends upon the previous batch) within a single transaction in PyODBC.
What I've Tried
Searching the PyODBC documentation for information on how PyODBC supports or doesn't support batch statements / the GO command. No references found.
Searching StackOverflow & Google for how to batch statements within PyODBC.
Introducing a small sleep between SQL fragment executions just in case there's some sort of race condition. Seemed unlikely to be a solution, and didn't change the behavior.
I've considered separating each batch of statements out into a separate transaction that is committed before the next batch is executed, but that would reduce/eliminate our ability to automatically roll back a schema migration that fails.
EDIT: I just found this question, which is pretty much exactly what I want to do. However, upon testing (in SSMS) the answer that recommends using EXEC I discovered that the second EXEC command (setting the value) fails because it isn't aware of the new foreign key. I'm bad at testing and it actually does succeed. This solution might work but isn't ideal since EXEC isn't compatible with parameters. Also, this won't work if variables are used across fragments.
BEGIN TRAN
EXEC('ALTER TABLE MyTable ADD NewForeignKeyID INT NULL FOREIGN KEY REFERENCES MyParentTable(ID)')
EXEC('UPDATE MyTable SET NewForeignKeyID = 1')
ROLLBACK TRAN
Invalid column name 'FK_TestID'.
If you are reading the SQL statements from a text file (such as one produced by scripting objects in SSMS) then you could just use Python's subprocess module to run the sqlcmd utility with that file as the input (-i). In its simplest form that would look like
server = "localhost"
port = 49242
uid = "scott"
pwd = "tiger^5HHH"
database = "myDb"
script_file = r"C:\__tmp\batch_test.sql"
"""contents of the above file:
DROP TABLE IF EXISTS so69020084;
CREATE TABLE so69020084 (src varchar(10), var_value varchar(10));
INSERT INTO so69020084 (src, var_value) VALUES ('1st batch', 'foo');
GO
INSERT INTO so69020084 (src, var_value) VALUES ('2nd batch', 'bar');
GO
"""
import subprocess
cmd = [
"sqlcmd",
"-S", f"{server},{port}",
"-U", uid,
"-P", pwd,
"-d", database,
"-i", script_file,
]
subprocess.run(cmd)
Setup:
Oracle DB running on a windows machine
Mac connected with the database, both in the same network
Problem:
When I created a sequence in SQL Developer, I can see and use the sequence in this session. If I logoff and login again the sequence is still there. But if I try to use the sequence via Python and cx_Oracle, it doesn't work. It also doesn't work the other way around.
[In SQL Developer: user: uc]
create SEQUENCE seq1;
select seq1.nextval from dual; ---> 1
commit; --> although the create statement is a DDL method, just in case
[login via Python, user: uc]
select seq1.currval from dual;--> ORA-08002 Sequence seq1.currval isn't defined in this session
The python code:
import cx_Oracle
cx_Oracle.init_oracle_client(lib_dir="/Users/benreisinger/Documents/testclients/instantclient_19_8", config_dir=None, error_url=None, driver_name=None)
# Connect as user "hr" with password "hr" to the "orclpdb" service running on a remote computer.
connection = cx_Oracle.connect("uc", "uc", "10.0.0.22/orcl")
cursor = connection.cursor()
cursor.execute("""
select seq1.currval from dual
""")
print(cursor)
for seq1 in cursor:
print(seq1)
The error says, that [seq1] wasn't defined in this session, but why does the following work:
select seq1.nextval from dual
--> returns 2
Even after issuing this, I can't use seq1.currval
Btw., select sequence_name from user_sequences returns seq1in Python
[as SYS user]
select * from v$session
where username = 'uc';
--> returns zero rows
Why is seq1 not in reach for the python program ?
Note: With tables, everything just works fine
EDIT:
also with 'UC' being upper case, no rows returned
first issuing
still doesn't work
Not sure how to explain this. The previous 2 answers are correct, but somehow you seem to miss the point.
First, take everything that is irrelevant out of the equation. Mac client on Windows db: doesn't matter. SQLDeveloper vs python: doesn't matter. The only thing that matters is that you connect twice to the database as the same schema. You connect twice, that means that you have 2 separate sessions and those sessions don't know about each other. Both sessions have access to the same database objects, so you if you execute ddl (eg create sequence), that object will be visible in the other session.
Now to the core of your question. The oracle documentation states
"To use or refer to the current sequence value of your session, reference seq_name.CURRVAL. CURRVAL can only be used if seq_name.NEXTVAL has been referenced in the current user session (in the current or a previous transaction)."
You have 2 different sessions, so according to the documentation, you should not be able to call seq_name.CURRVAL in the other session. That is exactly the behaviour you are seeing.
You ask "Why is seq1 not in reach for the python program ?". The answer is: you're not correct, it is in reach for the python program. You can call seq1.NEXTVAL from any session. But you cannot invoke seq1.NEXTVAL from one session (SQLDeveloper) and then invoke seq1.CURRVAL from another session (python) because that is just how sequences works as stated in documentation.
Just to confirm you're not in the same session, execute the following statement for both clients (SQLDeveloper and python):
select sys_context('USERENV','SID') from dual;
You'll notice that the session id is different.
CURRVAL returns the last allocated sequence number in the current session. So it only works when we have previously executed a NEXTVAL. So these two statements will return the same value when run in the same session:
select seq1.nextval from dual
/
select seq1.currval from dual
/
It's not entirely clear what you're trying to achieve, but it looks like your python code is executing a single statement for the connection, so it's not tapping into an existing session.
This statement returns zero rows ...
select * from v$session
where username = 'uc';
... because database objects in Oracle are stored in UPPER case (at least by default, but it's wise to stick with that default. So use where username = 'UC' instead.
Python established a new session. In it, sequence hasn't been invoked yet, so its currval doesn't exist. First you have to select nextval (which, as you said, returned 2) - only then currval will make sense.
Saying that
Even after issuing this, I can't use seq1.currval
is hard to believe.
This: select * From v$session where username = 'uc' returned nothing because - by default - all objects are stored in uppercase, so you should have ran
.... where username = 'UC'
Finally:
commit; --> although the create statement is a DDL method, just in case
Which case? There's no case. DDL commits. Moreover, commits twice (before and after the actual DDL statement). And there's nothing to commit either. Therefore, what you did is unnecessary and pretty much useless.
I have a stored procedure.
calling it via MySQL workbench as follows working;
CALL `lobdcapi`.`escalatelobalarm`('A0001');
But not from the python program. (means it is not throwing any exception, process finish execution silently) if I make any error in column names, then at python I get an error. So it calls my stored procedure but not working as expected. (it is an update query .it needs SAFE update )
Why through the python sqlalchemy this update didn't update any records?
CREATE DEFINER=`lob`#`%` PROCEDURE `escalatelobalarm`(IN client_id varchar(50))
BEGIN
SET SQL_SAFE_UPDATES = 0;
update lobdcapi.alarms
set lobalarmescalated=1
where id in (
SELECT al.id
from (select id,alarmoccurredhistoryid from lobdcapi.alarms where lobalarmpriorityid=1 and lobalarmescalated=0 and clientid=client_id
and alarmstatenumber='02' ) as al
inner join lobdcapi.`alarmhistory` as hi on hi.id=al.alarmoccurredhistoryid
and hi.datetimestamp<= current_timestamp() )
);
SET SQL_SAFE_UPDATES = 1;
END
I call it like;
from sqlalchemy import and_, func,text
db.session.execute(text("CALL escalatelobalarm(:param)"), {'param': clientid})
I suspect the param I pass via code didn't get bind properly?
I haven't called stored procs from SQLAlchemy, but it seems possible that this could be within a transaction because you're using the session. Perhaps calling db.session.commit() at the end would help?
If that fails, SQLAlchemy calls out calling stored procs here. Perhaps try their method of using callproc. Adapting to your use-case, something like:
connection = db.session.connection()
try:
cursor = connection.cursor()
cursor.callproc("escalatelobalarm", [clientid])
results = list(cursor.fetchall())
cursor.close()
connection.commit()
finally:
connection.close()
I've read some q&a about "syntax error at or near" but none could solve my issue.
A sample of the error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
psycopg2.ProgrammingError: syntax error at or near "UPDATE"
LINE 1: DECLARE "teste" CURSOR WITHOUT HOLD FOR UPDATE applicant SET...
^
Please, note the ^ right on the UPDATE. I've tested the update script on pgadmin 4 and everything worked fine.
The script is really simple:
UPDATE applicant
SET cv_entities = %s
WHERE id = %s
My code is basically:
def _connect_database():
return psy.connect(
dbname=settings.DATABASE['DBNAME'],
host=settings.DATABASE['HOST'],
port=settings.DATABASE['PORT'],
user=settings.DATABASE['USER'],
password=settings.DATABASE['PASSWORD'],
application_name=settings.env
)
# Connects to database
conn = _connect_database()
# Creats a named cursor
cur = conn.cursor('test')
# Execute
cur.execute(update_script, ('{"json": "test"}', '11125ba0-748f-11e8-b1d0-d8108ee3at21'))
I've explicitly written the script on the execute method as well as the parameters, but I still get the same error:
cur.execute("UPDATE applicant SET cv_entities = 'json here' WHERE id = '11125ba0-748f-11e8-b1d0-d8108ee3at21'")
Note that I even removed the double quotes from the first parameter ('json here')
Am I missing something?!
With
cur = conn.cursor('test')
you are trying to open a server-side cursor. Per the documentation:
cursor(name=None, cursor_factory=None, scrollable=None, withhold=False)
Return a new cursor object using the connection.
If name is specified, the returned cursor will be a server side cursor (also known as named cursor). Otherwise it will be a regular client side cursor.
Use just
cur = conn.cursor()
Server side (named) cursors can be used only for SELECT or VALUES queries. They implement Postgres cursors:
query
A SELECT or VALUES command which will provide the rows to be returned by the cursor.
With named cursors the resulting data is gathered on the server and sent (maybe partialy) to the client when needed (fetched). The data is stored only during the transaction in which the cursor was opened, so commit or rollback releases it.
Client side (unnamed) cursors give the possibility to execute any valid SQL query. A possible resultset is sent to the client immediately after executing the query.
It's possible to use named and unnamed cursors utilizing a single connection but if you want to do this concurrently, you should use two separate connections.
I have the following code:
conn = pymssql.connect(server, user, password, database)
cursor = conn.cursor()
id_new_field = pymssql.output(int)
res = cursor.callproc('NewField', ('Test',id_new_field))
conn.commit()
conn.close()
print(id_new_field.value)
print(res)
Unfortunately my output parameter isn't populated with the id of the field, although the stored procedure is executed correctly. The output ist always 'None'.
I think the problem is, that autocommit ist set to false in my example, am i right?
But i can't set autocommit to true because the stored procedure is raising an error when it isn't executed inside a transaction (It's a vendor SP and i can't alter the SP).
So, my workarount for now is, to use .execute() instead of .callproc() and writing raw sql into my python script. It's awful. ;)
Is there any chance to use .callproc() with autocommit = false?
Or do i have to do it completely different?
I also got None when I did
print(id_new_field.value)
but I was able to retrieve the value by simply indexing into res tuple:
print(res[1])
(Tested with Python 2.7.11 and pymssql 2.1.1.)