Develop Reference

Python + cx_Oracle : ORA-01017: invalid username/password; logon denied

I usually connect to OraClient18Home1 through Toad 12.6.0.53 so I can explore data and run my queries. We used TNSname.ora file to set up my connection in Toad(there is not syntax errors in TSNname)
Now, I would like to be able to connect to the Oracle database through a python IDE (I'm using Spyder). I have already installed cx_oracle. I'm using Python 3.9
Here are the TNSnames
FINDB=
(DESCRIPTION_LIST=
(FAILOVER=on)
(LOAD_BALANCE=off)
(DESCRIPTION=
(ADDRESS_LIST=
(LOAD_BALANCE=off)
(ADDRESS=
(PROTOCOL=TCP)
(HOST=finprd01-csscan.ca3.ocm.s8529456.oraclecloudatcustomer.com)
(PORT=1521)
)
)
(CONNECT_DATA=
(SERVICE_NAME=FINDB_rw.ca3.ocm.s8529456.oraclecloudatcustomer.com)
)
)
(DESCRIPTION=
(ADDRESS_LIST=
(LOAD_BALANCE=off)
(ADDRESS=
(PROTOCOL=TCP)
(HOST=finprd02-csscan.ca1.ocm.s7896852.oraclecloudatcustomer.com)
(PORT=1521)
)
)
(CONNECT_DATA=
(SERVICE_NAME=FINDB_rw.ca1.ocm.s7896852.oraclecloudatcustomer.com)
)
)
)
you can read the TSNname sript more clearly in the image
I used the same Username/password/port/host/service_name as the ones I use in Toad.
Here is my python code :
import cx_Oracle
p_password = "111111"
con = cx_Oracle.connect(user="User101", password=p_password ,dsn="finprd01-csscan.ca3.ocm.s8529456.oraclecloudatcustomer.com / FINDB_rw.ca3.ocm.s8529456.oraclecloudatcustomer.com", encoding="UTF-8")
print("Database version:", con.version)
print("Oracle Python version:", cx_Oracle.version)
When I run the code I get; ORA-01017: invalid username/password; logon denied.
I should mention that in the Cx_Oracle documentation, they use one Host and one Service_name. but if you look at my TNSname description I have 2 hosts and 2 service_names. I'm not a data architect I don't really understand why we have 2 hosts and 2 service_names whereas all the tutorials and blogs use just one each.
In my python connection, I selected randomly just one host and one service_name.
By the way, I was using this documentation https://cx-oracle.readthedocs.io/en/latest/user_guide/connection_handling.html
I used Easy Connect Syntax for Connection Strings, from the link above, this is the format
connection = cx_Oracle.connect(user="hr", password=userpwd,dsn="dbhost.example.com/orclpdb1",encoding="UTF-8")
What am I doing wrong?
I assumed that since I have access to the Oracle database through Toad. I should just re-use the same connection parameters from Toad (Username/password/port/host/service_name) and it will be easy to establish the connection from Python IDE (I'm using Spyder).
Do I need to contact admin for approval? I can connect the database through Toad, Maybe connecting through python will require permission?
You might suggest contacting Admin for help but they are short-staffed and I could only get an answer or someone over the phone in 2 weeks at the earliest. Therefore, I’m trying to do this myself. if you think contacting them is the only solution, I will have to do it then!
I would be very thankful for any tips or help.
Thanks

PyODBC: how to replicate behavior of batch separator (GO) in a transaction?

Background
I maintain a Python application that automatically applies SQL schema migrations (adding/removing tables and columns, adjusting the data, etc) to our database (SQL2016). Each migration is executed via PyODBC within a transaction so that it can be rolled back if something goes wrong. Sometimes a migration requires one or more batch statements (GO) to execute correctly. Since GO is not actually a T-SQL command but rather a special keyword in SSMS, I've been splitting each SQL migration on GO and executing each SQL fragment separately within the same transaction.
import pyodbc
import re
conn_args = {
'driver': '{ODBC Driver 17 for SQL Server}',
'hostname': 'MyServer',
'port': 1298,
'server': r'MyServer\MyInstance',
'database': 'MyDatabase',
'user': 'MyUser',
'password': '********',
'autocommit': False,
}
connection = pyodbc.connect(**conn_args)
cursor = connection.cursor()
sql = '''
ALTER TABLE MyTable ADD NewForeignKeyID INT NULL FOREIGN KEY REFERENCES MyParentTable(ID)
GO
UPDATE MyTable
SET NewForeignKeyID = 1
'''
sql_fragments = re.split(r'^\s*GO;?\s*$', sql, flags=re.IGNORECASE|re.MULTILINE)
for sql_frag in sql_fragments:
cursor.execute(sql_frag)
# Wait for the command to complete. This is necessary for some database system commands
# (backup, restore, etc). Probably not necessary for schema migrations, but included
# for completeness.
while cursor.nextset():
pass
connection.commit()
Problem
SQL statement batches aren't being executed like I expected. When the above schema migration is executed in SSMS, it succeeds. When executed in Python, the first batch (adding the foreign key) executes just fine, but the second batch (setting the foreign key value) fails because it isn't aware of the new foreign key.
('42S22', "[42S22] [FreeTDS][SQL Server]Invalid column name 'NewForeignKeyID'. (207) (SQLExecDirectW)")
Goal
Execute a hierarchy of SQL statement batches (i.e. where each statement batch depends upon the previous batch) within a single transaction in PyODBC.
What I've Tried
Searching the PyODBC documentation for information on how PyODBC supports or doesn't support batch statements / the GO command. No references found.
Searching StackOverflow & Google for how to batch statements within PyODBC.
Introducing a small sleep between SQL fragment executions just in case there's some sort of race condition. Seemed unlikely to be a solution, and didn't change the behavior.
I've considered separating each batch of statements out into a separate transaction that is committed before the next batch is executed, but that would reduce/eliminate our ability to automatically roll back a schema migration that fails.
EDIT: I just found this question, which is pretty much exactly what I want to do. However, upon testing (in SSMS) the answer that recommends using EXEC I discovered that the second EXEC command (setting the value) fails because it isn't aware of the new foreign key. I'm bad at testing and it actually does succeed. This solution might work but isn't ideal since EXEC isn't compatible with parameters. Also, this won't work if variables are used across fragments.
BEGIN TRAN
EXEC('ALTER TABLE MyTable ADD NewForeignKeyID INT NULL FOREIGN KEY REFERENCES MyParentTable(ID)')
EXEC('UPDATE MyTable SET NewForeignKeyID = 1')
ROLLBACK TRAN
Invalid column name 'FK_TestID'.

If you are reading the SQL statements from a text file (such as one produced by scripting objects in SSMS) then you could just use Python's subprocess module to run the sqlcmd utility with that file as the input (-i). In its simplest form that would look like
server = "localhost"
port = 49242
uid = "scott"
pwd = "tiger^5HHH"
database = "myDb"
script_file = r"C:\__tmp\batch_test.sql"
"""contents of the above file:
DROP TABLE IF EXISTS so69020084;
CREATE TABLE so69020084 (src varchar(10), var_value varchar(10));
INSERT INTO so69020084 (src, var_value) VALUES ('1st batch', 'foo');
GO
INSERT INTO so69020084 (src, var_value) VALUES ('2nd batch', 'bar');
GO
"""
import subprocess
cmd = [
"sqlcmd",
"-S", f"{server},{port}",
"-U", uid,
"-P", pwd,
"-d", database,
"-i", script_file,
]
subprocess.run(cmd)

Sequence nextval/currval in two sessions

Setup:
Oracle DB running on a windows machine
Mac connected with the database, both in the same network
Problem:
When I created a sequence in SQL Developer, I can see and use the sequence in this session. If I logoff and login again the sequence is still there. But if I try to use the sequence via Python and cx_Oracle, it doesn't work. It also doesn't work the other way around.
[In SQL Developer: user: uc]
create SEQUENCE seq1;
select seq1.nextval from dual; ---> 1
commit; --> although the create statement is a DDL method, just in case
[login via Python, user: uc]
select seq1.currval from dual;--> ORA-08002 Sequence seq1.currval isn't defined in this session
The python code:
import cx_Oracle
cx_Oracle.init_oracle_client(lib_dir="/Users/benreisinger/Documents/testclients/instantclient_19_8", config_dir=None, error_url=None, driver_name=None)
# Connect as user "hr" with password "hr" to the "orclpdb" service running on a remote computer.
connection = cx_Oracle.connect("uc", "uc", "10.0.0.22/orcl")
cursor = connection.cursor()
cursor.execute("""
select seq1.currval from dual
""")
print(cursor)
for seq1 in cursor:
print(seq1)
The error says, that [seq1] wasn't defined in this session, but why does the following work:
select seq1.nextval from dual
--> returns 2
Even after issuing this, I can't use seq1.currval
Btw., select sequence_name from user_sequences returns seq1in Python
[as SYS user]
select * from v$session
where username = 'uc';
--> returns zero rows
Why is seq1 not in reach for the python program ?
Note: With tables, everything just works fine
EDIT:
also with 'UC' being upper case, no rows returned
first issuing
still doesn't work

Not sure how to explain this. The previous 2 answers are correct, but somehow you seem to miss the point.
First, take everything that is irrelevant out of the equation. Mac client on Windows db: doesn't matter. SQLDeveloper vs python: doesn't matter. The only thing that matters is that you connect twice to the database as the same schema. You connect twice, that means that you have 2 separate sessions and those sessions don't know about each other. Both sessions have access to the same database objects, so you if you execute ddl (eg create sequence), that object will be visible in the other session.
Now to the core of your question. The oracle documentation states
"To use or refer to the current sequence value of your session, reference seq_name.CURRVAL. CURRVAL can only be used if seq_name.NEXTVAL has been referenced in the current user session (in the current or a previous transaction)."
You have 2 different sessions, so according to the documentation, you should not be able to call seq_name.CURRVAL in the other session. That is exactly the behaviour you are seeing.
You ask "Why is seq1 not in reach for the python program ?". The answer is: you're not correct, it is in reach for the python program. You can call seq1.NEXTVAL from any session. But you cannot invoke seq1.NEXTVAL from one session (SQLDeveloper) and then invoke seq1.CURRVAL from another session (python) because that is just how sequences works as stated in documentation.
Just to confirm you're not in the same session, execute the following statement for both clients (SQLDeveloper and python):
select sys_context('USERENV','SID') from dual;
You'll notice that the session id is different.

CURRVAL returns the last allocated sequence number in the current session. So it only works when we have previously executed a NEXTVAL. So these two statements will return the same value when run in the same session:
select seq1.nextval from dual
/
select seq1.currval from dual
/
It's not entirely clear what you're trying to achieve, but it looks like your python code is executing a single statement for the connection, so it's not tapping into an existing session.
This statement returns zero rows ...
select * from v$session
where username = 'uc';
... because database objects in Oracle are stored in UPPER case (at least by default, but it's wise to stick with that default. So use where username = 'UC' instead.

Python established a new session. In it, sequence hasn't been invoked yet, so its currval doesn't exist. First you have to select nextval (which, as you said, returned 2) - only then currval will make sense.
Saying that
Even after issuing this, I can't use seq1.currval
is hard to believe.
This: select * From v$session where username = 'uc' returned nothing because - by default - all objects are stored in uppercase, so you should have ran
.... where username = 'UC'
Finally:
commit; --> although the create statement is a DDL method, just in case
Which case? There's no case. DDL commits. Moreover, commits twice (before and after the actual DDL statement). And there's nothing to commit either. Therefore, what you did is unnecessary and pretty much useless.

SqlAlchemy+pymssql. Will raw parametrized queries use same execution plan?

In my application I have parametrized queries like this:
res = db_connection.execute(text("""
SELECT * FROM Luna_gestiune WHERE id_filiala = :id_filiala AND anul=:anul AND luna = :luna
"""),
id_filiala=6, anul=2010, luna=7).fetchone()
Will such query use same query execution plan if I run it in loop with different parameter values?

It seems unlikely. pymssql uses FreeTDS, and FreeTDS performs the parameter substitution before sending the query to the server, unlike some other mechanisms that send the query "template" and the parameters separately (e.g., pyodbc with Microsoft's ODBC drivers, as described in this answer).
That is, for the query you describe in your question, pymssql/FreeTDS will not send a query string like
SELECT * FROM Luna_gestiune WHERE id_filiala = #P1 AND anul = #P2 AND luna = #P3
along with separate values for #P1 = 6, #P2 = 2010, etc.. Instead it will build the literal query first, and then send
SELECT * FROM Luna_gestiune WHERE id_filiala = 6 AND anul = 2010 AND luna = 7
So for each parameteized query you send, the SQL command text will be different, and my understanding is that database engines will only re-use a cached execution plan if the current command text is identical to the cached version.
Edit: Subsequent testing confirms that pymssql apparently does not re-use cached execution plans. Details in this answer.

twisted adbapi turn off autocommit (psycopg2)

I am inserting a lot of rows and it seems that postgress can't keep up. I've googled a bit and it is suggested that you can turn off autocommit. I don't need to commit the values right away. It's data that I can fetch again if something goes wrong.
Now when I search for turning off autocommit I'm not finding what I'm looking for. I've tried supplying autocommit=False in the dbpool constructor:
dbpool = adbapi.ConnectionPool('psycopg2', user="xxx", password="xxx", database="postgres", host="localhost", autocommit=False)
2013-01-27 18:24:42,254 - collector.EventLogItemController - WARNING - [Failure instance: Traceback: : invalid connection option "autocommit"

psycopg2 does not claim to support an autocommit keyword argument to connect:
connect(dsn=None, database=None, user=None, password=None, host=None, port=None, connection_factory=None, async=False, **kwargs)
Create a new database connection.
The connection parameters can be specified either as a string:
conn = psycopg2.connect("dbname=test user=postgres password=secret")
or using a set of keyword arguments:
conn = psycopg2.connect(database="test", user="postgres", password="secret")
The basic connection parameters are:
- *dbname*: the database name (only in dsn string)
- *database*: the database name (only as keyword argument)
- *user*: user name used to authenticate
- *password*: password used to authenticate
- *host*: database host address (defaults to UNIX socket if not provided)
- *port*: connection port number (defaults to 5432 if not provided)
Using the *connection_factory* parameter a different class or connections
factory can be specified. It should be a callable object taking a dsn
argument.
Using *async*=True an asynchronous connection will be created.
Any other keyword parameter will be passed to the underlying client
library: the list of supported parameter depends on the library version.
The current postgresql documentation doesn't discuss any "autocommit" parameter either:
http://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
So perhaps the problem is that this is not the correct way to disable autocommit for a psycopg2 connection. Apart from that, you won't find that turning off autocommit actually helps you at all. adbapi.ConnectionPool will begin and commit explicit transactions for you, side-stepping any behavior autocommit mode might give you.

The problem with adbapi is that:
1) its missing features specific to the some of the database backends
2) its fake asynchronous api. Under the hoods its using the thread pool and to call the blocking methods.
For postgres I'd suggest using txpostgres library (source is here: https://github.com/wulczer/txpostgres). It is using asynchronous api of psycopg2 and it lets you specify the connection string.
You can find an example here: http://txpostgres.readthedocs.org/en/latest/usage.html#customising-the-connection-and-cursor-factories

Use the cp_openfun option of the twisted.enterprise.adbapi.ConnectionPool constructor
This function is called with the connection as a parameter.
In case of psycopg2 you can then set the autocommit property of that connection to True or False as stated here