I'm trying to implement a server side cursor in order to "bypass" Django ORM weakness when it comes to fetch an huge amount of data from the database.
But I don't understand how named cursor are supposed to be defined, since my current code seems not working properly. I define the cursor in this way:
id = 'cursor%s' % uuid4().hex
connection = psycopg2.connect('my connection string here')
cursor = connection.cursor(id, cursor_factory=psycopg2.extras.RealDictCursor)
The cursor seems to work in that it can be iterated and returns expected records as python dictionary, but when I try to close it (cursor.close()) I get the exception:
psycopg2 OperationalError: cursor *the generated cursor id* does not exist
WTF?! So what is the object I'm using to retrieve stuff from the database?
Is psycopg2 using a fallback default (unnamed) cursor since the one I defined is not found in my database (and if so... my big question: it's mandatory to define a cursor at the db level before using psycopg2?) I'm a lot confused, can you help me?
I made a really simple and silly mistake of forgetting to run ./manage.py makemigrations and ./manage.py migrate before running ./manage.py test which caused this error.
(I'm aware this doesn't answer the original question, but since this is the first result from Google I thought I would contribute. Hopefully that's okay)
I've had this problems when playing around with my models and launching the test with Pytest.
What resolved the problem for me was to reset the database of my test unit. I used --create-db like so:
pytest backend/test_projects/partners/test_actions.py --create-db
I had similar problem and found the solution. Just disable server side cursors like described here: https://docs.djangoproject.com/en/2.2/ref/settings/#disable-server-side-cursors
'default': {
...
'USER': DB_USER,
'PASSWORD': DB_PASSWORD,
'NAME': DB_NAME,
'DISABLE_SERVER_SIDE_CURSORS': True,
...
},
From the psycopg2 documentation:
"Named cursors are usually created WITHOUT HOLD, meaning they live only as long as the current transaction. Trying to fetch from a named cursor after a commit() or to create a named cursor when the connection transaction isolation level is set to AUTOCOMMIT will result in an exception."
Which is to say that these cursors do not need to be explicitly closed.
http://initd.org/psycopg/docs/usage.html#server-side-cursors
Related
I am attempting to execute a raw sql insert statement in Sqlalchemy, SQL Alchemy throws no errors when the constructed insert statement is executed but the lines do not appear in the database.
As far as I can tell, it isn't a syntax error (see no 2), it isn't an engine error as the ORM can execute an equivalent write properly (see no 1), it's finding the table it's supposed to write too (see no 3). I think it's a problem with a transaction not being commited and have attempted to address this (see no 4) but this hasn't solved the issue. Is it possible to create a nested transaction and what would start the 'first' so to speak?
Thankyou for any answers.
Some background:
I know that the ORM facilitates this and have used this feature and it works, but is too slow for our application. We decided to try using raw sql for this particular write function due to how often it's called and the ORM for everything else. An equivalent method using the ORM works perfectly, and the same engine is used for both, so it can't be an engine problem right?
I've issued an example of the SQL that the method using raw sql constructs to the database directly and that reads in fine, so I don't think it's a syntax error.
it's communicating with the database properly and can find the table as any syntax errors with table and column names throw a programmatic error so it's not just throwing stuff into the 'void' so to speak.
My first thought after reading around was that it was transaction error and that a transaction was being created and not closed, and so constructed the execute statement as such to ensure a transaction was properly created and commited.
with self.Engine.connect() as connection:
connection.execute(Insert_Statement)
connection.commit
The so called 'Insert Statement' has been converted to text using the sqlalchemy 'text' function, I don't quite understand why it won't execute if I pass the constructed string directly to the execute statement but mention it in case it's relevant.
Other things that may be relevant:
Python3 is running on an individual ec2 instance the postgres database on another. The table in particular is a timescaledb hypertable taking realtime data, hence the need for very fast writes, but probably not relevant.
Currently using pg8000 as dialect for no particular reason other than psycopg2 was throwing errors when trying the execute an equivalent method using the ORM.
Just so this question is answered in case anyone else ends up here:
The issue was a failure to call commit as a method, as #snakecharmerb pointed out. Gord Thompson also provided an alternate method using 'begin' which automatically commits rather than connection which is a 'commit as you go' style transaction.
Currently using cx_Oracle module in Python to connect to my Oracle database. I would like to only allow the user of the program to do read only executions, like Select, and NOT INSERT/DELETE queries.
Is there something I can do to the connection/cursor variables once I establish the connection to prevent writable queries?
I am using the Python Language.
Appreciate any help.
Thanks.
One possibility is to issue the statement "set transaction read only" as in the following code:
import cx_Oracle
conn = cx_Oracle.connect("cx_Oracle/welcome")
cursor = conn.cursor()
cursor.execute("set transaction read only")
cursor.execute("insert into c values (1, 'test')")
That will result in the following error:
ORA-01456: may not perform insert/delete/update operation inside a READ ONLY transaction
Of course you'll have to make sure that you create a Connection class that calls this statement when it is first created and after each and every commit() and rollback() call. And it can still be circumvented by calling a PL/SQL block that performs a commit or rollback.
The only other possibility that I can think of right now is to create a restricted user or role which simply doesn't have the ability to insert, update, delete, etc. and make sure the application uses that user or role. This one at least is fool proof, but a lot more effort up front!
I have this code to insert data to Database using MySQL. But when I ran that code using Python, there's no error. But when I checked the Database, the data isn't inserted. Is there anyone who can help me? I would appreciate it. :)
This is the code:
import MySQLdb
db=MySQLdb.connect(host="localhost", user="root", passwd="", db="try")
cursor=db.cursor()
insert="INSERT INTO `try`.`try` (`nomor`, `nama`) VALUES (NULL, 'bismillah')"
cursor.execute(insert)
You're not doing a COMMIT anywhere. So, if auto-commit is not on, all you've done is create a transaction that, if later committed, will insert this row.
Since you haven't done a SET AUTOCOMMIT anywhere, whether auto-commit is on depends on how you created the database. With at least some storage types (in particular, InnoDB), you can change the default at creation time, and, because you often want auto-commit disabled with those storage types, your GUI design tool, or the sample code you copied and pasted, or whatever may have done so for you. Also, the server variable that provides the default can itself be set to a different value at server startup/configuration. (See System Server Variables.)
If you want to make sure that auto-commit is on, just execute SET autocommit=1 before any other statements.
If you want to find out whether auto-commit is on, execute SHOW VARIABLES. (And if it's disabled, you may want to try SHOW GLOBAL VARIABLES LIKE 'autocommit' and SHOW SESSION VARIABLES like 'autocommit' to see which context you've disabled it in.)
If you cannot insert into mysql, there are several ways to solve it:
1: check the log
2: check the structure of your table, maybe it must not be null for nomor or other field
3: the last, when insert into mysqldb using program, you need to commit before you close the connection. for here: db.commit()
I'm using Elixir in a project that connects to a postgres database. I want to run the following query on the database I'm connected to, but I'm not sure how to do it as I'm rather new to Elixir and SQLAlchemy. Anyone know how?
VACUUM FULL ANALYZE table
Update
The error is: "UnboundExecutionError: Could not locate a bind configured on SQL expression or this Session". And the same result with session.close() issued before. I did try doing metadata.bind.execute() and that worked for a simple select. But for the VACUUM it said - "InternalError: (InternalError) VACUUM cannot run inside a transaction block", so now I'm trying to figure out how to turn that off.
Update 2
I can get the query to execute, but I'm still getting the same error - even when I create a new session and close the previous one.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# ... insert stuff
old_session.commit()
old_session.close()
new_sess = sessionmaker(autocommit=True)
new_sess.configure(bind=create_engine('postgres://user:pw#host/db', echo=True))
sess = new_sess()
sess.execute('VACUUM FULL ANALYZE table')
sess.close()
and the output I get is
2009-12-10 10:00:16,769 INFO sqlalchemy.engine.base.Engine.0x...05ac VACUUM FULL ANALYZE table
2009-12-10 10:00:16,770 INFO sqlalchemy.engine.base.Engine.0x...05ac {}
2009-12-10 10:00:16,770 INFO sqlalchemy.engine.base.Engine.0x...05ac ROLLBACK
finishing failed run, (InternalError) VACUUM cannot run inside a transaction block
'VACUUM FULL ANALYZE table' {}
Update 3
Thanks to everyone who responded. I wasn't able to find the solution I wanted, but I think I'm just going to go with the one described here PostgreSQL - how to run VACUUM from code outside transaction block?. It's not ideal, but it works.
Dammit. I knew the answer was going to be right under my nose. Assuming you setup your connection like I did.
metadata.bind = 'postgres://user:pw#host/db'
The solution to this was as simple as
conn = metadata.bind.engine.connect()
old_lvl = conn.connection.isolation_level
conn.connection.set_isolation_level(0)
conn.execute('vacuum analyze table')
conn.connection.set_isolation_level(old_lvl)
This is similar to what was suggested here PostgreSQL - how to run VACUUM from code outside transaction block?
because underneath it all, sqlalchemy uses psycopg to make the connection to postgres. Connection.connection is a proxy to the psycopg connection. Once I realized this, this problem came back to mind and I decided to take another whack at it.
Hopefully this helps someone.
You need to bind the session to an engine
session.bind = metadata.bind
session.execute('YOUR SQL STATEMENT')
UnboundExecutionError says that your session is not bound to an engine and there is no way to discover engine from query passed to execute(). You can either use engine.execute() directly or pass additional mapper parameter (either mapper or mapped model corresponding to table used in query) to session.execute() to help SQLAlchemy discover proper engine.
The InternalError says that you are trying to execute this statement inside explicitly (with BEGIN statement) started transaction. Have you issued some statements before it without calling commit()? If so, just call commit() or rollback() method to close transaction before doing VACUUM. Also note, that there are several parameter to sessionmaker() that tell SQLAlchemy when transaction should be started.
If you have access to SQLAlchemy session, you can execute arbitrary SQL statements via its execute method:
session.execute("VACUUM FULL ANALYZE table")
(Depending on the Postgres version) you most likely do not want to run "VACUUM FULL".
I'm new to Python and Python's MySQL adapter. I'm not sure if I'm missing something obvious here:
db = MySQLdb.connect(# db details omitted)
cursor = self.db.cursor()
# WORKS
cursor.execute("SELECT site_id FROM users WHERE username=%s", (username))
record = cursor.fetchone()
# DOES NOT SEEM TO WORK
cursor.execute("DELETE FROM users WHERE username=%s", (username))
Any ideas?
I'd guess that you are using a storage engine that supports transactions (e.g. InnoDB) but you don't call db.commit() after the DELETE. The effect of the DELETE is discarded if you don't commit.
See http://mysql-python.sourceforge.net/FAQ.html#my-data-disappeared-or-won-t-go-away:
Starting with 1.2.0, MySQLdb disables
autocommit by default, as required by
the DB-API standard (PEP-249). If you
are using InnoDB tables or some other
type of transactional table type,
you'll need to do connection.commit()
before closing the connection, or else
none of your changes will be written
to the database.
See also this similar SO question: Python MySQLdb update query fails
Perhaps you are violating a foreign key constraint.
To your code above,
just add a call to self.db.commit().
The feature is far from an annoyance:
It saves you from data corruption issues
when there are errors in your queries.
The problem might be that you are not committing the changes. it can be done by
conn.commit()
read more on this here