Develop Reference

Sql Alchemy Insert Statement failing to insert, but no error

I am attempting to execute a raw sql insert statement in Sqlalchemy, SQL Alchemy throws no errors when the constructed insert statement is executed but the lines do not appear in the database.
As far as I can tell, it isn't a syntax error (see no 2), it isn't an engine error as the ORM can execute an equivalent write properly (see no 1), it's finding the table it's supposed to write too (see no 3). I think it's a problem with a transaction not being commited and have attempted to address this (see no 4) but this hasn't solved the issue. Is it possible to create a nested transaction and what would start the 'first' so to speak?
Thankyou for any answers.
Some background:
I know that the ORM facilitates this and have used this feature and it works, but is too slow for our application. We decided to try using raw sql for this particular write function due to how often it's called and the ORM for everything else. An equivalent method using the ORM works perfectly, and the same engine is used for both, so it can't be an engine problem right?
I've issued an example of the SQL that the method using raw sql constructs to the database directly and that reads in fine, so I don't think it's a syntax error.
it's communicating with the database properly and can find the table as any syntax errors with table and column names throw a programmatic error so it's not just throwing stuff into the 'void' so to speak.
My first thought after reading around was that it was transaction error and that a transaction was being created and not closed, and so constructed the execute statement as such to ensure a transaction was properly created and commited.
with self.Engine.connect() as connection:
connection.execute(Insert_Statement)
connection.commit
The so called 'Insert Statement' has been converted to text using the sqlalchemy 'text' function, I don't quite understand why it won't execute if I pass the constructed string directly to the execute statement but mention it in case it's relevant.
Other things that may be relevant:
Python3 is running on an individual ec2 instance the postgres database on another. The table in particular is a timescaledb hypertable taking realtime data, hence the need for very fast writes, but probably not relevant.
Currently using pg8000 as dialect for no particular reason other than psycopg2 was throwing errors when trying the execute an equivalent method using the ORM.

Just so this question is answered in case anyone else ends up here:
The issue was a failure to call commit as a method, as #snakecharmerb pointed out. Gord Thompson also provided an alternate method using 'begin' which automatically commits rather than connection which is a 'commit as you go' style transaction.

Why does having no autocommit facility mean that all queries execute within a transaction in PostgreSQL?

From https://wiki.postgresql.org/wiki/Psycopg2_Tutorial
PostgreSQL does not have an autocommit facility which means that all
queries will execute within a transaction.
Execution within a transaction is a very good thing, it ensures data
integrity and allows for appropriate error handling. However there are
queries that can not be run from within a transaction. Take the
following example.
#/usr/bin/python2.4
#
#
import psycopg2
# Try to connect
try:
conn=psycopg2.connect("dbname='template1' user='dbuser' password='mypass'")
except:
print "I am unable to connect to the database."
cur = conn.cursor()
try:
cur.execute("""DROP DATABASE foo_test""")
except:
print "I can't drop our test database!"
This code would actually fail with the printed message of "I can't
drop our test database!" PostgreSQL can not drop databases within a
transaction, it is an all or nothing command. If you want to drop the
database you would need to change the isolation level of the database
this is done using the following.
conn.set_isolation_level(0)
You would place the above immediately preceding the DROP DATABASE
cursor execution.
I was wondering why
"PostgreSQL does not have an autocommit facility which means that all queries will execute within a transaction."
"PostgreSQL can not drop databases within a transaction"
"If you want to drop the database you would need to change the isolation level of the database"
Thanks.
Update:
What does autocommit mean in postgresql and psycopg2? answer my question

All the 3 are related to Python and its DB connector library, not the PostgreSQL itself:
PostgreSQL has an autocommit and it is active by default - which means that every SQL statement is immediately executed. When you start a transaction block, this autocommit mode is disabled until you finish the transaction (either by COMMIT or ROLLBACK)
The operation of destroying a database is implemented in a way where you can not run it from inside a transaction block. Also keep in mind that unlike most other databases PostgreSQL allows almost all DDL statements (obviously not the DROP DATABASE one) to be executed inside a transaction.
Actually you can not drop a database if anyone (including you) is currently connected to this database - so it does not matter what is your isolation level, you still have to connect to another database (e.g. postgres)

No Error, but can't input data to MySQL

I have this code to insert data to Database using MySQL. But when I ran that code using Python, there's no error. But when I checked the Database, the data isn't inserted. Is there anyone who can help me? I would appreciate it. :)
This is the code:
import MySQLdb
db=MySQLdb.connect(host="localhost", user="root", passwd="", db="try")
cursor=db.cursor()
insert="INSERT INTO `try`.`try` (`nomor`, `nama`) VALUES (NULL, 'bismillah')"
cursor.execute(insert)

You're not doing a COMMIT anywhere. So, if auto-commit is not on, all you've done is create a transaction that, if later committed, will insert this row.
Since you haven't done a SET AUTOCOMMIT anywhere, whether auto-commit is on depends on how you created the database. With at least some storage types (in particular, InnoDB), you can change the default at creation time, and, because you often want auto-commit disabled with those storage types, your GUI design tool, or the sample code you copied and pasted, or whatever may have done so for you. Also, the server variable that provides the default can itself be set to a different value at server startup/configuration. (See System Server Variables.)
If you want to make sure that auto-commit is on, just execute SET autocommit=1 before any other statements.
If you want to find out whether auto-commit is on, execute SHOW VARIABLES. (And if it's disabled, you may want to try SHOW GLOBAL VARIABLES LIKE 'autocommit' and SHOW SESSION VARIABLES like 'autocommit' to see which context you've disabled it in.)

If you cannot insert into mysql, there are several ways to solve it:
1: check the log
2: check the structure of your table, maybe it must not be null for nomor or other field
3: the last, when insert into mysqldb using program, you need to commit before you close the connection. for here: db.commit()

Execute sql query with Elixir

I'm using Elixir in a project that connects to a postgres database. I want to run the following query on the database I'm connected to, but I'm not sure how to do it as I'm rather new to Elixir and SQLAlchemy. Anyone know how?
VACUUM FULL ANALYZE table
Update
The error is: "UnboundExecutionError: Could not locate a bind configured on SQL expression or this Session". And the same result with session.close() issued before. I did try doing metadata.bind.execute() and that worked for a simple select. But for the VACUUM it said - "InternalError: (InternalError) VACUUM cannot run inside a transaction block", so now I'm trying to figure out how to turn that off.
Update 2
I can get the query to execute, but I'm still getting the same error - even when I create a new session and close the previous one.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# ... insert stuff
old_session.commit()
old_session.close()
new_sess = sessionmaker(autocommit=True)
new_sess.configure(bind=create_engine('postgres://user:pw#host/db', echo=True))
sess = new_sess()
sess.execute('VACUUM FULL ANALYZE table')
sess.close()
and the output I get is
2009-12-10 10:00:16,769 INFO sqlalchemy.engine.base.Engine.0x...05ac VACUUM FULL ANALYZE table
2009-12-10 10:00:16,770 INFO sqlalchemy.engine.base.Engine.0x...05ac {}
2009-12-10 10:00:16,770 INFO sqlalchemy.engine.base.Engine.0x...05ac ROLLBACK
finishing failed run, (InternalError) VACUUM cannot run inside a transaction block
'VACUUM FULL ANALYZE table' {}
Update 3
Thanks to everyone who responded. I wasn't able to find the solution I wanted, but I think I'm just going to go with the one described here PostgreSQL - how to run VACUUM from code outside transaction block?. It's not ideal, but it works.

Dammit. I knew the answer was going to be right under my nose. Assuming you setup your connection like I did.
metadata.bind = 'postgres://user:pw#host/db'
The solution to this was as simple as
conn = metadata.bind.engine.connect()
old_lvl = conn.connection.isolation_level
conn.connection.set_isolation_level(0)
conn.execute('vacuum analyze table')
conn.connection.set_isolation_level(old_lvl)
This is similar to what was suggested here PostgreSQL - how to run VACUUM from code outside transaction block?
because underneath it all, sqlalchemy uses psycopg to make the connection to postgres. Connection.connection is a proxy to the psycopg connection. Once I realized this, this problem came back to mind and I decided to take another whack at it.
Hopefully this helps someone.

You need to bind the session to an engine
session.bind = metadata.bind
session.execute('YOUR SQL STATEMENT')

UnboundExecutionError says that your session is not bound to an engine and there is no way to discover engine from query passed to execute(). You can either use engine.execute() directly or pass additional mapper parameter (either mapper or mapped model corresponding to table used in query) to session.execute() to help SQLAlchemy discover proper engine.
The InternalError says that you are trying to execute this statement inside explicitly (with BEGIN statement) started transaction. Have you issued some statements before it without calling commit()? If so, just call commit() or rollback() method to close transaction before doing VACUUM. Also note, that there are several parameter to sessionmaker() that tell SQLAlchemy when transaction should be started.

If you have access to SQLAlchemy session, you can execute arbitrary SQL statements via its execute method:
session.execute("VACUUM FULL ANALYZE table")

(Depending on the Postgres version) you most likely do not want to run "VACUUM FULL".

Using SQLite in a Python program

I have created a Python module that creates and populates several SQLite tables. Now, I want to use it in a program but I don't really know how to call it properly. All the tutorials I've found are essentially "inline", i.e. they walk through using SQLite in a linear fashion rather than how to actually use it in production.
What I'm trying to do is have a method check to see if the database is already created. If so, then I can use it. If not, an exception is raised and the program will create the database. (Or use if/else statements, whichever is better).
I created a test script to see if my logic is correct but it's not working. When I create the try statement, it just creates a new database rather than checking if one already exists. The next time I run the script, I get an error that the table already exists, even if I tried catching the exception. (I haven't used try/except before but figured this is a good time to learn).
Are there any good tutorials for using SQLite operationally or any suggestions on how to code this? I've looked through the pysqlite tutorial and others I found but they don't address this.

Don't make this more complex than it needs to be. The big, independent databases have complex setup and configuration requirements. SQLite is just a file you access with SQL, it's much simpler.
Do the following.
Add a table to your database for "Components" or "Versions" or "Configuration" or "Release" or something administrative like that.
CREATE TABLE REVISION(
RELEASE_NUMBER CHAR(20)
);
In your application, connect to your database normally.
Execute a simple query against the revision table. Here's what can happen.
The query fails to execute: your database doesn't exist, so execute a series of CREATE statements to build it.
The query succeeds but returns no rows or the release number is lower than expected: your database exists, but is out of date. You need to migrate from that release to the current release. Hopefully, you have a sequence of DROP, CREATE and ALTER statements to do this.
The query succeeds, and the release number is the expected value. Do nothing more, your database is configured correctly.

AFAIK an SQLITE database is just a file.
To check if the database exists, check for file existence.
When you open a SQLITE database it will automatically create one if the file that backs it up is not in place.
If you try and open a file as a sqlite3 database that is NOT a database, you will get this:
"sqlite3.DatabaseError: file is encrypted or is not a database"
so check to see if the file exists and also make sure to try and catch the exception in case the file is not a sqlite3 database

SQLite automatically creates the database file the first time you try to use it. The SQL statements for creating tables can use IF NOT EXISTS to make the commands only take effect if the table has not been created This way you don't need to check for the database's existence beforehand: SQLite can take care of that for you.
The main thing I would still be worried about is that executing CREATE TABLE IF EXISTS for every web transaction (say) would be inefficient; you can avoid that by having the program keep an (in-memory) variable saying whether it has created the database today, so it runs the CREATE TABLE script once per run. This would still allow for you to delete the database and start over during debugging.

As #diciu pointed out, the database file will be created by sqlite3.connect.
If you want to take a special action when the file is not there, you'll have to explicitly check for existance:
import os
import sqlite3
if not os.path.exists(mydb_path):
#create new DB, create table stocks
con = sqlite3.connect(mydb_path)
con.execute('''create table stocks
(date text, trans text, symbol text, qty real, price real)''')
else:
#use existing DB
con = sqlite3.connect(mydb_path)
...

Sqlite doesn't throw an exception if you create a new database with the same name, it will just connect to it. Since sqlite is a file based database, I suggest you just check for the existence of the file.
About your second problem, to check if a table has been already created, just catch the exception. An exception "sqlite3.OperationalError: table TEST already exists" is thrown if the table already exist.
import sqlite3
import os
database_name = "newdb.db"
if not os.path.isfile(database_name):
print "the database already exist"
db_connection = sqlite3.connect(database_name)
db_cursor = db_connection.cursor()
try:
db_cursor.execute('CREATE TABLE TEST (a INTEGER);')
except sqlite3.OperationalError, msg:
print msg

Doing SQL in overall is horrible in any language I've picked up. SQLalchemy has shown to be easiest from them to use because actual query and committing with it is so clean and absent from troubles.
Here's some basic steps on actually using sqlalchemy in your app, better details can be found from the documentation.
provide table definitions and create ORM-mappings
load database
ask it to create tables from the definitions (won't do so if they exist)
create session maker (optional)
create session
After creating a session, you can commit and query from the database.

See this solution at SourceForge which covers your question in a tutorial manner, with instructive source code :
y_serial.py module :: warehouse Python objects with SQLite
"Serialization + persistance :: in a few lines of code, compress and annotate Python objects into SQLite; then later retrieve them chronologically by keywords without any SQL. Most useful "standard" module for a database to store schema-less data."
http://yserial.sourceforge.net

Yes, I was nuking out the problem. All I needed to do was check for the file and catch the IOError if it didn't exist.
Thanks for all the other answers. They may come in handy in the future.