I just downloaded sqlite3.exe. It opens up as a command prompt. I created a table test & inserted a few entries in it. I used .backup test just in case. After I exit the program using .exit and reopened it I don't find the table listed under .tables nor can I run any query on it.
I need to quickly run an open source python program that makes use of this table & although I have worked with MySQL, I have no clue about sqlite. I need the minimal basics of sqlite. Can someone guide me through this or at least tell me how to permanently store my tables.
I have put this sqlite3.exe in Python folder assuming that python would then be able to read the sqlite files. Any ideas on this?
sqlite is built in to Python.
You should be able to access your table like this:
import sqlite3
conn = sqlite3.connect('/path/to/my.db')
curs = conn.cursor()
curs.execute("SELECT a_column FROM my_table;").fetchone()
curs.close()
conn.close()
You can execute your DDL statements from Python as well.
Make sure to commit the changes.
curs.execute("CREATE TABLE my_table (a_column text);")
conn.commit()
Why did you download some sqlite3.exe at all? Python should ship with sqlite3 already on board. import sqlite3 is all you need to do as long as you have a recent Python distribution.
To your problem: I would guess that sqlite3 creates a table in memory by default. Using Python, you need to dbConn = sqlite3.connect("somefile") to connect to a database. Then you can use dbCursor = dbConn.cursor() to connect to this file. The cursor can execute SQL commands by calling its execute(command) method. For example:
dbConn.execute("create table test (row text, otherrow real)")
Finally, you need to call dbConn.commit() to save everything you changed in the database.
The Python documentation knows everything else:
http://docs.python.org/library/sqlite3.html
Just execute sqlite3 foo.db? This will permanently store everything you do afterwards in this file. (No need for .backup.)
Related
I have been using cx_Oracle to perform SQL queries on an Oracle database in Python. So far I have been pasting these queries into strings and then running them using the cursor.execute() function that comes with cx_Oracle:
#simple example
query = """SELECT *
FROM my_table"""
cursor.execute(query)
However, my select queries have gotten quite complex, and the code is starting to look a bit messy. I was wondering if there were any way to simply save the SQL code into a .sql file and for Python or cx_Oracle to call that file? I thought something like that might be simple to find using Google, but my searches are oddly coming up dry.
Well, you can certainly save SQL code to a file and load it:
query = open('foo.sql', 'r').read()
cursor.execute(query)
I can't find any reference to saved queries in cx_Oracle, so that may be your best bet.
If I want to create a table and insert a new entry in another table, can this
be made atomic in the sqlite module?
Refering to the docs at http://docs.python.org/2/library/sqlite3.html:
By default, the sqlite3 module opens transactions implicitly before a
Data Modification Language (DML) statement (i.e.
INSERT/UPDATE/DELETE/REPLACE), and commits transactions implicitly
before a non-DML, non-query statement (i. e. anything other than
SELECT or the aforementioned).
So if you are within a transaction and issue a command like CREATE
TABLE ..., VACUUM, PRAGMA, the sqlite3 module will commit implicitly
before executing that command. There are two reasons for doing that.
The first is that some of these commands don’t work within
transactions. The other reason is that sqlite3 needs to keep track of
the transaction state (if a transaction is active or not).
I'm not sure if this second paragraph is meant to apply to automatically started
transactions or to both manual and automatic ones.
Sqlite docs http://www.sqlite.org/lang_transaction.html tell us that manual transactions
would not commit until an explicit COMMIT:
Transactions can be started manually using the BEGIN command. Such
transactions usually persist until the next COMMIT or ROLLBACK
command.
So suppose we have something like this:
con = sqlite3.connect(fdb)
cur = con.cursor()
sql = 'begin transaciton'
cur.execute(sql)
sql = 'CREATE TABLE some-table ...
cur.execute(sql)
# *** is there an implicit commit at this point ?! ***
sql = 'INSERT INTO another-table ...
cur.execute(sql)
con.commit()
Would this be atomic, or would python sqlite make a commit after the create table statement?
Is there a way to make it atomic?
You cannot do this atomically. The Python SQLite library implicitly issues a COMMIT whenever you execute a CREATE TABLE .. statement, because SQLite does not support executing the CREATE TABLE .. statement while a transaction is active.
You can test this by opening the database in both the python interpreter and the sqlite3 command line tool. As soon as you issue the CREATE TABLE .. statement, you can run a .schema command in the sqlite3 command line tool and see the result of that statement.
Note that this means that anything you did in the transaction before the CREATE TABLE .. statement will also have been committed. To look it in another way, the CREATE TABLE .. statement first commits, then starts a completely new transaction.
The Python SQLite3 libary inserts automatic commits even where none are needed.
To make your entire transaction atomic, use any other Python SQLite wrapper, such as, e.g., APSW.
I am writing python code to get movies data from dbpedia and put in MySql database.
presently, my application is having issues and it is hanging in between or if it aborts, then next time I restart, there some issues occur when droping the table and creating table in database.
Is it because last time connection with the database didnt get close?
If so then what can be possible solution to avoid this issue when restarting the application
What python library are you using to access MySQL? If you are using MySQLdb, then to be sure everything is written correctly, you need to use the "close" method of your cursor and the "commit" method of the connection. For example,
import MySQLdb
conn = MySQLdb.connect(user="username",pass="password",db="dbname")
cur = conn.cursor()
# Work with your cur object to do what you want
cur.close()
conn.commit()
Ok first, please assume that this is the first time I will be writing a single alphabet of code in SQL.. so please please please... it would be very kind if you can explain me each step. :)
I am using mac for this development.
So in the desktop, I have a huge (1.5GB) dumpfile of a database.
I have mysqlworkbench (yes I have never worked in that either :( )
And I have installed MySQLdb.
So, basically what I want to do is use the data in the database from python.
So I have python.. mysqldb to connect to db.. huge db and mysqlworkbench..
Now I assume, that this dumpfile is useless as its not in form of database..
so how do I "mount" this dump to a database..(mysqlworkbench)
After that how do I query from python..
(like the port number.. username , password!! ? )
It would be of huge help if anyone can help me.. I have been stuck in this since morning.. :(
In order to use the data in the file, it needs to be imported into the database. (Think of the file as a recipe for MySQL to make a database. Now you need to tell MySQL to make the database from the recipe).
You can import the database using command line, a python script or even MySQL workbench. However, due to the size of the file, using MySQL workbench could be a problem (Note: I am not familiar with MySQL workbench).
To import the database with the command line use the command:
mysql -u <username> -p -h localhost king_tracking < <filename>
To actually use the database with python, there are several step by step tutorials available, easily accessible from a google search. A basic script is (from here)
import MySQLdb
conn = MySQLdb.connect (host = "localhost",
user = "testuser",
passwd = "testpass",
db = "test")
cursor = conn.cursor ()
cursor.execute ("SELECT VERSION()")
row = cursor.fetchone ()
print "server version:", row[0]
cursor.close ()
conn.close ()
This just shows the version of MySQL, however if
"SELECT VERSION()"
is replaced by your own query, like
"SELECT * FROM <tablename> LIMIT 0,30"
you can execute your own queries.
In order to understand the tutorials on interfacing python and MySQL, you should be familiar with both separately. A good SQL tutorial is the one at W3schools.
The problem with creating a table is separate from the original question, but may need to be addressed before trying to import the file into the database.
I have created a Python module that creates and populates several SQLite tables. Now, I want to use it in a program but I don't really know how to call it properly. All the tutorials I've found are essentially "inline", i.e. they walk through using SQLite in a linear fashion rather than how to actually use it in production.
What I'm trying to do is have a method check to see if the database is already created. If so, then I can use it. If not, an exception is raised and the program will create the database. (Or use if/else statements, whichever is better).
I created a test script to see if my logic is correct but it's not working. When I create the try statement, it just creates a new database rather than checking if one already exists. The next time I run the script, I get an error that the table already exists, even if I tried catching the exception. (I haven't used try/except before but figured this is a good time to learn).
Are there any good tutorials for using SQLite operationally or any suggestions on how to code this? I've looked through the pysqlite tutorial and others I found but they don't address this.
Don't make this more complex than it needs to be. The big, independent databases have complex setup and configuration requirements. SQLite is just a file you access with SQL, it's much simpler.
Do the following.
Add a table to your database for "Components" or "Versions" or "Configuration" or "Release" or something administrative like that.
CREATE TABLE REVISION(
RELEASE_NUMBER CHAR(20)
);
In your application, connect to your database normally.
Execute a simple query against the revision table. Here's what can happen.
The query fails to execute: your database doesn't exist, so execute a series of CREATE statements to build it.
The query succeeds but returns no rows or the release number is lower than expected: your database exists, but is out of date. You need to migrate from that release to the current release. Hopefully, you have a sequence of DROP, CREATE and ALTER statements to do this.
The query succeeds, and the release number is the expected value. Do nothing more, your database is configured correctly.
AFAIK an SQLITE database is just a file.
To check if the database exists, check for file existence.
When you open a SQLITE database it will automatically create one if the file that backs it up is not in place.
If you try and open a file as a sqlite3 database that is NOT a database, you will get this:
"sqlite3.DatabaseError: file is encrypted or is not a database"
so check to see if the file exists and also make sure to try and catch the exception in case the file is not a sqlite3 database
SQLite automatically creates the database file the first time you try to use it. The SQL statements for creating tables can use IF NOT EXISTS to make the commands only take effect if the table has not been created This way you don't need to check for the database's existence beforehand: SQLite can take care of that for you.
The main thing I would still be worried about is that executing CREATE TABLE IF EXISTS for every web transaction (say) would be inefficient; you can avoid that by having the program keep an (in-memory) variable saying whether it has created the database today, so it runs the CREATE TABLE script once per run. This would still allow for you to delete the database and start over during debugging.
As #diciu pointed out, the database file will be created by sqlite3.connect.
If you want to take a special action when the file is not there, you'll have to explicitly check for existance:
import os
import sqlite3
if not os.path.exists(mydb_path):
#create new DB, create table stocks
con = sqlite3.connect(mydb_path)
con.execute('''create table stocks
(date text, trans text, symbol text, qty real, price real)''')
else:
#use existing DB
con = sqlite3.connect(mydb_path)
...
Sqlite doesn't throw an exception if you create a new database with the same name, it will just connect to it. Since sqlite is a file based database, I suggest you just check for the existence of the file.
About your second problem, to check if a table has been already created, just catch the exception. An exception "sqlite3.OperationalError: table TEST already exists" is thrown if the table already exist.
import sqlite3
import os
database_name = "newdb.db"
if not os.path.isfile(database_name):
print "the database already exist"
db_connection = sqlite3.connect(database_name)
db_cursor = db_connection.cursor()
try:
db_cursor.execute('CREATE TABLE TEST (a INTEGER);')
except sqlite3.OperationalError, msg:
print msg
Doing SQL in overall is horrible in any language I've picked up. SQLalchemy has shown to be easiest from them to use because actual query and committing with it is so clean and absent from troubles.
Here's some basic steps on actually using sqlalchemy in your app, better details can be found from the documentation.
provide table definitions and create ORM-mappings
load database
ask it to create tables from the definitions (won't do so if they exist)
create session maker (optional)
create session
After creating a session, you can commit and query from the database.
See this solution at SourceForge which covers your question in a tutorial manner, with instructive source code :
y_serial.py module :: warehouse Python objects with SQLite
"Serialization + persistance :: in a few lines of code, compress and annotate Python objects into SQLite; then later retrieve them chronologically by keywords without any SQL. Most useful "standard" module for a database to store schema-less data."
http://yserial.sourceforge.net
Yes, I was nuking out the problem. All I needed to do was check for the file and catch the IOError if it didn't exist.
Thanks for all the other answers. They may come in handy in the future.