I'm looking to run the following test.sql located in a folder on my C: drive. I've been playing with cx_Oracle and just can't get it to work.
test.sql contains the following.
CREATE TABLE MURRAYLR.test
( customer_id number(10) NOT NULL,
customer_name varchar2(50) NOT NULL,
city varchar2(50)
);
CREATE TABLE MURRAYLR.test2
( customer_id number(10) NOT NULL,
customer_name varchar2(50) NOT NULL,
city varchar2(50)
);
This is my code:
import sys
import cx_Oracle
connection = cx_Oracle.connect('user,'password,'test.ora')
cursor = connection.cursor()
f = open("C:\Users\desktop\Test_table.sql")
full_sql = f.read()
sql_commands = full_sql.split(';')
for sql_command in sql_commands:
cursor.execute(sql_command)
cursor.close()
connection.close()
This answer is relevant only if your test.sql file contains new lines '\n\' characters (like mine which I got from copy-pasting your sql code). You will need to remove them in your code, if they are present. To check, do
print full_sql
To fix the '\n's,
sql_commands = full_sql.replace('\n', '').split(';')[:-1]
The above should help.
It removes the '\n's and removes the empty string token at the end when splitting the sql string.
MURRAYLR.test is not acceptable table name in any DBMS I've used. The connection object the cx_oracle.connect returns should already have a schema selected. To switch to a different schema set the current_schema field on the connection object or add using <Schemaname>; in your sql file.
Obviously make sure that the schema exists.
Related
I'm making a car rental console base program in Python where I need to save data about cars I store (such as brand, registration number etc).
What would be the ideal type of file for such a thing, and how to iniciate it?
You can use sqlite3 to store the information.
You can create a table with columns such as brand,registration number etc.
If the registration number is unique to single type of car you can also take care of that condition in sqlite3
syntax is as simple as:
For creating table:
import sqlite3
conn = sqlite3.connect('test.db')
print "Opened database successfully";
conn.execute('''CREATE TABLE COMPANY
(REGISTRATION_NO INT PRIMARY KEY NOT NULL,
BRAND TEXT NOT NULL
);''')
print "Table created successfully";
conn.close()
For insertion:
import sqlite3
conn = sqlite3.connect('test.db')
print "Opened database successfully";
conn.execute("INSERT INTO COMPANY (REGISTRATION_NO,BRAND) \
VALUES (1, 'PAGANI')");
conn.commit()
conn.close()
For more information:
https://docs.python.org/2/library/sqlite3.html
I want to use sqlite3 to deal with data in Ubuntu with python. But I always failed and get errors. Codes related to database are as follows:
sqlite = "%s.db" % name
#connnect to the database
conn = sqlite3.connect(sqlite)
print "Opened database successfully"
c = conn.cursor()
#set default separator to "\t" in database
c.execute(".separator "\t"")
print "Set separator of database successfully"
#create table data_node
c.execute('''create table data_node(Time int,Node Text,CurSize int,SizeVar int,VarRate real,Evil int);''')
print "Table data_node created successfully"
node_info = "%s%s.txt" % (name,'-PIT-node')
c.execute(".import %\"s\" data_node") % node_info
print "Import to data_node successfully"
#create table data_face
data_info = "%s%s.txt" % (name,'-PIT-face')
c.execute('''create table data_face(Time int,Node Text,TotalEntry real,FaceId int,FaceEntry real,Evil int);''')
c.execute(".import \"%s\" data_face") % face_info
#get the final table : PIT_node
c.execute('''create table node_temp as select FIRST.Time,FIRST.Node,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio from tracer_temp FIRST,tracer_temp SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.Type='InData' AND SECOND.Type='OutInterests';''')
c.execute('''create table PIT_node as select A.Time,A.Node,B.SatisRatio,A.CurSize,A.SizeVar,A.VarRate,A.Evil from data_node A,node_temp B WHERE A.Time=B.Time AND A.Node=B.Node;''')
#get the final table : PIT_face
c.execute('''create table face_temp as select FIRST.Time,FIRST.Node,FIRST.FaceId,ROUND(FIRST.PacketsRaw/SECOND.PacketsRaw,4) as SatisRatio,SECOND.Packets from data_tracer FIRST,data_tracer SECOND WHERE FIRST.Time=SECOND.Time AND FIRST.Node=SECOND.Node AND FIRST.FaceId=SECOND.FaceId AND FIRST.Type='OutData' AND SECOND.Type='InInterests';''')
c.execute('''create table PIT_face as select A.Time,A.Node,A.FaceId,B.SatisRatio,B.Packets,ROUND(A.FaceEntry/A.TotalEntry,4),A.Evil from data_face as A,face_temp as B WHERE A.Time=B.Time AND A.Node=B.Node AND A.FaceId = B.FaceId;''')
conn.commit()
conn.close()
These sql-commands are right. When I run the code, it always shows sqlite3.OperationalError: near ".": syntax error. So how to change my code and are there other errors in other commands such as create table?
You have many problems in your code as posted, but the one you're asking about is:
c.execute(".separator "\t"")
This isn't valid Python syntax. But, even if you fix that, it's not valid SQL.
The "dot-commands" are special commands to the sqlite3 command line shell. It intercepts them and uses them to configure itself. They mean nothing to the actual database, and cannot be used from Python.
And most of them don't make any sense outside that shell anyway. For example, you're trying to set the column separator here. But the database doesn't return strings, it returns row objects—similar to lists. There is nowhere for a separator to be used. If you want to print the rows out with tab separators, you have to do that in your own print statements.
So, the simple fix is to remove all of those dot-commands.
However, there is a problem—at least one of those dot-commands actually does something:
c.execute(".import %\"s\" data_node") % node_info
You will have to replace that will valid calls to the library that do the same thing as the .import dot-command. Read what it does, and it should be easy to understand. (You basically want to open the file, parse the columns for each row, and do an executemany on an INSERT with the rows.)
As per Unload to S3 with Python using IAM Role credentials, the unload statement worked perfectly. So did other commands I tried, like copy and select statements.
However, I also tried to run a query which creates a table.. The create table query runs without error, but when it gets to the select statement, it throws an errors that relation "public.test" does not exist.
Any idea why is the table not created properly? Query below:
import sqlalchemy as sa
from sqlalchemy.orm import sessionmaker
import config
import pandas as pd
#>>>>>>>> MAKE CHANGES HERE >>>>>>>>
DATABASE = "db"
USER = "user"
PASSWORD = getattr(config, 'password') #see answer by David Bern https://stackoverflow.com/questions/43136925/create-a-config-file-to-hold-values-like-username-password-url-in-python-behave/43137301
HOST = "host"
PORT = "5439"
SCHEMA = "public" #default is "public"
########## connection and session creation ##########
connection_string = "redshift+psycopg2://%s:%s#%s:%s/%s" % (USER,PASSWORD,HOST,str(PORT),DATABASE)
engine = sa.create_engine(connection_string)
session = sessionmaker()
session.configure(bind=engine)
s = session()
SetPath = "SET search_path TO %s" % SCHEMA
s.execute(SetPath)
--create table example
query2 = '''\
create table public.test (
id integer encode lzo,
user_id integer encode lzo,
created_at timestamp encode delta32k,
updated_at timestamp encode delta32k
)
distkey(id)
sortkey(id)
'''
r2 = s.execute(query2)
--select example
query4 = '''\
select * from public.test
'''
r4 = s.execute(query4)
########## create DataFrame from SQL query output ##########
df = pd.read_sql_query(query4, connection_string)
print(df.head(50))
########## close session in the end ##########
s.close()
If I run the same directly in Redshift, it works just fine..
--Edit--
Some of the things tried:
Removing "\" from query string
adding ";" at the end of query string
changing "public.test" to "test"
removing SetPath = "SET search_path TO %s" % SCHEMA and s.execute(SetPath)
breaking the create statement- generates expected error
adding copy from S3 command after create- runs without error, but again no table created
adding a column to create statement that doesnt exist in the file that is generated from the copy command- generates expected error
adding r4 = s.execute(query4)- runs without error, but again created table not in Redshift
Apparently need to add s.commit() in order to create the table.. If you are populating it via copy command or insert into: then add it after the copy command (after the create table is optional). Basically, it does not auto commit for create/alter commands!
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#session-faq-whentocreate
http://docs.sqlalchemy.org/en/latest/core/connections.html#understanding-autocommit
I have written a small test application using SQLite with Python 3.3:
import sqlite3
MDB = sqlite3.connect('D:\MDB.db') # create the db object
cursor = MDB.cursor() # assign a cursor
cursor.execute('''CREATE TABLE IF NOT EXISTS section (
Code INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
Description TEXT )
''')
cursor.execute('''DELETE FROM section''') # delete contents for reruns
cursor.execute('''INSERT INTO section
(Description)
VALUES (?)
''', ('Abdul, Paula',))
cursor.execute('''INSERT INTO section
(Description)
VALUES (?)
''', ('ABWH',))
print('Results:\n')
cursor.execute('''SELECT * FROM section''')
selection = cursor.fetchall()
for row in selection:
print('\t', row)
The SELECT statement shows the results expected (seeming to indicate that the row exists), but if I connect to the database with SQLite-Manager, the table exists but is empty, and if I try the same query with another script connected to the database, nothing is returned. Can anyone please explain what I am doing wrong?
You're not saving changes (calling MDB.commit).
I'm having troubles with creating a database and tables. The database needs to be created within a Python script.
#connect method has 4 parameters:
#localhost (where mysql db is located),
#database user name,
#account password,
#database name
db1 = MS.connect(host="localhost",user="root",passwd="****",db="test")
returns
_mysql_exceptions.OperationalError: (1049, "Unknown database 'test'")
So clearly, the db1 needs to be created first, but how? I've tried CREATE before the connect() statement but get errors.
Once the database is created, how do I create tables?
Thanks,
Tom
Here is the syntax, this works, at least the first time around. The second time naturally returns that the db already exists. Now to figure out how to use the drop command properly.
db = MS.connect(host="localhost",user="root",passwd="****")
db1 = db.cursor()
db1.execute('CREATE DATABASE test1')
So this works great the first time through. The second time through provides a warning "db already exists". How to deal with this? The following is how I think it should work, but doesn't. OR should it be an if statement, looking for if it already exists, do not populate?
import warnings
warnings.filterwarnings("ignore", "test1")
Use CREATE DATABASE to create the database:
db1 = MS.connect(host="localhost",user="root",passwd="****")
cursor = db1.cursor()
sql = 'CREATE DATABASE mydata'
cursor.execute(sql)
Use CREATE TABLE to create the table:
sql = '''CREATE TABLE foo (
bar VARCHAR(50) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1
'''
cursor.execute(sql)
There are a lot of options when creating a table. If you are not sure what the right SQL should be, it may help to use a graphical tool like phpmyadmin to create a table, and then use SHOW CREATE TABLE to discover what SQL is needed to create it:
mysql> show create table foo \G
*************************** 1. row ***************************
Table: foo
Create Table: CREATE TABLE `foo` (
`bar` varchar(50) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
phpmyadmin can also show you what SQL it used to perform all sorts of operations. This can be a convenient way to learn some basic SQL.
Once you've experimented with this, then you can write the SQL in Python.
I think the solution is a lot easier, use "if not":
sql = "CREATE DATABASE IF NOT EXISTS test1"
db1.execute(sql)
import MySQLdb
# Open database connection ( If database is not created don't give dbname)
db = MySQLdb.connect("localhost","yourusername","yourpassword","yourdbname" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# For creating create db
# Below line is hide your warning
cursor.execute("SET sql_notes = 0; ")
# create db here....
cursor.execute("create database IF NOT EXISTS yourdbname")
# create table
cursor.execute("SET sql_notes = 0; ")
cursor.execute("create table IF NOT EXISTS test (email varchar(70),pwd varchar(20));")
cursor.execute("SET sql_notes = 1; ")
#insert data
cursor.execute("insert into test (email,pwd) values('test#gmail.com','test')")
# Commit your changes in the database
db.commit()
# disconnect from server
db.close()
#OUTPUT
mysql> select * from test;
+-----------------+--------+
| email | pwd |
+-----------------+--------+
| test#gmail.com | test |
+-----------------+--------+