I'm currently trying to create a database using SQLlite3 with Python, however I'm having trouble setting up a Primary Key. I'm aware of what one is, and how it uniquely identifies the table, but I want to change it from the standard "rowid" it comes with to the current date. When I try to add things into the table however it comes up with this:
File "Economic_Analyser.py", line 258, in <module>
startup()
File "Economic_Analyser.py", line 243, in startup
c.execute("INSERT INTO economicdata VALUES (dateandtime, up_GDPgrowthRate, up_GDP, up_GNP, up_GDPperCapita, up_GDPagriculture, up_GDPconstruction, up_GDPmanufacturing, up_GDPmining, up_GDPpublicadmin, up_GDPservices, up_GDPtransport, up_GDPtourism, up_UnemploymentRate, up_EmploymentRate, up_InflationRate, up_CPI, up_InterestRate, up_BalanceOfTrade, up_CurrentAccount, up_Imports, up_Exports, up_FDI, up_GovernmentSpending, up_GovernmentDebt, up_BusinessConfidence, up_Bankruptcies, up_CompetitiveRank, up_CorruptionRank, up_ConsumerConfidence, up_CorporateTaxRate, up_IncomeTaxRate)")
sqlite3.OperationalError: no such column: dateandtime
As you can see from my actual code below, I've declared that the date is the Primary Key but cannot add the data to it. I've changed the code multiple times based on what I've seen other people do but it hasn't worked. Just to clarify this isn't all of my code - just the parts that I think matter. Any help would be appreciated!!
try:
data_attempt = open("Economic_Analyser.db")
except:
print("- Database not found. Creating 'Economic_Analyser.db' .")
databasevariables()
c.execute("""CREATE TABLE economicdata (
dateandtime text NOT NULL PRIMARY KEY,
GDPgrowthRate decimal NOT NULL,
GDP decimal NOT NULL,
GNP decimal NOT NULL,
GDPperCapita decimal NOT NULL,
GDPagriculture decimal NOT NULL,
GDPconstruction decimal NOT NULL,
GDPmanufacturing decimal NOT NULL,
GDPmining decimal NOT NULL,
GDPpublicadmin decimal NOT NULL,
GDPservices decimal NOT NULL,
GDPtransport decimal NOT NULL,
GDPtourism decimal NOT NULL,
UnemploymentRate decimal NOT NULL,
EmploymentRate decimal NOT NULL,
InflationRate decimal NOT NULL,
CPI decimal NOT NULL,
InterestRate decimal NOT NULL,
BalanceOfTrade decimal NOT NULL,
CurrentAccount decimal NOT NULL,
Imports decimal NOT NULL,
Exports decimal NOT NULL,
FDI decimal NOT NULL,
GovernmentSpending decimal NOT NULL,
GovernmentDebt decimal NOT NULL,
BusinessConfidence decimal NOT NULL,
Bankruptcies decimal NOT NULL,
CompetitiveRank decimal NOT NULL,
CorruptionRank decimal NOT NULL,
ConsumerConfidence decimal NOT NULL,
CorporateTaxRate decimal NOT NULL,
IncomeTaxRate decimal NOT NULL
)""")
conn.commit()
c.execute("""CREATE TABLE users (
username text,
password text
)""")
conn.commit()
conn.close()
if internet_access == True:
databasevariables()
c.execute("INSERT INTO economicdata VALUES (dateandtime, up_GDPgrowthRate, up_GDP, up_GNP, up_GDPperCapita, up_GDPagriculture, up_GDPconstruction, up_GDPmanufacturing, up_GDPmining, up_GDPpublicadmin, up_GDPservices, up_GDPtransport, up_GDPtourism, up_UnemploymentRate, up_EmploymentRate, up_InflationRate, up_CPI, up_InterestRate, up_BalanceOfTrade, up_CurrentAccount, up_Imports, up_Exports, up_FDI, up_GovernmentSpending, up_GovernmentDebt, up_BusinessConfidence, up_Bankruptcies, up_CompetitiveRank, up_CorruptionRank, up_ConsumerConfidence, up_CorporateTaxRate, up_IncomeTaxRate)")
conn.commit()
conn.close()
print("- Most recent data has been saved.")
else:
print("- Failed.")
def databasevariables():
global conn
conn = sqlite3.connect("Economic_Analyser.db")
global c
c = conn.cursor()
You're putting the column names in the wrong place in your INSERT statement. They should go immediately after the tablename enclosed in parens/brackets, then you need a placeholder for each value to be inserted.
Instead you may do something like this:
columns = ['dateandtime',
'up_GDPgrowthRate',
'up_GDP',
'up_GNP',
'up_GDPperCapita',
'up_GDPagriculture',
'up_GDPconstruction',
'up_GDPmanufacturing',
'up_GDPmining',
'up_GDPpublicadmin',
'up_GDPservices',
'up_GDPtransport',
'up_GDPtourism',
'up_UnemploymentRate',
'up_EmploymentRate',
'up_InflationRate',
'up_CPI',
'up_InterestRate',
'up_BalanceOfTrade',
'up_CurrentAccount',
'up_Imports',
'up_Exports',
'up_FDI',
'up_GovernmentSpending',
'up_GovernmentDebt',
'up_BusinessConfidence',
'up_Bankruptcies',
'up_CompetitiveRank',
'up_CorruptionRank',
'up_ConsumerConfidence',
'up_CorporateTaxRate',
'up_IncomeTaxRate']
placeholders = ",".join('?'*len(columns))
insert_stmt = f"""INSERT INTO economicdata ({columns})
VALUES ({placeholders});"""
c.execute(insert_stmt)
Related
The following code results in None () across the row in every attempt. The query.values() code below is just a shortened line so as to keep things less complicated. Additionally I have problems inserting a dict as JSON in the address fields but that's another question.
CREATE TABLE public.customers (
id SERIAL,
email character varying(255) NULL,
name character varying(255) NULL,
phone character varying(16) NULL,
address jsonb NULL,
shipping jsonb NULL,
currency character varying(3) NULL,
metadata jsonb[] NULL,
created bigint NULL,
uuid uuid DEFAULT uuid_generate_v4() NOT NULL,
PRIMARY KEY (uuid)
);
from sqlalchemy import *
from sqlalchemy.orm import Session
# Create engine, metadata, & session
engine = create_engine('postgresql://postgres:password#database/db', future=True)
metadata = MetaData(bind=engine)
session = Session(engine)
# Create Table
customers = Table('customers', metadata, autoload_with=engine)
query = customers.insert()
query.values(email="test#test.com", \
name="testy testarosa", \
phone="+12125551212", \
address='{"city": "Cities", "street": "123 Main St", \
"state": "CA", "zip": "10001"}')
session.execute(query)
session.commit()
session.close()
# Now to see results
stmt = text("SELECT * FROM customers")
response = session.execute(stmt)
for result in response:
print(result)
# Results in None in the fields I explicitly attempted
(1, None, None, None, None, None, None, None, 1, None, None, None, None, UUID('9112a420-aa36-4498-bb56-d4129682681c'))
Calling query.values() returns a new insert instance, rather than modifying the existing instance in-place. This return value must be assigned to a variable otherwise it will have no effect.
You could build the insert iteratively
query = customers.insert()
query = query.values(...)
session.execute(query)
or chain the calls as Karolus K. suggests in their answer.
query = customers.insert().values(...)
Regarding the address column, you are inserting a dict already serialised as JSON. This value gets serialised again during insertion, so the value in the database ends up looking like this:
test# select address from customers;
address
══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
"{\"city\": \"Cities\", \"street\": \"123 Main St\", \"state\": \"CA\", \"zip\": \"10001\"}"
(1 row)
and is not amenable to being queried as a JSON object (because it's a JSONified string)
test# select address->'state' AS state from customers;
state
═══════
¤
(1 row)
You might find it better to pass the raw dict instead, resulting in this value being stored in the database:
test# select address from customers;
address
════════════════════════════════════════════════════════════════════════════
{"zip": "10001", "city": "Cities", "state": "CA", "street": "123 Main St"}
(1 row)
which is amenable to being queried as a JSON object:
test# select address->'state' AS state from customers;
state
═══════
"CA"
(1 row)
I am not sure what do you mean with
The query.values() code below is just a shortened line so as to keep
things less complicated.
So maybe I am not understanding the issue properly.
At any case the problem here is that you execute the insert() and the values() separately, while it is meant to be "chained".
Doing something like:
query = customers.insert().values(email="test#test.com", name="testy testarosa", phone="+12125551212", address='{"city": "Cities", "street": "123 Main St", "state": "CA", "zip": "10001"}')
will work.
Documentation: https://docs.sqlalchemy.org/en/14/core/selectable.html#sqlalchemy.sql.expression.TableClause.insert
PS: I did not faced any issues with the JSON field as well. Perhaps something with PG version?
I have a tuple as below
data = ({'weather station name': 'Substation', 'wind': '6 km/h', 'barometer': '1010.3hPa', 'humidity': '42%', 'temperature': '34.5 C', 'place_id': '001D0A00B36E', 'date': '2016-05-10 09:48:58'})
I am trying to push the values from the above tuple to the postgres table using the code below:
try:
con = psycopg2.connect("dbname='WeatherForecast' user='postgres' host='localhost' password='****'")
cur = con.cursor()
cur.executemany("""INSERT INTO weather_data(temperature,humidity,wind,barometer,updated_on,place_id) VALUES (%(temperature)f, %(humidity)f, %(wind)f, %(barometer)f, %(date)s, %(place_id)d)""", final_weather_data)
ver = cur.fetchone()
print(ver)
except psycopg2.DatabaseError as e:
print('Error {}'.format(e))
sys.exit(1)
finally:
if con:
con.close()
Where datatype of each field in the DB is as below:
id serial NOT NULL,
temperature double precision NOT NULL,
humidity double precision NOT NULL,
wind double precision NOT NULL,
barometer double precision NOT NULL,
updated_on timestamp with time zone NOT NULL,
place_id integer NOT NULL,
When i run the code to push the data into postgres table using psycopg2, it is raising an error "ValueError: unsupported format character 'f'"
I hope the issue is in formatting. Am using Python3.4
Have a look at the documentation:
The variables placeholder must always be a %s, even if a different placeholder (such as a %d for integers or %f for floats) may look more appropriate:
>>> cur.execute("INSERT INTO numbers VALUES (%d)", (42,)) # WRONG
>>> cur.execute("INSERT INTO numbers VALUES (%s)", (42,)) # correct
While, your SQL query contains all type of placeholders:
"""INSERT INTO weather_data(temperature,humidity,wind,barometer,updated_on,place_id)
VALUES (%(temperature)f, %(humidity)f, %(wind)f, %(barometer)f, %(date)s, %(place_id)d)"""
I'm trying to import a simple CSV file that I downloaded from Quandl into a MySQL table with the odo python package
t = odo('C:\ProgramData\MySQL\MySQL Server 5.6\Uploads\WIKI_20160725.partial.csv', 'mysql+pymysql://' + self._sql._user+':'
+ self._sql._password +'#localhost/testDB::QUANDL_DATA_WIKI')
The first row looks like this in the CSV:
A 7/25/2016 46.49 46.52 45.92 46.14 1719772 0 1 46.49 46.52 45.92 46.14 1719772
The MySQL table is defined as follows:
Ticker varchar(255) NOT NULL,
Date date NOT NULL,
Open numeric(15,2) NULL,
High numeric(15,2) NULL,
Low numeric(15,2) NULL,
Close numeric(15,2) NULL,
Volume bigint NULL,
ExDividend numeric(15,2),
SplitRatio int NULL,
OpenAdj numeric(15,2) NULL,
HighAdj numeric(15,2) NULL,
LowAdj numeric(15,2) NULL,
CloseAdj numeric(15,2) NULL,
VolumeAdj bigint NULL,
PRIMARY KEY(Ticker,Date)
It throws an exception 1292 with the following info:
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1292, "Incorrect date value: '7/25/2016' for column 'Date' at row 1") [SQL: 'LOAD DATA INFILE %(path)s\n INTO TABLE QUANDL_DATA_WIKI\n CHARACTER SET %(encoding)s\n FIELDS\n TERMINATED BY %(delimiter)s\n ENCLOSED BY %(quotechar)s\n ESCAPED BY %(escapechar)s\n LINES TERMINATED BY %(lineterminator)s\n IGNORE %(skiprows)s LINES\n '] [parameters: {'path': 'C:\ProgramData\MySQL\MySQL Server 5.6\Uploads\WIKI_20160725.partial.csv', 'quotechar': '"', 'skiprows': 0, 'lineterminator': '\r\n', 'escapechar': '\', 'delimiter': ',', 'encoding': 'utf8'}]
Does anyone have an idea what is wrong with the date in the first row? It doesn't seem to match it to the MySql database
mysql has problems with date conversions. I noticed when I defined the date field as a varchar
Date varchar(255) NOT NULL
then the csv file was read properly
in my SQL the conversion of the string to date format then looks like this:
STR_TO_DATE(Date, "%m/%d/%Y")
This is my code. It will read a bunch of files with SQL commands in identical formats (i.e. comments prefaced with -, along with some blank lines, hence the if condition in my loop)
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import csv,sqlite3,os
conn = sqlite3.connect('db_all.db')
c = conn.cursor()
files = os.listdir('C:\\Users\\ghb\\Desktop\\database_schema')
for file in files:
string = ''
with open(file, 'rb') as read:
for line in read.readlines():
if line[0]!='-' and len(line)!=0: string = string + line.rstrip() #error also occurs if I skip .rstrip
print string #for debugging purposes
c.executescript(string)
string=''
conn.close()
Error:
Traceback (most recent call last):
File "C:/Users/ghb/Desktop/database_schema/database_schema.py", line 16, in <module>
c.executescript(string)
sqlite3.OperationalError: near "SET": syntax error
For fear of clutter, here is the output of the string variable (without the INSERT INTO data, that is confidential):
SET SQL_MODE="NO_AUTO_VALUE_ON_ZERO";
CREATE TABLE IF NOT EXISTS `career` (
`carKey` int(11) NOT NULL AUTO_INCREMENT,
`persID` bigint(10) unsigned zerofill NOT NULL,
`persKey` int(6) unsigned NOT NULL,
`wcKey` int(2) unsigned NOT NULL,
`wtKey` int(2) unsigned DEFAULT NULL,
`pwtKey` int(2) unsigned DEFAULT NULL,
`dptId` bigint(10) unsigned NOT NULL,
`dptNr` int(4) unsigned NOT NULL,
`dptalias` varchar(10) COLLATE utf8_icelandic_ci NOT NULL,
`class` enum('A','B') COLLATE utf8_icelandic_ci NOT NULL,
`getfilm` enum('yes','no') COLLATE utf8_icelandic_ci NOT NULL DEFAULT 'yes',
`finished` enum('true','false') COLLATE utf8_icelandic_ci NOT NULL DEFAULT 'false',
`startDate` date NOT NULL,
`endDate` date DEFAULT NULL,
`regDate` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`user` tinyint(4) NOT NULL,
`status` set('LÉST','BRFL','BRFD','BRNN') COLLATE utf8_icelandic_ci DEFAULT NULL,
`descr` text COLLATE utf8_icelandic_ci,
PRIMARY KEY (`carKey`),
KEY `pwtKey` (`pwtKey`),
KEY `wtKey` (`wtKey`),
KEY `dptId` (`dptId`),
KEY `user` (`user`),
KEY `persID` (`persID`),
KEY `persKey` (`persKey`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_icelandic_ci AUTO_INCREMENT=2686 ;
Input sample:
INSERT INTO `career` (`carKey`, `persID`, `persKey`, `wcKey`, `wtKey`, `pwtKey`, `dptId`, `dptNr`, `dptalias`, `class`, `getfilm`, `finished`, `startDate`, `endDate`, `regDate`, `user`,
(5, 34536, 346, 22, 44, 34, 3454356, 33, 'asdasd', 'ASDASD', 'ASDSD', 'true', '1991-02-04', '2010-05-02', '2009-05-02 00:01:02', 1, NULL, 'HH:
'),
I have executed this code to insert a dictionary into my table in database,
d = {'err': '0', 'tst': '0', 'Type o': 'FTP', 'recip': 'ADMIN', 'id': '101', 'origin': 'REPORT', 'Type recip': 'SMTP', 'date': '2010-01-10 18:47:52'}
db = MySQLdb.connect("localhost","admin","password","database")
cursor = db.cursor()
cursor.execute("""INSERT INTO mytable(ID, ERR, TST, DATE, ORIGIN, TYPE_O, RECIP, TYPE_RECIP) VALUES (%(id)s, %(err)s, %(tst)s, %(date)s, %(origin)s, %(Type o)s, %(recip)s, %(Type recip)s)""", d)
db.commit()
db.close()
Create statement of my table:
CREATE TABLE mytable (
`ID` tinyint unsigned NOT NULL,
`ERR` tinyint NOT NULL,
`TST` tinyint unsigned NOT NULL,
`DATE` datetime NOT NULL,
`ORIGIN` varchar(30) NOT NULL,
`TYPE_O` varchar(10) NOT NULL,
`RECIP` varchar(30) NOT NULL,
`TYPE_RECIP` varchar(10) NOT NULL,
PRIMARY KEY (`ID`,`DATE`)
) ENGINE = InnoDB;
But i have an error, it says:
1064, "you have an error in your SQL syntax; check the manual that
corresponds to you MySQL server version... )
Be aware of SQL injections and use the second argument to execute for inserting your query parameters:
cursor.execute("""
INSERT INTO
table
(name, age, origin, date)
VALUES
(%(name)s, %(age)s, %(origin)s, %(date)s)
""", d)