I am a beginner in Python. I am trying to insert the following data in sqlite db using Python 3.4.
('config.xml', '09/12/2017 10:33:55 PM', 466, 'C:Users\ron\Downloads\folder');
But I am getting an error
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 23-26: truncated \UXXXXXXXX escape
I think this is because of this character - \
How can I tell sqlite to escape this character.
Code --
def create_filedetail(conn, filedetail):
"""
Create a new file detail into the filedetail table
:param conn:
:param filedetail:
:return: filedetail id
"""
sql = ''' INSERT INTO filedetail(filename,modified_date,filesize,filepath)
VALUES(?,?,?,?) '''
cur = conn.cursor()
cur.execute(sql, filedetail)
return cur.lastrowid
def main():
database = r"C:\Users\ron\Documents\sqlitedb\filedb.db"
sql_create_file_detail_table = """ CREATE TABLE IF NOT EXISTS filedetail (
id integer PRIMARY KEY,
filename text NOT NULL,
modified_date text,
filesize integer,
filepath text
); """
conn = create_connection(database)
if conn is not None:
# create filedetail table
create_table(conn, sql_create_file_detail_table)
with conn:
# create a new filedetail
filedetail = ('config.xml', '09/12/2017 10:33:55 PM', 466, "C:Users\ron\Downloads\folder");
filedetail_id = create_filedetail(conn, filedetail)
else:
print("Error! cannot create the database connection.")
Any help is highly appreciated. Thanks in advance.
You can see that there is a similar issue in this post:
What exactly do "u" and "r" string flags do, and what are raw string literals?
Basically, you can create a string literal by prepending an r to your string.
Look at this example. It returns an invalid character:
>>> mypath = "C:Users\ron\Downloads\folder"
>>> mypath
'C:Users\ron\\Downloads\x0colder'
However, if you use the string literals:
>>> mypath = r"C:Users\ron\Downloads\folder"
>>> mypath
'C:Users\\ron\\Downloads\\folder'
You can then insert this new string into your SQL table.
Related
I have created a database and I am trying to fetch data from it. I have a class Query and inside the class I have a function that calls a table called forecasts. The function is as follows:
def forecast(self, provider: str, zone: str='Mainland',):
self.date_start = date_start)
self.date_end = (date_end)
self.df_forecasts = pd.DataFrame()
fquery = """
SELECT dp.name AS provider_name, lf.datetime_from AS date, fr.name AS run_name, lf.value AS value
FROM load_forecasts lf
INNER JOIN bidding_zones bz ON lf.zone_id = bz.zone_id
INNER JOIN data_providers dp ON lf.provider_id = dp.provider_id
INNER JOIN forecast_runs fr ON lf.run_id = fr.run_id
WHERE bz.name = '{zone}'
AND dp.name = '{provider}'
AND date(lf.datetime_from) BETWEEN '{self.date_start}' AND '{self.date_end}'
"""
df_forecasts = pd.read_sql_query(fquery, self.connection)
return df_forecasts
In the scripts that I run I am calling the Query class giving it my inputs
query = Query(date_start, date_end)
And the function
forecast_df = query.forecast(provider='Meteologica')
I run my script in the command line in the classic way
python myscript.py '2022-11-10' '2022-11-18'
My script shows the error
sqlalchemy.exc.DataError: (psycopg2.errors.InvalidDatetimeFormat) invalid input syntax for type date: "{self.date_start}"
LINE 9: AND date(lf.datetime_from) BETWEEN '{self.date_start...
when I use this syntax, but when I manually input the string for date_start and date_end it works.
I cannot find a way to solve the problem with sqlalchemy, so I opened a cursor with psycopg2.
# Returns the datetime, value and provider name and issue date of the forecasts in the load_forecasts table
# The dates range is specified by the user when the class is called
def forecast(self, provider: str, zone: str='Mainland',):
# Opens a cursor to get the data
cursor = self.connection.cursor()
# Query to run
query = """
SELECT dp.name, lf.datetime_from, fr.name, lf.value, lf.issue_date
FROM load_forecasts lf
INNER JOIN bidding_zones bz ON lf.zone_id = bz.zone_id
INNER JOIN data_providers dp ON lf.provider_id = dp.provider_id
INNER JOIN forecast_runs fr ON lf.run_id = fr.run_id
WHERE bz.name = %s
AND dp.name = %s
AND date(lf.datetime_from) BETWEEN %s AND %s
"""
# Execute the query, bring the data and close the cursor
cursor.execute(query, (zone, provider, self.date_start, self.date_end))
self.df_forecasts = cursor.fetchall()
cursor.close()
return self.df_forecasts
If anyone finds the answer with sqlalchemy, I would love to see it!
I'm trying to pull data from an Excel spreadsheet to MySQL. My script can't find the path to the Excel file, and my IDE (Spyder) is giving an error on this line:
def read_excel(r'C:\\Users\\ParaSystems Limited\\Desktop\\main.xlsx'):
invalid syntax
import openpyxl
import pymysql as mdb
def read_excel(r'C:\\Users\\ParaSystems Limited\\Desktop\\main.xlsx'):
masterdict = {}
wb = openpyxl.load_workbook('main.xlsx')
for sheet in wb:
for arow in range(2, sheet.max_row+1):
if sheet['A'+str(arow)].value:
masterdict[sheet['A'+str(arow)].value] = {
'Equipment Number':sheet['B'+str(arow)].value,
'Number':sheet['C'+str(arow)].value,
'Description':sheet['D'+str(arow)].value,
'Manufacturer':sheet['E'+str(arow)].value,
'Serial Number':sheet['F'+str(arow)].value,
'Country of Manufacturer':sheet['G'+str(arow)].value,
'Functional Location Description':sheet['H'+str(arow)].value,
'Functional Location Number (Short)':sheet['I'+str(arow)].value,
'Functional Location Number':sheet['J'+str(arow)].value,
'COST OF SERVICING AND MAINTENANCE':sheet['K'+str(arow)].value,
'Office Location':sheet['L'+str(arow)].value
}
return masterdict
def inputIntoMySQL(masterdict):
con = mdb.connect(host= '127.0.0.1', user = 'root', password =None,db='scraping')
cur = con.cursor()
with con:
cur.execute("DROP TABLE IF EXISTS main")
cur.execute("CREATE TABLE main (rid INT PRIMARY KEY, EquipmentNumber VARCHAR(75), Description VARCHAR(75),\
Manufacturer VARCHAR(50), SerialNumber INT,CountryOfManufacturer VARCHAR(25), \
FunctionalLocationDescription VARCHAR(50), FunctionalLocationNumberShort VARCHAR(75), FunctionalLocationNumber VARCHAR(25),\
CostOfServicingAndMaintenance DECIMAL(15,2),OfficeLocation VARCHAR(35))")
for i in masterdict:
cur.execute('INSERT INTO DISTRIBUTORS_NESTLE(rid, EquipmentNumber,Description,Manufacturer,SerialNumber,\
CountryOfManufacturer,FunctionalLocationDescription, FunctionalLocationNumberShort,FunctionalLocationNumber\
CostOfServicingAndMaintenance,OfficeLocation) VALUES("%s", "%s", "%s","%s","%s","%s","%s","%s","%s","%s","%s")'
%(i,masterdict[i]['Equipment Number'],masterdict[i]['Description'],
masterdict[i]['Manufacturer'],masterdict[i]['Serial Number'],masterdict[i]['Country of Manufacturer'],
masterdict[i]['Functional Location Description'], masterdict[i]['Functional Location Number (Short)'], masterdict[i]['Functional Location Number'],
masterdict[i]['COST OF SERVICING AND MAINTENANCE'], masterdict[i]['Office Location']))
con.commit()
con.close()
The syntax error is because you're defining a function (read_excel) and you're putting the excel filepath directly in the function definition - with this syntax you the excel filepath isnt assigned to a variable so you wouldn't be able to use it within the function.
def read_excel(r'C:\Users\ParaSystems Limited\Desktop\main.xlsx')#Syntax error
To fix this you could create a parameter and make that particular filepath the default value:
def read_excel(excel_file_path = r'C:\Users\ParaSystems Limited\Desktop\main.xlsx')
Then when you call the function, you can call it without any parameters and the excel_file_path will default to that e.g.
read_excel()#Calls with excel_file_path as your default value
or
read_excel(excel_file_path = r'path\to\another\excel.xlsx') #Calls with excel_file_path as the passed parameter value
If there really isn't any need to call this function on any other excel, just declare it in the read_excel function and leave the parameters blank. e.g.
def read_excel():
excel_file_path = r'C:\Users\ParaSystems Limited\Desktop\main.xlsx'
This is not a valid function definition:
def read_excel(r'C:\\Users\\ParaSystems Limited\\Desktop\\main.xlsx'):
masterdict = {}
wb = openpyxl.load_workbook('main.xlsx')
...
You don't have any named parameters inside the parentheses, just a raw string.
It looks like you actually meant something like:
def read_excel(fname=r'C:\Users\ParaSystems Limited\Desktop\main.xlsx'):
masterdict = {} # [unchanged]
wb = openpyxl.load_workbook(fname) # _Uses_ the parameter.
...
Also, since you are using a raw string (r'...'), you shouldn't need to double the backslashes.
Single backslashes should work.
(You'll have to verify this yourself.
I don't have access to a Windows system, so I can't test this.)
I have the following code:
#!/usr/bin/python
# coding=UTF-8
import cx_Oracle
def oracle_connection(user, passwd, host, port, service):
oracle_con_details = user+'/'+passwd+'#'+host+':'+port+'/'+service
try:
oracle_connection = cx_Oracle.connect(oracle_con_details)
except cx_Oracle.DatabaseError as e:
error, = e.args
if error.code == 1017:
log.warning('Please check your credentials.')
else:
log.error('Database connection error: ')
log.error(e)
return oracle_connection
user_oracle = "user"
passw_oracle = "pass"
host_oracle = "host"
port_oracle = "port"
service_oracle = "service"
con_oracle = oracle_connection(user_oracle, passw_oracle, host_oracle, port_oracle, service_oracle)
query = """ SELECT COUNT(*) FROM TABLE WHERE MYDATA = 'REUNIÓN'"""
cursor_oracle = con_oracle.cursor()
cursor_oracle.execute(query)
data_tuple = cursor_oracle.fetchall()
Of course, Oracle credentials and query are just examples. Notice the query has 'Ó' character. This is the one giving me the following error:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xd3' in
position 49: ordinal not in range(128)
I've tried the solutions from some other questions here:
query.decode('utf-8')
query.encode('utf-8')
query.decode('unicode')
query.encode('unicode')
I understand my string (query) is encoded in unicode but I just don't understand why decoding it in utf-8 doesn't work.
Because of this my query doesn't get to Oracle how it should.
ADDITIONAL INFO:
Based on this answer I thought mystring.encode('utf-8) would work.
I cheked the type of my string with this method just in case and the result is 'ordinary string'.
Adding this to my python code solved it.
import os
os.environ["NLS_LANG"] = "SPANISH_SPAIN.UTF8"
Here's a question for you mysql + python folks out there.
Why does this mysql sql sequence of commands not work when I execute it through Python, but it does when I execute it via the mysql CLI?
#!/usr/bin/env python
import oursql as mysql
import sys, traceback as tb
import logging
# some other stuff...
class MySqlAuth(object):
def __init__(self, host = None, db = None, user = None, pw = None, port = None):
self.host = 'localhost' if host is None else host
self.db = 'mysql' if db is None else db
self.user = 'root' if user is None else user
self.pw = pw
self.port = 3306 if port is None else port
#property
def cursor(self):
auth_dict = dict()
auth_dict['host'] = self.host
auth_dict['user'] = self.user
auth_dict['passwd'] = self.pw
auth_dict['db'] = self.db
auth_dict['port'] = self.port
conn = mysql.connect(**auth_dict)
cur = conn.cursor(mysql.DictCursor)
return cur
def ExecuteNonQuery(auth, sql):
try:
cur = auth.cursor
log.debug('SQL: ' + sql)
cur.execute(sql)
cur.connection.commit()
return cur.rowcount
except:
cur.connection.rollback()
log.error("".join(tb.format_exception(*sys.exc_info())))
finally:
cur.connection.close()
def CreateTable(auth, table_name):
CREATE_TABLE = """
CREATE TABLE IF NOT EXISTS %(table)s (
uid VARCHAR(120) PRIMARY KEY
, k VARCHAR(1000) NOT NULL
, v BLOB
, create_ts TIMESTAMP NOT NULL
, mod_ts TIMESTAMP NOT NULL
, UNIQUE(k)
, INDEX USING BTREE(k)
, INDEX USING BTREE(mod_ts) );
"""
ExecuteNonQuery(auth, CREATE_TABLE % { 'table' : table_name })
CREATE_BEFORE_INSERT_TRIGGER = """
DELIMITER //
CREATE TRIGGER %(table)s_before_insert BEFORE INSERT ON %(table)s
FOR EACH ROW
BEGIN
SET NEW.create_ts = NOW();
SET NEW.mod_ts = NOW();
SET NEW.uid = UUID();
END;// DELIMIETER ;
"""
ExecuteNonQuery(auth, CREATE_BEFORE_INSERT_TRIGGER % { 'table' : table_name })
CREATE_BEFORE_INSERT_TRIGGER = """
DELIMITER //
CREATE TRIGGER %(table)s_before_update BEFORE UPDATE ON %(table)s
FOR EACH ROW
BEGIN
SET NEW.mod_ts = NOW();
END;// DELIMIETER ;
"""
ExecuteNonQuery(auth, CREATE_BEFORE_UPDATE_TRIGGER % { 'table' : table_name })
# some other stuff
The error that I get when I run the python is this:
2012-01-15 11:53:00,138 [4214 MainThread mynosql.py] DEBUG SQL:
DELIMITER //
CREATE TRIGGER nosql_before_insert BEFORE INSERT ON nosql
FOR EACH ROW
BEGIN
SET NEW.create_ts = NOW();
SET NEW.mod_ts = NOW();
SET NEW.uid = UUID();
END;// DELIMIETER ;
2012-01-15 11:53:00,140 [4214 MainThread mynosql.py] ERROR Traceback (most recent call last):
File "./mynosql.py", line 39, in ExecuteNonQuery
cur.execute(sql)
File "cursor.pyx", line 120, in oursql.Cursor.execute (oursqlx/oursql.c:15856)
File "cursor.pyx", line 111, in oursql.execute (oursqlx/oursql.c:15728)
File "statement.pyx", line 157, in oursql._Statement.prepare (oursqlx/oursql.c:7750)
File "statement.pyx", line 127, in oursql._Statement._raise_error (oursqlx/oursql.c:7360)
ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER //\n CREATE TRIGGER nosql_before_insert BEFORE INSERT ON nosql\n F' at line 1", None)
Although the error you are getting seems to be generated by the first DELIMITER // statement, you have a typo at the last mention of DELIMITER - you wrote it DELIMIETER ; - try to change that and see if that solves your issue.
Update
You have 2 typos for the same DELIMIETER ; - I believe you are getting the error just after the interpreter finds the first one:
DELIMITER //
CREATE TRIGGER %(table)s_before_insert BEFORE INSERT ON %(table)s
FOR EACH ROW
BEGIN
SET NEW.create_ts = NOW();
SET NEW.mod_ts = NOW();
SET NEW.uid = UUID();
END;// DELIMIETER ; <-- this one is wrong, it should be DELIMITER
You can only pass queries to mysql one at a time; it's up to the client to ensure that the query text is just one valid statement.
The MySQL client does this by tokenizing the entered query and looking for statement separators. In the case of a trigger definition, this doesn't work because the definition can contain semicolons (the default statement separator), and so you have to tell the cli to separate statements in another way, using the DELIMITER command.
The MySQLdb (and other) python api's require no such statement separation; the programmer is obligated to pass statements on at a time to query.
Try removing the DELIMITER statements altogether from your queries (when passed through the python api).
i have the following:
ora_wet = oracle_connection()
cursor = ora_wet.cursor()
sqlQuery = u"SELECT * FROM web_cities WHERE cty_name = 'София'"
cursor.execute(sqlQuery)
sqlResult = cursor.fetchone()
When I do this I get the following error:
TypeError: expecting None or a string on line 18 which is the cursor.execute(sqlQuery)
If I make the query non-unicode (without the u) it goes through but it returns nothing
edit: in reply to first comment:
NLS_LANGUAGE is BULGARIAN,
NLS_CHARACTERSET is CL8MSWIN1251
language is Python...
yes there is a record with cty_name = 'София'
connection is just:
def oracle_connection():
return cx_Oracle.connect('user/pass#server')
ora_wet = oracle_connection()