Exporting with xslxwriter and adding titles from SQlite tables - python

I am needing to either add or grab column names from a DB file using Sqlite and Xslx Writer.
The code I have (Shown Below) is my attempt on putting column names in the workbook manually. I am unable to figure out how to alter the loop to write on the second row to allow the titles to stay. I am hoping someone can help me figure this out or have a better way to pull the titles directly from my DB file.
I've tried adjusting "write_row(i, 0, row)" to "write_row(1,0,row)". Which causes it to write under the titles but it only grabs the last line of data from the db. I'm aware that the loop needs i somewhere in that, but I have no idea where.
def export():
todays_date = "Log "+str(datetime.datetime.now().strftime("%Y-%m-%d_%H_%M") )+ '.xlsx'
workbook = xlsxwriter.Workbook(todays_date)
worksheet = workbook.add_worksheet()
conn = sqlite3.connect("logging.db")
cur = conn.cursor()
cur.execute("SELECT * FROM ML")
mysel = cur.execute("SELECT * FROM ML")
worksheet.write("A1", 'ID')
worksheet.write("B1", 'Model')
worksheet.write("C1", 'Serial')
worksheet.write("D1", 'Test')
worksheet.write("E1", 'Before')
worksheet.write("F1", 'After')
worksheet.write("G1", 'Duration')
worksheet.write("H1", 'TimeStamp')
for i, row in enumerate(mysel):
worksheet.write_row(i, 0, row)
workbook.close()
os.startfile(todays_date)

Related

SQLITE3: Find which one are the double records when importing from excel

I'm using python to import from Excel information to my sqlite3 database. I need to avoid the same record to be added twice, but I'd also need to know which are the records added succeffully and which are the one that were already presented in the database. This is how I'm importing with pandas:
def import(self):
file = filedialog.askopenfilename(filetypes=(("Excel File", "*.csv"), ("All Files","*.*")), title="Select Excel")
with open(file, "r", encoding='utf-8') as f:
conn = sqlite3.connect('database/save.db')
double = []
ok = []
c = conn.cursor()
c.execute("SELECT rowid, * FROM record")
r = c.fetchall()
imported = pd.read_csv(f, keep_default_na=False, sep=';')
imported.to_sql('record', conn, if_exists='replace', index = False)
for row in r:
if row in imported:
double.append(row)
else:
ok.append(row)
conn.commit()
conn.close()
Using the for loop I'm able to save the duplicate row to the list but not the one inserted correctly which is always empty
Thanks for the help!

Python Loop writing to CSV files

I am trying to generate a csv file for each query output. I have multiple select queries (queries.sql) in a single SQL file and i am looping through it to execute in database and write each query output to its own csv file. When i execute the code all queries are executing in the database but only the last query result set is being written to csv file, rest all csv files are with no records. Any help is appreciated.
col_pattern = "(^|[_-])SSN#?($|[_-])|^SS#?$|(^|[_-])(SSN|SOC.*SEC.*).?(ID|NO|NUMBERS?|NUM|NBR|#)($|[_-])|^SOCIAL.?SEC(URITY)?#?$"
SQL = "select OWNER||'_'||TABLE_NAME||'_'||column_name from ALL_TAB_COLS where REGEXP_LIKE (column_name, :1) and owner NOT IN ('SYS','SYSMAN') order by table_name,column_name"
cursor.execute(SQL,(col_pattern,))
for row_data in cursor:
if not row_data[0].startswith('BIN$'):
fileName = row_data[0]
csv_file_dest = "/u01/exp/test/identity/csvdata/"+ fileName + ".csv"
outputFile = open(csv_file_dest,'w') # 'wb'
output = csv.writer(outputFile, dialect='excel')
f = open('/u01/exp/test/identity/queries.sql')
full_sql = f.read()
sql_commands = full_sql.replace('\n', "").split(';')[:-1]
#print(sql_commands)
for sql_command in sql_commands:
curs2 = cursor.execute(sql_command)
if printHeader: # add column headers if requested
cols = []
for col in curs2.description:
cols.append(col[0])
output.writerow(cols)
for row_data in curs2: # add table rows
output.writerow(row_data)
outputFile.close()
Looks like it is because all the SQL data is being written to whatever the last thing is that the variable "output" was set to. So the file that the variable "output" is set to is just being constantly overwritten.

How to insert all rows of Excel into a db table when column index's are in a variable when using xlrd and Python

I am working with xlrd, Python, and Postgres.
I pull data from Excels using xlrd and insert into a postgres database table.
I understand using index numbers per each column header, but there is 1 Excel where the column index numbers change(Meaning the data for column A would show in column A one day, but will switch to a different column on a different day). So instead of using the column index number, I am looping the Excel based on the text value for the header names and assigning the column index number to a variable. The issue I am having is: If there are 20 rows of data, the very last row of data from the Excel will insert into the DB table only, and I need all rows inserted.
How am I able to adjust my code so all rows of data inserts into the database table?
All help is greatly appreciated.
import psycopg2
import xlrd
try:
conn = psycopg2.connect(user = 'Username', password = 'Password', host = 'Host_name', database = 'DB_name', port = Port_number)
mycursor = conn.cursor()
print('DB connection open')
print('XLRD Data inserting into DB Table')
#Open the excel
book = xlrd.open_workbook('excel_name.xlsx')
#Use Index # for Which worksheet or use by_sheet_name......
sheet = book.sheet_by_index(0)
#Loop row index's
for rowidx in range(sheet.nrows):
row = sheet.row(rowidx)
#Loop Column index's
for colidx, cell in enumerate(row):
#Cell value Text for Header name
if cell.value == "Header_Name_Column_A":
#Varablize header index #
header_name_a_index = colidx
if cell.value == "Header_Name_Column_B":
#Varablize header index #
header_name_b_index = colidx
if cell.value == "Header_Name_Column_C":
#Varablize header index #
header_name_c_index = colidx
if cell.value == "Header_Name_Column_D":
#Varablize header index #
header_name_d_index = colidx
#Insert generic values into DB table columns
#I have tried moving the sql and excel for loop within the different loops above.
sql = """INSERT INTO db_table_name(
first_column_name_db,
second_column_name_db,
third_column_name_db,
fourth_column_name_db
)
VALUES(
%s,
%s,
%s,
%s)"""
#loop through all rows and cells
for r in range(1, sheet.nrows):
first_column_name_db = sheet.cell(r, header_name_a_index).value
#Index number prints for each row, but only the last row inserts into the db table.
print(header_name_a_id_index)
second_column_name_db = sheet.cell(r, header_name_b_index).value
third_column_name_db = sheet.cell(r, header_name_c_index).value
fourth_column_name_db = sheet.cell(r, header_name_d_index).value
#Assign values to each row
values = (
first_column_name_db,
second_column_name_db,
third_column_name_db,
fourth_column_name_db
)
mycursor.execute(sql, values)
#Commit to the DB. Close the mycursor and conn.
mycursor.close()
#Just a thought - Do I need to commit each row???
conn.commit()
conn.close()
except Exception as e:
#Close cursor and connection if error
mycursor.close()
conn.close()
print('Error')
print(e)

cx_oracle reading from CSV

I have a cx_oracle connection and I am looking to run a 'batch' of sorts trying to gather ids from last names from a CSV file. Below I have my code, in which I am getting a cx_Oracle.DatabaseError: ORA-01756: quoted string not properly terminated error.
It is pointing to the line
and spriden_change_ind is null'''.format(lname,fname)
However I know this is working as you will see my commented code uses the format in this way and it works just fine. The rows_to_dict_list is a nice function I found here sometime ago to basically add the column names to the output.
Any direction would be nice! thank you
import csv, cx_Oracle
def rows_to_dict_list(cursor):
columns = [i[0] for i in cursor.description]
new_list = []
for row in cursor:
row_dict = dict()
for col in columns:
row_dict[col] = row[columns.index(col)]
new_list.append(row_dict)
return new_list
connection = cx_Oracle.connect('USERNAME','PASSWORD','HOSTNAME:PORTNUMBER/SERVICEID')
cur = connection.cursor()
printHeader = True
with open('nopnumber_names.csv')as csvfile:
reader = csv.DictReader(csvfile)
for row in reader:
lname = row['Last']
fname = row['First']
cur.execute('''select spriden_pidm as PIDM,
spriden_last_name as Last,
spriden_first_name as First,
spriden_mi as Middle,
spriden_ID as ID
from spriden
where upper(spriden_last_name) = '{0}'
and upper(spriden_first_name) = '{1}'
and spriden_change_ind is null'''.format(lname,fname)
)
# THIS RECORD RUNS FINE
# cur.execute('''select spriden_pidm as PIDM,
# spriden_ID as ID,
# spriden_last_name as Last,
# spriden_first_name as First
# from spriden
# where spriden_pidm = '{}'
# and spriden_change_ind is null'''.format(99999)
# )
data = rows_to_dict_list(cur)
for row in data:
print row
cur.close()
connection.close()
My best guess is that a first name or surname somewhere in your CSV file has a ' character in it.
You really shouldn't be building SQL by concatenating strings or using string formatting. You are at risk of SQL injection if you do so. What happens if someone has put a record in your CSV file with surname X' OR 1=1 --?
Instead, use bind parameters to send the values of your variables to the database. Try the following:
cur.execute('''select spriden_pidm as PIDM,
spriden_last_name as Last,
spriden_first_name as First,
spriden_mi as Middle,
spriden_ID as ID
from spriden
where upper(spriden_last_name) = :lname
and upper(spriden_first_name) = :fname
and spriden_change_ind is null''',
{"lname": lname, "fname": fname}
)

I seem to only be able to create a 3 column table with python-docx

I want to create a word file using python-docx.
This word file will have a table where cells are populated by a sqlite3 query.
it all works fine with 1-3 columns, but when i try to add a fourth it does not save anymore. I can add a fourth heading, but as soon as i populate the cells for that fourth heading it does not save anymore. I have tried looking up this problem, but could not find anything. So my question to you is: Is my code correct? Am I overlooking something? If so, what?
from docx import Document
import sqlite3
document = Document()
conn = sqlite3.connect('Database.db')
c = conn.cursor()
table = document.add_table(1, 4)
heading_cells = table.rows[0].cells
heading_cells[0].text = 'Description'
heading_cells[1].text = 'Hours/units'
heading_cells[2].text = 'Rate'
heading_cells[3].text = 'Total' #all works well when this is active
Records = (c.execute("SELECT * FROM My_List")).fetchall()
for item in Records:
cells = table.add_row().cells
cells[0].text = str(item[1])
cells[1].text = str(item[2])
cells[2].text = str(item[3])
cells[3].text = str(item[4]) #nothing works when this is active
document.save('Demo.docx')

Categories

Resources