I created a small application to export data from my mysql database in csv, it works, but if I want to create another report is presented the following error:
pymysql.err.InternalError: (1086, "File '/TEMP/.CSV' already exists")
Yes, the file already exists. My question is, how do I generate two reports, even with the same name. Ex. hi.csv, and following hi.csv (1)
Following is the code below:
import tkinter as tk
import pymysql
root = tk.Tk()
root.geometry("")
root.title("excel teste")
conn = pymysql.connect(host="localhost", port=3306, user="root", password="", database="omnia")
with conn:
print("connect successfull!")
cursor = conn.cursor()
with cursor:
cursor.execute("SELECT VERSION()")
versao = cursor.fetchone()
print("Versão do gerenciador Maria DB: %s" % versao)
def exp_rel_con_pag():
conn = pymysql.connect(host="localhost", port=3306, user="root", password="", database="omnia")
with conn:
statm = "SELECT * FROM omniacademp INTO OUTFILE '/TEMP/"".CSV' FIELDS TERMINATED BY ',' ENCLOSED BY ''"
cursor = conn.cursor()
with cursor:
cursor.execute(statm)
results = cursor.fetchone()
print(results)
tk.Button(root, width=15, text="run", command=exp_rel_con_pag).place(x=10, y=10)
root.mainloop()
You could import the error class:
from pymysql.err import InternalError
Add a counter:
fileIndex = 0
Then see if the file already exists:
try:
statm = "SELECT * FROM omniacademp INTO OUTFILE '/TEMP/HI.CSV' FIELDS TERMINATED BY ',' ENCLOSED BY ''"
cursor.execute(statm)
except InternalError:
statm = "SELECT * FROM omniacademp INTO OUTFILE '/TEMP/HI ({}).CSV' FIELDS TERMINATED BY ',' ENCLOSED BY ''".format(fileIndex)
cursor.execute(statm)
fileIndex += 1
You need to add some level of dynamic naming. Personally I use timestamps.
For example I use openpyxl to write my excel files and datetime for my timestamp.
By using a timestamp down to seconds There is very little chance you will ever have a problem with the filename.
Here is the code I use once I have data to write.
import os
import openpyxl
from datetime import datetime as dt
list_of_data = [['row1'], ['row2'], ['row3'], ['row4']]
wb = openpyxl.Workbook() # create workbook
main_ws = wb.worksheets[0] # designate what worksheet I am working on.
for sub_list in list_of_data:
main_ws.append(sub_list) # writing data to each row.
# creating timestamp while removing special characters.
time_stamp = ''.join([{'-': '', ' ': '', ':': '', '.': ''}.get(c, c) for c in str(dt.now())])[0:12]
# build file name.
file_name = '{} - {}.xlsx'.format('report', time_stamp)
# using os library to build path to my local documents folder.
path = os.path.join(os.environ['USERPROFILE'], 'Documents', file_name)
# saving wb.
wb.save(filename=path)
As you can see I now have an excel file in my docs folder with a timestamp.
Related
I am trying to use Regex cleaning steps in Python to test to see if a pattern matches and if so, clean it to the specified carrier.
For instance, if re.match("\bA\.?X\.?A\.?\b", Carrier): Carrier = CarrierMatch
I've tried this by running a for loop on the number of raw carrier fields followed by another for loop on all of the match descriptions (just printing for now) and it takes FOREVER to run. Hoping someone out there has a better method.
Ideally I would like to see if it's possible to compile all match descriptions for Carrier I have in SQL (~2,000) and pull out the regex match pattern(s) to then use to append the carrier field.
For reference the SQL data fields are [raw_pattern], [Carrier]
import sys
import re
import pyodbc
import sys
import os
import pandas as pd
from datetime import datetime
import time
regexlist = list()
carrierlist = list()
rpt_id = 1234
#rpt_id = sys.argv[1]
plan_typs = list()
try:
conn = pyodbc.connect('Driver={SQL Server};'
'Server=xxxxxxxxx;'
'Database=xxxxxxxxx;'
'Trusted_Connection=xxxxx;')
except:
print('Connection Failed')
sys.exit()
cursor = conn.cursor()
sql = "delete from [dbo].[python_test1] where rpt_id = '""" + str(rpt_id) + """'"""
cursor.execute(sql)
conn.commit()
cursor = conn.cursor()
sql = "insert into [dbo].[python_test1](rpt_id, raw_carr_nm) select distinct rpt_id, raw_carr_nm from [dbo].[wrk_data] where rpt_id = '""" + str(rpt_id) + """'"""
cursor.execute(sql)
conn.commit()
sql = "SELECT [raw_pattern], [Carrier] FROM [dbo].[ref_regex_t]"
regex1 = pd.read_sql(sql, conn)
sql = "select * from [dbo].[python_test1] where rpt_id = '""" + str(rpt_id) + """'"""
carriers = pd.read_sql(sql, conn)
for index, row in regex1.iterrows():
regexlist.append(row['raw_pattern'])
for index, row in carriers.iterrows():
carrierlist.append(row['Carrier'])
for i in carrierlist:
print('"' + i + '"')
for i in regexlist:
print('"' + i + '"')
So I am trying to create an auto update to SQL from another excel file, by unique value, as to know what is the new data to add to the database..
There's different in columns names between the database and the excel file as in the database and names without spaces...
I tried to do it with pandas it gave me the same error
So here's my simple code tried with xlrd
import xlrd
from sqlalchemy import create_engine
def insert():
book = xlrd.open_workbook(r"MNM_Rotterdam_5_Daily_Details-20191216081027 - Copy (2).xlsx")
sheet = book.sheet_by_name("GSM Details")
database = create_engine(
'mssql+pyodbc://WWX542337CDCD\SMARTRNO_EXPRESS/myDB?driver=SQL+Server+Native+Client+11.0') # name of database
cnxn = database.raw_connection
cursor = cnxn.cursor()
query = """Insert INTO [myDB].[dbo].[mnm_rotterdam_5_daily_details-20191216081027] (Date, SiteName, CellCI, CellLAC, CellName, CellIndex) values (?,?,?,?,?,?)"""
for r in range(1, sheet.nrows):
date = sheet.cell(r,0).value
site_name = sheet.cell(r,3).value
cell_ci = sheet.cell(r,4).value
cell_lac = sheet.cell(r,5).value
cell_name = sheet.cell(r,6).value
cell_index = sheet.cell(r,7).value
values = (date, site_name, cell_ci, cell_lac, cell_name, cell_index)
cursor.execute(query, values)
cnxn.commit()
# Close the cursor
cursor.close()
# Commit the transaction
database.commit()
# Close the database connection
database.close()
# Print results
print ("")
print ("")
columns = str(sheet.ncols)
rows = str(sheet.nrows)
print ("Imported", columns,"columns and", rows, "rows. All Done!")
insert()
and this is the error:
I tried to change the range I found another error:
Traceback (most recent call last):
File "D:/Tooling/20200207/uniquebcon.py", line 48, in <module>
insert()
File "D:/Tooling/20200207/uniquebcon.py", line 37, in insert
database.commit()
AttributeError: 'Engine' object has no attribute 'commit'
I think this is related to SQL-Alchemy in the connection
Instead of creating the cursor directly with
cursor = database.raw_connection().cursor()
you can create a connection object, then create the cursor from that, and then call .commit() on the connection:
cnxn = database.raw_connection()
crsr = cnxn.cursor()
# do stuff with crsr ...
cnxn.commit()
I've been trying for many days to find a solution to this problem. I need to write a small jpg image for each record in an sqlite database.
Finally I managed to insert the file but judging from the size it was written in the database as raw instead of a (compressed) jpg.
The code I used is:
imgobj = Image.open('./fotocopies/checks/633.jpg')
con = sqlite3.connect("pybook.db")
cur = con.cursor()
cur.execute("UPDATE data_fotocopies SET fotocopy=? WHERE refid=633 and reftype=0", [ buffer(imgobj.tobytes()) ] )
If I try to open the file it cannot be inserted in the database so, the following code:
imgobj = open('./fotocopies/checks/632.jpg')
con = sqlite3.connect("pybook.db")
cur = con.cursor()
cur.execute("UPDATE data_fotocopies SET fotocopy=? WHERE refid=632 and reftype=0", [sqlite3.Binary(imgobj)] )
gives the following error:
cur.execute("UPDATE data_fotocopies SET fotocopy=? WHERE refid=632 and reftype=0", [sqlite3.Binary(imgobj)] )
TypeError: buffer object expected
Unfortunately no previous answer in stackoverflow covers me as I've tried them all. Furthermore all the storing retrieving has to be done via a gtk3 interface which I suspect will mean another (series of) problem(s) i.e. how to set an existing image to get its data from the db response etc.
Can anyone help?
Storing and retrieving BLOBs
import sqlite3
import os.path
from os import listdir, getcwd
from IPython.core.display import Image
def get_picture_list(rel_path):
abs_path = os.path.join(os.getcwd(),rel_path)
print 'abs_path =', abs_path
dir_files = os.listdir(abs_path)
return dir_files
def create_or_open_db(db_file):
db_is_new = not os.path.exists(db_file)
conn = sqlite3.connect(db_file)
if db_is_new:
print 'Creating schema'
sql = '''create table if not exists PICTURES(
ID INTEGER PRIMARY KEY AUTOINCREMENT,
PICTURE BLOB,
TYPE TEXT,
FILE_NAME TEXT);'''
conn.execute(sql) # shortcut for conn.cursor().execute(sql)
else:
print 'Schema exists\n'
return conn
def insert_picture(conn, picture_file):
with open(picture_file, 'rb') as input_file:
ablob = input_file.read()
base=os.path.basename(picture_file)
afile, ext = os.path.splitext(base)
sql = '''INSERT INTO PICTURES
(PICTURE, TYPE, FILE_NAME)
VALUES(?, ?, ?);'''
conn.execute(sql,[sqlite3.Binary(ablob), ext, afile])
conn.commit()
def extract_picture(cursor, picture_id):
sql = "SELECT PICTURE, TYPE, FILE_NAME FROM PICTURES WHERE id = :id"
param = {'id': picture_id}
cursor.execute(sql, param)
ablob, ext, afile = cursor.fetchone()
filename = afile + ext
with open(filename, 'wb') as output_file:
output_file.write(ablob)
return filename
conn = create_or_open_db('picture_db.sqlite')
picture_file = "./pictures/Chrysanthemum50.jpg"
insert_picture(conn, picture_file)
conn.close()
conn = create_or_open_db('picture_db.sqlite')
cur = conn.cursor()
filename = extract_picture(cur, 1)
cur.close()
conn.close()
Image(filename='./'+filename)
Finally I got it working thanks to Andrej Kesely's comment. The working solution is
imgobj = base64.b64encode(open('./fotocopies/checks/624.jpg').read())
con = sqlite3.connect("pybook.db")
cur = con.cursor()
qf="UPDATE data_fotocopies SET fotocopy='%s' WHERE refid=%d AND reftype=0"%(lite.Binary(fotocopy_blob),id)
cur.execute(qf) #yes, it is dangerous for injection`
and retrieving the image from the database is done as:
qf="SELECT fotocopy FROM data_fotocopies WHERE refid=%d and reftype=0"%self.check_id
self.cur.execute(qf)
try:
fd=base64.b64decode(self.cur.fetchall()[0][0])
byting = GLib.Bytes(fd)
self.fotocopy = Gio.MemoryInputStream.new_from_bytes(byting)
...
self.fotocopy_ent=self.builder.get_object("fotocopy") # as it is made in glade
pixbuf = GdkPixbuf.Pixbuf.new_from_stream(self.fotocopy,None) #finally the pixbuf although
#it produces errors if I have
#no stream/image to "feed" it.
self.fotocopy_ent.set_from_pixbuf(pixbuf)
Still can't figure out why all other solutions I've found don't work. I use Python 2.7.6 ang gtk3, but this one I subit does.
Thank you all for your help.
My usecase is to write create a temp table in the postgres database and fetch records from it and insert into a different table.
The code i used is:
import psycopg2
import sys
import pprint
from __future__ import print_function
from os.path import join,dirname,abspath
import xlrd
import os.path
newlist = []
itemidlist = []
def main():
conn_string = "host='prod-dump.cvv9i14mrv4k.us-east-1.rds.amazonaws.com' dbname='ebdb' user='ebroot' password='*********'"
# print the connection string we will use to connect
# print "Connecting to database" % (conn_string)
# get a connection, if a connect cannot be made an exception will be raised here
conn = psycopg2.connect(conn_string)
# conn.cursor will return a cursor object, you can use this cursor to perform queries
cursor = conn.cursor()
dealer_id = input("Please enter dealer_id: ")
group_id = input("Please enter group_id: ")
scriptpath = os.path.dirname('__file__')
filename = os.path.join(scriptpath, 'Winco - Gusti.xlsx')
xl_workbook = xlrd.open_workbook(filename, "rb")
xl_sheet = xl_workbook.sheet_by_index(0)
print('Sheet Name: %s' % xl_sheet.name)
row=xl_sheet.row(0)
from xlrd.sheet import ctype_text
print('(Column #) type:value')
for idx, cell_obj in enumerate(row):
cell_type_str = ctype_text.get(cell_obj.ctype, 'unknown type')
#print('(%s) %s %s' % (idx, cell_type_str, cell_obj.value))
num_cols = xl_sheet.ncols
for row_idx in range(0, xl_sheet.nrows): # Iterate through rows
num_cols = xl_sheet.ncols
id_obj = xl_sheet.cell(row_idx, 1) # Get cell object by row, col
itemid = id_obj.value
#if itemid not in itemidlist:
itemidlist.append(itemid)
# execute our Query
'''
cursor.execute("""
if not exists(SELECT 1 FROM model_enable AS c WHERE c.name = %s);
BEGIN;
INSERT INTO model_enable (name) VALUES (%s)
END;
""" %(itemid,itemid))
'''
cursor.execute("drop table temp_mbp1")
try:
cursor.execute("SELECT p.model_no, pc.id as PCid, g.id AS GROUPid into public.temp_mbp1 FROM products p, \
model_enable me, products_clients pc, groups g WHERE p.model_no = me.name \
and p.id = pc.product_id and pc.client_id = %s and pc.client_id = g.client_id and g.id = %s"\
% (dealer_id,group_id)
except (Exception, psycopg2.DatabaseError) as error:
print(error)
cursor.execute("select count(*) from public.temp_mbp1")
# retrieve the records from the database
records = cursor.fetchall()
# print out the records using pretty print
# note that the NAMES of the columns are not shown, instead just indexes.
# for most people this isn't very useful so we'll show you how to return
# columns as a dictionary (hash) in the next example.
pprint.pprint(records)
if __name__ == "__main__":
main()
The try except block in between the program is not throwing any error but the table is not getting created in the postgres database as i see in the data admin.
The output shown is:
Please enter dealer_id: 90
Please enter group_id: 13
Sheet Name: Winco Full 8_15_17
(Column #) type:value
[(3263,)]
Thanks,
Santosh
You didn't commit the changes, so they aren't saved in the database. Add to the bottom, just below the pprint statement:
conn.commit()
How can I use the current date to name my db file so when it runs it creates a db file which is named after the current date. This is what I have so far:
import sqlite3
import time
timedbname = time.strftime("%d/%m/%Y")
# Connecting to the database file
conn = sqlite3.connect(???)
with this error its the same with '/' or '-' or '.' in "%d/%m/%Y":
conn = sqlite3.connect(timedbname, '.db')
TypeError: a float is required
27.01.2016
Try using:
time.strftime("%d-%m-%Y")
I guess it doesn't work because of the slashes in the generated date.
You can't have dashes in your table name. Also it can't start with a digit.
import sqlite3
from datetime import date
timedbname = '_' + str(date.today()).replace('-','_')
# Connecting to the database file
conn = sqlite3.connect(':memory:')
cursor = conn.cursor()
cursor.execute('''CREATE TABLE %s (col1 int, col2 int)''' % (timedbname))
cursor.execute('''INSERT INTO %s VALUES (1, 2)''' % (timedbname))
cursor.execute('''SELECT * FROM %s'''%timedbname).fetchall()
This worked:
import sqlite3
import time
timedbname = time.strftime("_" + "%d.%m.%Y")
conn = sqlite3.connect(timedbname + '.db')