Develop Reference

parsing .log files then sort in access

I'm writing a parsing program that that searches through 100+ .log files after some keyword, then puts the words in different array´s and separates the words in to columns in excel. Now I want to sort them in Access automatically so that I can process the different .log file combinations. I can "copy paste" from my Excel file to Access, but that so inefficient and gives some errors... I would like it to be "automatic". I'm new to Access and don´t know how to link from python to Access, I have tried doing it as I did to Excel but that didn't work and started looking in to OBDC but had some problems there to...
import glob # includes
import xlwt # includes
from os import listdir # includes
from os.path import isfile, join # includes
def logfile(filename, tester, createdate,completeresponse):
# Arrays for strings
response = []
message = []
test = []
date = []
with open(filename) as filesearch: # open search file
filesearch = filesearch.readlines() # read file
for line in filesearch:
file = filename[39:] # extract filename [file]
for lines in filesearch:
if createdate in lines: # extract "Create Time" {date}
date.append(lines[15:34])
if completeresponse in lines:
response.append(lines[19:])
print('pending...')
i = 1 # set a number on log {i}
d = {}
for name in filename:
if not d.get(name, False):
d[name] = i
i += 1
if tester in line:
start = '-> '
end = ':\ ' # |<----------->|
number = line[line.find(start)+3: line.find(end)] #Tester -> 1631 22 F1 2E :\ BCM_APP_31381140 AJ \ Read Data By Identifier \
test.append(number) # extract tester {test}
# |<--------------------------------------------
text = line[line.find(end)+3:] # Tester -> 1631 22 F1 2E :\ BCM_APP_31381140 AJ \ Read Data By Identifier \
message.append(text)
with open('Excel.txt', 'a') as handler: # create .txt file
for i in range(len(message)):
# A B C D E
handler.write(f"{file}|{date[i]}|{i}|{test[i]}|{response[i]}")
# A = filename B = create time C = number in file D = tester E = Complete response
# open with 'w' to "reset" the file.
with open('Excel.txt', 'w') as file_handler:
pass
# ---------------------------------------------------------------------------------
for filename in glob.glob(r'C:\Users\Desktop\Access\*.log'):
logfile(filename, 'Sending Request: Tester ->', 'Create Time:','Complete Response:','Channel')
def if_number(s): # look if number or float
try:
float(s)
return True
except ValueError:
return False
# ----------------------------------------------
my_path = r"C:\Users\Desktop\Access" # directory
# search directory for .txt files
text_files = [join(my_path, f) for f in listdir(my_path) if isfile(join(my_path, f)) and '.txt' in f]
for text_file in text_files: # loop and open .txt document
with open(text_file, 'r+') as wordlist:
string = [] # array ot the saved string
for word in wordlist:
string.append(word.split('|')) # put word to string array
column_list = zip(*string) # make list of all string
workbook = xlwt.Workbook()
worksheet = workbook.add_sheet('Tab')
worksheet.col(0) # construct cell
first_col = worksheet.col(0)
first_col.width = 256*50
second_col = worksheet.col(1)
second_col.width = 256*25
third_col = worksheet.col(2)
third_col.width = 256*10
fourth_col = worksheet.col(3)
fourth_col.width = 256*50
fifth_col = worksheet.col(4)
fifth_col.width = 256*100
i = 0 # choose column 0 = A, 3 = C etc
for column in column_list:
for item in range(len(column)):
value = column[item].strip()
if if_number(value):
worksheet.write(item, i, float(value)) # text / float
else:
worksheet.write(item, i, value) # number / int
i += 1
print('File:', text_files, 'Done')
workbook.save(text_file.replace('.txt', '.xls'))
Is there a way to automate the "copy paste"-command, if so how would that look like and work? and if that's something that can´t be done, some advice would help a lot!
EDIT
Thanks i have done som googling and thanks for your help! but now i get a a error... i still can´t send the information to the Access file, i get a syntax error. and i know it exist because i would want to uppdate the existing file... is there a command to "uppdate an exising Acces file"?
error
pyodbc.ProgrammingError: ('42S01', "[42S01] [Microsoft][ODBC Microsoft Access Driver] Table 'tblLogfile' already exists. (-1303) (SQLExecDirectW)")
code
import pyodbc
UDC = r'C:\Users\Documents\Access\UDC.accdb'
# DSN Connection
constr = " DSN=MS Access Database; DBQ={0};".format(UDC)
# DRIVER connection
constr = "DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};UID=admin;UserCommitSync=Yes;Threads=3;SafeTransactions=0;PageTimeout=5;MaxScanRows=8;MaxBufferSize=2048;FIL={MS Access};DriverId=25;DefaultDir=C:/USERS/DOCUMENTS/ACCESS;DBQ=C:/USERS/DOCUMENTS/ACCESS/UDC.accdb"
# Connect to database UDC and open cursor
db = pyodbc.connect(constr)
cursor = db.cursor()
sql = "SELECT * INTO [tblLogfile]" +\
"FROM [Excel 8.0;HDR=YES;Database=C:/Users/Documents/Access/Excel.xls.[Tab];"
cursor.execute(sql)
db.commit()
cursor.close()
db.close()

First, please note, MS Access, a database management system, is not MS Excel, a spreadsheet application. Access sits on top of a relational engine and maintains strict rules in data and relational integrity whereas in Excel anything can be written across cells or ranges of cells with no rules. Additionally, the Access object library (tabledefs, querydefs, forms, reports, macros, modules) is much different than the Excel object library (workbooks, worksheets, range, cell, etc.), so there is no one-to-one translation in code.
Specifically, for your Python project, consider pyodbc using a make-table query that runs a direct connection to the Excel workbook. Since MS Access' database is the ACE/JET engine (Windows .dll files, available on Windows machines regardless of Access install). One feature of this data store is the ability to connect to workbooks even text files! So really, MSAccess.exe is just a GUI console to view .mdb/.accdb files.
Below creates a new database table that replicates the specific workbook sheet data, assuming the sheet maintains:
tabular format beginning in A1 cell (no merged cells/repeating labels)
headers in top row (no spaces before or after or special characters !#$%^~<>)))
columns of consistent data type format (i.e., data integrity rules)
Python code
import pyodbc
databasename = 'C:\\Path\\To\\Database\\File.accdb'
# DSN Connection
constr = "DSN=MS Access Database;DBQ={0};".format(databasename)
# DRIVER CONNECTION
constr = "DRIVER={{Microsoft Access Driver (*.mdb, *.accdb)}};DBQ={0};".format(databasename)
# CONNECT TO DATABASE AND OPEN CURSOR
db = pyodbc.connect(constr)
cur = db.cursor()
# RUN MAKE-TABLE QUERY FROM EXCEL WORKBOOK SOURCE
# OLDER EXCEL FORMAT
sql = "SELECT * INTO [myNewTable]" + \
" FROM [Excel 8.0;HDR=Yes;Database=C:\Path\To\Workbook.xls].[SheetName$];"
# CURRENT EXCEL FORMAT
sql = "SELECT * INTO [myNewTable]" + \
" FROM [Excel 12.0 Xml;HDR=Yes;Database=C:\Path\To\Workbook.xlsx].[SheetName$];"
cursor.execute(sql)
db.commit()
cur.close()
db.close()

Almost certainly the answer from Parfait above is a better way to go, but for fun I'll leave my answer below
If you are willing to put in the time I think you need 3 things to complete the automation of what you want to do:
1) Send a string representation of your data to the windows Clipboard There is windows specific code for this, or you can just save yourself some time and use pyperclip
2) Learn VBA and use VBA to grab the string from the clipboard and process it. Here is some example VBA code that I used in excel the past to grab text from the Clipboard
Function GetTextFromClipBoard() As String
Dim MSForms_DataObject As New MSForms.DataObject
MSForms_DataObject.GetFromClipboard
GetTextFromClipBoard = MSForms_DataObject.GetText()
End Function
3) use pywin32 (I believe available easily with Anaconda) to automate the vba access calls from Python. This is probably going to be the hardest part as the specific call trees are (in my opinion) not well documented and takes a lot of poking and digging to figure out what exactly you need to do. Its painful to say the least, but use IPython to help you with visual cues of what methods your pywin32 objects have available.
As I look at the instructions above, I realize it may also be possible to skip the clipboard and just send the information directly from python to access via pywin32. If you do the clipboard route however, you can break the steps up.
send one dataset to the clipboard
grab and process the data using the VBA editor in Access
after you figure out 1 and 2, use pywin32 to bridge the gap
Good luck, and maybe write a blog post about it if you figure it out to share the details.

win32com Excel PasteSpecial

I'm having some trouble with PasteSpecial in python. Here's the sample code:
import win32com.client as win32com
from win32com.client import constants
xl = win32com.gencache.EnsureDispatch('Excel.Application')
xl.Visible = True
wb = xl.Workbooks.Add ()
Sheet1 = wb.Sheets("Sheet1")
# Fill in some summy formulas
for i in range(10):
Sheet1.Cells(i+1,1).Value = "=10*"+str(i+1)
Sheet1.Range("A1:A16").Copy()
Sheet1.Range("C1").Select()
Sheet1.PasteSpecial(Paste=constants.xlPasteValues)
I'm getting the following error:
TypeError: Paste() got an unexpected keyword argument 'Paste'
I know that paste is a keyword argument because of the MSDN here:
http://msdn.microsoft.com/en-us/library/office/ff839476(v=office.15).aspx
Any idea why it won't let me do this? Can't really find much on the web.
Edit for solution(s):
import win32com.client as win32com
from win32com.client import constants
xl = win32com.gencache.EnsureDispatch('Excel.Application')
xl.Visible = True
wb = xl.Workbooks.Add ()
Sheet1 = wb.Sheets("Sheet1")
# Fill in some summy formulas
for i in range(10):
Sheet1.Cells(i+1,1).Value = "=10*"+str(i+1)
Sheet1.Range("A1:A16").Copy()
Sheet1.Range("C1").PasteSpecial(Paste=constants.xlPasteValues)
# OR this I just found right after I posted this works as well:
xl.Selection.PasteSpecial(Paste=constants.xlPasteValues)

You can get value for xlPasteFormats by execute macro in Excel vb:
Sub Macro2()
Range("A7").Select
ActiveCell.FormulaR1C1 = xlPasteFormats
End Sub
The value for xlPasteFormats is -4122
In Python script you can use
xlSheet.Range("A7:H7").Copy()
xlSheet.Range("A%s:H%s"%(r,r)).PasteSpecial(Paste=-4122)

I don't work with python but to do a PasteSpecial in Excel-VBA, you have to mention the cell where you want to perform the pastespecial, so try like
Sheet1.Range("C1").PasteSpecial(Paste=constants.xlPasteValues)
If you want a simple paste then I guess this should work
Sheet1.Paste

AutoFilter method of Range class failed (Dispatch vs EnsureDispatch)

This code fails with error: "AutoFilter method of Range class failed"
from win32com.client.gencache import EnsureDispatch
excel = EnsureDispatch('Excel.Application')
excel.Visible = 1
workbook = excel.Workbooks.Add()
sheet = workbook.ActiveSheet
sheet.Cells(1, 1).Value = 'Hello world'
sheet.Columns.AutoFilter()
This code also fails although it used to work:
from win32com.client import Dispatch
excel = Dispatch('Excel.Application')
excel.Visible = 1
workbook = excel.Workbooks.Add()
sheet = excel.ActiveSheet
sheet.Cells(1, 1).Value = 'Hello world'
sheet.Columns.AutoFilter()

Python uses win32com to communicate directly with Windows applications, and can work with (via EnsureDispatch) or without (via Dispatch) prior knowledge of the application's API. When you call EnsureDispatch, the API is fetched and written into win32com.gen_py., thereby permanently adding the application's API into your Python library.
Once you've initialised an application with EnsureDispatch, any time that a script uses Dispatch for that application, it will be given the pre-fetched API. This is good, because you can then make use of the predefined application constants (from win32com.client import constants).
However, sometimes previously working code will break. For example, in the following code, AutoFilter() will work without an argument as long as the Excel API has never previously been cached in the library...
# ExcelAutoFilterTest1
# Works unless you ever previously called EnsureDispatch('Excel.Application')
from win32com.client import Dispatch
excel = Dispatch('Excel.Application')
excel.Visible = 1
workbook = excel.Workbooks.Add()
sheet = workbook.ActiveSheet
sheet.Cells(1, 1).Value = 'Hello world'
sheet.Columns.AutoFilter()
The following code will always fail because now the Excel API has been fetched and written to win32com.gen_py.00020813-0000-0000-C000-000000000046x0x1x7 in your Python library, it will no longer accept AutoFilter() without an argument.
# ExcelAutoFilterTest2
# Always fails with error: AutoFilter method of Range class failed
from win32com.client.gencache import EnsureDispatch
excel = EnsureDispatch('Excel.Application')
excel.Visible = 1
workbook = excel.Workbooks.Add()
sheet = workbook.ActiveSheet
sheet.Cells(1, 1).Value = 'Hello world'
sheet.Columns.AutoFilter()
The following code always works because we're now providing the VisibleDropDown argument (1=on, 0=off).
# ExcelAutoFilterTest3
# Always succeeds
from win32com.client.gencache import EnsureDispatch
excel = EnsureDispatch('Excel.Application')
excel.Visible = 1
workbook = excel.Workbooks.Add()
sheet = workbook.ActiveSheet
sheet.Cells(1, 1).Value = 'Hello world'
sheet.Columns.AutoFilter(1)
This seems to be a bug, because the Excel API documentation claims that all arguments to AutoFilter are optional:
"If you omit all the arguments, this method simply toggles the display
of the AutoFilter drop-down arrows in the specified range."

Can you use the same instance of a xlwt workbook in different functions?

I have to code a really annoying script that uses one excel file to update another, but because you cannot directly edit a xls file, nor insert row, I have had to improvise.
Now my question is:
Using the xlwt module for Python (using 2.7x), when you create a workbook and are working on it, how does one write to the same worksheet that was created in a different function? Can I just pass the workbook back and forth with its variable name? If so, how do I access the first worksheet I made, workbook[0]?
I have multiple functions that need to interact with this xlwt xls file I am making, so I just want to be sure I can pass it around different functions.
Thanks!

...yes
import xlwt
class MyWorkbook:
''' allow access to a workbooks sheets'''
def __init__(self,*args,**kwargs):
self.wb = xlwt.Workbook(*args,**kwargs)
self.sheets = []
def add_sheet(self,sheet_name):
self.sheets.append(self.wb.add_sheet(sheet_name))
return self.sheets[-1]
def GetSheetByIndex(self,n):
return self.sheets[n]
def save(self,fname_or_stream):
return self.wb.save(fname_or_stream)
def CreateWB():
''' return a MyWorkbook instance with 1 sheet'''
m= MyWorkbook()
m.add_sheet("first_sheet")
return m
def ModifySheet0(mwb):
'''uses instance of MyWorkbook and modifies sheet0'''
s = mwb.GetSheetByIndex(0)
s.write(0,0,"Hello World!")
def DoItAll()
'''passing around MyWorkbook'''
wb = CreateWB()
ModifySheet0(wb)
wb.save("somefile.xls")

Problem using Python comtypes library to add a querytable to Excel

I'm trying to create a QueryTable in an excel spreadsheet using the Python comtypes library, but getting a rather uninformative error...
In vba (in a module within the workbook), the following code works fine:
Sub CreateQuery()
Dim con As ADODB.Connection
Dim rs As ADODB.Recordset
Dim ws As Worksheet
Dim qt As QueryTable
Set ws = ActiveWorkbook.Sheets(1)
Set con = New ADODB.Connection
con.Open ("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\Path\to\Db.mdb;")
Set rs = New ADODB.Recordset
rs.Open "Select * from [tbl Base Data];", con
Set qt = ws.QueryTables.Add(rs, ws.Range("A1"))
qt.Refresh
End Sub
But the following Python code:
import sys
import comtypes.client as client
def create_querytable():
constring = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\\Path\\to\\Db.mdb"
conn = client.CreateObject("ADODB.Connection", dynamic = True)
rs = client.CreateObject("ADODB.Recordset", dynamic = True)
SQL = "Select * from [tbl Base Data];"
conn.Open(constring)
rs.Open(SQL, conn)
excel = client.CreateObject("Excel.Application", dynamic = True)
excel.Visible = True
ws = excel.Workbooks.Add().Sheets(1)
qt = ws.QueryTables.Add(rs, ws.Range["A1"])
qt.Refresh()
rs.Close()
conn.Close()
Throws the unhelpful error message:
Traceback (most recent call last):
File "<pyshell#34>", line 1, in <module>
create_querytable()
File "C:/Documents and Settings/cvmne250/Desktop/temp.py", line 17, in create_querytable
qt = ws.QueryTables.Add(rs, ws.Range["A1"])
File "G:\ISA\SPSS\comtypes\lib\comtypes\client\lazybind.py", line 160, in caller
File "G:\ISA\SPSS\comtypes\lib\comtypes\automation.py", line 628, in _invoke
COMError: (-2147352567, 'Exception occurred.', (None, None, None, 0, None))
Any ideas on what's happening here?
Thanks!

I simplified your code and this should work fine (I'll explain the changes below):
def create_querytable2():
constring = "OLEDB;Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\path\to\db.mdb;"
SQL = "Select * from tblName;"
excel = client.CreateObject("Excel.Application", dynamic=True)
excel.Visible = True
ws = excel.Workbooks.Add().Worksheets(1)
ws.QueryTables.Add(constring, ws.Range["A1"], SQL).Refresh()
The QueryTables.Add() function can create the Connection and Recordset objects for you, so that simplifies a lot of things... you just need to add what type of connection it is in the conneciton string (the "OLEDB" part).
Letting Excel do most of the work seems to solve your problem :)

It looks like your error is on this line:
qt = ws.QueryTables.Add(rs, ws.Range["A1"])
I think your problem is that you are using python syntax to look up a value in a VBA Collection. Try changing your square brackets to parentheses.
i.e.
qt = ws.QueryTables.Add(rs, ws.Range("A1"))
The reason being that in VBA when you invoke a Collection like this, Range("A1"), you are actually calling it's default method, Range.Item("A1"). Basically, VBA Collections do not translate to python dictionaries.
I'm getting this from this forum thread, and my experience with VBA.
Edit due to comment:
Unfortunately, I've tried both: as
noted in your link, they sometimes
don't do the same thing, but my gut
feeling here is that the '[' is more
likely to be what I want. – mavnn
Do you know if comtypes.client.CreateObject works the same as win32com.client.Dispatch? You might try creating your com object with the win32com package and see if that makes a difference.