I have data in an excel spreadsheet (*.xlsx) that consists of 1,213 rows of sensitive information (so, I'm sorry I can't share the data) and 35 columns. Every entry is a string (I don't know if that is screwing it up or not). The first row is the column names and I've never had a problem importing it with the column names embedded before (it's just easier to click that they're embedded so I don't have to name every column by hand). I put the path to the data in the quick start wizard and hit the next button and it doesn't do anything. I hit it again and it turns the mouse into the loader as if it's loading. I've waited for it for 15 minutes before, but every time I click on QlikView the program just crashes.
I have a deadline I have to meet here and I can't afford to not finish this project. It's extremely important that I get it working.
Just as a NB, I used Python to merge two Excel spreadsheets together so I don't know if that may be what's causing the problem either. I can open the file perfectly fine in Excel though.
I was using the walkthrough when creating a new file when I should have just made a script. First click on edit script in the menu bar. Click Table Files.... Choose your file, then make sure that the Labels section has Embedded Labels in the dropdown. It will create a query like the following-
LOAD [Resource name],
[Employee ID],
Vertical,
[Contract Type *],
Notes
FROM
[D:\path\to\file\*.xlsx]
(ooxml, embedded labels, table is Sheet1);
That's one part of the solution, but then I ran into a new problem. It said that it was able to fetch all of the rows, but when I started making the charts and graphs it was only showing 6 data points. I recreated the file, did exactly what I did above but also added the transformation step. Now the problem is solved.
I have had QlikView crash when importing an Excel spreadsheet that was exported with the SQuirreL SQL client (from a Firebird database). Opening the spreadsheet in Excel, and saving it again solved the problem.
I know that this is no longer relevant to your problem, but hopefully it can help someone with a similarly appearing issue.
I see that you do not include a 'header is 0 lines,' which could be cauing an issue?
Follows a snippet from my standard Excel file import - just the FROM section. My setup is done through variables but they follow this form:
Set vTableW = 'WIP Metrics' ;
Set vPathData = '..\Raw Data Reports\' ;
Set vFile08 = 'Misc Transactions VCB*.xlsx' ;
Set vHeader08 = 2 ;
Set vSheet08 = 'Misc Trans' ;
Set vWhere08 = ( Len(Trim([Date Received])) > 0
And Len(Trim([Lot Number])) > 0
And Len(Trim([Y/N])) > 0
And Len(Trim([Initials])) > 0 ) ;
'$(vTableW)':
Load
AutoNumber(RowNo(), 1) As [_Load WIPWO ID],
...
additional columns
...
If(IsNull([Comments]), '', Trim([Comments])) As [VCB Comments]
From
'$(vPathData)$(vFile08)'
(ooxml, embedded labels, header is $(#vHeader08) lines, table is '$(vSheet08)')
Where($(vWhere08)) ;
Regarding the point made above relating to syn keys, add the line
Exit Script ;
Before your import to check load is OK to that point. Then move it immediately after the Excel load and repeat. Use the debug facility in the Load Script process.
Hope this helps!
Related
I have the following script to fetch dynamically changing data in cell F60 on a worksheet. It is a stock price which constantly changes. This data is fetched by a Python code and is working.
I need to record this F60 data on another sheet within the same workbook every time it changes, without deleting the previously recorded data.
When I enter values manually in F60, I see all these values one by one on the destination sheet.
If the data is updated automatically (by Python code), I see the time stamp and not the data.
Private Sub Worksheet_Change(ByVal Target As Range)
On Error GoTo Whoa
Application.EnableEvents = False
If Not Intersect(Target, Range("F60")) Is Nothing Then
Dim x
Dim NR As Long
With Application
.EnableEvents = False
x = Target.Value
On Error Resume Next
With Sheets("Sheet2")
NR = .Range("A" & Rows.Count).End(xlUp).Row + 1
.Range("A" & NR).Value = Now
.Range("B" & NR).Value = x
End With
On Error GoTo 0
.EnableEvents = True
End With
End If
Letscontinue:
Application.EnableEvents = True
Exit Sub
Whoa:
MsgBox Err.Description
Resume Letscontinue
End Sub
Let me post my comment as answer:
The Worksheet_Change-event is triggered only if data is modified within Excel (the program itself), either because a user typed or because a VBA code wrote something into one or more cells.
It can't be triggered if you write it via Python (or any other program). Python doesn't open (the Application) Excel and therefore no VBA environment is available. VBA needs it's host program (in your case that's Excel) to run. If Python writes the data, it also needs to write the data into your "history" sheet.
Think of it vice versa: You have a text file (or any other file). If you write something into this file using Excel/VBA, you can't expect that a piece of Python code runs automatically and does something with the data.
I am building an internal plugin for Sublime Text 3 to automate some changes we need done in our HTML.
I'm close to finishing it so we can start using it but I just have one small issue I can't seem to find a solution for.
The plugin will search for some specific HTML code, save them in a file and then modify the code in the view to reference some text in a json file. Then the plugin will open a new view and paste the contents of the file we saved some code in step 1. After which, I need to do some formatting in the new view (replace commas, quotes and brackets with a new line) and then run another internal sublime command in the new view.
The issue is after replacing the commas, quotes and brackets with a new line, the new view changes to an "Instant File Search" which messes with how the view runs the next command and inserts some garbage characters.
Example of the code:
#Open new tab and paste the copied subheads
newView = self.view.window().new_file(syntax="HTML.sublime-syntax")
newView.insert(edit, 0, json.dumps(sections))
# Clear any current selections in the new view
newView.sel().clear()
# Remove quotes, commas and square brackets and separate into new lines
q = newView.find_all(r"(\x{0022}\,\s\x{0022})|(\[\x{0022})|(\x{0022}\])")
for region in reversed(q):
newView.sel().add(region)
newView.replace(edit, region, "\n")
# Clear any current selections in the new view
newView.sel().clear()
q = newView.find_all("{% r")
for region in reversed(q):
newView.sel().add(region)
newView.replace(edit, region, "<p>{% r")
# Clear any current selections in the new view
newView.sel().clear()
q = newView.find_all("endreplace %}")
for region in reversed(q):
newView.sel().add(region)
newView.replace(edit, region, "endreplace %}</p>")
This is what it outputs in the new view:
If I click undo, I get the expected output which seems to mean sublime does something extra after running the last find_all code.
I haven't found anything online regarding this issue so I'm unable to try things to fix this.
Does anyone have an idea on how I should proceed?
P.S. One possible solution is to manually open a new view and run some commands but I'm trying to do all of that in 1 step using my plugin.
I tested the openpyxl .remove() function and it's working on multiple empty file.
Problem: I have a more complex Excel file with multiple sheet that I need to remove. If I remove one or two it works, when I try to remove three or more, Excel raise an error when I open the file.
Sorry, we have troubles getting info in file bla bla.....
logs talking about pictures troubles
logs about error105960_01.xml ?
The strange thing is that it's talking about pictures trouble but I don't have this error if I don't remove 3 or more sheet. And I don't try to remove sheet with images !
Even more strange, It's always about the number, every file can be deleted without trouble but if I remove 3 or more, Excel yell at me.
The thing is that, it's ok when Excel "repair" the "error" but sometimes, excel reinitialize the format of the sheets (size of cell, bold and length of the characters, etc...) and everything fail :(
bad visual that I want to avoid
If someone have an idea, i'm running out of creativity !
For the code, I only use basic functions (simplify here but it would be long to present more...).
INPUT_EXCEL_PATH = "my_excel.xlsx"
OUTPUT_EXCEL_PATH = "new_excel.xlsx"
wb = openpyxl.load_workbook(INPUT_EXCEL_PATH)
ws = wb["sheet1"]
wb.remove(ws)
ws = wb["sheet2"]
wb.remove(ws)
ws = wb["sheet3"]
wb.remove(ws)
wb.save(OUTPUT_EXCEL_PATH)
In my case it was some left over empty CalculationChainPart. I used DocxToSource to investigate the corrupted file. Excel will attempt to fix the file on load. Save this file and compare it's structure to the original file. To delete descendant parts you can use the DeletePart() method.
using (SpreadsheetDocument doc = SpreadsheetDocument .Open(document, true)) {
MainDocumentPart mainPart = doc.MainDocumentPart;
if (mainPart.DocumentSettingsPart != null) {
mainPart.DeletePart(mainPart.DocumentSettingsPart);
}
}
CalculationChainPart can be also removed anytime.
While calculation chain information can be loaded by a spreadsheet application, it is not required. A calculation chain can be constructed in memory at load-time (source)
I am trying to open an excel file from python, get it to recalculate and then save it with the newly calculated values.
The spreadsheet is large and opens fine in LibreOffice with GUI, and initially shows old values. If I then do a Data->Calculate->Recalculate Hard I see the correct values, and I can of course saveas and all seems fine.
But, there are multiple large spreadsheets I want to do it from so I don't want to use a GUI instead I want to use Python. The following all seems to work to create a new spreadsheet but it doesn't have the new values (unless I again manually do a recalculate hard)
I'm running on Linux. First I do this:
soffice --headless --nologo --nofirststartwizard --accept="socket,host=0.0.0.0,port=8100,tcpNoDelay=1;urp"
Then, here is sample python code:
import uno
local = uno.getComponentContext()
resolver = local.ServiceManager.createInstanceWithContext("com.sun.star.bridge.UnoUrlResolver", local)
context = resolver.resolve("uno:socket,host=localhost,port=8100;urp;StarOffice.ServiceManager")
remoteContext = context.getPropertyValue("DefaultContext")
desktop = context.createInstanceWithContext("com.sun.star.frame.Desktop", remoteContext)
document = desktop.getCurrentComponent()
file_url="file://foo.xlsx"
document = desktop.loadComponentFromURL(file_url, "_blank", 0, ())
controller=document.getCurrentController()
sheet=document.getSheets().getByIndex(0)
controller.setActiveSheet(sheet)
document.calculateAll()
file__out_url="file://foo_out.xlsx"
from com.sun.star.beans import PropertyValue
pv_filtername = PropertyValue()
pv_filtername.Name = "FilterName"
pv_filtername.Value = "Calc MS Excel 2007 XML"
document.storeAsURL(file__out_url, (pv_filtername,))
document.dispose()
After running the above code, and opening foo_out.xlsx it shows the "old" values, not the recalculated values. I know that the calculateAll() is taking a little while, as I would expect for it to do the recalculation. But, the new values don't seem to actually get saved.
If I open it in Excel it does an auto-recalculate and shows the correct values and if I open in LibreOffice and do Recalculate Hard it shows the correct values. But, what I need is to save it, from python like above, so that it already contains the recalculated values.
Is there any way to do that?
Essentially, what I want to do from python is:
open, recalculate hard, saveas
It seems that this was a problem with an older version of LibreOffice. I was using 5.0.6.2, on Linux, and even though I was recalculating, the new values were not even showing up when I extracted the cell values directly.
However, I upgraded to 6.2 and the problem has gone away, using the same code and the same input files.
I decided to just answer my own question, instead of deleting it, as this was leading to a frustration until I solved it.
I'll try to give a brief background here. I recently received a large amount of data that was all digitized from paper maps. Each map was saved as an individual file that contains a number of records (polygons mostly). My goal is to merge all of these files into one shapefile or geodatabase, which is an easy enough task. However, other than spatial information, the records in the file do not have any distinguishing information so I would like to add a field and populate it with the original file name to track its provenance. For example, in the file "505_dmg.shp" I would like each record to have a "505_dmg" id in a column in the attribute table labeled "map_name". I am trying to automate this using Python and feel like I am very close. Here is the code I'm using:
# Import system module
import arcpy
from arcpy import env
from arcpy.sa import *
# Set overwrite on/off
arcpy.env.overwriteOutput = "TRUE"
# Define workspace
mywspace = "K:/Research/DATA/ADS_data/Historic/R2_ADS_Historical_Maps/Digitized Data/Arapahoe/test"
print mywspace
# Set the workspace for the ListFeatureClass function
arcpy.env.workspace = mywspace
try:
for shp in arcpy.ListFeatureClasses("","POLYGON",""):
print shp
map_name = shp[0:-4]
print map_name
arcpy.AddField_management(shp, "map_name", "TEXT","","","20")
arcpy.CalculateField_management(shp, "map_name","map_name", "PYTHON")
except:
print "Fubar, It's not working"
print arcpy.GetMessages()
else:
print "You're a genius Aaron"
The output I receive from running this script:
>>>
K:/Research/DATA/ADS_data/Historic/R2_ADS_Historical_Maps/Digitized Data/Arapahoe/test
505_dmg.shp
505_dmg
506_dmg.shp
506_dmg
You're a genius Aaron
Appears successful, right? Well, it has been...almost: a field was added and populated for both files, and it is perfect for 505_dmg.shp file. Problem is, 506_dmg.shp has also been labeled "505_dmg" in the "map_name" column. Though the loop appears to be working partially, the map_name variable does not seem to be updating. Any thoughts or suggestions much appreciated.
Thanks,
Aaron
I received a solution from the ESRI discussion board:
https://geonet.esri.com/thread/114520
Basically, a small edit in the Calculate field function did the trick. Here is the new code that worked:
arcpy.CalculateField_management(shp, "map_name","\"" + map_name + "\"", "PYTHON")