I have a python script and used it to create a dataframe in Power BI.
Now I want to edit that dataframe in Power BI but don´t enter from scratch as new data because I want to keep all the charts inside my Power BI model.
For example in my old dataframe i specified some dates inside my script so the information was limited to those dates. Now i want to change the dates to new ones but dont want to lose all the model.
df = df
You can edit the python scripts doing the following steps:
Open Query Editor
At 'Applied steps', the first one, source, contains a small gear symbol just on the right side, click on it.
You can change the script direct into Power Query.
I hope you're not dong this in a PowerBI Python Visual. If you're using Python under the Transform tab in the Power Query Editor, the key to your problem lies not in Python itself, but rather in the reference function available to you if you right-click the table under queries in the Power Query Editor:
Try this:
1: Save the following sample data in a csv as C:\pbidata\src.csv file and load it into PowerBI using Get Data > Text/Csv
A,B,C
1,1*0,100
2,20,200
3,30,300
2: Display it as a table:
3: Open the Power Query Editor through Edit Queries
4: Add some Python
Here you can insert a Python snippet after the Changed type step under Applied steps with Transform > Run Python Script. Inserting the following example code:
# 'dataset' holds the input data for this script
import pandas as pd
df=dataset.copy(deep=True)
df['D']=df['C']*2
... will give you this:
5: And let's say that you're happy with this for now and that you'd like to make a plot out of it back on the Power BI Desktop. I'm using a clustered bar chart to get this:
6: Now, like you're saying, if you'd like to have df['D']=df['C']/4 instead, but retain the same dataset, Python script and figure Plot 1, Py script 1, go back to the Power Query Editor and use the functionality that I mentioned in the beginning:
7: And add another Python snippet:
# 'dataset' holds the input data for this script
import pandas as pd
df=dataset.copy(deep=True)
df['D']=df['D']/4
And there we go:
Now you have two different Python snippets that build on the same dataset. You still have the data from the first snippet, and you can do whatever you want with the second snippet without messing up your data source.
8: Insert another chart to verify:
9: Maybe have some fun with the whole thing by changing the source file:
Data:
A,B,C
100,10,100
2,20,200
3,30,150
New plots:
Related
TLDR: How can I make a notebook cell save its own python code to a file so that I can reference it later?
I'm doing tons of small experiments where I make adjustments to Python code to change its behaviour, and then run various algorithms to produce results for my research. I want to save the cell code (the actual python code, not the output) into a new uniquely named file every time I run it so that I can easily keep track of which experiments I have already conducted. I found lots of answers on saving the output of a cell, but this is not what I need. Any ideas how to make a notebook cell save its own code to a file in Google Colab?
For example, I'm looking to save a file that contains the entire below snippet in text:
df['signal adjusted'] = df['signal'].pct_change() + df['baseline']
results = run_experiment(df)
All cell codes are stored in a List variable In.
For example you can print the lastest cell by
print(In[-1]) # show itself
# print(In[-1]) # show itself
So you can easily save the content of In[-1] or In[-2] to wherever you want.
Posting one potential solution but still looking for a better and cleaner option.
By defining the entire cell as a string, I can execute it and save to file with a separate command:
cell_str = '''
df['signal adjusted'] = df['signal'].pct_change() + df['baseline']
results = run_experiment(df)
'''
exec(cell_str)
with open('cell.txt', 'w') as f:
f.write(cell_str)
Background
Right now I'm creating a macro to help automate the creation of some graphs in VBA. However, the creation of the graphs requires specific tasks to be done, for example, certain points in a series to be larger depending on previous instances. I would much rather do this data manipulation in python.
Problem
I want to use excel for its user-friendly interface but want to handle all the data manipulation within Python. How can I send data I create in VBA to python. To clarify I'm not trying to read specific cells in the excel sheet.
If I define a string in VBA say...
Dim example_string as String
example_string = "Hello, 1, 2, 3, Bye"
How can I send this information I created within VBA to Python for manipulation?
More Specifics
I have a textbox in excel that is filled by the user, which I read using VBA. I want to send that txt data from VBA to python. The user highlights the desired cells, which are not necessarily the same each time, clicks a button and fills a textbox. I don't want to use range or specific cell selection since this would require the user to specifically enter all the desired data into cells (too time-consuming).
I want to understand the basic procedure of how to send data between VBA and python.
You can do the whole thing in python, it will be more efficient and you can either use excel or sqlite3 as database, go here to read about graphic interfaces with tkinter, use pandas and numpy to process your data.
If you insist in sending data to python, import sys to your python script to read parameters and then run it from vba with the shell() method.
EDIT: You wanted an example, here it is =>
Open a new excel file, create a procedure like this (VBA CODE):
Sub sendToPython()
Dim shell As Object
Dim python As String
Dim callThis As String
Dim passing
Set shell = VBA.CreateObject("Wscript.Shell")
'/* This is where you installed python (Notice the triple quotes and use your own path *always)*/
python = """C:\Users\yourUserName\appdata\local\programs\python\python37\python.exe"""
'/* This is the data you'll be passing to python script*/
passing = "The*eye*of*the*tiger"
callThis = "C:\Users\yourUserName\desktop\yourScriptName.py " & passing & ""
shell.Run python & callThis
End Sub
The idea is to create some kind of a parser in python, this is my silly example (PYTHON CODE):
import sys
f = open("log.txt", "w")
arg = (sys.argv[1]).split("*")
s = " "
arg = s.join(arg)
print("This is the parameter i've entered: " + arg, file=f)
Notice how i used sys to read a parameter and i exported to actually see some results because otherwise you'll just see a black screen popping up for like a millisecond.
I also found this article, but it requires you to wrap the python script in a class and i don't know if that works for you
I am trying to open an excel file from python, get it to recalculate and then save it with the newly calculated values.
The spreadsheet is large and opens fine in LibreOffice with GUI, and initially shows old values. If I then do a Data->Calculate->Recalculate Hard I see the correct values, and I can of course saveas and all seems fine.
But, there are multiple large spreadsheets I want to do it from so I don't want to use a GUI instead I want to use Python. The following all seems to work to create a new spreadsheet but it doesn't have the new values (unless I again manually do a recalculate hard)
I'm running on Linux. First I do this:
soffice --headless --nologo --nofirststartwizard --accept="socket,host=0.0.0.0,port=8100,tcpNoDelay=1;urp"
Then, here is sample python code:
import uno
local = uno.getComponentContext()
resolver = local.ServiceManager.createInstanceWithContext("com.sun.star.bridge.UnoUrlResolver", local)
context = resolver.resolve("uno:socket,host=localhost,port=8100;urp;StarOffice.ServiceManager")
remoteContext = context.getPropertyValue("DefaultContext")
desktop = context.createInstanceWithContext("com.sun.star.frame.Desktop", remoteContext)
document = desktop.getCurrentComponent()
file_url="file://foo.xlsx"
document = desktop.loadComponentFromURL(file_url, "_blank", 0, ())
controller=document.getCurrentController()
sheet=document.getSheets().getByIndex(0)
controller.setActiveSheet(sheet)
document.calculateAll()
file__out_url="file://foo_out.xlsx"
from com.sun.star.beans import PropertyValue
pv_filtername = PropertyValue()
pv_filtername.Name = "FilterName"
pv_filtername.Value = "Calc MS Excel 2007 XML"
document.storeAsURL(file__out_url, (pv_filtername,))
document.dispose()
After running the above code, and opening foo_out.xlsx it shows the "old" values, not the recalculated values. I know that the calculateAll() is taking a little while, as I would expect for it to do the recalculation. But, the new values don't seem to actually get saved.
If I open it in Excel it does an auto-recalculate and shows the correct values and if I open in LibreOffice and do Recalculate Hard it shows the correct values. But, what I need is to save it, from python like above, so that it already contains the recalculated values.
Is there any way to do that?
Essentially, what I want to do from python is:
open, recalculate hard, saveas
It seems that this was a problem with an older version of LibreOffice. I was using 5.0.6.2, on Linux, and even though I was recalculating, the new values were not even showing up when I extracted the cell values directly.
However, I upgraded to 6.2 and the problem has gone away, using the same code and the same input files.
I decided to just answer my own question, instead of deleting it, as this was leading to a frustration until I solved it.
I am developing a production model and a very long list of constraints (ca. 600). How can I export the list to an excel file for analysis?
so far i am using model.cons.pprint() to generate a visualisation of the constraints at runtime. However its very cumbersome to review 100s of lines in a terminal window.
This is the current solution that produces an output in the terminal window at runtime.
# define constraints
model.cons = pyomo.ConstraintList()
#
# ....
#
model.cons.pprint()
#
this produces
Size=577, Index=cons_index, Active=True, Key : Lower : Body : Upper : Active
and then 577 lines of constraints. which is fine to get started.
i have tried the following to generate a dataframe,
df_cons = pd.DataFrame.from_dict(model.cons.extract_values(), orient='index', columns=[str(model.cons)])
print(df_cons)
print("--------------")
However this did not work.
I am looking for a command to export the constraint list to a dataframe or write directly to an excel file.
I have data in an excel spreadsheet (*.xlsx) that consists of 1,213 rows of sensitive information (so, I'm sorry I can't share the data) and 35 columns. Every entry is a string (I don't know if that is screwing it up or not). The first row is the column names and I've never had a problem importing it with the column names embedded before (it's just easier to click that they're embedded so I don't have to name every column by hand). I put the path to the data in the quick start wizard and hit the next button and it doesn't do anything. I hit it again and it turns the mouse into the loader as if it's loading. I've waited for it for 15 minutes before, but every time I click on QlikView the program just crashes.
I have a deadline I have to meet here and I can't afford to not finish this project. It's extremely important that I get it working.
Just as a NB, I used Python to merge two Excel spreadsheets together so I don't know if that may be what's causing the problem either. I can open the file perfectly fine in Excel though.
I was using the walkthrough when creating a new file when I should have just made a script. First click on edit script in the menu bar. Click Table Files.... Choose your file, then make sure that the Labels section has Embedded Labels in the dropdown. It will create a query like the following-
LOAD [Resource name],
[Employee ID],
Vertical,
[Contract Type *],
Notes
FROM
[D:\path\to\file\*.xlsx]
(ooxml, embedded labels, table is Sheet1);
That's one part of the solution, but then I ran into a new problem. It said that it was able to fetch all of the rows, but when I started making the charts and graphs it was only showing 6 data points. I recreated the file, did exactly what I did above but also added the transformation step. Now the problem is solved.
I have had QlikView crash when importing an Excel spreadsheet that was exported with the SQuirreL SQL client (from a Firebird database). Opening the spreadsheet in Excel, and saving it again solved the problem.
I know that this is no longer relevant to your problem, but hopefully it can help someone with a similarly appearing issue.
I see that you do not include a 'header is 0 lines,' which could be cauing an issue?
Follows a snippet from my standard Excel file import - just the FROM section. My setup is done through variables but they follow this form:
Set vTableW = 'WIP Metrics' ;
Set vPathData = '..\Raw Data Reports\' ;
Set vFile08 = 'Misc Transactions VCB*.xlsx' ;
Set vHeader08 = 2 ;
Set vSheet08 = 'Misc Trans' ;
Set vWhere08 = ( Len(Trim([Date Received])) > 0
And Len(Trim([Lot Number])) > 0
And Len(Trim([Y/N])) > 0
And Len(Trim([Initials])) > 0 ) ;
'$(vTableW)':
Load
AutoNumber(RowNo(), 1) As [_Load WIPWO ID],
...
additional columns
...
If(IsNull([Comments]), '', Trim([Comments])) As [VCB Comments]
From
'$(vPathData)$(vFile08)'
(ooxml, embedded labels, header is $(#vHeader08) lines, table is '$(vSheet08)')
Where($(vWhere08)) ;
Regarding the point made above relating to syn keys, add the line
Exit Script ;
Before your import to check load is OK to that point. Then move it immediately after the Excel load and repeat. Use the debug facility in the Load Script process.
Hope this helps!