I am trying to develop a Python program for a task. The task is to find all routes in a state/county which are not reachable within 60 minutes from certain locations.
The task will take data (route info, speed limit, etc.) from excel. The excel data will be changed because there are many states/counties in USA/Canada.
I want to link the python code with excel such that we can get answers by clicking on the excel sheet.
Can you tell me how this can be done? Any ideas?
So I have task of automating a workflow such that:
Whenever an excel file (a.xlsx) is added/modified to the SharePoint Folder ->
My custom data extractor code will process this excel file ->
Extracted data will be stored as a new excel file (b.xlsx) in another folder on SharePoint.
This has to be achieved using Power Automate or Logic Apps with Azure Functions. But I am not able to wrap my head around how to go about this.
Has anyone implemented something like this before?
PS: My code is in Python.
So, when a.xlsx is created or updated, you want to perform some action to that file before save as b.xlsx in another folder.
If it is something that cannot be done just using Power Automate/Logic Apps, you can insert a azure function to your flow in 2 different ways:
Using an Azure Function Action (more here)
Using an Http Action (more here)
You will need an azure function of type http trigger
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-http-webhook-trigger?tabs=in-process%2Cfunctionsv2&pivots=programming-language-python
If you can share what you need to do before save as b.xlsx I may be able to help more
I have a Power BI template set up and I need to create weekly reports using this template. I only use a single CSV file per report and the structures of the files are all identical (same number of columns, same headers, etc).
NOTE: A new CSV file is downloaded and used every time I make a report. In other words, every week I download a new CSV file and make a new report from that file.
Right now, my Power BI template has a parameter that asks the user to input the file path to the CSV file before loading the report. I want to know if there is a way I can automate the following:
Opening the Power BI template file
Inputting the file path into the parameter field
Pressing the "Load" button
I understand that I could use Python and PyAutoGUI to control my desktop and, by extension, Power BI. Just wanted to know if there's another way to automate?
I understood your task. Here is number of step you need to take.
Download and install the PowerBi personal gateway on you machine.
https://powerbi.microsoft.com/en-us/gateway/
Go to settings of your dataset in PowerBi service (in your browser)
https://i.stack.imgur.com/vE9Bc.png
The gateway installed on your machine should be detected like in this picture https://i.stack.imgur.com/f3MS1.png
Setup a refresh schedule in this part of the dataset settings https://i.stack.imgur.com/LByMO.png
Once all of this is done, you do not need to manually do upload/publishing from your machine. Gateway will upload your new and fresh data to the report on its own.
Cheers!!
I'm so sorry for the vague question here, but I'm hoping an SPSS expert will be able to help me out here. We have some surveys that are done via SPSS, from which we extract data for an internal report. Right now the process is very cumbersome and requires going to the SPSS Data Collection Interviewer Server Administration page and manually exporting data from two different projects (which takes hours at a time!). We then take that data, massage it, and upload it to another database that drives the internal report.
My question is, does anyone out there know how to automate this process? Is there a SQL Server database behind the SPSS data? Where does the .mdd file come in to play? Can my team (who is well-versed in extracting data from various sources) tap into the SQL Server database behind SPSS to get our data? Or do we need some sort of Python script and plugin?
If I'm missing information that would be helpful in answering the question, please let me know. I'm happy to provide it; I just don't know what to provide.
Thanks so much.
As mentioned by other contributors, there are a few ways to achieve this. The simplest I can suggest is using the DMS (data management script) and windows scheduler. Ideally you should follow below steps.
Prerequisite:
1. You should have access to the server running IBM Data collection
2. Basic knowledge of windows task scheduler
3. Knowledge of DMS scripting
Approach:
1. Create a new DMS script from the template
2. If you want to perform only data extract / transformation, you only need input and output data source
3. In the input data source, create/build the connection string pointing to your survey on IBM Data collection server. Use the data source as SQL
4. In the select query: use "Select * from VDATA" if you want to export all variables
5. Set the output data connection string by selecting the output data format as SPSS (if you want to export it in SPSS)
6. run the script manually and see if the SPSS export is what is expected
7. Create batch file using text editor (save with .bat extension). Add below lines
cd "C:\Program Files\IBM\SPSS\DataCollection\6\DDL\Scripts\Data Management\DMS"
Call DMSRun YOURDMSFILENAME.dms
Then add a line to copy (using XCOPY) the data / files extracted to the location where you want to further process it.
Save the file and open windows scheduler to schedule the execution of this batch file for data extraction.
If you want to do any further processing, you create an mrs or dms file and add to the batch file.
Hope this helps!
There are a number of different ways you can accomplish easing this task and even automate it completely. However, if you are not an IBM SPSS Data Collection expert and don't have access to somebody who is or have the time to become one, I'd suggest getting in touch with some of the consultants who offer services on the platform. Internally IBM doesn't have many skilled SPSS resources available, so they rely heavily on external partners to do services on a lot of their products. This goes for IBM SPSS Data Collection in particular, but is also largely true for SPSS Statistics.
As noted by previous contributors there is an approach using Python for data cleaning, merging and other transformations and then loading that output into your report database. For maintenance reasons I'd probably not suggest this approach. Though you are most likely able to automate the export of data from SPSS Data Collection to a sav file with a simple SPSS Syntax (and an SPSS add-on data component), it is extremely error prone when upgrading either SPSS Statistics or SPSS Data Collection.
From a best practice standpoint, you ought to use the SPSS Data Collection Data Management module. It is very flexible and hardly requires any maintenance on upgrades, because you are working within the same data model framework (e.g. survey metadata, survey versions, labels etc. is handled implicitly) right until you load your transformed data into your reporting database.
Ideally the approach would be to build the mentioned SPSS Data Collection Data Management script and trigger it at the end of each completed interview. In this way your reporting will be close to real-time (you can make it actual real-time by triggering the DM script during the interview using the interview script events - just a FYI).
All scripting on the SPSS Data Collection platform including Data Management scripting is very VB-like, so for most people knowing VB, it is very easy to get started and it is documented very well in the SPSS Data Collection DDL. There you'll also be able to find examples of extracting survey data from SPSS Data Collection surveys (as well as reading and writing data to/from other databases, files etc.). There are also many examples of data manipulation and transformation.
Lastly, to answer your specific questions:
Yes, there is always an MS SQL Server behind SPSS Data Collection -
no exceptions. However, generally speaking the data model is way to
complex to read out data directly from it. If you have a look in it,
you'll quickly realize this.
The MDD file (short for Meta Data Document) is containing all survey meta
data including data source specifications, version history etc.
Without it you'll not be able to make anything of the survey data in
the database, which is the main reason I'd suggest to stay within the
SPSS Data Collection platform for as large part of your data handling
as possible. However, it is indeed just a readable XML file.
Note that the SPSS Data Collection Data Management Module requires a separate license and if the scripting needed is large or complex, you'd probably want base professional too, if that's not what you already use for developing the questionnaires and handling the surveys.
Hope that helps.
This isn't as clean as working directly with whatever database is holding the data, but you could do something with an exported data set:
There may or may not be a way for you to write and run an export script from inside your Admin panel or whatever. If not, you could write a simple Python script using Selenium WebDriver which logs into your admin panel and exports all data to a *.sav data file.
Then you can use the Python SPSS extensions to write your analysis scripts. Note that these scripts have to run on a machine that has a copy of SPSS installed.
Once you have your data and analysis results accessible to Python, you should be able to easily write that to your other database.
How do you build a Excel RTDServer in Python/IronPython. (IE I want to implement the IRTDServer interface in Python/IronPython. So I can push data into Excel in real time from Python/IronPython)
I have looked all over the place but the only examples that I'm able to find are in C#.
You can use pyrtd that implements an Excel RTD Server.
You can find the code on Google Code page of the project