I have connected Python to Google Sheet through API under Google Cloud Platform . My project requires me to retrieve the new data whenever it is added to Google Sheet. Is there a way to trigger Python code to run to get the last row of the Google Sheet?
This depends on how your python script is run.
For example, if it's a cloud function, you can run it pretty easily with something like
function executePythonFunction() {
UrlFetchApp.fetch('<YOUR-PYTHON-CLOUD-FUNCTION-URL>');
}
by creating installable trigger for Change event
Related
What I basically want to happen is my on demand scheduled query will run when a new file lands in my google cloud storage bucket. This query will load the CSV file into a temporary table, perform some transformation/cleaning and then append to a table.
Just to try and get the first part running, my on demand scheduled query looks like this. The idea being it will pick up the CSV file from the bucket and dump it into a table.
LOAD DATA INTO spreadsheep-20220603.Case_Studies.loading_test
from files
(
format='CSV',
uris=['gs://triggered_upload/*.csv']
);
I was in the process of setting up a Google Cloud Function that triggers when a file lands in the storage bucket, that seems to be fine but I haven't had luck working out how that function will trigger the scheduled query.
Any idea what bit of python code is needed in the function to trigger the query?
It seems to me that it's not really a scheduled query you want at all. You don't want one to run at regular intervals, you want to run a query in response to a certain event.
Now, you've rigged up a cloud function to execute some code whenever a new file is added to a bucket. What this cloud function needs is the BigQuery python client library. Here's an example of how it's used.
All that remains is to wrap this code in an appropriate function and specify dependencies and permissions using the cloud functions python framework. Here is a guide on how to do that.
I have data running from a server to Google BigQuery. I would like the data to be analysed in R or Python and have the results presented in Google Data Studio or have resulting tables returned to BigQuery. I've read about the packages bigrquery and googleCloudStorageR, but I don't want to manually run the scripts through R studio every time new data is pushed to the server.
Is there a way to have a R/Python script connected to BigQuery which runs every time new data is pushed to BigQuery. I read this is possible in Power BI, but can't find a solution for Google Data Studio. Summarising; I would like a dashboard with live (or frequently updated) data that needs some analysis in R/Python, but without running the code constantly.
Thanks!
This is now possible via the googleCloudRunner package, which lets you set up pub/sub messages that can be triggered from BigQuery table updates, to run R code then push to another BigQuery table. Its one of the example use cases on the website.
I've stored some records on my datastore console. Now I wanna manipulate them to optimize Big Data analytics.
How can I write a Python cloud routine to make some transformations on my data? Can I trigger it with Datastore events?
Thanks a lot.
I have coded a little bit myself. You can find my code in GitHub.
What it is doing:
It is an HTTP Cloud Function
Establishing connection to Google Cloud Datastore through client()
Updates a value of specific entry in the entity using ID number and entry's column name
How to use:
Test this Cloud Function and get the idea of how it is working. Then manipulate according to your needs. I have tested this my self and it is working.
Create an HTTP trigger Python Cloud Function.
Set the name to updateDatastore
Copy and paste the code from GitHub.
Add this line google-cloud-datastore to the requirements.txt file.
In main code assign ENTITY_KIND your entity's kind value
In main code assign ENTITY_KEY your entity's key value
When clicked on HTTP trigger URL, after your Cloud Function's execution current time will be written in the column.
I'd like to publish many workbooks to tableau server. When I publish the same workbook again, it will overwrite it. What I want is that I'm going to not publish the same workbook again without overwriting it. Is there any way to do that in python?
Using tabcmd, the publish command has an overwrite option.
https://onlinehelp.tableau.com/current/server/en-us/tabcmd_cmd.htm#iddf805b62-18ff-4497-9245-adc6905b2084
An example from the documentation:
tabcmd publish "analysis_sfdc.hyper" -n "Sales Analysis"
You can use subprocess to call tabcmd.
That satisfies the question of how to not overwrite a workbook when publishing. If you want to check if the workbook exists, you could do a tabcmd get and check for a 404 error. I'm not aware of a command that gets a list of all published workbooks.
You can use the REST API. See https://onlinehelp.tableau.com/current/api/rest_api/en-us/help.htm#REST/rest_api_ref.htm#Publish_Workbook%3FTocPath%3DAPI%2520Reference%7C_____54.
The REST API allows you to run your python code from a computer separate from the Tableau Server so that your python code isn't competing with Server for CPU cycles.
The REST API includes methods to check if a workbook already exists and to overwrite it. The REST API is very robust and easily used in python code.
In excel you can create user defined functions with python using pyxll. I have been moving to Google spreadsheets and using their Google app script, but the libraries are so much bigger and better in python, I wish there was a way to build user defined functions using python from Google spreadsheets. There are ways to interact python with Google sheets like gspread. Is there a way to run python on Google app engine then get sheet to trigger that code? What other ways is there to trigger python code from Google spreadsheets?
You should create a webservice in GAE which then can be called using Google Apps Script UrlFetch class.
This is how I usually do to integrate a third party app with Apps Script App.
In a Spreadsheet container script you can create a code like
function myFunction(){
//your code
//Call the webservice
var response = UrlFetchApp.fetch('my_webservice_url', {payload:'...', method:'POST'});
Logger.log(response.getContentText());
// your code based on response
}
Above code can be triggered by a time driven trigger in Apps Script based on some conditions
Deploy your python code as a cloud function:
https://cloud.google.com/functions/docs/writing/http.
Then call your function with URL Fetch as shown above.
One way is to have some code that reads the spreadsheet all the time, then runs some other code when a condition is met.
Without GAE, you could use the following code:
#http://code.google.com/p/gdata-python-client/downloads/list
import gdata.spreadsheet.service as s
spreadsheet_key = 'spreadsheetkey'# https://docs.google.com/spreadsheet/ccc?key=<spreadsheet key>&usp=sharing#gid=0
worksheet_key = 'od6' #first tab
gd_client = s.SpreadsheetsService(spreadsheet_key, worksheet_key)
gd_client.email = 'user#gmail.com'
gd_client.password = 'password'
gd_client.ProgrammaticLogin()
list_feed = gd_client.GetListFeed(spreadsheet_key, worksheet_key)
for entry in list_feed.entry:
#read cell values and then do something if the condition is met
If you wanted to have the spreadsheet run code in a GAE app, then you could publish the spreadsheet and construct the URL of the spreadsheet (JSON) like this: https://spreadsheets.google.com/feeds/list/(spreadsheetkey)/od6/public/values?alt=json
This address can be accessed via the app, the cell values can be read, and some code can be triggered.
The approach is the same with both ideas: some code monitors the spreadsheet and when some condition is met, some other code is triggered. I'm not sure how you could run the code (in a GAE app, say) when the condition is met purely from the Google Spreadsheet.