I just learned about Google Apps Script and am wondering whether this is a solution for me.
I have a Python script on my desktop which eventually creates a CSV file stored on my computer. In the end it would be great to have the values from this CSV file appended to an existing Google Spreadsheet.
So now I'm wondering: is it possible to create a Google Apps Script which fetches these values from the locally stored CSV, and ideally even to call this Google Apps Script from within my Python script?
Python scripts can invoke Google Apps Scripts via its Execution API. (You'll find a Python Quickstart at that link.)
To go the other way (have Google Apps Script pull the file from your computer) is impossible... but if your Python script puts the csv file into your local Google Drive folder (which syncs to the cloud service), a Google Apps Script can access the sync'd file. You cannot trigger this based on any event - instead, it would need to be time based. (Fine if you know the file is generated periodically, e.g. daily at 3 AM, say.)
Yes. You can go down the execution API route. For a simpler approach, you could have a doGet() function in your script, make a HTTP GET request from Python to the script's URL and pass on the data as an input. Example here: GAS as backend service
Related
To start I managed to successfully run pywin32 locally where it opened the Excel workbooks and refreshed the SQL Query then saved and close them.
I had to download those workbooks locally from Sharepoint and have them sync to apply the changes using one drive.
My Question is would this be possible to do within Sharepoint itself ? Have a python script scheduled on a server and have the process occur there in the backend through a command.
I use this program called Alteryx where I can have batch files execute scripts and maybe I could use an API of some sort to accomplish this on a scheduled basis since thats the only server I have access to.
I have tried looking on this site and other sources but I can't find a post where it would reference this specifically.
I use Jupyter Notebooks to write my scripts and Alteryx to build a workflow with those scripts but I can use other IDEs if I need to.
I am developing a python script that downloads some excel files from a web service. These two files are combined with another one stored in my computer locally to produce the final file. This final file is loaded to some database and PowerBI dashboard to finally visualize data.
My question is: How can I schedule this to run it daily if my computer is turned off? As I said, two files are web scraped (so no problem to schedule) but one file is stored locally.
One solution that comes to my mind: Store the local file in Google Drive/OneDrive and download it with the API so my script is not dependent of my computer. But if this was the case, how can I schedule that? What service would you use? Heroku,...?
I am not entirely sure about your context, but I think you could look into using AWS Lambda for this. It is reasonably easy to set it up and also create a schedule for running code.
It is even easier to achieve this using the serverless framework. This link shows an example built with Python that will run on a schedule.
I am running the schedule package for exactly something like that.
It’s easy to setup and works very well.
I have been using Google CLoud for a few weeks now and I am facing a big problem for my limited GCP knowledge.
I have a python project whos goal is to "scrape" datas from a website using it's API. My project run a few tens of thousands of requests during executions and it can take very long (few hours, maybe more)
I have 4 python scripts in my project and it's all orchestrated by a bash script.
The execution is as follow :
The first script check a CSV file with all the instructions for the requests, and exeute the requests, save all the results from the requests in CSV files
Second script check the previously created CSV files and recreate an other CSV instruction file
The first script run again but with the new instructions and again save results in CSV files
Second script checks again and do the same again ...
... and so on a few times
Third script cleans the datas, delete duplicates and create an unique CSV file
Fourth script upload the final CSV file to bucket storage
Now I want to get ride of that bash script and I would like to automatize execution of thos scripts approx. once a week.
The problem here is the execution time. Here is what I already tested :
Google App Engine : The timeout of a request on GAE is limited to 10 minutes, and my functions can run for few hours. GAE is not usable here.
Google Compute Engine : My scripts will run max. 10-15 hours a week, keeping a compute engine up during all that time would be too pricey.
What could I do to automatize the execution of my scripts in a cloud environment ? What could be solutions I didn't though about, without changing my code ?
Thank you
A simple way to accomplish this without the need to get rid of the existing bash script that orchestrates everything would be:
Include the bash script on the startup script for the instance.
At the end of the bash script, include a shutdown command.
Schedule the starting of the instance using Cloud Scheduler. You'll have to make an authenticated call to the GCE API to start the existing instance.
With that, your instance will start on a schedule, it will run the startup script (that will be your existing orchestrating script), and it will shut down once it's finished.
I have a Python 2.7 script at https://github.com/jhsu802701/dopplervalueinvesting . When I run the screen.py script locally, the end result is a new screen-output sub-directory (within the root directory) and a results.csv file within it.
What I'm trying to do is put this script on a remote server, run this screen.py script every night, and make the results.csv file publicly readable.
I've tried to do this on Google App Engine, but I can't get it to work. The Google App Engine tutorial revolves around trying to dynamically create a web site, and I haven't been able to figure out how to make anything other than an index.html file in the root directory work. HOW DO I MAKE OTHER FILES PUBLICLY READABLE?
Is Google App Engine the way to go, or am I barking up the wrong tree? I understand that another route is using WebFaction, a web hosting provider that offers a whole Linux system. (Running my app on my current web host, MDDHosting, is not an option because lxml is not available without a much more costly VPS.)
In summary, my questions are:
How do I run my Python script in Google App Engine and make the output results.csv file publicly available?
If Google App Engine isn't the solution for me, should I use WebFaction? (I already tried Heroku, and it didn't work for me.)
What are my other options?
I'm willing to pay for a solution, but only if I get web hosting as well. (I'm not willing to pay for MDDHosting for my dopplervalueinvesting.com web site AND another host for running my script.)
I think GAE should be good for what you want, but you may need to work differently because, as a comment pointed out, you can't write to the file system but have to use the datastore instead.
So you need in your app.yaml list of handlers, something like
- url: /results.csv
script: deliver_results_file.py
- url: /screen
login: admin
script: screen.py
screen.py needs to save the results to the datastore in some convenient format. deliver_results_file.py then queries the datastore, and if the results are not already in CSV format then it converts them accordingly. It then writes the formatted data directly to the output (usually using self.response.out.write within a webapp request handler) as if it was a dynamically generated webpage.
Finally you want to schedule it to run once a night - I believe this is possible using cron jobs.
I have a Python 2.7 script that produces *.csv files. I'd like to run this Python script on a remote server and make the *.csv files publicly available to read.
Can this be done on Heroku? I've gone through the tutorial, but it seems to be geared towards people who want to create a whole web site.
If Heroku isn't the solution for me, what are the alternatives? I tried Google App Engine, but it requires Python 2.5 and won't work with 2.7.
MORE DETAILS:
I have a Python 2.7 script that analyzes all stocks that trade on the AMEX, NYSE, and NASDAQ exchanges and writes the output into *.csv files that can be read with a spreadsheet application. I want the script to automatically run every night on a remote server, and I want the *.csv files it produces to be publicly available.
Web hosting
Ok so you should be able to achieve what you need pretty simply. There are many webhosts that have python support. Your requirement is pretty simple. Just upload your python scripts to the web server. Then you can schedule a cron job to call your script at a specific time every day. Your script will run as scheduled and should save the csv files in the web servers document root. Keep in mind you don't need to your script to run in the web server, just on the same server. The web server will just serve your static csv files for you once you place them in the webserver's document root.
Desktop with dropbox
Another maybe easier option is take any desktop and schedule your python script to run on it each night you can do this in windows, Linux, Mac. Also install dropbox it gives you 2GB free online storage. Then your scripts just have to save the csv fies to the Dropbox/Public directory. When they do this they will automatically get synced to the dropbox servers and can be accessed through your public url like any other web page on the internet. You get 2GB for free which should be more then enough for a whole bunch of CSV files.