I have a JSON containing users' information:
[
{
'id': 1
'password': 'abc'
'email': 'abc#def.com'
},
...
]
and a function:
foo(user_id)
I would like to write a Python script to run on my GCE server, which can send emails to all of my users containing a link each.
When e.g. user with id = 2 clicks his/her link, my server will run
foo(2)
What is the simplest way of generating the URLs and making them trigger the function on my server?
I don't know how to use HTTP but would love to take this opportunity to learn, I just have no idea where to start, as most python-HTTP tutorials I found dealt with client-side.
Rather than using a GCE server, I recommend looking at Google App Engine, since Google will manage a lot more for you. This will let you focus on just the server code, rather than the server configuration.
Check out this tutorial that shows how to deploy a simple python app to GCP and do something similar to what you're asking about.
If you must use GCE, you can setup a simple HTTP server in Python like Flask or Bottle to run you code.
Related
I am trying to create Jira issues with data populated in a row in google sheet, I plan to put a button to read the contents of the row and create Jira issues, I have figured the Jira API wrote the script for it and also the Google sheets API to read the row values to put in the Jira API.
How do I link the button to the python script in my local machine in a simple manner, I went through other similar asks here, but they are quite old and hoping now some new way might be available.
Please help me achieve this in a simple way, any help is greatly appreciated.
Thank You and Stay Safe.
Google sheets cannot run code on your local machine. That means you have a few options:
Click the button locally
Instead of clicking a button on the google sheet, you can run the script yourself from the command line. This is probably how you tested it, and not what you want.
Watch the spreadsheet
You could have your python script setup to run every few minutes. This has the benefit of being very straightforward to setup (google for cron jobs), but does not have a button, and may be slower to update. Also, it stops working if you turn off your computer.
Make script available for remote execution
You can make it so that your script can be run remotely, but it requries extra work. You could buy a website domain, and point it towards your computer (using dynamic dns), and then make the google sheet request your new url. This is a lot of work, and costs real money. This is probably not the best way
Move the script into the cloud
This is probably what you want: cut your machine out of the loop. You can use Google AppScripts, and rewrite your jira code there. You can then configure the google AppScript to run on a button click.
Unfortunately, you really can't get a button press in a Google Sheet to launch a local Python script-- Google Sheets / your browser cannot access your local files and programs in that way.
You can create a button that runs a Google Apps Script (GAS). This is some code based on JavaScript, attached to the spreadsheet, hosted/run by Google. Here's a tutorial on how to run via button press.
If you can port your script into GAS, that is one solution.
If you want to keep the script in Python, you basically need to deploy it and then use GAS to call your Python script. The simplest way I can think of (which is not super simple, but is totally doable!) is as follows:
1. Make your Python script into an API.
Use something like Flask or FastAPI to setup your own API. The aim that when a certain URL is visited, it will trigger your Python program to run a function which does all the work. With FastAPI it might look like this:
from fastapi import FastAPI
app = FastAPI()
def main():
print("Access Google Sheet via API...")
# your code here
print("Upload to JIRA via API...")
# your code here
#app.get("/")
def root():
main()
return {"message": "Done"}
Here, "/" is the API endpoint. When you visit (or make a "get" request) to the URL of the deployed app, simply ending in "/", the root function will get called, which calls your main function. (You could set up different URL endings to do different things).
We can test this locally. If you follow the setup instructions for FastAPI, you should be able to run the command uvicorn main:app --reload which launches a server at http://127.0.0.1:8000. If you visit that URL in your browser, the script should get run and the message "Done" should appear in your browser.
2. Deploy your Python app
There are many services that can host your Python program, such as Heroku or Google Cloud. They may offer free trials but this generally costs money. FastAPI has instructions for deploying to Deta which seems to currently have a free tier.
When your app is app and running, there should be an associated web address such as "https://1kip8d.deta.dev/". If you access this in the browser it will run your script and return the "Done" message.
3. Hit your Python API from Google Sheets, using GAS
The last step it to "hit" that URL using GAS, instead of visiting it manually in the browser. Following the tutorial mentioned above, create a GAS script linked to your spreadsheet, and a button which is "assigned" to your script. The script will look something like this:
function myFunction() {
var response = UrlFetchApp.fetch("https://1kip8d.deta.dev/");
Logger.log(response.getContentText());
}
Now, whenever you press the button, GAS will visit that URL, which will cause your Python script to execute.
You might want to check out Google Colaboratory. It's a service by Google that can host your Python code (called a "notebook"), connect with your Google Drive (and other Google services), and make calls out to web endpoints (which would be your Jenkins server). I think those are the three pieces you're dealing with here.
Just to be clear... your code wouldn't be local anymore (if that's really important to you). Instead, it would be hosted by Google. The notebooks are saved to your Google Drive account, so you get the security that provides.
I am currently building a web-app that requires a Spotify user to login using their credentials in order to access their playlists
I'm using the Spotipy python wrapper for Spotify's Web API and generating an access token using,
token = util.prompt_for_user_token(username,scope,client_id,client_secret,redirect_uri)
The code runs without any issues on my local machine. But, when I deploy the web-app on AWS, it does not proceed to the redirected uri and allow for user login.
I have tried transferring the ".cache-username" file via SCP to my AWS machine instance and gotten it to work in limited fashion.
Is there a solution to this issue? I'm fairly new to AWS and hence don't have much to go on or any idea where to look. Any help would be greatly appreciated. Thanks in advance!!
The quick way
Run the script locally so the user can sign in once
In the local project folder, you will find a file .cache-{userid}
Copy this file to your project folder on AWS
It should work
The database way
There is currently an open feature request on Github that suggests to store tokens in a DB. Feel free to subscribe to the issue or to contribute https://github.com/plamere/spotipy/issues/51
It's also possible to write a bit of code to persist new tokens into a DB and then read from it. That's what I'm doing as part of an AWS Lambda using DynamoDB, it's not very nice but it works perfectly https://github.com/resident-archive/resident-archive/blob/a869b73f1f64538343be1604d43693b6165cc58a/functions/to-spotify/main.py#L129..L157
The API way
This is probably the best way, as it allows multiple users to sign in simultaneously. However it is a bit more complex and requires you host a server that's accessible by URL.
This example uses Flask but one could adapt it to Django for example https://github.com/plamere/spotipy/blob/master/examples/app.py
I have a simple python script that gets the local weather forecast and sends an email with the data
I want to run this script daily, i found out that cron is used for this purpose but online cron jobs require a url
I wanted to ask how to host my python scripts so that they run online through a url, if possible that is...
I would recommend using Heroku with a python buildpack as a starting point. Using the flask library, you can very minimally start a web container and expose the endpoint online which can then be queried from your cron service. Heroku also provides a free account which ideally should fit your need.
As a peek into how easy it is to setup flask, well..
from flask import Flask
app = Flask(__name__)
#app.route('/cron-me')
def cron_me():
call_my_function_here()
return 'Success'
.. and you're done ¯\_(ツ)_/¯
Try setting up a simple Flask app at www.pythonanywhere.com, I think it will do the job for you even with the free account.
EDIT: And for sending e-mails, you can use Mailgun, with the free version you can send e-mails to a small number of addresses that need to be validated from the recipient side (to avoid people using it for spam :-))
I created a Flask-Webservice with Python that runs independently inside a docker container. I then uploaded the docker image to an Azure Container Registry. From there I can create a WebService (for Containers) with some few clicks in the Azure Portal, that runs this container. So far so good. It behaves just as I want it to.
But of course I don't want anyone to access the service. So I need some kind if authentication. Luckily (or so I thought) there is a built-in authentication-mechanism (I think it is based on OAuth ... I am not that well versed in security issues). Its documentation is a bit sparse on what actually happens and also concentrates on solutions in C#.
I first created a project with Google as described here and then configured the WebApp-Authentication with the Client-Id and Secret. I of course gave Google a java script source and callback-url, too.
When I now log off my Google account and try a GET-Request to my Webservice in the Browser (the GET should just return a "hello world"-String), I am greeted with a Login Screen ... just as I expected.
When I now login to Google again, I am redirected to the callback-url in the browser with some kind of information in the parameters.
a token perhaps? It looks something like this:
https://myapp.azurewebsites.net/.auth/login/google/callback?state=redirxxx&code=xxx&authuser=xxx&session_state=xxx&prompt=xxx).
Here something goes wrong, because an error appears.
An error occurred.
Sorry, the page you are looking for is currently unavailable.
Please try again later.
If you are the system administrator of this resource then you should check the error log for details.
Faithfully yours, nginx.
As far as I now, nginx is a server software that hosts my code. I can imagine that it also should handle the authentication process. It obviously lets all requests through to my code when authentication is turned off, but blocks un-authenticated accesses otherwise and redirects to the google login. Google then checks if your account is authorized for the application and redirects you to the callback with the access token along with it. This then returns a cookie which should grant my browser access to the app. (I am just reproducing the documentation here).
So my question is: What goes wrong. Does my Browser not accept the cookie. Did I something wrong when configuring Google+ or the Authentication in the WebApp. Do I have to use a certain development stack to use the authentication. Is it not supported for any of the technologies I use (Python, Flask...).
EDIT
#miknik:
In Microsofts documentation of the authentication/authorization it says
The authentication and authorization module runs in the same sandbox
as your application code. When it's enabled, every incoming HTTP
request passes through it before being handled by your application
code.
...
The module runs separately from your application code and is
configured using app settings. No SDKs, specific languages, or changes
to your application code are required.
So while you are probably right that the information in the callback-redirect is the authorization grant/code and that after that this code should now be used to get an access token from Google, I don't quite understand how this would work in my situation.
As far as I can see it Microsofts WebApp for Container-Resource on Azure should take care of getting the token automatically and return it as part of the response to the callback-request. The documentation states 4 steps:
Sign user in: Redirects client to /.auth/login/.
Post-authentication: Provider redirects client to /.auth/login//callback.
Establish authenticated session: App Service adds authenticated cookie to response.
Serve authenticated content: Client includes authentication cookie in subsequent requests (automatically handled by browser).
It seems to me that step 2 fails and that that would be exactly what you wrote: that the authorization grant is to be used by the server to get the access token but isn't.
But I also don't have any control over that. Perhaps someone could clear things up by correcting me on some other things:
First I can't quite figure out which parts of my problem represent which role in the OAuth-scheme.
I think I am the Owner, and by adding users to the list in the Google+-Project I authorize them to use my service.
Google is obviously the authorization server
my WebService (or better yet my WebApp for Containers) is the resource server
and finally an application or postman that does the requests is the Client
In the descriptions of OAuth I read the problematic step boils down to: the resource server gets the access token from the authorization server and passes it along to the client. And Azures WebApps Resource is prompted (and enabled) to do so by being called with the callback-url. Am I right somewhere in this?
Alas, I agree that I don't quite understand the whole protocol. But I find most descriptions on the net less than helpful because they are not specific to Azure. If anyone knows a good explanation, general or Azure-specific, please make a comment.
I found a way to make it work and I try to explain what went wrong as good as I can. Please correct me if I go wrong or use the wrong words.
As I suspected the problem wasn't so much that I didn't understand OAuth (or at least how Azure manages it) but the inner workings of the Azure WebApp Service (plus some bad programming on my part). Azure runs an own Server and is not using the built-in server of flask. The actual problem was that my flask-program didn't implement a WSGI-Interface. As I could gather this is another standard for python scripts to interact with any server. So while rudimentary calls from the server (I think Azure uses nginx) were possible, more elaborate calls, like the redirect to the callback url went to dev/null.
I build a new app following this tutorial and then secured it by following the authentication/authorization-tutorial and everything worked fine. The code in the tutorial implements WSGI and is probably more conform to what Azure expects. My docker solution was too simple.
My conclusion: read up on this WSGI-standard that flask always warned me about and I didn't listen and implement it in any code that goes beyond fiddeling around in development.
To use Google API's, after activating them from the Google Developers Console, one needs to generate credentials. In my case, I have a backend that is supposed to consume the API server side. For this purpose, there is an option to generate what the Google page calls "Key for server applications". So far so good.
The problem is that in order to generate the key, one has to mention IP addresses of servers that would be whitelisted. But GAE has no static IP address that I could use there.
There is an option to manually get the IP's by executing:
dig -t TXT _netblocks.google.com #ns1.google.com
However there is no guarantee that the list is static (further more, it is known to change from time to time), and there is no programatic way I could automate the use of adding IP's that I get from dig into the Google Developers Console.
This leaves me with two choices:
Forget about GAE for this project, ironically, GAE cannot be used as a backend for Google API's (better use Amazon or some other solution for that). or
Program something like a watchdog over the output of the dig command that would notify me if there's a change, and then I would manually update the whitelist (no way I am going to do this - too dangerous), or allow all IP's to use the Google API granted it has my API key. Not the most secure solution but it works.
Is there any other workaround? Can it be that GAE does not support consuming Google API's server side?
You can use App Identity to access Google's API from AppEngine. See: https://developers.google.com/appengine/docs/python/appidentity/. If you setup your app using the cloud console, it should have already added your app's identity with permission to your project, but you can always check that out. From the "Permissions" Tab in cloud console for your project, make sure your service account is added under "Service Accounts" (in the form of your_app_id#appspot.gserviceaccount.com)
Furthermore, if you use something like the JSON API Libs available for python, you can use the bundled oauth2 library to do all of this for you using AppAssertionCredentials to authorize the API you wish to use. See: https://developers.google.com/api-client-library/python/guide/google_app_engine#ServiceAccounts
Yes, you should use App Identity. Forget about getting an IP or giving up on GAE :-) Here is an example of how to use Big Query, for example, inside a GAE application:
static {
// initializes Big Query
JsonFactory jsonFactory = new JacksonFactory();
HttpTransport httpTransport = new UrlFetchTransport();
AppIdentityCredential credential = new AppIdentityCredential(Arrays.asList(Constants.BIGQUERY_SCOPE));
bigquery = new Bigquery.Builder(httpTransport, jsonFactory, credential)
.setApplicationName(Constants.APPLICATION_NAME).setHttpRequestInitializer(credential)
.setBigqueryRequestInitializer(new BigqueryRequestInitializer(Constants.API_KEY)).build();
}