In my company we want to build an application in Google app engine which will manage user provisioning to Google apps. But we do not really know what data source to use?
We made two propositions :
spreadsheet which will contains users' data and we will use spreadsheet API to get this data and use it for user provisioning
Datastore which will contains also users' data and this time we will use Datastore API.
Please note that my company has 3493 users and we do not know too many advantages and disadvantages of each solution.
Any suggestions please?
If you use the Datastore API, you will also need to build out a way to manage users data in the system.
If you use Spreadsheets, that will serve as your way to manage users data, so in that way managing the data would be taken care of for you.
The benefits to use the Datastore API would be if you'd like to have a seamless integration of managing the user data into your application. Spreadsheet integration would remain separate from your main application.
Related
I am currently trying to help a small business transition from using Google Sheets as a database to something more robust and scalable - preferably staying within Google services. I've looked into Google Cloud Storage and BigQuery, however - there are employees that need to manually update new data so anything in GCP won't be user friendly for non technical persons. I was thinking of employees still manually updating the Google sheets, and write a Python program to automatically update GCS or BigQuery, but the issue is that Google sheets is extremely slow and cannot handle the amount of data that's currently stored in there now.
Has anyone faced a similar issue and have any ideas/suggestions? Thank you so much in advance :)
What you might be able to do is to save the Google Sheet file as a .csv file and then import it in BigQuery. From there, maybe the employes can use simple commands to insert data. Please note that this question is very opinion based and anyone can suggest various ways to achieve what you want.
Create a web app hosted on App engine for the front end and data entry and connect it to Cloud SQL or BQ as a backend. You can create a UI in your web app where employees can access data from CloudSQL/BQ if needed. Alternatively you can use Google forms for data entry and connect it to Cloud SQL.
I am willing to show Google Analytics and Google Search Console data directly into Superset through their API.
Make direct queries to Google Analytics API in JSON (instead of storing the results into my database then showing them into Superset) and show the result in Superset
Make direct queries to Google Search Console API in JSON and show the result in Superset
Make direct queries to other amazing JSON APIs and show the result in Superset
How can I do so?
I couldn't find a Google Analytics datasource. I couldn't find a Google Search Console datasource either.
I can't find a way to display in Superset data retrieved from an API, only data stored in a database. I must be missing something, but I can't find anything in the docs related to authenticating & querying external APIs.
Superset can’t query external data API’s directly. Superset has to work with a supported database or data engine (https://superset.incubator.apache.org/installation.html#database-dependencies). This means that you need to find a way to fetch data out of the API and store it in a supported database / data engine. Some options:
Build a little Python pipeline that will query the data API, flatten the data to something tabular / relational, and upload that data to a supported data source - https://superset.incubator.apache.org/installation.html#database-dependencies - and set up Superset so it can talk to that database / data engine.
For more robust solutions, you may want to work with an devops / infrastructure to stand up a workflow scheduler like Apache Airflow (https://airflow.apache.org/) to regularly ping this API and store it in a database of some kind that Superset can talk to.
If you want to regularly query data from a popular 3rd party API, I also recommend checking out Meltano and learning more about Singer taps. These will handle some of the heavy lifting of fetching data from an API regularly and storing it in a database like Postgres. The good news is that there's a Singer tap for Google Analytics - https://github.com/singer-io/tap-google-analytics
Either way, Superset is just a thin layer above your database / data engine. So there’s no way around the reality that you need to find a way to extract data out of an API and store it in a compatible data source.
There is this project named shillelagh by one of Superset's contributors. This gives a SQL interface to REST APIs. This same package is used in Apache Superset to connect with gsheets.
New adapters are relatively easy to implement. There's a step-by-step tutorial that explains how to create a new adapter to an API or filetype in shillelagh.
The package shillelagh underlying uses SQLite Virtual Tables by using the SQLite wrapper APSW
Redash is an alternative to Superset for that task, but it doesn't have the same features. Here is a compared list of integrations for both tools: https://discuss.redash.io/t/a-comparison-of-redash-and-superset/1503
A quick alternative is paying for a third party service like: https://www.stitchdata.com/integrations/google-analytics/superset/
There is no such connector available by default.
A recommended solution would be storing your Google Analytics and Search Console data in a database, you could write a script that pulls data every 4 hours or whichever interval works for you.
Also, you shouldn't store all data but only the dimension/metrics you wish to see in your reports.
I am looking for some clarification on how the Google OAuth 2.0 architecture handles the management of multiple Google Analytics accounts.
The scenario is straightforward: I am developing an analytics tool that hooks into a Google Analytics account and generates a report. I have a service account set up and verified and working with my own analytics account. My question is how to elegantly expand this so that incoming users can simply share permissions to their Google Analytics accounts with me, I can collect their data and pull up a report for them?
The confusion comes from the service account/OAuth 2.0 documentation. It seems that simplest thing I can do is create a client id and credentials, have users click a button, collect an authorization code and use it to generate an access and refresh token, then store the refresh token for later use. However, there also appears to be a means of using a service account that has permissions granted to is and simply changing the account id that I embed in the analytics query. There's also documentation of using Flow and Credential objects to perform authentication and access procedures.
I am mixing all of these up and can't get my bearings on what actually does what and with what limitations. Any help outlining the use cases and proper usages of these systems would be greatly appreciated. Thanks!
I was going through googles's api python-client-library and google analytics api . I was able to do all steps mentioned in official docs but then I got some doubts. Since I've never done this kind of thing before, so I need your valuable suggestions/tips.
My Goal:
Want to design a web application in Python(using django/flask) and google-api-python-client. I have few matrices(coming from my web ecommerce product that is using GA.) and I'm not sure if google analytics dashboard by default support at that deep level. so I will use Google's analytic api to customize data according to my need and show in my analytic web app(which can be accessed by any one).
Doubts/Queries:
1) first of all which reporting api I would be needing for this mentioned here. core api or metadata api?
2) while I was setting up the project and client key, I chose 2nd option(OAuth 2.0 client ID)
is that ok or should I chose service account? once I selected 2nd options there were couple of radio buttons(web, android, ios, other, etc.) I chose other or should I chose web?
3) once i chose other option from radio button list, I executed my script and it prompt a browser to ask for permission, I allowed. here my question is if I put my application on production would there not be any browser, what would happen in that case?
I would really appreciate if you can help me in these queries, sorry for long question, this is my first question.
PS: Bottom line is how one should structure and develop there analytic web application in general.
The key thing to understand is that Google Analytics is an authenticated API. It is designed make it easy to allow the end user to access their own data. It is designed to be hard to allow the end user to access data they do not own.
If you are building a web application to allow your users to access their own private data It is recommended that you use a client side authentication method, such as in this example or this example.
If you are trying to build a web application that shares your private data with your users there are a few ways to go about it:
You could collect the data server side in python using a service account (note you will have to add the service account the GA account you wish it to have access).
You can take a hybrid approach and have a service account generate an access token and use the embed api to actually make the query.
In the end I would encourage you to spend some time to read Using OAuth 2.0 to Access Google APIs, and understand senarios descussed and ask yourself which of these senerios will work best for my application.
I intend to create an event management system, where users may login, view a list of events, sign up to volunteer at events, cancel registration, and view a calendar view of events of the day. In addition, users should receive emails, notifying them when the event(s) they have signed up for is coming up.
Primarily, it should be accessible through a browser, but I would like, in the future, for it to become an organization-wide iPad app.
I have a few questions. How should I store my data? Will MySQL be sufficient? Is it better if I sign up for a service like Heroku, Appfog, or GAE? If I don't need these services, would it be able to run if I used my shared hosting plan over at GoDaddy?
I was thinking about using Google Drive Spreadsheets to store data, and using Google's spreadsheet API to access and write to it. Are there any advantages or disadvantages of taking this route?
Only 50 or so people would need to access it.
I like Python, but haven't done much on the web. Is web2py a good idea, or is it overkill for my purposes?
For UI, I was leaning towards HTML + jQuery + AJAX? + Bootstrap.
Thanks!
I would recommend you to use Django, which is a macro framework written in Python to develop web applications.
Database is of your choice like mysq, postgresql or by default you can use the sqlite3 database offered by Django.
The same goes with UI. You can use the technologies like html, js, CSS, jQuery, bootstrap etc.
Hope this helps.