We have an online company. The website contains many features and we would like to analyse which customers visit which sites, and how many times.
PROBLEM:
I am trying to write a program that should use certain Google Analytics data to create an HTML table (using pandas), that can be viewed anytime with the most recent Google Analytics data.
WHAT I HAVE DONE:
I have managed to get authenticated and have all permissions (I believe so because I haven't received a permission error message yet) and get in return a service object, which I don't know how to use/open?
#!/usr/bin/env python3
"""Script that does the following:
1) Initialise a Google Analytics Reporting API service object
"""
import os
import argparse
from apiclient.discovery import build
import httplib2
from oauth2client import client
from oauth2client import file
from oauth2client import tools
import yaml
import pandas as pd
scopes = ['https://www.googleapis.com/auth/analytics.readonly']
# Path to client_secrets.json file.
client_secrets_path = 'credentials/client_secret_xx.apps.googleusercontent.com.json'
def initialise_analyticsreporting():
"""Initializes the analyticsreporting service object.
Returns:
an authorized analyticsreporting service object.
"""
# Parse command-line arguments.
parser = argparse.ArgumentParser(
formatter_class=argparse.RawDescriptionHelpFormatter,
parents=[tools.argparser])
flags = parser.parse_args([])
# Set up a Flow object to be used if we need to authenticate.
flow = client.flow_from_clientsecrets(
"credentials/client_secret_xx.apps.googleusercontent.com.json",
scope='https://www.googleapis.com/auth/analytics.readonly',
message=tools.message_if_missing(client_secrets_path))
# Prepare credentials, and authorize HTTP object with them.
# If the credentials don't exist or are invalid run through the native client
# flow. The Storage object will ensure that if successful the good
# credentials will get written back to a file.
storage = file.Storage('credentials/analyticsreporting.dat')
credentials = storage.get()
if credentials is None or credentials.invalid:
credentials = tools.run_flow(flow, storage, flags)
http = credentials.authorize(http=httplib2.Http())
# Build the service object.
analytics = build('analyticsreporting', 'v4', http=http)
return analytics
This returns analytics, looking like this <googleapiclient.discovery.Resource object at 0x00000XOXOXOXOX>
At the end of the day, I would just want to have the Google Analytics data in a pandas data frame so that I can manipulate and work with it. I am no expert with Google Analytics. This is crucial for our business, any help would be appreciated. I am really scratching my head.
EXPECTED OUTPUT (to serve as a guideline to what I want to achieve, I am fairly skilled with Pandas. The problem is to get the data from GA):
user_id site visits
123 abc.com/something 12
234 abc.com/smthngelse 7
Thanks, I am happy to answer questions
Your analytics object is just a service object - you can use it to access the methods that return the data, but it does not, by itself, contain Google Analytics data. As you are using version 4 of the core reporting API, you can just look at this example from the documentation:
def get_report(analytics):
# Use the Analytics Service Object to query the Analytics Reporting API V4.
return analytics.reports().batchGet(
body={
'reportRequests': [
{
'viewId': VIEW_ID,
'dateRanges': [{'startDate': '7daysAgo', 'endDate': 'today'}],
'metrics': [{'expression': 'ga:sessions'}]
}]
}
).execute()
Change the metrics and add dimensions to your liking (not every combination works or makes sense, though), enter your view id and you should be good to go.
Related
I'm trying to consume Google's Workspace APIs, but I'm having extreme difficulty with the documentation to create my first code, following the first steps I did the following
I created a project within Google Cloud
I enabled the Admin SDK API
I created a service account
I created a key in Json format
in the Workspace dashboard under delegation across domain I added the unique id and the following scope
[
'https://www.googleapis.com/auth/apps.order',
'https://www.googleapis.com/auth/siteverification',
'https://www.googleapis.com/auth/directory.readonly',
'https://www.googleapis.com/auth/admin.directory.user',
'https://www.googleapis.com/auth/admin.reports.usage.readonly',
'https://www.googleapis.com/auth/admin.reports.audit.readonly',
'https://www.googleapis.com/auth/gmail.send'
]
I would like to use the document from the link https://developers.google.com/admin-sdk/reports/reference/rest to consult the activities of a specific user but I can't find an example code to consume this API using these credentials in Python , I'm new in this area and would like some help.
Generate a token and when I tried to use an api it didn't work and it was unauthorized, below is the code I used
import requests
url = "https://admin.googleapis.com/admin/reports/v1/activity/users/usuario#exemplo.com/applications/calendar"
payload = ""
headers = {"Authorization": "Bearer xptoz_exemple_test=PHQbcdddx3xxxxxxxxxxxxddddddddd"}
response = requests.request("GET", url, data=payload, headers=headers)
print(response.text)
You are getting unauthorized because the service account doesn't have permission to do what ever it is you are trying to do. To get permission you need to be using the proper scope, and the service account needs to not only have permission to use this scope but it must have delegated to a user on the domain that has access to the data.
First you need to be sure that your authorization code is delegating to a user on your domain.
#!/usr/bin/python
from oauth2client.service_account import ServiceAccountCredentials
from httplib2 import Http
from apiclient.discovery import build
scopes = [ 'https://www.googleapis.com/auth/admin.reports.usage.readonly' ]
credentials = ServiceAccountCredentials.from_json_keyfile_name('/path/to/my/key.json', scopes)
delegated_credentials = credentials.create_delegated('me#mydomain.com')
http_auth = credentials.authorize(Http())
service = build('admin', 'directory_v1', credentials=creds)
Then you should consider having a look at Python quickstart the auth in this is set up to use an installed app. However the rest of the code should show you how to use the Client library rather then sending all the requests manually like you are now.
So I'm trying to create a new calendar, but i want to be able to specify what google account to create it in, assuming i have the credentials for said account, which i do. The code bellow creates it on the currently signed in user, or requires user interaction to allow access. Is there a way to specify an user and run the command on the background. I essentially just want to add a calendar to my account when the program runs, but i cant guarantee that my account will be logged in at the time.
I believe this was possible with the version 2 of the google api through ClientLogin, but i'm trying to use version 3.
import gflags
import httplib2
from apiclient.discovery import build
from oauth2client.file import Storage
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.tools import run
FLAGS = gflags.FLAGS
FLOW = OAuth2WebServerFlow(
client_id='MY_CLIENT_KEY',
client_secret='MY_CLIENT_SECRET',
scope='https://www.googleapis.com/auth/calendar',
user_agent='MyApp/v1')
storage = Storage('calendar.dat')
credentials = storage.get()
if credentials is None or credentials.invalid == True:
credentials = run(FLOW, storage)
http = httplib2.Http()
http = credentials.authorize(http)
service = build(serviceName='calendar', version='v3', http=http)
calendar = {
'summary': 'Test3',
'timeZone': 'America/New_York'
}
created_calendar = service.calendars().insert(body=calendar).execute()
With V3, you'll need to use a Service Account in order to act as the user. The process is described in the Google Drive documentation, you just need to use Calendar API v3 scopes and references instead of Google Drive API.
Another option would be to store the OAuth2 refresh token and use it to grab valid access tokens even if the user is not logged in. See my reply to google Calendar api v3 Auth only for first time
I would like to develop an app engine application that directly stream data into a BigQuery table.
According to Google's documentation there is a simple way to stream data into bigquery:
http://googlecloudplatform.blogspot.co.il/2013/09/google-bigquery-goes-real-time-with-streaming-inserts-time-based-queries-and-more.html
https://developers.google.com/bigquery/streaming-data-into-bigquery#streaminginsertexamples
(note: in the above link you should select the python tab and not Java)
Here is the sample code snippet on how streaming insert should be coded:
body = {"rows":[
{"json": {"column_name":7.7,}}
]}
response = bigquery.tabledata().insertAll(
projectId=PROJECT_ID,
datasetId=DATASET_ID,
tableId=TABLE_ID,
body=body).execute()
Although I've downloaded the client api I didn't find any reference to a "bigquery" module/object referenced in the above Google's example.
Where is the the bigquery object (from snippet) should be located?
Can anyone show a more complete way to use this snippet (with the right imports)?
I've Been searching for that a lot and found documentation confusing and partial.
Minimal working (as long as you fill in the right ids for your project) example:
import httplib2
from apiclient import discovery
from oauth2client import appengine
_SCOPE = 'https://www.googleapis.com/auth/bigquery'
# Change the following 3 values:
PROJECT_ID = 'your_project'
DATASET_ID = 'your_dataset'
TABLE_ID = 'TestTable'
body = {"rows":[
{"json": {"Col1":7,}}
]}
credentials = appengine.AppAssertionCredentials(scope=_SCOPE)
http = credentials.authorize(httplib2.Http())
bigquery = discovery.build('bigquery', 'v2', http=http)
response = bigquery.tabledata().insertAll(
projectId=PROJECT_ID,
datasetId=DATASET_ID,
tableId=TABLE_ID,
body=body).execute()
print response
As Jordan says: "Note that this uses the appengine robot to authenticate with BigQuery, so you'll to add the robot account to the ACL of the dataset. Note that if you also want to use the robot to run queries, not just stream, you need the robot to be a member of the project 'team' so that it is authorized to run jobs."
Here is a working code example from an appengine app that streams records to a BigQuery table. It is open source at code.google.com:
http://code.google.com/p/bigquery-e2e/source/browse/sensors/cloud/src/main.py#124
To find out where the bigquery object comes from, see
http://code.google.com/p/bigquery-e2e/source/browse/sensors/cloud/src/config.py
Note that this uses the appengine robot to authenticate with BigQuery, so you'll to add the robot account to the ACL of the dataset.
Note that if you also want to use the robot to run queries, not just stream, you need to robot to be a member of the project 'team' so that it is authorized to run jobs.
I'm looking for a good way to retrieve every emails address of my contacts from a google account for a "desktop" application in Python.
In a first time, I created an app via Google Code. I toggled Google Plus API, retrieving most of my user data, but not any of my contacts.
I started investigate, and I found a lot of stuff, but most of them was outdated.
I found a good way to retrieve my contacts, using gdata library but granting me a full read/write access on it, via https://www.google.com/m8/feeds with no feedback.
self.gd_client = gdata.contacts.client.ContactsClient(source='MyAppliName')
self.gd_client.ClientLogin(email, password, self.gd_client.source)
According to the official 'google contact api' google group, which migrated to stackoverflow, read only access is broken.
By the way, I'm not a huge fan of 'Trust my application, I use read only access, I swear."
I found the google api playground at https://developers.google.com/oauthplayground in which they use OAuth2.0 token with most of apis, including contact, toggling a webpage:
Google OAuth 2.0 Playground is requesting permission to:
Manage your contacts
According to this playground, it's possible to use OAuth2.0 with google contact api, but I have no idea how to add https:// www.google.com/m8/feeds to my scope, which doesn't appear on the list.
Is there an other way to do that ?
If this question is still open for you, here is some sample code how to use oauth2 and Google Contact API v3:
import gdata.contacts.client
from gdata.gauth import AuthSubToken
from oauth2client import tools
from oauth2client.client import flow_from_clientsecrets
from oauth2client.file import Storage
def oauth2_authorize_application(client_secret_file, scope, credential_cache_file='credentials_cache.json'):
"""
authorize an application to the requested scope by asking the user in a browser.
:param client_secret_file: json file containing the client secret for an offline application
:param scope: scope(s) to authorize the application for
:param credential_cache_file: if provided or not None, the credenials will be cached in a file.
The user does not need to be reauthenticated
:return OAuth2Credentials object
"""
FLOW = flow_from_clientsecrets(client_secret_file,
scope=scope)
storage = Storage(credential_cache_file)
credentials = storage.get()
if credentials is None or credentials.invalid:
# Run oauth2 flow with default arguments.
credentials = tools.run_flow(FLOW, storage, tools.argparser.parse_args([]))
return credentials
SCOPES = ['https://www.google.com/m8/feeds/', 'https://www.googleapis.com/auth/userinfo.email']
credentials = oauth2_authorize_application('client-secret.json', scope=SCOPES)
token_string = credentials.get_access_token().access_token
# deprecated!
# auth_token = AuthSubToken(token_string, SCOPES)
with open('client-secret.json') as f:
oauth2_client_secret = json.load(f)
auth_token = gdata.gauth.OAuth2Token(
client_id=oauth2_client_secret['web']['client_id'],
client_secret=oauth2_client_secret['web']['client_secret'],
scope=SCOPES,
user_agent='MyUserAgent/1.0',
access_token=credentials.get_access_token().access_token,
refresh_token=credentials.refresh_token)
client = gdata.contacts.client.ContactsClient(auth_token=auth_token)
query = gdata.contacts.client.ContactsQuery()
The request should look like:
https://accounts.google.com/o/oauth2/auth?
scope=https%3A%2F%2Fwww.google.com%2Fm8%2Ffeeds&
state=<myState>&
redirect_uri=<Redirect URI>&
response_type=code&
client_id=<my Client ID>&approval_prompt=force
This will obtain read/write access to the user's contacts.
I'm trying to connect to the google doubeclick api through a service account (client email and p12 certificate), using the python client library as in the following example:
http://code.google.com/p/google-api-python-client/source/browse/samples/service_account/tasks.py
It's returning me an empty access_token:
In [9]: type(credentials.access_token)
Out[9]: <type 'NoneType'>
What is the significance of this? Is there something I am likely doing wrong? I have also tried accessing the tasks api as in the example (thinking that possibly the doubleclick api is not a supported scope) but same result.
UPDATE (example code):
from oauth2client.client import SignedJwtAssertionCredentials
import httplib2
from adspygoogle.dfp import DfpClient
f = file('/path/to/.ssh/google-api.p12', 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials('<email>', key, scope='https://www.google.com/apis/ads/publisher')
credentials.refresh(http)
http = httplib2.Http()
http = credentials.authorize(http)
client = DfpClient.DfpClient(headers={'networkCode': '<code>', 'applicationName': 'test', 'userAgent': 'test', 'oauth2credentials': credentials})
inventory_service = client.GetInventoryService()
inventory_service.getAdUnitsByStatement({'query':''})
ERROR:
DfpAuthenticationError: [AuthenticationError.NO_NETWORKS_TO_ACCESS # ]
The NO_NETWORKS_TO_ACCESS error specifically means that you did authenticate to the API endpoint but that your account isn't associated with a network. Search for that error on this page https://developers.google.com/doubleclick-publishers/docs/reference/v201203/InventoryService?hl=en.
You either need to have the Google account you are authenticating as invited to the network via the DoubleClick User Interface or you need to use impersonation.
A more specific writeup on DFP API and service accounts was recently posted https://developers.google.com/doubleclick-publishers/docs/service_accounts to the documentation. I suggest you look at the alternative section of that documentation to determine if you might prefer an OAuth 2.0 installed flow.