I have two BigQuery projects and I want to copy a view from Project 1 to Project 2:
from google.cloud import bigquery
proj_1 = bigquery.Client.from_service_account_json(<path>, project='Project 1')
dataset_1 = proj_1.dataset(<dataset_name>)
view_1 = dataset_1.table(<view_name>) # View to copy, already existing
proj_2 = bigquery.Client.from_service_account_json(<path>, project='Project 2')
dataset_2 = proj_2.dataset(<dataset_name>)
view_2 = dataset_2.table(<view_name>) # Destination for copied view
# Start copy job like Google says
# https://cloud.google.com/bigquery/docs/tables#copyingtable
I get the following error:
RuntimeError: [{'message': 'Using table <project>:<dataset>.<view_name> is not allowed for this operation because of its type. Try using a different table that is of type TABLE.', 'reason': 'invalid'}]
I already know that if I set the attribute view_query, view_2 will be recognized as a view. If I set it manually, it works. But the second (automated) solution does not, because the attribute view_1.view_query is always None.
view_2.view_query = 'SELECT * FROM ...' # works
view_2.view_query = view_1.view_query # Won't work, because view_1.view_query is always None
How can I access the query of view_1?
Call of view_1.reload() loads the attribute view_query.
See https://googlecloudplatform.github.io/google-cloud-python/latest/bigquery-usage.html
So
view_1.reload()
view_2.view_query = view_1.view_query
view_2.create() # No need for a copy job, because there is no data copied
does the trick now.
Related
I have a python program that loads an order to a treeview, this order is loaded in the form of documents in a firestore collection from firebase. When I press the order that I want to load, I call the function loadOrder with the necessary id to filter them. But for some reason they don't load.
This is my code:
def loadPedido(idTarget):
docs = db.collection(u'slots').where(u'slotId', u'==', idTarget).stream()
for doc in docs:
docu = doc.to_dict()
nombre = (docu.get('SlotName'))
entero = (docu.get('entero'))
valor = (docu.get('slotPrecio'))
print(f'{doc.id} => {nombre}')
trvPedido.insert("",'end',iid= doc.id, values=(doc.id,nombre, entero, valor))
idTarget is the id to filter, and check with a print that it arrives correctly.
i tried this:
If I write the result of the varable directly in the code, it loads correctly, like so:
...
docs = db.collection(u'slots').where(u'slotId', u'==', u"2996gHQ32CNFMp5vyieu").stream()
...
I'm currently developing locally an Azure function that communicates with Microsoft Sentinel, in order to fetch the alert rules from it, and more specifically their respective querys :
credentials = AzureCliCredential()
alert_rules_operations = SecurityInsights(credentials, SUBSCRIPTION_ID).alert_rules
list_alert_rules = alert_rules_operations.list(resource_group_name=os.getenv('RESOURCE_GROUP_NAME'), workspace_name=os.getenv('WORKSPACE_NAME'))
The issue is that when I'm looping over list_alert_rules, and try to see each rule's query, I get an error:
Exception: AttributeError: 'FusionAlertRule' object has no attribute 'query'.
Yet, when I check their type via the type() function:
list_alert_rules = alert_rules_operations.list(resource_group_name=os.getenv(
'RESOURCE_GROUP_NAME'), workspace_name=os.getenv('WORKSPACE_NAME'))
for rule in list_alert_rules:
print(type(rule))
##console: <class 'azure.mgmt.securityinsight.models._models_py3.ScheduledAlertRule'>
The weirder issue is that this error appears only when you don't print the attribute. Let me show you:
Print:
for rule in list_alert_rules:
query = rule.query
print('query', query)
##console: query YAY I GET WHAT I WANT
No print:
for rule in list_alert_rules:
query = rule.query
...
##console: Exception: AttributeError: 'FusionAlertRule' object has no attribute 'query'.
I posted the issue on the GitHub repo, but I'm not sure whether it's a package bug or a runtime issue. Has anyone ran into this kind of problems?
BTW I'm running Python 3.10.8
TIA!
EDIT:
I've tried using a map function, same issue:
def format_list(rule):
query = rule.query
# print('query', query)
# query = query.split('\n')
# query = list(filter(lambda line: "//" not in line, query))
# query = '\n'.join(query)
return rule
def main(mytimer: func.TimerRequest) -> None:
# results = fetch_missing_data()
credentials = AzureCliCredential()
alert_rules_operations = SecurityInsights(
credentials, SUBSCRIPTION_ID).alert_rules
list_alert_rules = alert_rules_operations.list(resource_group_name=os.getenv(
'RESOURCE_GROUP_NAME'), workspace_name=os.getenv('WORKSPACE_NAME'))
list_alert_rules = list(map(format_list, list_alert_rules))
I have tried with same as you used After I changed like below; I get the valid response.
# Management Plane - Alert Rules
alertRules = mgmt_client.alert_rules.list_by_resource_group('<ResourceGroup>')
for rule in alertRules:
# Try this
test.query = rule.query //Get the result
#print(rule)
if mytimer.past_due:
logging.info('The timer is past due!')
Instead of this
for rule in list_alert_rules:
query = rule.query
Try below
for rule in list_alert_rules:
# Try this
test.query = rule.query
Sorry for the late answer as I've been under tons of work these last few days.
Python has an excellent method called hasattr that checks if the object contains a specific key.
I've used it in the following way:
for rule in rules:
if hasattr(rule, 'query'):
...
The reason behind using this is because the method returns object of different classes, however inherited from the one same mother class.
Hope this helps.
I'm trying to authorize a view programmatically in BigQuery and I have the following issue: I just tried the code proposed in the Google docs (https://cloud.google.com/bigquery/docs/dataset-access-controls) but when it comes the part of getting the current access entries for the dataset the result is always empty. I don't want to overwrite the current configuration. Any idea about this behavior?
def authorize_view(dataset_id, view_name):
dataset_ref = client.dataset(dataset_id)
view_ref = dataset_ref.table(view_name)
source_dataset = bigquery.Dataset(client.dataset('mydataset'))
access_entries = source_dataset.access_entries # This returns []
access_entries.append(
bigquery.AccessEntry(None, 'view', view_ref.to_api_repr())
)
source_dataset.access_entries = access_entries
source_dataset = client.update_dataset(
source_dataset, ['access_entries']) # API request
I'm running into the following error when trying to load a table from google cloud storage:
BadRequest: 400 Load configuration must specify at least one source URI (POST https://www.googleapis.com/bigquery/v2/projects/fansidata/jobs)
Meanwhile my uri is valid (ie: i can see it in the gcs web app)
uris = ['gs://my-bucket-name/datastore_backup_analytics_2016_12_21_2_User/1569751766512529035929A5AA9742/output-0']
job_name = 'Load_User'
destinationTable = dataset.table('Transfer')
job = bigquery_client.load_table_from_storage(job_name, destinationTable, uris)
job.begin()
I could be wrong, but it looks like load_table_from_storage in the Python API expects a single string for the third argument as opposed to a list. If you want to match multiple files, you can use a * at the end. For example,
uri = 'gs://my-bucket-name/datastore_backup_analytics_2016_12_21_2_User/1569751766512529035929A5AA9742/output-*']
job_name = 'Load_User'
destinationTable = dataset.table('Transfer')
job = bigquery_client.load_table_from_storage(job_name, destinationTable, uri)
job.begin()
Client.load_table_from_storage takes one or more source URIs (that is what *soure_uris means in python). E.g.:
job = client.load_table_from_storage(
'load-job-123', my_table_object,
'gs://my-bucket-name/table_one',
'gs://my-bucket-name/table_two')
I'm brand new at Python and I'm trying to write an extension to an app that imports GA information and parses it into MySQL. There is a shamfully sparse amount of infomation on the topic. The Google Docs only seem to have examples in JS and Java...
...I have gotten to the point where my user can authenticate into GA using SubAuth. That code is here:
import gdata.service
import gdata.analytics
from django import http
from django import shortcuts
from django.shortcuts import render_to_response
def authorize(request):
next = 'http://localhost:8000/authconfirm'
scope = 'https://www.google.com/analytics/feeds'
secure = False # set secure=True to request secure AuthSub tokens
session = False
auth_sub_url = gdata.service.GenerateAuthSubRequestUrl(next, scope, secure=secure, session=session)
return http.HttpResponseRedirect(auth_sub_url)
So, step next is getting at the data. I have found this library: (beware, UI is offensive) http://gdata-python-client.googlecode.com/svn/trunk/pydocs/gdata.analytics.html
However, I have found it difficult to navigate. It seems like I should be gdata.analytics.AnalyticsDataEntry.getDataEntry(), but I'm not sure what it is asking me to pass it.
I would love a push in the right direction. I feel I've exhausted google looking for a working example.
Thank you!!
EDIT: I have gotten farther, but my problem still isn't solved. The below method returns data (I believe).... the error I get is: "'str' object has no attribute '_BecomeChildElement'" I believe I am returning a feed? However, I don't know how to drill into it. Is there a way for me to inspect this object?
def auth_confirm(request):
gdata_service = gdata.service.GDataService('iSample_acctSample_v1.0')
feedUri='https://www.google.com/analytics/feeds/accounts/default?max-results=50'
# request feed
feed = gdata.analytics.AnalyticsDataFeed(feedUri)
print str(feed)
Maybe this post can help out. Seems like there are not Analytics specific bindings yet, so you are working with the generic gdata.
I've been using GA for a little over a year now and since about April 2009, i have used python bindings supplied in a package called python-googleanalytics by Clint Ecker et al. So far, it works quite well.
Here's where to get it: http://github.com/clintecker/python-googleanalytics.
Install it the usual way.
To use it: First, so that you don't have to manually pass in your login credentials each time you access the API, put them in a config file like so:
[Credentials]
google_account_email = youraccount#gmail.com
google_account_password = yourpassword
Name this file '.pythongoogleanalytics' and put it in your home directory.
And from an interactive prompt type:
from googleanalytics import Connection
import datetime
connection = Connection() # pass in id & pw as strings **if** not in config file
account = connection.get_account(<*your GA profile ID goes here*>)
start_date = datetime.date(2009, 12, 01)
end_data = datetime.date(2009, 12, 13)
# account object does the work, specify what data you want w/
# 'metrics' & 'dimensions'; see 'USAGE.md' file for examples
account.get_data(start_date=start_date, end_date=end_date, metrics=['visits'])
The 'get_account' method will return a python list (in above instance, bound to the variable 'account'), which contains your data.
You need 3 files within the app. client_secrets.json, analytics.dat and google_auth.py.
Create a module Query.py within the app:
class Query(object):
def __init__(self, startdate, enddate, filter, metrics):
self.startdate = startdate.strftime('%Y-%m-%d')
self.enddate = enddate.strftime('%Y-%m-%d')
self.filter = "ga:medium=" + filter
self.metrics = metrics
Example models.py: #has the following function
import google_auth
service = googleauth.initialize_service()
def total_visit(self):
object = AnalyticsData.objects.get(utm_source=self.utm_source)
trial = Query(object.date.startdate, object.date.enddate, object.utm_source, ga:sessions")
result = service.data().ga().get(ids = 'ga:<your-profile-id>', start_date = trial.startdate, end_date = trial.enddate, filters= trial.filter, metrics = trial.metrics).execute()
total_visit = result.get('rows')
<yr save command, ColumnName.object.create(data=total_visit) goes here>