Develop Reference

Dataplex API Tag Policies

I'm on exploration Dataplex API with Python in Google Documentation, there is documentation to Get Lake, Zone, Assets, etc. I've explored that documentation, but I didn't find any documentation related to Tag Policies, for example, I need to attach my Tag Template and add Policy Tag to my BigQuery Table via API.
Is it possible to attach Tag Template and add Policy Tag into BigQuery Table via API?
Here is the link that I've explored:
Dataplex API Service
Dataplex API Metadata Service
Data Catalog API

for attaching tag templates to BigQuery table, first you will have to lookup the entry in dataplex using api
https://cloud.google.com/python/docs/reference/datacatalog/latest/google.cloud.datacatalog_v1.types.LookupEntryRequest
and then attach is to table using api
https://cloud.google.com/python/docs/reference/datacatalog/latest/google.cloud.datacatalog_v1.types.CreateTagRequest
here's sample code, this creates tag template and also attaches it to table in same code base
https://cloud.google.com/data-catalog/docs/samples/data-catalog-quickstart
and to attach policy, use this api
https://cloud.google.com/python/docs/reference/datacatalog/latest/google.cloud.datacatalog_v1.types.CreatePolicyTagRequest
hope this helps

again.
To simulate the pythonic api's behaviour, I used google cloud api explorer to explain in detail. see below.
The entry lookup is to search for object(s) you want to attach a tag/tag templates
Basically here's how I simulated the api calls using api explorer
To attach a tag to a BigQuery table, first step is to search the table up using Datacatalog api url below
LookupEntryRequest
The parameters I passed to get below response is
sqlResource: "bigquery.table.myproject.zz_DataSet.tblOne"
Above should give you output as
{
"name": "projects/myproject/locations/australia-southeast2/entryGroups/#bigquery/entries/mykey",
"type": "TABLE",
"schema": {
"columns": [
{
"type": "STRING",
"mode": "NULLABLE",
"column": "firstname"
},
{
"type": "STRING",
"mode": "NULLABLE",
"column": "lastname"
}
]
},
"sourceSystemTimestamps": {
"createTime": "2023-01-16T04:22:49.397Z",
"updateTime": "2023-01-16T04:22:49.397Z"
},
"linkedResource": "//bigquery.googleapis.com/projects/myproject/datasets/zz_DataSet/tables/tblOne",
"bigqueryTableSpec": {
"tableSourceType": "BIGQUERY_TABLE"
},
"usageSignal": {
"updateTime": "2023-02-05T07:59:59.928Z",
"usageWithinTimeRange": {
"30D": {
"totalCompletions": 7,
"totalFailures": 1,
"totalExecutionTimeForCompletionsMillis": 7385
}
}
},
"integratedSystem": "BIGQUERY",
"fullyQualifiedName": "bigquery:myproject.zz_DataSet.tblOne"
}
The search gives you ability to query multiple tables or attach tags at Dataset level too, see parameters section on link above.
This is why I suggest you use entry look up first as its more scalable code.
API Call two: This is how i simulated the attach tag to resource. If you go to link below
CreateTagRequest
As an example: I pre created an tag template from console and then used the template-id value to pass as a parameter to the request
Input:
parent: projects/myproject/locations/australia-southeast2/entryGroups/#bigquery/entries/mykey from above name element
request body:
{
"template": "projects/myproject/locations/australia-southeast1/tagTemplates/api_call_test_tag_template",
"fields": {
"name": {
"stringValue": "apitestcall"
}
}
}
Output:
Below is the response generated and if you see in data catalog console, you will see bigquery table with the tag template attached to it with value to name field as "apitestcall" attached to it. see image attached
{
"name": "projects/myproject/locations/australia-southeast2/entryGroups/#bigquery/entries/mykey/tags/tagsKey",
"template": "projects/myproject/locations/australia-southeast1/tagTemplates/api_call_test_tag_template",
"fields": {
"name": {
"displayName": "name",
"stringValue": "apitestcall"
}
},
"templateDisplayName": "api-call-test-tag-template"
}
Finally, please do make sure you have all the correct IAM permissions required for this task.

Azure Functions - Kafka Trigger and Output Binding - Consume exactly-once

I'm testing the Kafka Trigger and Output Binding in Azure Functions to consume a topic and write the message to another topic, very simple code.
But when I enable the auto-scale feature and the function provision new instances, I'm losing the 'exactly-once' feature and apparently some messages are being delivered to more than one instance.
host.json
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[3.6.0, 4.0.0)"
}
}
function.json
{
"scriptFile": "__init__.py",
"bindings": [
{
"type": "kafkaTrigger",
"name": "kafkaTrigger",
"direction": "in",
"brokerList": "%PEP_BEES_KAFKA_BOOTSTRAP%",
"topic": "%PEP_BEES_KAFKA_SOURCE_TOPIC%",
"consumerGroup": "%PEP_BEES_KAFKA_SOURCE_TOPIC_CONSUMER_GROUP%"
},
{
"type": "kafka",
"direction": "out",
"name": "kafkaOutput",
"brokerList": "%PEP_BEES_KAFKA_BOOTSTRAP%",
"topic": "%PEP_BEES_KAFKA_DESTINATION_TOPIC%"
}
]
}
init.py
import logging, json
from azure.functions import KafkaEvent
import azure.functions as func
import typing
def main(kafkaTrigger : typing.List[KafkaEvent], kafkaOutput: func.Out[str]):
message = json.loads(str(kafkaTrigger.get_body().decode('utf-8')))
input_msg = str(message['Value'])
kafkaOutput.set(input_msg)
As you can see in the image below, the 'test-topic-output' (destination) has more messages than the 'test-topic' (source), indicating that sometimes more than one instance is consuming a message:
Message count
If I disable the auto-scaling feature, this behavior does not happen.
I just need that the 'exactly-once' feature works even with the function auto-scale enabled, to have an elastic environment.
EDIT
I just found out some erros during the Kafka Trigger processing:
Kafka Trigger errors
Confluent.Kafka.KafkaException:
at Confluent.Kafka.Impl.SafeKafkaHandle.StoreOffsets (Confluent.Kafka, Version=1.9.0.0, Culture=neutral, PublicKeyToken=12c514ca49093d1e)
at Confluent.Kafka.Consumer2.StoreOffset (Confluent.Kafka, Version=1.9.0.0, Culture=neutral, PublicKeyToken=12c514ca49093d1e) at Microsoft.Azure.WebJobs.Extensions.Kafka.AsyncCommitStrategy2.Commit (Microsoft.Azure.WebJobs.Extensions.Kafka, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: /mnt/vss/_work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Trigger/AsyncCommitStrategy.cs:28)
at Microsoft.Azure.WebJobs.Extensions.Kafka.FunctionExecutorBase`2.Commit (Microsoft.Azure.WebJobs.Extensions.Kafka, Version=3.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: /mnt/vss/_work/1/s/src/Microsoft.Azure.WebJobs.Extensions.Kafka/Trigger/FunctionExecutorBase.cs:87)

If I disable the auto-scaling feature, this behavior does not happen
If you scale up, that wouldn't prevent records that were just consumed from existing inputs from being sent to output function.
You'd need a way to catch a consumer group rebalance within the function, and upon that action, then don't send to the output... Given that Azure doesn't expose Kafka native API, then its unlikely that is possible. You also would need to control when the function commits its consumer offsets (assuming PEP_BEES_KAFKA_SOURCE_TOPIC_CONSUMER_GROUP is a constant value); e.g. if it commits before kafkaTrigger.get_body(), then you should get less "extras", but if it commits after kafkaOutput.set(), then you'll have more upon scaling up...
Rather than use Python or serverless functions for replicating topics, you can use MirrorMaker or Kafka Streams instead.

I had noticed similar behavior of the kafka triggers consuming duplicate events when the scaling up the instance count (and surprisingly also during scale down). In batch mode with batch size of 50, its always the same 50 events that get read by 2 instance. I have also noticed this happens only once per instance scale up or (down).
One of the potential solution you can explore is using a warmup trigger that run during scale-out operation.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-warmup?tabs=in-process&pivots=programming-language-python

Can i build an application in python that will auto populate docusign fields from a google sheet?

I use docusign every day and have set up a google sheet that auto populates the email subject and body.
I'm still having to copy and paste this data into the 'email subject' & 'email message'. See screenshot example.
Is there a way i can reduce this process to the click of a button?

Absolutely. Here's are two rough flows that would be helpful to pursue.
You could make use of the Google Apps Script to push data directly to DocuSign. Downside here is that you'd be writing the code in JavaScript.
You can write a short python app yourself, which uses the Google Sheets API (via python sdk) to retrieve your data, then pass along the appropriate info to the DocuSign eSignature API (via python sdk).
In either case, once you have access to the data you wish to forward to DocuSign, you will want to first authenticate your account to make API requests on your behalf.
Then you can make a request like this:
POST /envelopes
{
"emailSubject": "Please sign these documents",
"emailBlurb": "Thank you for subscribing. Click the link to sign",
"documents": [
{
"documentBase64": "JVBER...iUlRU9GCg==",
"name": "Test Doc",
"fileExtension": "pdf",
"documentId": "1"
}
],
"recipients": {
"signers": [
{
"email": "test#test.com",
"name": "Test User",
"recipientId": "1",
"routingOrder": "1",
"tabs": {
"numberTabs": [
{
"tabLabel": "PO #",
"locked": "false",
"xPosition": "200",
"yPosition": "200",
"documentId": "1",
"pageNumber": "1"
}
]
}
}
]
},
"status": "sent"
}

How to access EventGrid trigger Event properties in Azure functions input bindings

I want to pull in cosmos documents into my azure function based on contents of eventgrid events that it triggers on (python worker runtime). Is it possible to do this?
I have below function.json:
{
"scriptFile": "__init__.py",
"bindings": [
{
"type": "eventGridTrigger",
"name": "event",
"direction": "in"
},
{
"name": "documents",
"type": "cosmosDB",
"databaseName": "Metadata",
"collectionName": "DataElementMappings",
"sqlQuery" : "SELECT * from c where c.id = {subject}",
"connectionStringSetting": "MyCosmosDBConnectionString",
"direction": "in"
}
]
}
I want to use properties of the event in the query for the cosmos input binding. I tried it with subject here. It fails with:
[6/4/2020 5:34:45 PM] System.Private.CoreLib: Exception while executing function: Functions.orch_taskchecker. System.Private.CoreLib: The given key 'subject' was not present in the dictionary.
So I am not using the right key name. The only key I got to work is {data} but unfortunately I was not able to access any properties inside the event data using the syntax {data.property} as it says in the docs. I would expect all of the event grid schema keys to be available to use in other bindings since an event is a JSON payload. I got none of them to work, e.g. eventTime, event_type, topic, etc.
Since I saw an example of the same idea for storage queues and they used for example {Queue.id} I tried things like {Event.subject}, {EventGridEvent.subject} all to no avail.
Nowhere in the docs can I find samples of event grid trigger data being used in other bindings. Can this be achieved?

For EventGridTrigger (C# script) can be used for input bindings a custom data object of the EventGridEvent class.
The following example shows a blob input bindings for event storage account:
{
"bindings": [
{
"type": "eventGridTrigger",
"name": "eventGridEvent",
"direction": "in"
},
{
"type": "blob",
"name": "body",
"path": "{data.url}",
"connection": "rk2018ebstg_STORAGE",
"direction": "in"
}
],
"disabled": false
}
Note, that only a property from the data object can be referenced.

There is the event.get_json() method that you can use to access the data field in the python code, I did not look at the bindings

Create a Drive File with a set locale in international settings through Drive API

I need to create a file using the Google Drive API (I'm using v3, the latest at the moment). Using python if it matters.
My code is below,
drive_service.files().create(supportsTeamDrives=True, body={
'name': 'test-file',
'mimeType': 'application/vnd.google-apps.spreadsheet',
'parents': [folder_id],
'properties': {'locale': 'en_GB',
'timeZone': 'Europe/Berlin'}
})
Following the documentation #here, I tried to set the properties key with the locale set to the wanted one but it keeps creating the file with the default locale of my account.
How can I make it work at the creation time? is there another parameter I can fill?

Your problem is that you are mixing up two different "properties".
The properties you are setting are user-defined properties which are only ever consumed by you yourself. They are of no significance to Google.
The properties you want to set are part of the Spreadsheet API. See https://developers.google.com/sheets/api/reference/rest/v4/spreadsheets#SpreadsheetProperties
The simplest solution is to not use the Drive API to create your spreadsheet. Instead use the Spreadsheet API as descibed https://developers.google.com/sheets/api/reference/rest/v4/spreadsheets/create

I just tested this in the Apis Explorer
Create file Request
POST https://www.googleapis.com/drive/v3/files?key={YOUR_API_KEY}
{
"properties": {
"test": "test"
},
"name": "Hello"
}
Response
{
"kind": "drive#file",
"id": "1CYFI5rootSO5cndBD2gFb1n8SVvJ7_jo",
"name": "Hello",
"mimeType": "application/octet-stream"
}
File get request
GET https://www.googleapis.com/drive/v3/files/1CYFI5rootSO5cndBD2gFb1n8SVvJ7_jo?fields=*&key={YOUR_API_KEY}
Response
"kind": "drive#file",
"id": "1CYFI5rootSO5cndBD2gFb1n8SVvJ7_jo",
"name": "Hello",
"mimeType": "application/octet-stream",
"starred": false,
"trashed": false,
"explicitlyTrashed": false,
"parents": [
"0AJpJkOVaKccEUk9PVA"
],
"properties": {
"test": "test"
},
It appears to be working just fine i suggest you try checking the following:
The file id that is returned in the response from creating the file. To ensure that you are checking the one you just uploaded. Every time you run that you are going t create a new file.
Also remember to add fields=* with file.get if that's what you are using to check the result of your properties.