Develop Reference

Authentification to kubernetes api via Azure Active Directory (AKS)

I would like to use python kubernetes-client to connect to my AKS cluster api.
To do that I try to use the example give by kubernetes:
config.load_kube_config()
v1 = client.CoreV1Api()
print("Listing pods with their IPs:")
ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))
It is supposed to load my local kubeconfig and get a pods list but I get the following error:
Traceback (most recent call last): File "test.py", line 4, in
config.load_kube_config() File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 661, in load_kube_config
loader.load_and_set(config) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 469, in load_and_set
self._load_authentication() File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 203, in _load_authentication
if self._load_auth_provider_token(): File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 221, in _load_auth_provider_token
return self._load_azure_token(provider) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 233, in _load_azure_token
self._refresh_azure_token(provider['config']) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 253, in _refresh_azure_token
refresh_token, client_id, '00000002-0000-0000-c000-000000000000') File
"/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/authentication_context.py",
line 236, in acquire_token_with_refresh_token
return self._acquire_token(token_func) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/authentication_context.py",
line 128, in _acquire_token
return token_func(self) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/authentication_context.py",
line 234, in token_func
return token_request.get_token_with_refresh_token(refresh_token, client_secret) File
"/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/token_request.py",
line 343, in get_token_with_refresh_token
return self._get_token_with_refresh_token(refresh_token, None, client_secret) File
"/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/token_request.py",
line 340, in _get_token_with_refresh_token
return self._oauth_get_token(oauth_parameters) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/token_request.py",
line 112, in _oauth_get_token
return client.get_token(oauth_parameters) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/oauth2_client.py",
line 291, in get_token
raise AdalError(return_error_string, error_response) adal.adal_error.AdalError: Get Token request returned http error: 400
and server response:
{"error":"invalid_grant","error_description":"AADSTS65001: The user or
administrator has not consented to use the application with ID
'' named 'Kubernetes AD Client
'. Send an interactive authorization request for this user and
resource.\r\nTrace ID:
\r\nCorrelation ID:
\r\nTimestamp: 2019-10-14
12:32:35Z","error_codes":[65001],"timestamp":"2019-10-14
12:32:35Z","trace_id":"","correlation_id":"","suberror":"consent_required"}
I really don't understand why it doesn't work.
When I use kubectl, all work fine.
I read some docs but I'm not sure to understand the adal error.
Thanks for your help

Login as a tenant admin to https://portal.azure.com
Open the registration for your app in the
Go to Settings then Required Permissions
Press the Grant Permissions button
If you are not a tenant admin, you cannot give admin consent
From https://github.com/Azure-Samples/active-directory-angularjs-singlepageapp-dotnet-webapi/issues/19

This is good post where you can find snippet to authenticate to AKS:
from azure.identity import AzureCliCredential
from azure.mgmt.resource import ResourceManagementClient
from azure.mgmt.containerservice import ContainerServiceClient
from azure.mgmt.containerservice.models import (ManagedClusterAgentPoolProfile,
ManagedCluster)
credential = AzureCliCredential()
subscription_id = "XXXXX"
resource_group= 'MY-RG'
resouce_client=ResourceManagementClient(credential,subscription_id)
container_client=ContainerServiceClient(credential,subscription_id)
resouce_list=resouce_client.resources.list_by_resource_group(resource_group)
Note: You need to install respective Az Python SKD libraries.

Speech-to-text: google.api_core.exceptions.PermissionDenied: 403

I am trying to use Google speech-to-text service, according to https://googleapis.github.io/google-cloud-python/latest/speech/index.html
I have created project, uploaded audio to gs: cloud, added permissions, downloaded json file named My First Project-7bb85a480131.json. https://console.cloud.google.com/storage/browser/mybucket?project=my-project
that is my file:
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="/home/joo/Документы/LocalRepository/robotze/My First Project-7bb85a480131.json"
from google.cloud import speech
client = speech.SpeechClient()
audio = speech.types.RecognitionAudio(
uri='gs://zaudio/audio.mp3')
config = speech.types.RecognitionConfig(
encoding=speech.enums.RecognitionConfig.AudioEncoding.LINEAR16,
language_code='ru-RU',
sample_rate_hertz=44100)
operation = client.long_running_recognize(config=config, audio=audio)
op_result = operation.result()
for result in op_result.results:
for alternative in result.alternatives:
print('=' * 20)
print(alternative.transcript)
print(alternative.confidence)
Issue: i got
google.api_core.exceptions.PermissionDenied: 403 my-service-account#my-project.iam.gserviceaccount.com does not have storage.objects.get access to mybucket/audio.mp3.
Full traceback
/home/joo/anaconda3/bin/python /home/joo/Документы/LocalRepository/robotze/speech-to-text-googlecloud.py
Traceback (most recent call last):
File "/home/joo/anaconda3/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/home/joo/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 565, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/joo/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "my-service-account#my-project.iam.gserviceaccount.com does not have storage.objects.get access to mybucket/audio.mp3."
debug_error_string = "{"created":"#1565253582.126380437","description":"Error received from peer ipv4:74.125.131.95:443","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"my-service-account#my-project.iam.gserviceaccount.com does not have storage.objects.get access to mybucket/audio.mp3.","grpc_status":7}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/joo/Документы/LocalRepository/robotze/speech-to-text-googlecloud.py", line 46, in <module>
operation = client.long_running_recognize(config=config, audio=audio)
File "/home/joo/anaconda3/lib/python3.6/site-packages/google/cloud/speech_v1/gapic/speech_client.py", line 341, in long_running_recognize
request, retry=retry, timeout=timeout, metadata=metadata
File "/home/joo/anaconda3/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
return wrapped_func(*args, **kwargs)
File "/home/joo/anaconda3/lib/python3.6/site-packages/google/api_core/retry.py", line 273, in retry_wrapped_func
on_error=on_error,
File "/home/joo/anaconda3/lib/python3.6/site-packages/google/api_core/retry.py", line 182, in retry_target
return target()
File "/home/joo/anaconda3/lib/python3.6/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/home/joo/anaconda3/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.PermissionDenied: 403 my-service-account#my-project.iam.gserviceaccount.com does not have storage.objects.get access to mybucket/audio.mp3.
Process finished with exit code 1
What i tried: gcloud auth application-default login - login in browser works, but still 403 error

From what i can see on your logs, you are able to authenticate your service account inside your code (you are currently authenticating with: starting-account-*******-239919.iam.gserviceaccount.com), however, that service account doesn't have "storage.objects.get" permission over the object "zaudio/audio.mp3".
So you can either:
A.- Give the proper permissions to that service account (may be the role "storage.objectViewer" inside that bucket would be enough, but you could also set it with the role "storage.admin" so it can have more control over that bucket and others).
B.- Authenticate using other service account that have the proper permissions.

I resolved the following issue:
“google.api_core.exceptions.PermissionDenied: 403 my-service-account#my-project.iam.gserviceaccount.com does not have storage.objects.get access to mybucket/audio.mp3.”
To resolve this issue: go to your bucket, click that three dots, choose “Edit permissions”, Entity set “User”, Name set your email (in this case, my-service-account#my-project.iam.gserviceaccount.com), Access: “Reader”. Save and try again. This should resolve this issue. Regardless whether you have created the bucket, whatnot, you have to do this step to explicitly set permission. Hopefully this is useful.

Python : connection failed after authentication success using PyDrive

I am using python version 3 with PyDrive to connect Google spreadsheet.
I succeeded to open browser and do authentication. However, after that I got an error and program does not continue.
this is an error message
Authentication successful.
Traceback (most recent call last):
File "/home/PycharmProjects/misc/.venv/lib/python3.5/site-packages/pydrive/files.py", line 369, in _FilesInsert
http=self.http)
File "/home/PycharmProjects/misc/.venv/lib/python3.5/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/PycharmProjects/misc/.venv/lib/python3.5/site-packages/googleapiclient/http.py", line 812, in execute
_, body = self.next_chunk(http=http, num_retries=num_retries)
File "/home/PycharmProjects/misc/.venv/lib/python3.5/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/PycharmProjects/misc/.venv/lib/python3.5/site-packages/googleapiclient/http.py", line 916, in next_chunk
raise ResumableUploadError(resp, content)
googleapiclient.errors.ResumableUploadError: <HttpError 403 "Project 895099846013 is not found and cannot be used for API calls. If it is recently created, enable Drive API by visiting then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.">
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "drive_control.py", line 10, in <module>
file.Upload()
File "/home/PycharmProjects/misc/.venv/lib/python3.5/site-packages/pydrive/files.py", line 285, in Upload
self._FilesInsert(param=param)
File "/home/PycharmProjects/misc/.venv/lib/python3.5/site-packages/pydrive/auth.py", line 75, in _decorated
return decoratee(self, *args, **kwargs)
File "/home/PycharmProjects/misc/.venv/lib/python3.5/site-packages/pydrive/files.py", line 371, in _FilesInsert
raise ApiRequestError(error)
pydrive.files.ApiRequestError: <HttpError 403 "Project 895099846013 is not found and cannot be used for API calls. If it is recently created, enable Drive API by visiting url If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.">
this is my code
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
gauth = GoogleAuth()
gauth.LocalWebserverAuth()
drive = GoogleDrive(gauth)
file = drive.CreateFile({'title': 'Hello.txt'})
file.SetContentString('Hello World!')
file.Upload()

this error have clear description in traceback:
403 "Project 895099846013 is not found and cannot be used for API
calls. If it is recently created, enable Drive API by visiting url
In your account, you must have the enabled status of the Google Drive API.
You can see it in the Console Developer toolbar
If you do not see it, follow the instructions

Google Cloud Shell HttpAccessTokenRefreshError: This can occur if a VM was created with no service account or scopes

I am trying to run a simple script from
https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/data_analysis/lab2/python/grepc.py
(this code is a Dataflow pipeline connecting to Google Storage)
It worked last week. But when I am running it now, I always get the same error:
> Traceback (most recent call last):
File "grepc.py", line 50, in <module>
run()
File "grepc.py", line 44, in run
| 'write' >> beam.io.WriteToText(output_prefix)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/textio.py", line 391, in __init__
skip_header_lines=skip_header_lines)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/textio.py", line 89, in __init__
validate=validate)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filebasedsource.py", line 105, in __init__
self._validate()
File "/usr/local/lib/python2.7/dist-packages/apache_beam/options/value_provider.py", line 109, in _f
return fnc(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filebasedsource.py", line 165, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filesystems.py", line 131, in match
return filesystem.match(patterns, limits)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/gcsfilesystem.py", line 138, in match
raise BeamIOError("Match operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions {'gs://{MY_BUCKET}/javahelp/*.java': HttpAccessTokenRefreshError(u' This can occur if a VM was created with no service account or scopes.',)}
I have no idea how to solve this. And a lot of Googling did not help neither.

Acquiring new user credentials to use for Application Default Credentials fixed my problem.
This is what I used
gcloud auth application-default login
It is well documented here https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login
and you can find the explanation here : This command is useful when you are developing code that would normally use a service account but need to run the code in a local development environment where it's easier to provide user credentials. The credentials will apply to all API calls that make use of the Application Default Credentials client library
Another solution I found was : download keyfile for the compute engine service account and export GOOGLE_APPLICATION_CREDENTIALS to point towards the keyfile

I'm assuming the {MYBUCKET} in the error message is not literal and was replaced by your bucket name.
If you're running this from a GCE VM instance, can you run this command and paste the output here?
gcloud compute instances describe {instance-name} --zone {instance-zone}
The above would tell you what service accounts and scopes your VM instance has. And also:
gcloud projects get-iam-policy {project-name}
This would tell you what service accounts your project has. Please wipe out the project number or any info that you deem sensitive.

Youtube Api: Python Search is not working throws Application Default Credentials are not available

Trying to search youtube videos using this code: https://github.com/youtube/api-samples/blob/master/python/search.py
But whatever I try I get below error even though I provided the api key:
Traceback (most recent call last):
File "search.py", line 56, in <module>
youtube_search(args)
File "search.py", line 18, in youtube_search
developerKey=DEVELOPER_KEY)
File "C:\Python27\Scripts\tvapp_env\lib\site-packages\oauth2client\_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Python27\Scripts\tvapp_env\lib\site-packages\googleapiclient\discovery.py", line 226, in build
credentials=credentials)
File "C:\Python27\Scripts\tvapp_env\lib\site-packages\oauth2client\_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Python27\Scripts\tvapp_env\lib\site-packages\googleapiclient\discovery.py", line 358, in build_from_document
credentials = _auth.default_credentials()
File "C:\Python27\Scripts\tvapp_env\lib\site-packages\googleapiclient\_auth.py", line 40, in default_credentials
return oauth2client.client.GoogleCredentials.get_application_default()
File "C:\Python27\Scripts\tvapp_env\lib\site-packages\oauth2client\client.py", line 1264, in get_application_default
return GoogleCredentials._get_implicit_credentials()
File "C:\Python27\Scripts\tvapp_env\lib\site-packages\oauth2client\client.py", line 1254, in _get_implicit_credentials
raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
Please help, im running this from windows machine.
Or is there better code to make youtube search.
Thanks

Seems like you are not running the app in either Google app engine or in Google compute engine. So there are not going to be any default credentials to use so you have to download the credentials from the google developer console first and use those credentials in your application. Please refer this document for further details.