I am trying to use MSI example provided in below link :
https://learn.microsoft.com/en-us/python/azure/python-sdk-azure-authenticate?view=azure-python#mgmt-auth-msi
To do that, I created a linux VM , installed MSI extension on it and running above code in a python application and when I run that python application I get the following error:
[azureuser#vish-redhat ~]$ python msi-auth.py
No handlers could be found for logger "msrestazure.azure_active_directory"
Traceback (most recent call last):
File "msi-auth.py", line 10, in <module>
subscription = next(subscription_client.subscriptions.list())
File "/usr/lib/python2.7/site-packages/msrest/paging.py", line 121, in __next__
self.advance_page()
File "/usr/lib/python2.7/site-packages/msrest/paging.py", line 107, in advance_page
self._response = self._get_next(self.next_link)
File "/usr/lib/python2.7/site-packages/azure/mgmt/resource/subscriptions/v2016_06_01/operations/subscriptions_operations.py", line 207, in internal_paging
request, header_parameters, **operation_config)
File "/usr/lib/python2.7/site-packages/msrest/service_client.py", line 191, in send
session = self.creds.signed_session()
File "/usr/lib/python2.7/site-packages/msrestazure/azure_active_directory.py", line 685, in signed_session
self.set_token()
File "/usr/lib/python2.7/site-packages/msrestazure/azure_active_directory.py", line 681, in set_token
self.scheme, _, self.token = get_msi_token(self.resource, self.port, self.msi_conf)
File "/usr/lib/python2.7/site-packages/msrestazure/azure_active_directory.py", line 590, in get_msi_token
result = requests.post(request_uri, data=payload, headers={'Metadata': 'true'})
File "/usr/lib/python2.7/site-packages/requests/api.py", line 108, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(111, 'Connection refused'))
[azureuser#vish-redhat ~]$
Code:
from msrestazure.azure_active_directory import MSIAuthentication
from azure.mgmt.resource import ResourceManagementClient, SubscriptionClient
# Create MSI Authentication
credentials = MSIAuthentication()
# Create a Subscription Client
subscription_client = SubscriptionClient(credentials)
subscription = next(subscription_client.subscriptions.list())
subscription_id = subscription.subscription_id
# Create a Resource Management client
resource_client = ResourceManagementClient(credentials, subscription_id)
# List resource groups as an example. The only limit is what role and policy are assigned to this MSI token.
for resource_group in resource_client.resource_groups.list():
print(resource_group.name)
You need install Python SDK in your Linux VM. Please refer to this official document.
pip install azure
Also, you need give Owner role for your VM on subscription level.
More information about this please refer to this link.
Now, you could use this code to test on VM. I test in my lab, it works for me.
Note: You need modify resource_client = ResourceManagementClient(credentials, subscription_id) to resource_client = ResourceManagementClient(credentials, str(subscription_id)), it requires a string type.
A connection error is usually because the extension is not yet available. You can try if the extension is available using the CLI with az login --msi
https://learn.microsoft.com/en-us/azure/active-directory/managed-service-identity/how-to-use-vm-sign-in
If it works, your VM is created correctly with MSI support. It it doesn't, probably your extension is not configured correctly.
Note that we changed the way to get a token with MSI from inside a VM. We now use IMDS:
https://learn.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service
Starting with the next release of the CLI (the first one of April 2018), CLI will authenticate with IMDS directly and not use the VM extension anymore. This is already shipped in the underlying library msrestazure in its 0.4.25 version. This one will bypass completely your VM extension to use IMDS and is the prefered scenario now. Could you try with this version of msrestazure? If it works with 0.4.25 but not in 0.4.24, this likely means your VM extension is not installed correctly, but you don't care since it's a deprecated scenario :)
Note that in order to get a token, your VM doesn't need any special permissions or ownership of subscription. However, for this token to be useful you need it :). But since your error is related to the "get a token" part and not permission, I would just kindly suggest that you might need this complementary info for later if you have permissions issues:
https://learn.microsoft.com/en-us/azure/active-directory/managed-service-identity/howto-assign-access-cli
(full disclosure, I work at MS in the SDK/CLI team and wrote the MSI support)
Related
Environment details
OS type and version:
Python version: 3.9.0
pip version: 22.0.4
google-api-python-client version: 2.48.0
Description
Hi, I'm running into an error when trying to fetch the Google Play Console reports of our mobile apps (such as installations, errors etc.). I first tried with this manual but it seems to be outdated and didn't work. So after some research I changed it similar to this one, that it fits to the current google api (see code snippet below).
Steps I have done:
Created a project on "console.cloud.google.com"
Created the service account
Created the json key file
Invited the service account on play.google.com/console and gave him full admin rights (normally "see app information and download bulk reports" should be enough)
Added the role "Storage Object Viewer" to the Service account in https://console.cloud.google.com/iam-admin/iam?authuser=1&project=myproject
waited for 24h to make sure there are no errors because of syncs or so.
(I anonymized some of the values below).
Code example
from googleapiclient.discovery import build
from google.oauth2 import service_account
scopes = ['https://www.googleapis.com/auth/devstorage.read_only','https://www.googleapis.com/auth/cloud-platform.read_only']
key_file_location = 'files/access_token/mykeyfile.json'
cloud_storage_bucket = r'pubsite_prod_rev_00123456789'
report_to_download = 'installs/installs_com.my.app_202201_country.csv'
creds = service_account.Credentials.from_service_account_file(key_file_location,scopes=scopes)
service = build('storage','v1', credentials=creds)
print(service.objects().get(bucket = cloud_storage_bucket, object= report_to_download).execute())
Stack trace
Traceback (most recent call last):
File "C:\Users\myuser\project\z_10_ext_google_play_store.py", line 46, in <module>
print(service.objects().get(bucket = cloud_storage_bucket, object= report_to_download).execute())
File "D:\Programs\Python\lib\site-packages\googleapiclient\_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "D:\Programs\Python\lib\site-packages\googleapiclient\http.py", line 923, in execute
resp, content = _retry_request(
File "D:\Programs\Python\lib\site-packages\googleapiclient\http.py", line 191, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "D:\Programs\Python\lib\site-packages\google_auth_httplib2.py", line 209, in request
self.credentials.before_request(self._request, method, uri, request_headers)
File "D:\Programs\Python\lib\site-packages\google\auth\credentials.py", line 133, in before_request
self.refresh(request)
File "D:\Programs\Python\lib\site-packages\google\oauth2\service_account.py", line 410, in refresh
access_token, expiry, _ = _client.jwt_grant(
File "D:\Programs\Python\lib\site-packages\google\oauth2\_client.py", line 199, in jwt_grant
six.raise_from(new_exc, caught_exc)
File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: ('No access token in response.', {'id_token': 'eyJ...'})
I hope that I provided enough information and I'm sorry in advance if I made a stupid mistake.
I started working with the Prefect Orchestration tool.
My goal is to set up a server managing my automation on different other PCs and servers.
I do not fully understand the architecture of Prefect yet (with all these Agents etc.) but I managed to start a server on a remote Ubuntu environment.
To access the UI remotely I created a config.toml and added following lines:
[server]
endpoint = "<IPofserver>:4200/graphql"
[server.ui]
apollo_url = "http://<IPofserver>:4200/graphql"
[telemetry]
[server.telemetry]
enabled = false
The telemetry part is just to disable sending analysis data to Prefect.
Afterswards it was possible to accesss the UI from another PC and also to start an Agent on another PC with:
prefect agent local start --api "http://<IPofserver>:4200/graphql"
But how can I deploy flows now? A do not find an option to set their api like for the agent.
Even if I try to register a flow on the machine where the server itself is runnig I get following error message:
Traceback (most recent call last): File "", line 1, in
File
"/usr/local/lib/python3.10/dist-packages/prefect/core/flow.py", line
1726, in register
registered_flow = client.register( File "/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 831, in register
project = self.graphql(query_project).data.project # type: ignore File
"/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 443, in graphql
result = self.post( File "/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 398, in post
response = self._request( File "/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 633, in _request
response = self._send_request( File "/usr/local/lib/python3.10/dist-packages/prefect/client/client.py",
line 497, in _send_request
response = session.post( File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line
635, in post
return self.request("POST", url, data=data, json=json, **kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py",
line 587, in request
resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line
695, in send
adapter = self.get_adapter(url=request.url) File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line
792, in get_adapter
raise InvalidSchema(f"No connection adapters were found for {url!r}") requests.exceptions.InvalidSchema: No connection adapters
were found for ':4200/graphql'
Used Example Code:
import prefect
from prefect import task, Flow
#task
def say_hello():
logger = prefect.context.get("logger")
logger.info("Hello, Cloud!")
with Flow("hello-flow") as flow:
say_hello()
# Register the flow under the "tutorial" project
flow.register(project_name="Test")
If you are getting started with Prefect, I'd recommend using Prefect 2.0 - check this documentation page on getting started and this one about the underlying architecture.
If you still need help with Prefect Server and Prefect 1.0, check this extensive troubleshooting guide and if that doesn't help, send us a message on Slack, and we'll try to help you there.
I am trying to get client of Azure DevOps by using python script.
Here is the code that I am using for this purpose:
from azure.devops.connection import Connection
git_client = p_context.connection.clients.get_git_client()
p_context is access token of my Azure DevOps organization.
Following is the error that I am facing:
Traceback (most recent call last):
File "/home/vsts/work/1/s/IdentifyGitBuildCommitItems.py", line 95, in <module>
t = get_commit_changes_for_build(context ,azdevops_projectid ,azdevops_build_id ,azdevops_build_number ,bld_commit.id ,azdevops_repository_name)
File "/home/vsts/work/1/s/IdentifyGitBuildCommitItems.py", line 51, in get_commit_changes_for_build
comt = git_client.get_changes(p_commit_id ,repository_id=p_repository_name ,project=p_project_id)
File "/opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages/azure/devops/released/git/git_client_base.py", line 250, in get_changes
query_parameters=query_parameters)
File "/opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages/azure/devops/client.py", line 104, in _send
response = self._send_request(request=request, headers=headers, content=content, media_type=media_type)
File "/opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages/azure/devops/client.py", line 68, in _send_request
self._handle_error(request, response)
File "/opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages/azure/devops/client.py", line 295, in _handle_error
status_code=response.status_code))
azure.devops.exceptions.AzureDevOpsClientRequestError: Page not found. Operation returned a 404 status code.
##[error]/opt/hostedtoolcache/Python/3.6.10/x64/bin/python failed with return code: 1
This is the full GitHub example that I am following for my use case: https://github.com/venkatra/dbt_hacks/blob/PERSISTENT_TABLE_MATERIALIZATION/IdentifyGitBuildCommitItems.py
You could check the Azure DevOps Python API and try the following code:
from azure.devops.connection import Connection
from msrest.authentication import BasicAuthentication
import pprint
# Fill in with your personal access token and org URL
personal_access_token = 'YOURPAT'
organization_url = 'https://dev.azure.com/YOURORG'
# Create a connection to the org
credentials = BasicAuthentication('', personal_access_token)
connection = Connection(base_url=organization_url, creds=credentials)
# Get a client
git_client = connection.clients.get_git_client()
I have figured out the issue and resolve the error.
Error was prompting due to my source code was present in my personal GitHub account not in the Azure Git repository. When I moved source code from personal Git to Azure Git repository error got resolved.
I am following this document: https://cloud.google.com/text-to-speech/docs/quickstart-client-libraries#client-libraries-install-python
I have followed the instructions. I get an error when I run the provided Python code ("Create audio data" section).
First Try:
It appears the problem is with authentication. I authenticated with gcloud init.
C:\webDev\pycharm\TTS-Nov-\Scripts\python.exe C:/webDev/pycharm/TTS-Nov-/TTS-Main.py
C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\auth\_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
Traceback (most recent call last):
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\grpc_helpers.py", line 59, in error_remapped_callable
return callable_(*args, **kwargs)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\grpc\_channel.py", line 533, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\grpc\_channel.py", line 467, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the texttospeech.googleapis.com. We recommend that most server applications use service accounts instead. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/."
debug_error_string = "{"created":"#1541901610.961000000","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the texttospeech.googleapis.com. We recommend that most server applications use service accounts instead. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.","grpc_status":7}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/webDev/pycharm/TTS-Nov-/TTS-Main.py", line 26, in <module>
response = client.synthesize_speech(synthesis_input, voice, audio_config)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\cloud\texttospeech_v1\gapic\text_to_speech_client.py", line 257, in synthesize_speech
request, retry=retry, timeout=timeout, metadata=metadata)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\gapic_v1\method.py", line 139, in __call__
return wrapped_func(*args, **kwargs)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\retry.py", line 260, in retry_wrapped_func
on_error=on_error,
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\retry.py", line 177, in retry_target
return target()
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\timeout.py", line 206, in func_with_timeout
return func(*args, **kwargs)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\grpc_helpers.py", line 61, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.PermissionDenied: 403 Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the texttospeech.googleapis.com. We recommend that most server applications use service accounts instead. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.
Second Try Update:
I ran this
gcloud auth activate-service-account test-2#tts-alphax-nov.iam.gserviceaccount.com --key-file="C:\webDev\keys\google-cloud\tts-alphax-nov-b101c56c70b7.json" --project=tts-alphax-nov
This resulted in:
Activated service account credentials for: [test-2#tts-alphax-nov.iam.gserviceaccount.com]
I then ran the python again and this was the error I got.
C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\auth\_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
Traceback (most recent call last):
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\grpc_helpers.py", line 59, in error_remapped_callable
return callable_(*args, **kwargs)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\grpc\_channel.py", line 533, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\grpc\_channel.py", line 467, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the texttospeech.googleapis.com. We recommend that most server applications use service accounts instead. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/."
debug_error_string = "{"created":"#1541903472.332000000","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the texttospeech.googleapis.com. We recommend that most server applications use service accounts instead. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.","grpc_status":7}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/webDev/pycharm/TTS-Nov-/TTS-Main.py", line 26, in <module>
response = client.synthesize_speech(synthesis_input, voice, audio_config)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\cloud\texttospeech_v1\gapic\text_to_speech_client.py", line 257, in synthesize_speech
request, retry=retry, timeout=timeout, metadata=metadata)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\gapic_v1\method.py", line 139, in __call__
return wrapped_func(*args, **kwargs)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\retry.py", line 260, in retry_wrapped_func
on_error=on_error,
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\retry.py", line 177, in retry_target
return target()
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\timeout.py", line 206, in func_with_timeout
return func(*args, **kwargs)
File "C:\webDev\pycharm\TTS-Nov-\lib\site-packages\google\api_core\grpc_helpers.py", line 61, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.PermissionDenied: 403 Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the texttospeech.googleapis.com. We recommend that most server applications use service accounts instead. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/.
This error message is usually thrown when the application is not being authenticated correctly due to several reasons such as missing files, invalid credential paths, incorrect environment variables assignations, among other causes. Keep in mind that when you set an environment variable value in a session, it is reset every time the session is dropped.
Based on this, I suggest you to validate that the credential file and file path are being correctly assigned (avoiding to include blank characters when setting the GOOGLE_APPLICATION_CREDENTIALS env variable), as well as try to explicitly specify your service account file directly into your code by adding the following credentials code to your solution:
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file('<serviceAccountPath>.json')
texttospeech.TextToSpeechClient(credentials=credentials)
I am trying to run a simple script from
https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/data_analysis/lab2/python/grepc.py
(this code is a Dataflow pipeline connecting to Google Storage)
It worked last week. But when I am running it now, I always get the same error:
> Traceback (most recent call last):
File "grepc.py", line 50, in <module>
run()
File "grepc.py", line 44, in run
| 'write' >> beam.io.WriteToText(output_prefix)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/textio.py", line 391, in __init__
skip_header_lines=skip_header_lines)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/textio.py", line 89, in __init__
validate=validate)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filebasedsource.py", line 105, in __init__
self._validate()
File "/usr/local/lib/python2.7/dist-packages/apache_beam/options/value_provider.py", line 109, in _f
return fnc(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filebasedsource.py", line 165, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/filesystems.py", line 131, in match
return filesystem.match(patterns, limits)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/gcsfilesystem.py", line 138, in match
raise BeamIOError("Match operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions {'gs://{MY_BUCKET}/javahelp/*.java': HttpAccessTokenRefreshError(u' This can occur if a VM was created with no service account or scopes.',)}
I have no idea how to solve this. And a lot of Googling did not help neither.
Acquiring new user credentials to use for Application Default Credentials fixed my problem.
This is what I used
gcloud auth application-default login
It is well documented here https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login
and you can find the explanation here : This command is useful when you are developing code that would normally use a service account but need to run the code in a local development environment where it's easier to provide user credentials. The credentials will apply to all API calls that make use of the Application Default Credentials client library
Another solution I found was : download keyfile for the compute engine service account and export GOOGLE_APPLICATION_CREDENTIALS to point towards the keyfile
I'm assuming the {MYBUCKET} in the error message is not literal and was replaced by your bucket name.
If you're running this from a GCE VM instance, can you run this command and paste the output here?
gcloud compute instances describe {instance-name} --zone {instance-zone}
The above would tell you what service accounts and scopes your VM instance has. And also:
gcloud projects get-iam-policy {project-name}
This would tell you what service accounts your project has. Please wipe out the project number or any info that you deem sensitive.