This error i am getting:
ERROR:boto:Unable to read instance data, giving up
Traceback (most recent call last):
File "<ipython-input-62-476f799f9e0f>", line 2, in <module>
conn = boto.connect_s3()
File "/usr/local/lib/python2.7/dist-packages/boto/__init__.py", line 141, in connect_s3
return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 191, in __init__
validate_certs=validate_certs, profile_name=profile_name)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 569, in __init__
host, config, self.provider, self._required_auth_capability())
File "/usr/local/lib/python2.7/dist-packages/boto/auth.py", line 993, in get_auth_handler
'Check your credentials' % (len(names), str(names)))
NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials
This error Message is coming while establishing connection with aws S3Connection.
I want to establish connection with AWS S3 and read CSV files.
please Help me out?
i am using Python 2.7.12
And Now i am using this below code:
import boto
import time
from boto.s3.connection import S3Connection
conn = S3Connection('<aws access key>','<aws secret key>')
print conn
from boto.s3.connection import Location
print '\n'.join(i for i in dir(Location) if i[0].isupper())
conn.create_bucket('egp-shared-prod/egp-prod-c2c1/',
location=Location.DEFAULT)
And, Its show Error:
File "<ipython-input-69-4b49d719d4ca>", line 15, in <module>
conn.create_bucket('egp-shared-prod/egp-prod-c2c1/', location=Location.DEFAULT)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 616, in create_bucket
data=data)
File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 668, in make_request
retry_handler=retry_handler
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1030, in _mexe
raise ex
gaierror: [Errno -2] Name or service not known
I tried your code and my testing has found that the error is related to your bucket name of egp-shared-prod/egp-prod-c2c1/.
The Bucket Restrictions and Limitations documentation says:
Bucket names can contain lowercase letters, numbers, and hyphens.
Slashes are not permitted. Also, they seem to be upsetting the boto code.
Boto (the official AWS Python bindings) that you are using, expect you to save your AWS_ACCESS_KEY_id and AWS_SECRET_ACCESS_KEY in environment variables like so:
export AWS_ACCESS_KEY_ID='AK123'
export AWS_SECRET_ACCESS_KEY='abc123'
you can pass aws cridential access like:
#Connection with s3 :
s3= boto3.resource(
service_name='s3',
region_name='us-east-1',
aws_secret_access_key='',
aws_access_key_id=''
)
Related
I am using a private key authentication to connect to Snowflake using python,
**This is working successfully when connecting directly using Java Client
import snowflake.connector
import os
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives.asymmetric import dsa
from cryptography.hazmat.primitives import serialization
with open("rsa_key.p8", "rb") as key:
p_key= serialization.load_pem_private_key(
key.read(),
password='XXXXX'.encode(),
backend=default_backend()
)
pkb = p_key.private_bytes(
encoding=serialization.Encoding.DER,
format=serialization.PrivateFormat.PKCS8,
encryption_algorithm=serialization.NoEncryption())
conn = snowflake.connector.connect(
user=XXXXX,
password=XXXXXXX,
account=XXXXXXXXX,
private_key=pkb,
warehouse=XXX,
database=XXXXXX,
schema=XXXX
)
Have masked real values where needed, but these are correct as same as work direct with Java client.
Error:
/usr/lib/python3/dist-packages/jwt/algorithms.py:179: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
self.hash_alg()
Traceback (most recent call last):
File "tryconnection.py", line 37, in <module>
schema='PUBLIC'
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/__init__.py", line 53, in Connect
return SnowflakeConnection(**kwargs)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/connection.py", line 189, in __init__
self.connect(**kwargs)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/connection.py", line 493, in connect
self.__open_connection()
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/connection.py", line 710, in __open_connection
self.__authenticate(auth_instance)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/connection.py", line 963, in __authenticate
session_parameters=self._session_parameters,
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/auth.py", line 217, in authenticate
socket_timeout=self._rest._connection.login_timeout)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 530, in _post_request
_include_retry_params=_include_retry_params)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 609, in fetch
**kwargs)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 711, in _request_exec_wrapper
raise e
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 653, in _request_exec_wrapper
method, full_url, headers, data, conn)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/network.py", line 758, in _handle_unknown_error
u'errno': ER_FAILED_TO_REQUEST,
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/errors.py", line 100, in errorhandler_wrapper
connection.errorhandler(connection, cursor, errorclass, errorvalue)
File "/usr/local/lib/python3.6/dist-packages/snowflake/connector/errors.py", line 73, in default_errorhandler
done_format_msg=errorvalue.get(u'done_format_msg'))
snowflake.connector.errors.OperationalError: 250003: None: Failed to get the response. Hanging? method: post, url:
Thank you in advance for your help.
I can not see the rest of the error message so I couldn't be sure. Are you sure enter your account name (and region and cloud if needed) as the account parameter, instead of full Snowflake URL?
https://docs.snowflake.com/en/user-guide/python-connector-example.html#connecting-to-snowflake
When connecting Jira, usually people uses JDBC connection string which includes full snowflake URL:
https://docs.snowflake.com/en/user-guide/python-connector-example.html#connecting-to-snowflake
I would like to use python kubernetes-client to connect to my AKS cluster api.
To do that I try to use the example give by kubernetes:
config.load_kube_config()
v1 = client.CoreV1Api()
print("Listing pods with their IPs:")
ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))
It is supposed to load my local kubeconfig and get a pods list but I get the following error:
Traceback (most recent call last): File "test.py", line 4, in
config.load_kube_config() File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 661, in load_kube_config
loader.load_and_set(config) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 469, in load_and_set
self._load_authentication() File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 203, in _load_authentication
if self._load_auth_provider_token(): File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 221, in _load_auth_provider_token
return self._load_azure_token(provider) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 233, in _load_azure_token
self._refresh_azure_token(provider['config']) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/kubernetes/config/kube_config.py",
line 253, in _refresh_azure_token
refresh_token, client_id, '00000002-0000-0000-c000-000000000000') File
"/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/authentication_context.py",
line 236, in acquire_token_with_refresh_token
return self._acquire_token(token_func) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/authentication_context.py",
line 128, in _acquire_token
return token_func(self) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/authentication_context.py",
line 234, in token_func
return token_request.get_token_with_refresh_token(refresh_token, client_secret) File
"/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/token_request.py",
line 343, in get_token_with_refresh_token
return self._get_token_with_refresh_token(refresh_token, None, client_secret) File
"/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/token_request.py",
line 340, in _get_token_with_refresh_token
return self._oauth_get_token(oauth_parameters) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/token_request.py",
line 112, in _oauth_get_token
return client.get_token(oauth_parameters) File "/Users//works/test-kube-api-python/env/lib/python2.7/site-packages/adal/oauth2_client.py",
line 291, in get_token
raise AdalError(return_error_string, error_response) adal.adal_error.AdalError: Get Token request returned http error: 400
and server response:
{"error":"invalid_grant","error_description":"AADSTS65001: The user or
administrator has not consented to use the application with ID
'' named 'Kubernetes AD Client
'. Send an interactive authorization request for this user and
resource.\r\nTrace ID:
\r\nCorrelation ID:
\r\nTimestamp: 2019-10-14
12:32:35Z","error_codes":[65001],"timestamp":"2019-10-14
12:32:35Z","trace_id":"","correlation_id":"","suberror":"consent_required"}
I really don't understand why it doesn't work.
When I use kubectl, all work fine.
I read some docs but I'm not sure to understand the adal error.
Thanks for your help
Login as a tenant admin to https://portal.azure.com
Open the registration for your app in the
Go to Settings then Required Permissions
Press the Grant Permissions button
If you are not a tenant admin, you cannot give admin consent
From https://github.com/Azure-Samples/active-directory-angularjs-singlepageapp-dotnet-webapi/issues/19
This is good post where you can find snippet to authenticate to AKS:
from azure.identity import AzureCliCredential
from azure.mgmt.resource import ResourceManagementClient
from azure.mgmt.containerservice import ContainerServiceClient
from azure.mgmt.containerservice.models import (ManagedClusterAgentPoolProfile,
ManagedCluster)
credential = AzureCliCredential()
subscription_id = "XXXXX"
resource_group= 'MY-RG'
resouce_client=ResourceManagementClient(credential,subscription_id)
container_client=ContainerServiceClient(credential,subscription_id)
resouce_list=resouce_client.resources.list_by_resource_group(resource_group)
Note: You need to install respective Az Python SKD libraries.
I want to create a bucket to upload some wav files in it. I am able to create a bucket manually with location required, but when I am trying to program in python to create a bucket with us-west-2 location
session = boto3.Session(aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
s3 = session.resource('s3')
s3.create_bucket(Bucket='test-asterisk1', CreateBucketConfiguration={'LocationConstraint': 'eu-central-1'})
I got the following error
Traceback (most recent call last):
File "create_bucket.py", line 10, in <module>
s3.create_bucket(Bucket='asterisk1', CreateBucketConfiguration={'LocationConstraint': 'ap-south-1'})
File "/home/dileep/.local/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/home/dileep/.local/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__ response = getattr(parent.meta.client, operation_name)(**params)
File "/home/dileep/.local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/dileep/.local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The ap-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.
I am on indian IP trying to create a 'us-west-2' end point is that creating a problem?
so I tried changing location constrain one by one time each
"LocationConstraint": "EU"|"eu-west-1"|"us-west-1"|"us-west-2"|"ap-south-1"|"ap-southeast-1"|"ap-southeast-2"|"ap-northeast-1"|"sa-east-1"|"cn-north-1"|"eu-central-1"
but whatever location I try it gives me the same error.
so I tried creating with boto instead of boto3
import boto
from boto.s3.connection import Location
s3 = boto.connect_s3(aws_access_key_id, aws_secret_access_key)
s3.create_bucket('test-asterisk1', location=Location.USWest2)
it throws error
File "s2t_amazon.py", line 27, in <module>
s3.create_bucket('test-asterisk2', location=Location.USWest2)
File "/home/dileep/.local/lib/python2.7/site-packages/boto/s3/connection.py", line 623, in create_bucket
response.status, response.reason, body)
boto.exception.S3CreateError: S3CreateError: 409 Conflict
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.
</Message><BucketName>test-asterisk2</BucketName>
<RequestId>EAF26BA152FD20A5</RequestId<HostId>ep0WFZEb1mIjEgbYIY4BGGuOTi5HSutYd3XTKgFjWmRMnGG0ajj5TLF4/t1amJQsOZdZQrqGnoE=</HostId></Error>
I have checked if the bucket is created, and it is not created by anyone of the methods. Can anyone suggest what could be the problem?
This works:
import boto3
s3_client = boto3.client('s3', region_name = 'eu-central-1')
s3_client.create_bucket(Bucket='my-bucket', CreateBucketConfiguration={'LocationConstraint': 'eu-central-1'})
The import thing to realise is that the command must be sent to the region where the bucket is being created. Thus, you'll need to specify the region when creating the client and also when creating the bucket.
Hello I am trying to connect Python to S3 in Frankfurt region using boto 2.43 where I want to print contents of bucket name...
Following is my code :
from boto.s3.connection import S3Connection
hostname='s3.eu-central-1.amazonaws.com'
conn = S3Connection(aws_access_key_id,aws_secret_access_key, host=hostname)
bucket_name = conn.get_bucket('jd-eu01-isg-analytics-data-from-us01', validate=False)
for key in bucket_name.list(prefix='EU_Scripts_For_Frankfurt/'):
print key
continue
When I am executing it,it throws following error :
File "/usr/lib/python2.7/site-packages/boto/s3/bucketlistresultset.py", line 34, in bucket_lister
encoding_type=encoding_type)
File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 473, in get_all_keys
'', headers, **params)
File "/usr/lib/python2.7/site-packages/boto/s3/bucket.py", line 399, in _get_all
query_args=query_args)
File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
retry_handler=retry_handler
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 927, in _mexe
request.authorize(connection=self)
File "/usr/lib/python2.7/site-packages/boto/connection.py", line 377, in authorize
connection._auth_handler.add_auth(self, **kwargs)
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 727, in add_auth
**kwargs)
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 546, in add_auth
string_to_sign = self.string_to_sign(req, canonical_request)
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 486, in string_to_sign
sts.append(self.credential_scope(http_request))
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 468, in credential_scope
region_name = self.determine_region_name(http_request.host)
File "/usr/lib/python2.7/site-packages/boto/auth.py", line 662, in determine_region_name
return region_name
UnboundLocalError: local variable 'region_name' referenced before assignment
How to resolve this issue ?? is this because of boto version ?? Any solutions please
Use boto3, it is easier. boto2 is not supported by AWS and boto already deprecated and there is no intention for features
The error pop up because it can't find the region. Boto/boto3
API will check the region name inside the boto/boto3 service initialization stage. If you didn't specify it, it will look for default region name define inside credential file or environment variable (e.g. ~/.aws/config).
This is true even you explicitly specify S3 endpoint URL. If you don't want to hard code the credential, region name, then you must setup AWS credential as specified here: credential configuration.
Making boto/boto3 using credential file/environment variable will make your code cleaner and more flexible, e.g. you can even using STS without changing the code. e.g.:
import boto3
# You can choose between service resource or service client.
s3 = boto3.client("s3")
response = s3.list_objects_v2(
Bucket="jd-eu01-isg-analytics-data-from-us01",
Prefix="EU_Scripts_For_Frankfurt"
)
for content in response["Contents"]:
print content["Key"]
Nevertheless, you can still hardcode access key id, secre_key, region name, etc by passing the parameter when you initialize boto/boto3 resources. (boto API is similar)
import boto3
s3 = boto3.client(
"s3",
region_name = "eu-central-1",
aws_access_key_id = 'xxxxxxxx",
aws_secret_access_key= = "yyyyyyy")
I want to make some very easy tasks on BigQuery via a python script. I found this package which does not work well. Indeed, when I try this code:
from bigquery import get_client
project_id = 'txxxxxxxxxxxxxxxxxx9'
# Service account email address as listed in the Google Developers Console.
service_account = '7xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com'
# PKCS12 or PEM key provided by Google.
key = '/home/fxxxxxxxxxxxx/Dropbox/access_keys/google_storage/xxxxxxxxxxxxxxxxxxxxx.pem'
client = get_client(project_id, service_account=service_account, private_key_file=key, readonly=True)
# Submit an async query.
results = client.get_table_schema('newdataset', 'newtable2')
print('results')
I get this error:
/home/xxxxxx/anaconda3/envs/snakes/bin/python2.7 /home/xxxxxx/Dropbox/Prog/bigQuery_daily_import/src/main.py
Traceback (most recent call last):
File "/home/xxxxxx/Dropbox/Prog/bigQuery_daily_import/src/main.py", line 9, in <module>
client = get_client(project_id, service_account=service_account, private_key_file=key, readonly=True)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/bigquery/client.py", line 83, in get_client
readonly=readonly)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/bigquery/client.py", line 101, in _get_bq_service
service = build('bigquery', 'v2', http=http)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/oauth2client/util.py", line 142, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/googleapiclient/discovery.py", line 196, in build
cache)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/googleapiclient/discovery.py", line 242, in _retrieve_discovery_doc
resp, content = http.request(actual_url)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/oauth2client/client.py", line 565, in new_request
self._refresh(request_orig)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/oauth2client/client.py", line 835, in _refresh
self._do_refresh_request(http_request)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/oauth2client/client.py", line 862, in _do_refresh_request
body = self._generate_refresh_request_body()
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/oauth2client/client.py", line 1541, in _generate_refresh_request_body
assertion = self._generate_assertion()
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/oauth2client/client.py", line 1670, in _generate_assertion
private_key, self.private_key_password), payload)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/oauth2client/_pycrypto_crypt.py", line 121, in from_string
pkey = RSA.importKey(parsed_pem_key)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 665, in importKey
return self._importKeyDER(der)
File "/home/xxxxxx/anaconda3/envs/snakes/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py", line 588, in _importKeyDER
raise ValueError("RSA key format is not supported")
ValueError: RSA key format is not supported
Process finished with exit code 1
My question: is there a tutorial in python which shows how to communicate easily with BigQuery: importing a dataset from google storage or S3, querying something, exporting the result to google storage.
A lot depends on your environment, and once you've figure that out everything should be super simple. I see the only problem on the error log you pasted is figuring out authentication.
Python pandas has had support for BigQuery for a while:
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.io.gbq.read_gbq.html
And I did a video with the creators of the module:
https://www.youtube.com/watch?v=gLeTDUMb7HY
Now, the simplest and fastest way these days to launch an Jupyter notebook with all of the Google Cloud goodies you mention is our new Google Datalab project:
https://cloud.google.com/datalab/
The only Datalab caveat is that it works on cloud servers, but if you want a fully managed Jupyter/IPython environment, totally secure, persistent, and ready to handle BigQuery, storage, etc... try it out.
Meanwhile, if you are writing a web application look at how other web applications solve this task.
For example, re:dash code to connect to BigQuery:
https://github.com/EverythingMe/redash/blob/master/redash/query_runner/big_query.py