I am starting with Microsoft Azure SDK for Python (https://github.com/Azure/azure-sdk-for-python), but I have problems.
I am using Scientific Linux and I have installed the SDK for Python 3.4 following the next steps:
(instead of the SDK directory)
python setup.py install
after that I created a simple script just to test the connection:
from azure.storage import BlobService
blob_service = BlobService(account_name='thename', account_key='Mxxxxxxx3w==' )
blob_service.create_container('testcontainer')
for i in blob_service.list_containers():
print(i.name)
following this documentation:
http://blogs.msdn.com/b/tconte/archive/2013/04/17/how-to-interact-with-windows-azure-blob-storage-from-linux-using-python.aspx
http://azure.microsoft.com/en-us/documentation/articles/storage-python-how-to-use-blob-storage/#large-blobs
but is not working, I always receive the same error:
python3 test.py
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/azure-0.9.0-py3.4.egg/azure/storage/storageclient.py", line 143, in _perform_request
File "/usr/local/lib/python3.4/site-packages/azure-0.9.0-py3.4.egg/azure/storage/storageclient.py", line 132, in _perform_request_worker
File "/usr/local/lib/python3.4/site-packages/azure-0.9.0-py3.4.egg/azure/http/httpclient.py", line 247, in perform_request
azure.http.HTTPError: The value for one of the HTTP headers is not in the correct format.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 21, in <module>
blob_service.create_container('testcontainer')
File "/usr/local/lib/python3.4/site-packages/azure-0.9.0-py3.4.egg/azure/storage/blobservice.py", line 192, in create_container
File "/usr/local/lib/python3.4/site-packages/azure-0.9.0-py3.4.egg/azure/__init__.py", line 905, in _dont_fail_on_exist
File "/usr/local/lib/python3.4/site-packages/azure-0.9.0-py3.4.egg/azure/storage/blobservice.py", line 189, in create_container
File "/usr/local/lib/python3.4/site-packages/azure-0.9.0-py3.4.egg/azure/storage/storageclient.py", line 150, in _perform_request
File "/usr/local/lib/python3.4/site-packages/azure-0.9.0-py3.4.egg/azure/storage/__init__.py", line 889, in _storage_error_handler
File "/usr/local/lib/python3.4/site-packages/azure-0.9.0-py3.4.egg/azure/__init__.py", line 929, in _general_error_handler
azure.WindowsAzureError: Unknown error (The value for one of the HTTP headers is not in the correct format.)
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.
RequestId:b37c5584-0001-002b-24b8-c2c245000000
Time:2014-11-19T14:54:38.9378626Z</Message><HeaderName>x-ms-version</HeaderName><HeaderValue>2012-02-12</HeaderValue></Error>
Thanks in advance and best regards.
I have this exact same issue. I believe it's a library bug, but the author/s haven't had their say yet.
It looks like the response states the version, but it's actually giving you the header that's wrong. Its value should be "2014-02-14", you can do the fix shown in https://github.com/Azure/azure-sdk-for-python/pull/289 .
Hopefully this will be fixed and nobody will ever read this answer. Cheers!
Related
I am trying to push the data to GCP DataStore, The below code snippet works fine in Jupyter Notebook but it is throwing error in VS Code.
def load_data_json(self, kind_name, data_with_qp_ID, qp_id):
#Load the data in JSON format to upload into the DataStore
data_with_qp_ID_as_JSON = self.convert_DF_to_JSON(data_with_qp_ID, qp_id)
#Loop to iterate through the JSON format and upload into the GCS Storage
for data in data_with_qp_ID_as_JSON.keys():
with self.client.transaction():
incomplete_key = self.client.key(kind_name)
task = datastore.Entity(key=incomplete_key)
task.update(data_with_qp_ID_as_JSON[data])
self.client.put(task)
return 'Ingestion Successful - Data Store Repository'
I have defined the name of the bucket in "Kind Name", data_with_qp_id is a pandas dataframe, qp_id is the name of the column name in pandas. Please see the error message that I get below,
Traceback (most recent call last):
File "/Users/ajaykrishnan/Desktop/Projects/Sprint 3/Data Migration/DataMigration_v1.1/main2.py", line 139, in <module>
write_datastore_db.load_data_json(ds_kindname, bookmarks_data_with_qp_ID, qp_id)
File "/Users/ajaykrishnan/Desktop/Projects/Sprint 3/Data Migration/DataMigration_v1.1/pkg/repository/ds_repository.py", line 50, in load_data_json
self.client.put(task)
File "/opt/anaconda3/lib/python3.9/site-packages/google/cloud/datastore/client.py", line 597, in put
self.put_multi(entities=[entity], retry=retry, timeout=timeout)
File "/opt/anaconda3/lib/python3.9/site-packages/google/cloud/datastore/client.py", line 634, in put_multi
current.put(entity)
File "/opt/anaconda3/lib/python3.9/site-packages/google/cloud/datastore/transaction.py", line 315, in put
super(Transaction, self).put(entity)
File "/opt/anaconda3/lib/python3.9/site-packages/google/cloud/datastore/batch.py", line 227, in put
_assign_entity_to_pb(entity_pb, entity)
File "/opt/anaconda3/lib/python3.9/site-packages/google/cloud/datastore/batch.py", line 373, in _assign_entity_to_pb
bare_entity_pb = helpers.entity_to_protobuf(entity)
File "/opt/anaconda3/lib/python3.9/site-packages/google/cloud/datastore/helpers.py", line 208, in entity_to_protobuf
key_pb = entity.key.to_protobuf()
File "/opt/anaconda3/lib/python3.9/site-packages/google/cloud/datastore/key.py", line 298, in to_protobuf
key.path.append(element)
TypeError: Parameter to MergeFrom() must be instance of same class: expected google.datastore.v1.Key.PathElement got PathElement.
My environment is as follows,
Mac OS Monterey V12.06
Python - Conda 3.9.12
I was able to clear this error. It was an issue with Protobuf library that my environment was using. I downgraded the version of protobuf from 4.x.x to 3.20.1 and it worked.
I'm following this data prediction using Cloud ML Engine with scikit-learn tutorial for GCP AI Platforms. I tried to make an API call to BigQuery with:
def query_to_dataframe(query):
import pandas as pd
import pkgutil
privatekey = pkgutil.get_data('trainer', 'privatekey.json')
print(privatekey[:200])
return pd.read_gbq(query,
project_id=PROJECT,
dialect='standard',
private_key=privatekey)
but got the following error:
Traceback (most recent call last):
[...]
TypeError: a bytes-like object is required, not 'str'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 66, in <module>
arguments['numTrees']
File "/root/.local/lib/python3.7/site-packages/trainer/model.py", line 119, in train_and_evaluate
train_df, eval_df = create_dataframes(frac)
File "/root/.local/lib/python3.7/site-packages/trainer/model.py", line 95, in create_dataframes
train_df = query_to_dataframe(train_query)
File "/root/.local/lib/python3.7/site-packages/trainer/model.py", line 82, in query_to_dataframe
private_key=privatekey)
File "/usr/local/lib/python3.7/dist-packages/pandas/io/gbq.py", line 149, in read_gbq
credentials=credentials, verbose=verbose, private_key=private_key)
File "/root/.local/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 846, in read_gbq
dialect=dialect, auth_local_webserver=auth_local_webserver)
File "/root/.local/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 184, in __init__
self.credentials = self.get_credentials()
File "/root/.local/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 193, in get_credentials
return self.get_service_account_credentials()
File "/root/.local/lib/python3.7/site-packages/pandas_gbq/gbq.py", line 413, in get_service_account_credentials
"Private key is missing or invalid. It should be service "
pandas_gbq.gbq.InvalidPrivateKeyFormat: Private key is missing or invalid. It should be service account private key JSON (file path or string contents) with at least two keys: 'client_email' and 'private_key'. Can be obtained from: https://console.developers.google.com/permissions/serviceaccounts
When the package runs in local environment, the private key loads fine, but when submitted as a ml-engine training job, the error occurs. Note that the private key fails to load only when I use GCP RUNTIME_VERSION="1.15" and PYTHON_VERSION="3.7", but can load with no problem when I use PYTHON_VERSION="2.7".
In case it's useful, the structure of my package is:
/babyweight
- setup.py
- trainer
- __init__.py
- model.py
- privatekey.json
- task.py
I'm not sure if the problem is due to a bug in Python, or where I placed privatekey.json.
I was able to solve the problem after I changed read_gbq's attribute for reading BigQuery access key from private_keys to credentials, as recommended by #rmesteves, and as shown here. I then set the value as the absolute path to privatekey.json, as shown here. Now the job is able to run without error.
Note: I only encountered this problem with Python 3+, but not with Python 2.7. I'm not sure why. It could possibly be due to the implementation of read_gbq.
I've been migrating some Python 2.7.11 code to 3.5.1 after running into trouble with unicode. This was the last straw - since I started using the venv module there's no reason to be on 2.7 just because someone doesn't like 3!
The problem occurs while trying to run a one-way sync (ie. downloading changes only).
Here is the full error message, paths shortened:
Traceback (most recent call last):
File "%SCRIPT%.py", line 209, in <module>
updated_schedules = dbx_sync.One_Way_Sync(config['Dropbox Parameters']['Directory'], config['Dropbox Parameters']['Base Path'])
File "%COMMON_PATH%\modules\dropbox_sync_schedules.py", line 62, in One_Way_Sync
result = client.delta(cursor, base_path)
File "%COMMON_PATH%\env-home\lib\site-packages\dropbox\client.py", line 569, in delta
return self.rest_client.POST(url, params, headers)
File "%COMMON_PATH%\env-home\lib\site-packages\dropbox\rest.py", line 322, in POST
return cls.IMPL.POST(*n, **kw)
File "%COMMON_PATH%\env-home\lib\site-packages\dropbox\rest.py", line 260, in POST
is_json_request=is_json_request)
File "%COMMON_PATH%\env-home\lib\site-packages\dropbox\rest.py", line 235, in request
raise ErrorResponse(r, r.read())
dropbox.rest.ErrorResponse: [400] 'Invalid "cursor" parameter: u"b\'\'"'
Searching for "invalid cursor parameter" wasn't any help, so I thought I'd come here.
u"b\'\'" is the key here. I just couldn't understand how that representation had ended up being sent as a string.
The issue was in reading the old cursor from a file (which for this example is empty): in Python 2 I had opened the file in mode rb - in Python 3 just r is all that's required, and everything works.
Hurrah!
I am new to jira-python and have run into an issue with using jirashell.
jira-python was installed from documentation given here: http://jira.readthedocs.org/en/latest/
When I try to start jirashell using:
ubuntu#ip-10-0-0-12:~$ jirashell -s https://jira.atlassian.com
I get the following error:
Traceback (most recent call last):
File "/usr/local/bin/jirashell", line 11, in <module> sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/jira/jirashell.py", line 248, in main
jira = JIRA(options=options, basic_auth=basic_auth, oauth=oauth)
File "/usr/local/lib/python2.7/dist-packages/jira/client.py", line 202, in __init__self._create_oauth_session(oauth)
File "/usr/local/lib/python2.7/dist-packages/jira/client.py", line 1871, in _create_oauth_session
oauth['consumer_key'],
KeyError: u'consumer_key'
I have also tried to get to a server using basic auth but that returns the same error. Using curl works fine, but I wanted to see the objects that are getting returned and get help as I develop by python-jira integration.
Thank you for any insight.
I am trying to delete a spreadsheet in Google Docs with this function:
def f_DeleteResource(xls_name):
"""Delete a resource"""
client=Auth()
for e1 in client.GetResources().entry:
e2 = client.GetResource(e1)
if xls_name==e2.title.text:
client.DeleteResource(e2.resource_id.text,True)
And I obtain different errors when I change the first parameter of client.DeleteResource(p1,p2):
client.DeleteResource(e2.resource_id.text,True):
Traceback (most recent call last):
File "C:\xmp\D6GDocsDeleteUpload.py", line 164, in <module> main()
File "C:\xmp\D6GDocsDeleteUpload.py", line 157, in main f_DeleteResource(sys.argv[2])
File "C:\xmp\D6GDocsDeleteUpload.py", line 144, in f_DeleteResource client.DeleteResource(e2.resource_id.text,True)
File "C:\Python27\lib\site-packages\gdata\docs\client.py", line 540, in delete_resource uri = entry.GetEditLink().href
AttributeError: 'str' object has no attribute 'GetEditLink'
client.DeleteResource(e2,True):
Traceback (most recent call last):
File "C:\xmp\D6GDocsDeleteUpload.py", line 164, in <module> main()
File "C:\xmp\D6GDocsDeleteUpload.py", line 157, in main f_DeleteResource(sys.argv[2])
File "C:\xmp\D6GDocsDeleteUpload.py", line 144, in f_DeleteResource client.DeleteResource(e2,True)
File "C:\Python27\lib\site-packages\gdata\docs\client.py", line 543, in delete_resource return super(DocsClient, self).delete(uri, **kwargs)
File "C:\Python27\lib\site-packages\gdata\client.py", line 748, in delete **kwargs)
File "C:\Python27\lib\site-packages\gdata\docs\client.py", line 66, in request return super(DocsClient, self).request(method=method, uri=uri, **kwargs)
File "C:\Python27\lib\site-packages\gdata\client.py", line 319, in request RequestError)
gdata.client.RequestError: Server responded with: 403, <errors xmlns='http://schemas.google.com/g/2005'><error><domain>GData</domain><code>matchHeaderRequired</code><location type='header'>If-Match|If-None-Match</location><internalReason>If-Match or If-None-Match header or entry etag attribute required</internalReason></error></errors>
Anyone can help me?
It seems to be a bug in Google API Python library. I checked gdata-2.0.16 and noticed that DeleteResource() function uses only URL of the resource (gdata/docs/client.py lines 540-543), but later checks for hasattr(entry_or_uri, 'etag') (gdata/client.py lines 737-741) and of course string value (uri) doesn't have etag attribute.
You may work around it using force keyword argument:
import gdata.docs.data
import gdata.docs.client
client = gdata.docs.client.DocsClient()
client.ClientLogin('xxxxxx#gmail.com', 'xxxxxx', 'XxX')
for doc in client.GetAllResources():
if doc.title.text == 'qpqpqpqpqpqp':
client.DeleteResource(doc, force=True)
break
If you want you may report an error to library maintainers (if it isn't already reported).
This issue has been fixed in this revision:
http://code.google.com/p/gdata-python-client/source/detail?r=f98fff494fb89fca12deede00c3567dd589e5f97
If you sync you client to the repository, you should be able to delete a resource without having to specify 'force=True'.