I have been learning how to use Amazon S3 API by using the open source Eucalyptus. So far I have been able to successfully use REST, but now I would also like to use SOAP. I seem to be having trouble generating the correct signature for my request. The service is giving me a 403 Forbidden error:
Traceback (most recent call last):
File "soap.py", line 31, in <module>
r = w.download_file('mybucket', 'test.txt')
File "soap.py", line 27, in download_file
r = self.client.service.ListAllMyBuckets(self.access_key, timestamp, signature)
File "/usr/lib/python2.6/site-packages/suds/client.py", line 521, in __call__
return client.invoke(args, kwargs)
File "/usr/lib/python2.6/site-packages/suds/client.py", line 581, in invoke
result = self.send(soapenv)
File "/usr/lib/python2.6/site-packages/suds/client.py", line 619, in send
description=tostr(e), original_soapenv=original_soapenv)
File "/usr/lib/python2.6/site-packages/suds/client.py", line 677, in process_reply
raise Exception((status, description))
Exception: (403, u'Forbidden')
My code is in Python 2 and uses the SUDS-Jurko library for sending SOAP requests:
from suds.client import Client
class WalrusSoap:
wsdl_url = 'https://s3.amazonaws.com/doc/2006-03-01/AmazonS3.wsdl'
server_url = 'https://localhost:8773/services/Walrus'
def __init__(self, access_key, secret_key):
self.access_key = access_key
self.secret_key = secret_key
self.client = Client(self.wsdl_url)
self.client.wsdl.services[0].setlocation(self.server_url)
#print self.client
def create_signature(self, operation, timestamp):
import base64, hmac, hashlib
h = hashlib.sha1(self.secret_key)
h.update("AmazonS3" + operation + timestamp)
#h = hmac.new(key=self.secret_key, msg="AmazonS3" + operation + timestamp, digestmod=hashlib.sha1)
return base64.encodestring(h.digest()).strip()
def download_file(self, bucket, filename):
from time import gmtime, strftime
timestamp = strftime('%Y-%m-%dT%H:%M:%S.001Z', gmtime())
print(timestamp)
signature = self.create_signature('ListAllMyBuckets', timestamp)
print(signature)
r = self.client.service.ListAllMyBuckets(self.access_key, timestamp, signature)
return r
w = WalrusSoap(access_key='MOBSE7FNS6OC5NYC75PG8', secret_key='yxYZmSLCg5Xw6rQVgoIuVLMAx3hZRlxDc0VOJqox')
r = w.download_file('mybucket', 'test.txt')
print(r)
I changed the server endpoint, because otherwise the WSDL points to the regular S3 servers at Amazon. I also have two different ways of creating the signature in my create_signature function. I was swapping between one and the other by simply commenting out the second one. Neither of the two seem to work. My question is what am I doing wrong?
SOAP Authentication: http://docs.aws.amazon.com/AmazonS3/latest/dev/SOAPAuthentication.html
SUDS-Jurko Documentation: https://bitbucket.org/jurko/suds/overview
Edit: I realized I forgot to include an example of what timestamp and signature is printed for debugging purposes.
2014-12-05T00:27:41.001Z
0h8vxE2+k10tetXZQJxXNnNUjjw=
Edit 2: Also, I know that the download_file function does not download a file :) I am still in testing/debug phase
Edit 3: I am aware that REST is better to use, at least according to Amazon. (Personally I think REST is better also.) I am also already aware that SOAP is deprecated by Amazon. However I would like to go down this path anyways, so please do me a favor and do not waste my time with links to the deprecation. I assure you that while writing this SOAP code, I was already well aware of the deprecation. In fact one of the links I posted has the deprecation notice printed at the top of its page. However, if you have evidence showing that Walrus completely ditches SOAP or that they stopped working on the SOAP portion, I would like to see something like that. But please do not tell me Amazon has deprecated SOAP.
The S3 SOAP API does not support "new" features so the REST API should be used where possible:
http://docs.aws.amazon.com/AmazonS3/latest/dev/SOAPAPI3.html
https://forums.aws.amazon.com/message.jspa?messageID=77821
IIRC recent versions of Eucalyptus do not support SOAP with S3.
That said, the signature looks good to me so I would check if the client/service hosts have the correct time, if there is a difference of more than 15 minutes authentication would fail.
You could also check the cloud-error.log on the Walrus service host as there may be more details on the failure there.
Eucalyptus does not support SOAP for S3 as of Eucalyptus version 4.0.0.
If you are using an older version of Eucalyptus (pre 4.0), then SOAP should work. Note, however, that the wsdl provided by S3 is not necessarily current or accurate for even their own service. S3 is notorious for changing the API without version or wsdl bumps, particularly since they stopped updating the SOAP API. So, there are likely responses from Walrus that do not conform to the published WSDL because our XML was updated based on the responses we see from S3 (via REST) and the SOAP and REST responses diverged. The signatures should be compatible, however, so worst case you would see 500 or 400 errors, but not 403 responses.
My recommendation is if you really want to learn the S3 SOAP API, you'll have to use S3 proper. The S3 SOAP support in Eucalyptus is there pre-4.0 but may not be compliant with the current state of S3 SOAP--we stopped testing against the S3 SOAP API directly when the AWS SDKs stopped using SOAP in favor of better REST support since that was the API moving forward.
Eucalyptus does support SOAP (see ZachH's comment about deprecation):
https://www.eucalyptus.com/docs/eucalyptus/4.0.2/schemas/aws-apis/s3/index.html
I decided to scrap using python, and I produced some working code with C# instead. Turns out C# has better SOAP libraries.
Related
I am moving my first steps with the upwork API in python, but I am stuck at the first steps.
I am following the tutorials but I cannot get the tokens from oauth.
upwork.ca_certs_locater.LINUX_PATH = 'C:\\Users\\somedir\\cacert.pem'
client = upwork.Client(upwork_key, upwork_secret)
print("Please to this URL (authorize the app if necessary):")
print(client.auth.get_authorize_url())
print("After that you should be redirected back to your app URL with " +
"additional ?oauth_verifier= parameter")
the upwork library seems installed correctly, the .pem file is also loaded correctly and I think I can connect to the server.
The API keys are correctly enabled, checked with the support.
However client.auth.get_authorize_url() returns empty parameters and auth.get_request_token() returns (null,null).
What can be wrong? How do I enable logging to check where all gets stuck?
client.auth.get_authorize_url() returns:
"https://www.upwork.com/services/api/auth?oauth_token=None"
client.auth.auth.get_request_token() returns:
(None, None)
I am using python 3.6
Unfortunately, this is a "known" issue that may happen when Python3 is in use - README says: "These are Python (2, and 3 which is "supported" via unofficial PR #27 and not guaranteed) bindings for Upwork Public API".
Could you please try with Python 2.x
According to Google's OAUTH API documentation, the userinfo.profile and userinfo.email scopes have been deprecated in favor of using profile and email. There is a lot of information from other API users about this switch as well.
However, when trying to use the People API, I get this error:
2016-02-22 13:01:25,044 :Exception on /admin/testbank/settings [GET]
Traceback (most recent call last):
(Flask traceback omitted...)
File "/home/somedev/testweb/views/admin_view.py", line 78, in admin_view
res_acct_info = people_service.people().get(userId='me').execute()
File "/home/somedev/env/lib/python3.4/site-packages/googleapiclient/discovery.py", line 676, in method
raise TypeError('Got an unexpected keyword argument "%s"' % name)
TypeError: Got an unexpected keyword argument "userId"
Looking at the Google API sample, this should be the right call. However, dumping parameters.argmap within the library's discovery.py shows that userId does not exist. What am I doing wrong?
(Note: I'm trying to tag google-people, since this is where the API pages suggest, but I don't have enough rep to tag this. Could someone else add this tag for me?)
It turns out that I've been blind to the exact code I'm reading... all of these examples (particularly Google's example) have been using the Google+ API, which does indeed have the userId argument.
For reference, the "old way" is by using the oauth2/userinfo service:
service = build('oauth2', 'v2', http=http)
user = users_service.userinfo().get().execute()
name = user.get('name')
email = user.get('email')
You can use the Google+ API to get the same information - it will work, even if the user does not have a Google+:
service = discovery.build("plus", "v1", http=http)
user = service.people().get(userId='me').execute()
# This assumes that user['emailAddresses'] exists and
# has at least one element...
name = user.get('displayName')
email = user.get('emailAddresses')[0].get("value")
At the time of writing, it seems that the People API was released recently (February 10th, 2016)! It makes sense that there wouldn't be much documentation about it...
To use the newer People API (and maybe claim cleanliness from Google+), this is the correct way to fetch the current user's information:
service = discovery.build('people', 'v1', http_auth)
user = people_service.people().get(resourceName='people/me').execute()
# This assumes that user['names'] and user['emailAddresses']
# exists and has at least one element...
name = user.get('names')[0].get("displayName")
email = user.get('emailAddresses')[0].get("value")
resourceName replaces userId in the People API. It has a similar purpose (to identify the current user or another user), but has a different format, as seen with using 'people/me' versus just 'me'.
Both the Google+ and the newer People API simply require the email and profile scopes. However, unlike the former userinfo API, you need to manually enable the Google+ API and/or the People API in order to use them.
tl;dr: Google+ API uses userId='me', new People API uses resourceName='people/me', you should use one of these supported APIs - both return the same information, just in a slightly different format!
I'm using chef server, configuring few different nodes / environments.
when asking for env attributes using the pychef api, few times in a row (when refreshing a web page using python server calling chef server) im getting ChefServerNotFoundError (the first few times are fine, and third exception is raised)
I guess that there is kind of firewall / anti ddos attacks on this server, but i can not figure out how to edit these settings.
anyone have any idea?
this is a part of the method (that is called 3 times and throws an exception):
env_nodes = Search('node').query('chef_environment: {0}'.format(env_name))
nodes_dict = {}
for n in env_nodes:
node = Node(n['name'])
nodes_dict[node.name] = node['ipaddress']`
and this is the traceback:
File "C:\env\lib\site-packages\chef\search.py", line 91, in __getitem__
row_value = self.data['rows'][value]
File "C:\env\lib\site-packages\chef\search.py", line 59, in data
self._data = self.api[self.url]
TypeError: 'NoneType' object is not subscriptable`
When using PyChef in a webapp or other multi-threaded system you should really pass in the API object explicitly. There is a system to track a default API target in a threadlocal for the purposes of making simple scripts easier, but in retrospect this was probably a mistake as it leads to these confusing issues. This would be a better version of that code, also faster:
nodes_dict = {row.object.name: row.object['ipaddress'] for row in Search('node', 'chef_environment:{}'.format(env_name), api=api)}
Where api is the return value of chef.autoconfigure() or some other ChefAPI object.
Using the GData Calendar API via App Engine in Python, when you create an event there are handy little helper methods to parse the response:
new_event = calendar_service.InsertEvent(event, '/calendar/feeds/default/private/full')
helper = new_event.GetEditLink().href
When you create a new calendar:
new_calendar = gd_client.InsertCalendar(new_calendar=calendar)
I was wondering if there might be related methods that I just can't find in the documentation (or that are--perhaps--undocumented)?
I need to store the new calendar's ID in the datastore, so I would like something along the lines of:
new_calendar = gd_client.InsertCalendar(new_calendar=calendar)
new_calendar.getGroupLink().href
In my code, the calendar is being created, and G is returning the Atom response with a 201, but before I get into using elementtree or atom.parse to extract the desired element, I was hoping someone here might be able to help.
Many thanks in advance :)
I've never used the GData API, so I could be wrong, but...
It looks like GetLink() will return the link object for any specified rel. Seems like GetEditLink() just calls GetLink(), passing in the rel of the Edit link. So you should be able to call GetLink() on the response from InsertCalendar(), and pass in the rel of the Group link.
Here's the pydoc info that I used to figure this out: http://gdata-python-client.googlecode.com/svn/trunk/pydocs/gdata.calendar_resource.data.html
Couple months ago I have asked the same question but in the context of older version of ZSI (How to display outcoming and incoming SOAP message for ZSI.ServiceProxy in Python?). Now, in the new version of ZSI 2.1 there is no tacefile parameter). I tried to find a documentation for the new version but I faild. Does anyone know how to display the SOAP messages generated and received by ZSI 2.1? Thank you in advance :-)
For debugging I have found less interfering solution using wireshark to trace the TCP packages. It looks like that:
I had this same problem. My workaround was to modify the dispatch.py file that comes with ZSI.
I created a logging function (logmessage) for my app that would store SOAP messages into a database and then added that function where necessary. I do not recall the ZSI version I was using however. You should be able to find these functions pretty easily in the code though. I ave approximate L numbers since i made other edits
in Dispatch.py file in your site-packages directory
L156 - logs SOAP responses
def _Dispatch(tons-of-args, **kw):
#several lines of code edited here#
#several lines of code edited here#
#several lines of code edited here#
sw = SoapWriter(nsdict=nsdict)
sw.serialize(result, tc)
logmessage( str(sw), 1, kw['request'].get_remote_host() ) #LOGGING HERE
L168 - logs SOAP errors
def _ModPythonSendFault(f, **kw):
logmessage( str(f.AsSOAP()), 1, kw['request'].get_remote_host() ) #LOGGING ADDED HERE
_ModPythonSendXML(f.AsSOAP(), 500, **kw)
L277 - logs requests
def AsHandler(request=None, modules=None, **kw):
'''Dispatch from within ModPython.'''
a = request.read(-1)
logmessage( a, 0, request.get_remote_host() ) #LOGGING ADDED HERE
ps = ParsedSoap(a)
kw['request'] = request
_Dispatch(ps, modules, _ModPythonSendXML, _ModPythonSendFault, **kw)