Develop Reference

Boto3 Cloudformation Error: Template format error: unsupported structure

I am unable to load parameters into the python boto3 Cloudformation client.
Below is my parameter file:
[
{"ParameterKey": "pVpcId", "ParameterValue": "vpc-XXXXXX"},
{"ParameterKey": "pContact", "ParameterValue": "XDXDXX"},
{"ParameterKey": "pCC", "ParameterValue": "XXXXX" },
{"ParameterKey": "pFormat", "ParameterValue": "True"}
]
I'm loading it into the program in the following way:
with open(parameter_file, 'r') as infile:
parameters=ast.literal_eval(infile.read())
client = boto3.client('cloudformation',aws_access_key_id=access_key,aws_secret_access_key=secret_key,aws_session_token=session_token,region_name=region)
response = client.create_stack(
StackName=stack_name,
TemplateURL=stack_url,
Parameters=parameters
)
When I establish a boto3 client with Cloudformation and call it, I get the error described below. The call works without the parameters, so it's definitely something to do with the parameter file.
Traceback (most recent call last):
File "cf_create_stack", line 85, in <module>
Parameters=parameters
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateStack operation: Template format error: unsupported structure.

So the TemplateBody param is expecting the content/str of the CloudFormation Template file and not just the filename.
The following should work satisfactorily.
cf_template = open('batch-job-cft.yml').read()
cf_client.create_stack(StackName='Batch Job', TemplateBody=cf_template)
# OR
# Optimal usage would be as below
with open('batch-job-cft.yml', 'r') as cf_file:
cft_template = cf_file.read()
cf_client.create_stack(StackName='Batch Job', TemplateBody=cft_template)

I believe it is impossible that you got that response based on the code sample you provided.
I think you might have tried TemplateBody rather than TemplateURL?
To reproduce the error message, try this simple example:
#!/usr/bin/env python
import boto3
import ast
parameter_file = 'parameters.json'
client = boto3.client('cloudformation')
with open(parameter_file, 'r') as infile:
parameters = ast.literal_eval(infile.read())
response = client.create_stack(
StackName='TestStack',
TemplateBody='file://cloudformation.yml',
Parameters=parameters
)
If you place the parameters file and template in the expected locations and run this, you should then see the exact error message you are seeing:
Traceback (most recent call last):
File "test.py", line 17, in <module>
Parameters=parameters
File "/Users/alexharvey/git/home/python-test/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/alexharvey/git/home/python-test/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 661, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateStack operation: Template format error: unsupported structure.
Note that this error:
Template format error: unsupported structure
Comes from the AWS API, not Boto3.
The error is caused when you pass in a file:// URI or URL to the TemplateBody parameter.
Moreover, I believe it is simply impossible to get that response if you really passed anything to TemplateURL.
See also:
this related answer, for reproducing this error message using the AWS CLI.
API docs where TemplateBody and TemplateURL are documented.

Here's how I was able to resolve this:
In order to load the parameter file (which loads as a list of dictionaries), I had the following code:
with open(parameter_file) as f:
parameters=json.load(f)
for l in parameters:
l['UsePreviousValue']=eval('False')
In order to pass the Cloudformation template, I used the following code (basically reading it as a string):
with open(cloudformation_template) as g:
template_body=g.read()
Finally, I passed both variables into the cloudformation client:
response = client.create_stack(
StackName=stack_name,
TemplateBody=template_body,
Parameters=parameters
)
The thing that was going wrong for me was that boto3 'Cloudformation' client expects a 'list of dictionaries' for parameters but expects a 'string' for the cloudformation template.
This is very frustrating and I'm trying to find a way to raise this to AWS.

Profiling CherryPy

I've been trying to start profiling my CherryPy webserver, but the documentation is lacking in detail in how this should be set up. I understand that I should be able to use cherrypy.lib.profiler as middleware to mount my initial server. Right now, I have code like the following:
server_app = ServerClass()
cherrypy.tree.mount(server_app, '/', '/path/to/config/file.cfg')
cherrypy.engine.start()
cherrypy.engine.block()
I want to mount the profiling middleware, and it seems that something like the following is required:
from cherrypy.lib import profiler
server_app = ServerClass()
server_cpapp = cherrypy.Application(server_app, '/', '/path/to/config/file.cfg')
server_profile_cpapp = profiler.make_app(server_cpapp, '/home/ken/tmp/cprofile', True)
#cherrypy.tree.mount(server_profile_cpapp)
cherrypy.tree.graft(server_profile_cpapp)
cherrypy.engine.start()
cherrypy.engine.block()
For some reason cherrypy.tree.mount doesn't work, but if I use cherrypy.tree.graft all seems to operate fine (I can make requests to the server as normal)
However, the above code generates a cp_0001.prof file under /home/ken/tmp/cprofile and I am not sure how to interpret it. I have tried using pyprof2calltree to read the data into KCacheGrind, but I get a parsing error. Does what I'm doing seem correct, and if so how do I interpret the output file?

It turns out that the profile files generated by CherryPy can be interpreted using the profiler.py script shipped as part of CherryPy. Simply run profiler.py in the <site-packages>/cherrypy/lib directory as follows:
python profiler.py /directory/containing/prof/files 8080
Then navigate to localhost:8080 in your browser and the profiling results for all .prof files in the target directory will be displayed in a simple text interface.
I would still prefer to be able to export the results into a calltree to profile using KCacheGrind, but this seems to do for basic profiling.
This is documented in the change log for v2.1 of CherryPy when the profiler was introduced (although the other details on that page describing how to set up the profiler has since become deprecated)

I am also trying to get profiling up and running for a cherrypy instance. I used the same code you have in your initial question, which seems to work in that it generates a cp_0001.prof file in the folder.
To answer your question, I am able to open this file in runsnakerun to see the profiling output in a tree view.
The problem I have is that every request I do to the server now fails, with the following output in the log:
[29/May/2013:16:39:32] ENGINE AssertionError('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
Traceback (most recent call last):
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 1302, in communicate
req.respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 831, in respond
self.server.gateway(self).respond()
File "<path>\packages\cherrypy\wsgiserver\wsgiserver2.py", line 2115, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "<path>\packages\cherrypy\_cptree.py", line 290, in __call__
return app(environ, start_response)
File "<path>\packages\cherrypy\lib\profiler.py", line 188, in __call__
return self.profiler.run(gather)
File "<path>\packages\cherrypy\lib\profiler.py", line 147, in run
result = self.profiler.runcall(func, *args)
File "<path>\python\lib\profile.py", line 472, in runcall
return func(*args, **kw)
File "<path>\packages\cherrypy\lib\profiler.py", line 183, in gather
def gather():
File "<path>\python\lib\profile.py", line 246, in trace_dispatch_i
if self.dispatch[event](self, frame, t):
File "<path>\python\lib\profile.py", line 301, in trace_dispatch_call
frame, frame.f_back)
AssertionError: ('Bad call', ('', 0, 'sleep'), <frame object at 0x08522400>, <frame object at 0x08522030>, <frame object at 0x08156748>, <frame object at 0x06D06F10>)
I am using python 2.6.6 and cherrypy 3.2.2
Any suggestions?

Python 3.x and TestLink xmlprc

Appreciate your helping first, I am new for the python 3.x.
When I try to use Python 3.x to parse the testlink xmlprc server. I got below error, but I can run the code under Python 2.x, any idea?
import xmlrpc.client
server = xmlrpc.client.Server("http://172.16.29.132/SITM/lib/api/xmlrpc.php") //here is my testlink server
print (server.system.listMethods()) //I can print the methods list here
print (server.tl.ping()) // Got error.
Here is the error:
['system.multicall', 'system.listMethods', 'system.getCapabilities', 'tl.repeat', 'tl.sayHello', 'tl.ping', 'tl.setTestMode', 'tl.about', 'tl.checkDevKey', 'tl.doesUserExist', 'tl.deleteExecution', 'tl.getTestSuiteByID', 'tl.getFullPath', 'tl.getTestCase', 'tl.getTestCaseAttachments', 'tl.getFirstLevelTestSuitesForTestProject', 'tl.getTestCaseCustomFieldDesignValue', 'tl.getTestCaseIDByName', 'tl.getTestCasesForTestPlan', 'tl.getTestCasesForTestSuite', 'tl.getTestSuitesForTestSuite', 'tl.getTestSuitesForTestPlan', 'tl.getLastExecutionResult', 'tl.getLatestBuildForTestPlan', 'tl.getBuildsForTestPlan', 'tl.getTotalsForTestPlan', 'tl.getTestPlanPlatforms', 'tl.getProjectTestPlans', 'tl.getTestPlanByName', 'tl.getTestProjectByName', 'tl.getProjects', 'tl.addTestCaseToTestPlan', 'tl.assignRequirements', 'tl.uploadAttachment', 'tl.uploadTestCaseAttachment', 'tl.uploadTestSuiteAttachment', 'tl.uploadTestProjectAttachment', 'tl.uploadRequirementAttachment', 'tl.uploadRequirementSpecificationAttachment', 'tl.uploadExecutionAttachment', 'tl.createTestSuite', 'tl.createTestProject', 'tl.createTestPlan', 'tl.createTestCase', 'tl.createBuild', 'tl.setTestCaseExecutionResult', 'tl.reportTCResult']
Traceback (most recent call last):
File "F:\SQA\Python\Testlink\Test.py", line 5, in <module>
print (server.tl.ping())
File "C:\Python31\lib\xmlrpc\client.py", line 1029, in __call__
return self.__send(self.__name, args)
File "C:\Python31\lib\xmlrpc\client.py", line 1271, in __request
verbose=self.__verbose
File "C:\Python31\lib\xmlrpc\client.py", line 1070, in request
return self.parse_response(resp)
File "C:\Python31\lib\xmlrpc\client.py", line 1164, in parse_response
p.feed(response)
File "C:\Python31\lib\xmlrpc\client.py", line 454, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: junk after document element: line 2, column 0

When I've seen this message before, it happened because the contents of the transported data wasn't escaped for XML transport. The solution was to wrap the data in an XMLRPC Binary object.
In your case, you don't control the server side, so the above isn't a solution for you but it may suggest what the actual problem is.
Also, the Python 2 versus Python 3 difference suggests that there is a text/bytes issue at work.
To help diagnose the issue, set verbose=True so you can see the actual HTTP request/response headers and the XML request/response. That may show you what is at line 2: column 0. You may find that the issue may be with the PHP script not wrapping up binary data in base64 encoding as required by the XMLRPC spec.

Thank you , I find out all the methods list, only 'tl.sayHello', 'tl.ping','tl.about' has this problem, and all of them are pass a string with a PHP automatic loader empty file *.class.php to the parser, other methods are pass a xml file. So I give up to use those methods and the script works fine.

AttributeError: 'search' object has no attribute 'status_code' in django project

I'm testing a django project using the test sever when it gives me the following exception
Traceback (most recent call last):
File "/usr/lib/pymodules/python2.6/django/core/servers/basehttp.py", line 280, in run
self.result = application(self.environ, self.start_response)
File "/usr/lib/pymodules/python2.6/django/core/servers/basehttp.py", line 674, in >call
return self.application(environ, start_response)
File "/usr/lib/pymodules/python2.6/django/core/handlers/wsgi.py", line 245, in call
response = middleware_method(request, response)
File "/usr/lib/pymodules/python2.6/django/middleware/common.py", line 84, in >process_response
if response.status_code == 404:
AttributeError: 'search' object has no attribute 'status_code'
Just from reading this traceback, I don't think the problem is in my code, but I'm not sure. Could someone look through my code and help me solve this problem? my code is hosted on GitHub and any other comments or suggestions are greatly appreciated. Thanks in advance for your time and consideration.

MusicGrep/musicgrep/views.py is not returning a HttpResponse instance

There are quite a few problems with your code, but the one that's causing the actual error you quote is that you don't instantiate your FormWizard subclass in your urlconf.
As the documentation shows, you need to actually call the search class in urls.py to instantiate it, so you pass an instance rather than a class.
(r'^MusicGrep/$', 'MusicGrep.musicgrep.forms.search()'),
It might help if you followed the PEP8 guidelines on naming - if your class was called Search with a capital S, it would be more obvious that it was actually a class and not a function.

_URLFetchResult object has no attribute 'body' when calling Freebase via Python Module

I've been using the Freebase Python module successfully to read data, and today I started testing writing data to Freebase.
But I keep getting an error that isn't making sense to me:
Within the same scope of code, I can perform an mqlread(query) without error.
But when I try:
freebase.mqlwrite(query)
I get an error like:
File "/Users/willmerydith/repos/supermeeple-sk/admin.py", line 96, in post
result = freebase.mqlwrite(query)
File "/Users/willmerydith/repos/supermeeple-sk/freebase/api/session.py",
line 745, in mqlwrite
form=dict(query=qstr))
File "/Users/willmerydith/repos/supermeeple-sk/freebase/api/session.py",
line 442, in _httpreq_json
resp, body = self._httpreq(*args, **kws)
File "/Users/willmerydith/repos/supermeeple-sk/freebase/api/session.py",
line 428, in _httpreq
return self._http_request(url, method, body, headers)
File "/Users/willmerydith/repos/supermeeple-sk/freebase/api/httpclients.py",
line 88, in __call__
self._raise_service_error(url, resp.status_code,
resp.headers['content-type'], resp.body)
AttributeError: '_URLFetchResult' object has no attribute 'body'
Does this mean that those calls are failing to reach Freebase? Or
that Freebase is not sending back a proper Result?

It turns out this is a bug in Freebase-Python. I've filed the bug and offered a solution: http://code.google.com/p/freebase-python/issues/detail?id=15