Develop Reference

Return Data from Lambda backed custom resource CDK

It cannot be this hard, I have a custom resource class and for the life of me cannot figure out how to access the data and move on.
from typing_extensions import runtime
import os
from aws_cdk import (
core as cdk, aws_iam as iam, aws_lambda as lamb, aws_route53 as route53, aws_logs as logs,
custom_resources as cr
)
import builtins
from jsii import python
class hz_with_delegation(cdk.Construct):
def __init__(self, scope: cdk.Construct, id: builtins.str, delSet: builtins.str, env: builtins.str) -> builtins.any:
super().__init__(scope, id)
self.hostedZoneLambdaRole = iam.Role(
self, "hzlambdarole", assumed_by=iam.ServicePrincipal('lambda.amazonaws.com'))
self.hostedZoneLambdaRole.add_to_policy(
iam.PolicyStatement(resources=["*"], actions=["route53:*"])
)
self.hzResourceLambda = lamb.Function(self, "hz_custom_resource_handler",
runtime=lamb.Runtime.PYTHON_3_8,
handler="main.on_event",
role=self.hostedZoneLambdaRole,
code=lamb.Code.from_asset(
os.path.dirname(__file__) + "/lambda"),
log_retention=logs.RetentionDays.ONE_DAY
)
self.hz_with_delegation_provider = cr.Provider(self, "hz_with_delegation_set",
on_event_handler=self.hzResourceLambda,
)
self.cr = cdk.CustomResource(self, "hz_with_delegation",
service_token=self.hz_with_delegation_provider.service_token,
properties={
"delSet": delSet,
"env": env
})
I cannot figure out how to get any information out of this sucker once it's made
class CdkPoCStack(cdk.Stack):
def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
# Custom resource to handle the delegation
test = custom_hosted_zone.hz_with_delegation(
self, id="some Id", delSet="N06079012Q8954F8DG91Y", env=environment)
Ultimately I want the physical resourceID, the way I'm going about it is passing it back in the provider lambda in the Data field of the response. In order to get the data I need to do a getAttr, in order to do that I need the logicalID
Cannot for the life of me figure out how to get either the physical or more importantly the logical ID of a custom resource construct.
I tried this
self.get_logical_id(test)
and I get this error
Traceback (most recent call last):
File "/Users/sahm.samarghandi/bbot/CDK_PoC/app.py", line 16, in <module>
CdkPoCStack(app, "DjangoAppStack",
File "/Users/sahm.samarghandi/bbot/CDK_PoC/.venv/lib/python3.9/site-packages/jsii/_runtime.py", line 83, in __call__
inst = super().__call__(*args, **kwargs)
File "/Users/sahm.samarghandi/bbot/CDK_PoC/cdk_poc/cdk_poc_stack.py", line 35, in __init__
print(self.get_logical_id(test.cr))
File "/Users/sahm.samarghandi/bbot/CDK_PoC/.venv/lib/python3.9/site-packages/aws_cdk/core/__init__.py", line 15945, in get_logical_id
return typing.cast(builtins.str, jsii.invoke(self, "getLogicalId", [element]))
File "/Users/sahm.samarghandi/bbot/CDK_PoC/.venv/lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 128, in wrapped
return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
File "/Users/sahm.samarghandi/bbot/CDK_PoC/.venv/lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 340, in invoke
response = self.provider.invoke(
File "/Users/sahm.samarghandi/bbot/CDK_PoC/.venv/lib/python3.9/site-packages/jsii/_kernel/providers/process.py", line 359, in invoke
return self._process.send(request, InvokeResponse)
File "/Users/sahm.samarghandi/bbot/CDK_PoC/.venv/lib/python3.9/site-packages/jsii/_kernel/providers/process.py", line 326, in send
raise JSIIError(resp.error) from JavaScriptError(resp.stack)
jsii.errors.JSIIError: Object of type #aws-cdk/core.CustomResource is not convertible to #aws-cdk/core.CfnElement

According to the documentation, you should be able to get the physical id using the ref attribute of the custom resource.
However, if you want the logical id, you'll need to use the defaultChild attribute on the node of the custom resource you're creating (since you're using a second-level construct). Once you get that default child you should be able to get the logical id from the id attribute.

TypeError: issubclass() arg 1 must be a class, while I am quite sure arg 1 is a class

Somewhere within my code I have the following line of code.
from inspect import isclass
if isclass(route.handler) and issubclass(route.handler, web.View):
Unfortunately this line of code gives the exception below in my production environment.
TypeError: issubclass() arg 1 must be a class
As far as I know, the Python (3.7.7) compiler will first check the first condition of the if statement and if this evaluates to false, it will not check the second condition. Therefore I must conclude that route.handler must be a class, and therefore the TypeError I am getting should not be occurring. Am I missing something here? Does someone know what might be causing this?
(Unfortunately I am not able to reproduce the error)
edit:
The error originates from the swagger-aiohttp package. Here's the entire traceback:
Traceback (most recent call last):
File "/var/www/app/main.py", line 249, in <module>
run_app(cfg)
File "/var/www/app/main.py", line 225, in run_app
setup_swagger(app, ui_version=SWAGGER_VERSION, swagger_url=SWAGGER_URL)
File "/home/webapp/.venv/lib/python3.7/site-packages/aiohttp_swagger/__init__.py", line 72, in setup_swagger
security_definitions=security_definitions
File "/home/webapp/.venv/lib/python3.7/site-packages/aiohttp_swagger/helpers/builders.py", line 163, in generate_doc_from_each_end_point
end_point_doc = _build_doc_from_func_doc(route)
File "/home/webapp/.venv/lib/python3.7/site-packages/aiohttp_swagger/helpers/builders.py", line 44, in _build_doc_from_func_doc
if isclass(route.handler) and issubclass(route.handler, web.View):
File "/home/webapp/.venv/lib/python3.7/abc.py", line 143, in __subclasscheck__
return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class
edit2:
The route.handler should be an aiohttp class-based view. For example this is how one would create one and build a swagger UI on top of that.
class PingHandler(web.View):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
async def get(request):
"""
---
description: This end-point allow to test that service is up.
tags:
- Health check
produces:
- text/plain
responses:
"200":
description: successful operation. Return "pong" text
"405":
description: invalid HTTP Method
"""
return web.Response(text="pong")
app = web.Application()
app.router.add_route('GET', "/ping", PingHandler)
setup_swagger(app, swagger_url="/api/v1/doc", ui_version=3)
In my current implementation I also have a decorator added to the Handler class.
edit3:
When debugging locally (where it's working fine), the route.handler seems to be a <class 'abc.ABCMeta'>.

I finally discovered the problem. The error is raised whenever a decorator from the wrapt library is used together with a abc.ABCMeta class. This is currently an open issue for the wrapt library. An example is shown below:
import abc
from inspect import isclass
import wrapt
class Base:
pass
class BaseWithMeta(metaclass=abc.ABCMeta):
pass
#wrapt.decorator
def pass_through(wrapped, instance, args, kwargs):
return wrapped(*args, **kwargs)
#pass_through
class B(BaseWithMeta):
pass
#pass_through
class C(Base):
pass
isclass(C)
>>>
True
issubclass(C, Base)
>>>
True
isclass(B)
>>>
True
issubclass(B, BaseWithMeta)
>>>
TypeError: issubclass() arg 1 must be a class

gae mapreduce generator error no attribute validate_bucket_name

This is my first GAE project. I got my serial code to work on the dev_app (I am using the GoogleAppEngineLauncher on Mac). Since my code takes too long to finish I am trying to use mapreduce to speed up the process. I tried the following code but keep getting the following error. I am not sure if this is because of some error in my code or if I am missing any statements in the *yaml files. Kindly help!
class ShuffleDictPipeline(base_handler.PipelineBase):
def run(self, *args, **kwargs):
""" run """
mapper_params = {
"entity_kind": "coremic.RandomDict",
"batch_size": 500,
"filters": [("idx", "=", ndb_custom_key)]
}
reducer_params = {
"mime_type": "text/plain"
}
output = yield mapreduce_pipeline.MapreducePipeline(
"calc_shuff_core_microb",
mapper_spec="coremic.shuffle_dict_coremic_map",
mapper_params=mapper_params,
reducer_spec="coremic.shuffle_dict_coremic_reduce",
reducer_params=reducer_params,
input_reader_spec="mapreduce.input_readers.DatastoreInputReader",
output_writer_spec="mapreduce.output_writers.BlobstoreOutputWriter",
shards=16)
yield StoreOutput(output)
Error:
ERROR 2016-03-05 20:03:21,706 pipeline.py:2432]
Generator mapreduce.mapper_pipeline.MapperPipeline(*(u'calc_shuff_core_microb-map', u'coremic.shuffle_dict_coremic_map', u'mapreduce.input_readers.DatastoreInputReader'), **{'output_writer_spec': u'mapreduce.output_writers._GoogleCloudStorageKeyValueOutputWriter', 'params': {u'batch_size': 500, u'bucket_name': u'app_default_bucket', u'entity_kind': u'coremic.RandomDict',... (324 bytes))#b96dd511c0454fd99413d267b7388857 raised exception. AttributeError: 'NoneType' object has no attribute 'validate_bucket_name'
Traceback (most recent call last):
File "/Users/rr/GAE/coremic/pipeline/pipeline.py", line 2156, in evaluate
self, pipeline_key, root_pipeline_key, caller_output)
File "/Users/rr/GAE/coremic/pipeline/pipeline.py", line 1110, in _run_internal
return self.run(*self.args, **self.kwargs)
File "/Users/rr/GAE/coremic/mapreduce/mapper_pipeline.py", line 102, in run
queue_name=self.queue_name,
File "/Users/rr/GAE/coremic/mapreduce/control.py", line 125, in start_map
in_xg_transaction=in_xg_transaction)
File "/Users/rr/GAE/coremic/mapreduce/handlers.py", line 1730, in _start_map
mapper_output_writer_class.validate(mapper_spec)
File "/Users/rr/GAE/coremic/mapreduce/output_writers.py", line 1075, in validate
return cls.WRITER_CLS.validate(mapper_spec)
File "/Users/rr/GAE/coremic/mapreduce/output_writers.py", line 723, in validate
super(_GoogleCloudStorageOutputWriter, cls).validate(mapper_spec)
File "/Users/rr/GAE/coremic/mapreduce/output_writers.py", line 604, in validate
cloudstorage.validate_bucket_name(
AttributeError: 'NoneType' object has no attribute 'validate_bucket_name'

I am still working on getting everything to work, but couple of things helped.
1.1 Install google cloud storage client lib on SDK to access the bucket. cloud google com appengine docs python googlecloudstorageclient
1.2 Set up (create) the bucket.
Then follow steps from https://plus.google.com/+EmlynORegan/posts/6NPaRKxMkf3
Note how the mapper params has changed.
2 - In mapreduce pipelines, replace
"mapreduce.output_writers.BlobstoreOutputWriter"
with
"mapreduce.output_writers.GoogleCloudStorageConsistentOutputWriter"
3 - update reducer params to:
{
"mime_type": "text/plain",
"output_writer": {
"bucket_name": ,
"tmp_bucket_name":
}
}
Other very useful link:
https://gist.github.com/nlathia/ab670053ed460c4ca02f/89178e132b894fe5467c09164d3827f70e4ae2f8

You can do 1 of 2 things. Either
Create a google cloud storage bucket associated with your project, because at the moment none is associated with it, hence the NoneType. Once done, you can add that to your mapper_params.
mapper_params = {
...
"bucket_name": "<your google cloud storage bucket name>",
...
}
OR
Create a default bucket by visiting your app engine's application settings in the cloud console https://console.cloud.google.com/appengine/settings?project=

Install GoogleAppEngineCloudStorageClient in your project.
output_writes.py does the following:
try:
# Check if the full cloudstorage package exists. The stub part is in runtime.
cloudstorage = None
import cloudstorage
if hasattr(cloudstorage, "_STUB"):
cloudstorage = None
# "if" is needed because apphosting/ext/datastore_admin:main_test fails.
if cloudstorage:
from cloudstorage import cloudstorage_api
from cloudstorage import errors as cloud_errors
except ImportError:
pass # CloudStorage library not available
So, when importing cloudstorage fails, the value of cloudstorage variable = None. And that causes the exception later.

i am trying to connect mysql using bottle.py, but it is showing error

I followed this link : "https://pypi.python.org/pypi/bottle-mysql/0.1.1"
and "http://bottlepy.org/docs/dev/"
this is my py file:
import bottle
from bottle import route, run, template
import bottle_mysql
app = bottle.Bottle()
# # dbhost is optional, default is localhost
plugin = bottle_mysql.Plugin(dbuser='root', dbpass='root', dbname='delhipoc')
app.install(plugin)
#route('/hai/<name>')
def show(name,dbname):
dbname.execute('SELECT id from poc_people where name="%s"', (name))
print "i am in show"
return template('<b>Hello {{name}}</b>!',name=name)
run(host='localhost', port=8080)
this is my code and it is throwing error like:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\bottle.py", line 764, i
return route.call(**args)
File "C:\Python27\lib\site-packages\bottle.py", line 1575,
rv = callback(*a, **ka)
TypeError: show() takes exactly 2 arguments (1 given)
please help me

Not familiar with bottle-mysql, but in the example you provided:
#app.route('/show/:<tem>')
In your code:
#route('/hai/<name>')
It might expect:
#route('/hai/:<name>')

Simple. Change this line:
def show(name,dbname):
to this:
def show(name, delhipoc):
Better yet, use dbname = 'db' and then
def show(name, db):
The MySQL plugin has chosen an unfortunate name for its db name param. It would've been more clear to call it something like db_parameter_name, because what it actually refers to is the name of the db parameter in the decorated Python function.

PermanentTaskFailure: 'module' object has no attribute 'Migrate'

I'm using Nick Johnson's Bulk Update library on google appengine (http://blog.notdot.net/2010/03/Announcing-a-robust-datastore-bulk-update-utility-for-App-Engine). It works wonderfully for other tasks, but for some reason with the following code:
from google.appengine.ext import db
from myapp.main.models import Story, Comment
import bulkupdate
class Migrate(bulkupdate.BulkUpdater):
DELETE_COMPLETED_JOBS_DELAY = 0
DELETE_FAILED_JOBS = False
PUT_BATCH_SIZE = 1
DELETE_BATCH_SIZE = 1
MAX_EXECUTION_TIME = 10
def get_query(self):
return Story.all().filter("hidden", False).filter("visible", True)
def handle_entity(self, entity):
comments = entity.comment_set
for comment in comments:
s = Story()
s.parent_story = comment.story
s.user = comment.user
s.text = comment.text
s.submitted = comment.submitted
self.put(s)
job = Migrate()
job.start()
I get the following error in my logs:
Permanent failure attempting to execute task
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 258, in post
run(self.request.body)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 122, in run
raise PermanentTaskFailure(e)
PermanentTaskFailure: 'module' object has no attribute 'Migrate'
It seems quite bizarre to me. Clearly that class is right above the job, they're in the same file and clearly the job.start is being called. Why can't it see my Migrate class?
EDIT: I added this update job in a newer version of the code, which isn't the default. I invoke the job with the correct URL (http://version.myapp.appspot.com/migrate). Is it possible this is related to the fact that it isn't the 'default' version served by App Engine?

It seems likely that your declaration of the 'Migrate' class is in the handler script (Eg, the one directly invoked by app.yaml). A limitation of deferred is that you can't use it to call functions defined in the handler module.
Incidentally, my bulk update library is deprecated in favor of App Engine's mapreduce support; you should probably use that instead.