Develop Reference

CDK WAF Python Multiple Statement velues error

I have AWS WAF CDK that is working with rules, and now I'm trying to add a rule in WAF with multiple statements, but I'm getting this error:
Resource handler returned message: "Error reason: You have used none or multiple values for a field that requires exactly one value., field: STATEMENT, parameter: Statement (Service: Wafv2, Status Code: 400, Request ID: 6a36bfe2-543c-458a-9571-e929142f5df1, Extended Request ID: null)" (RequestToken: b751ae12-bb60-bb75-86c0-346926687ea4, HandlerErrorCode: InvalidRequest)
My Code:
{
'name': 'ruleName',
'priority': 3,
'statement': {
'orStatement': {
'statements': [
{
'iPSetReferenceStatement': {
'arn': 'arn:myARN'
}
},
{
'iPSetReferenceStatement': {
'arn': 'arn:myARN'
}
}
]
}
},
'action': {
'allow': {}
},
'visibilityConfig': {
'sampledRequestsEnabled': True,
'cloudWatchMetricsEnabled': True,
'metricName': 'ruleName'
}
},

There are two things going on there:
Firstly, your capitalization is off. iPSetReferenceStatement cannot be parsed and creates an empty statement reference. The correct key is ipSetReferenceStatement.
However, as mentioned here, there is a jsii implementation bug causing some issues with the IPSetReferenceStatementProperty. This causes it not to be parsed properly resulting in a jsii error when synthesizing.
You can fix it by using the workaround mentioned in the post.
Add to your file containing the construct:
import jsii
from aws_cdk import aws_wafv2 as wafv2 # just for clarity, you might already have this imported
#jsii.implements(wafv2.CfnRuleGroup.IPSetReferenceStatementProperty)
class IPSetReferenceStatement:
#property
def arn(self):
return self._arn
#arn.setter
def arn(self, value):
self._arn = value
Then define your ip reference statement as follows:
ip_set_ref_stmnt = IPSetReferenceStatement()
ip_set_ref_stmnt.arn = "arn:aws:..."
ip_set_ref_stmnt_2 = IPSetReferenceStatement()
ip_set_ref_stmnt_2.arn = "arn:aws:..."
Then in the rules section of the webacl, you can use it as follows:
...
rules=[
{
'name': 'ruleName',
'priority': 3,
'statement': {
'orStatement': {
'statements': [
wafv2.CfnWebACL.StatementProperty(
ip_set_reference_statement=ip_set_ref_stmnt
),
wafv2.CfnWebACL.StatementProperty(
ip_set_reference_statement=ip_set_ref_stmnt_2
),
]
}
},
'action': {
'allow': {}
},
'visibilityConfig': {
'sampledRequestsEnabled': True,
'cloudWatchMetricsEnabled': True,
'metricName': 'ruleName'
}
}
]
...
This should synthesize your stack as expected.

Trigger python script on EMR from Lambda on S3 Object arrival with Object details [duplicate]

I want to execute spark submit job on AWS EMR cluster based on the file upload event on S3. I am using AWS Lambda function to capture the event but I have no idea how to submit spark submit job on EMR cluster from Lambda function.
Most of the answers that i searched talked about adding a step in the EMR cluster. But I do not know if I can add add any step to fire "spark submit --with args" in the added step.

You can, I had to same thing last week!
Using boto3 for Python (other languages would definitely have a similar solution) you can either start a cluster with the defined step, or attach a step to an already up cluster.
Defining the cluster with the step
def lambda_handler(event, context):
conn = boto3.client("emr")
cluster_id = conn.run_job_flow(
Name='ClusterName',
ServiceRole='EMR_DefaultRole',
JobFlowRole='EMR_EC2_DefaultRole',
VisibleToAllUsers=True,
LogUri='s3n://some-log-uri/elasticmapreduce/',
ReleaseLabel='emr-5.8.0',
Instances={
'InstanceGroups': [
{
'Name': 'Master nodes',
'Market': 'ON_DEMAND',
'InstanceRole': 'MASTER',
'InstanceType': 'm3.xlarge',
'InstanceCount': 1,
},
{
'Name': 'Slave nodes',
'Market': 'ON_DEMAND',
'InstanceRole': 'CORE',
'InstanceType': 'm3.xlarge',
'InstanceCount': 2,
}
],
'Ec2KeyName': 'key-name',
'KeepJobFlowAliveWhenNoSteps': False,
'TerminationProtected': False
},
Applications=[{
'Name': 'Spark'
}],
Configurations=[{
"Classification":"spark-env",
"Properties":{},
"Configurations":[{
"Classification":"export",
"Properties":{
"PYSPARK_PYTHON":"python35",
"PYSPARK_DRIVER_PYTHON":"python35"
}
}]
}],
BootstrapActions=[{
'Name': 'Install',
'ScriptBootstrapAction': {
'Path': 's3://path/to/bootstrap.script'
}
}],
Steps=[{
'Name': 'StepName',
'ActionOnFailure': 'TERMINATE_CLUSTER',
'HadoopJarStep': {
'Jar': 's3n://elasticmapreduce/libs/script-runner/script-runner.jar',
'Args': [
"/usr/bin/spark-submit", "--deploy-mode", "cluster",
's3://path/to/code.file', '-i', 'input_arg',
'-o', 'output_arg'
]
}
}],
)
return "Started cluster {}".format(cluster_id)
Attaching a step to an already running cluster
As per here
def lambda_handler(event, context):
conn = boto3.client("emr")
# chooses the first cluster which is Running or Waiting
# possibly can also choose by name or already have the cluster id
clusters = conn.list_clusters()
# choose the correct cluster
clusters = [c["Id"] for c in clusters["Clusters"]
if c["Status"]["State"] in ["RUNNING", "WAITING"]]
if not clusters:
sys.stderr.write("No valid clusters\n")
sys.stderr.exit()
# take the first relevant cluster
cluster_id = clusters[0]
# code location on your emr master node
CODE_DIR = "/home/hadoop/code/"
# spark configuration example
step_args = ["/usr/bin/spark-submit", "--spark-conf", "your-configuration",
CODE_DIR + "your_file.py", '--your-parameters', 'parameters']
step = {"Name": "what_you_do-" + time.strftime("%Y%m%d-%H:%M"),
'ActionOnFailure': 'CONTINUE',
'HadoopJarStep': {
'Jar': 's3n://elasticmapreduce/libs/script-runner/script-runner.jar',
'Args': step_args
}
}
action = conn.add_job_flow_steps(JobFlowId=cluster_id, Steps=[step])
return "Added step: %s"%(action)

AWS Lambda function python code if you want to execute Spark jar using spark submit command:
from botocore.vendored import requests
import json
def lambda_handler(event, context):
headers = { "content-type": "application/json" }
url = 'http://ip-address.ec2.internal:8998/batches'
payload = {
'file' : 's3://Bucket/Orchestration/RedshiftJDBC41.jar
s3://Bucket/Orchestration/mysql-connector-java-8.0.12.jar
s3://Bucket/Orchestration/SparkCode.jar',
'className' : 'Main Class Name',
'args' : [event.get('rootPath')]
}
res = requests.post(url, data = json.dumps(payload), headers = headers, verify = False)
json_data = json.loads(res.text)
return json_data.get('id')

Specifying a Custom Date Range in Googleads API TargetingIdeaService

I am trying to use the Googleads API TargetingIdeaService to plan budget and get forecast for a list of keywords, as in Keyword Planner:
Keyword Planner Screenshot
I want to specify the date range as next year (for example, 20180301~20190228), but I can't find any parameters or syntax that allow me to do so. The MonthlySearchVolumeAttribute is for selecting data for the past 12 months, so that doesn't seem to work for the situation.
Below is my python script for defining the selector:
selector = {
'searchParameters': [
{
'xsi_type': 'RelatedToQuerySearchParameter',
'queries': keyword_list
},
{
# Location setting (optional)
'xsi_type': 'LocationSearchParameter',
'locations': [{'id': '2840'}] #US
},
{
# Language setting (optional).
'xsi_type': 'LanguageSearchParameter',
'languages': [{'id': '1000'}] #English
},
{
# Network search parameter (optional)
'xsi_type': 'NetworkSearchParameter',
'networkSetting': {
'targetGoogleSearch': True,
'targetSearchNetwork': False,
'targetContentNetwork': False,
'targetPartnerSearchNetwork': True
}
}
],
'ideaType': 'KEYWORD',
'requestType': 'STATS',
'requestedAttributeTypes': ['KEYWORD_TEXT', 'SEARCH_VOLUME','COMPETITION','AVERAGE_CPC'],
# Configure the selector's Paging to limit the number of results returned by a single request
'paging': {
'startIndex': str(offset),
'numberResults': str(PAGE_SIZE)
},
}
Is there anything I can do to achieve my goal? Thanks!

i want place order virtual server with mutiple quantity ,how i can generate order container

Hi all i have used softlayer api using python
i need place virtual server with multiple quantity ,how generate order container ??
i have like this,
import SoftLayer
client = SoftLayer.Client(username='XXXXXX',api_key='xxxxxx')
vmorderparmers = {
'hostname':'testhost',
'domain': 'exampledomain.com',
'datacenter': 'sjc01',
'startCpus':1,
'maxMemory': 1024,
'localDiskFlag': True,
'hourlyBillingFlag': True,
'operatingSystemReferenceCode':'CENTOS_6_64',
"blockDevices": [
{
"device": "0",
"diskImage": {
"capacity": 100
}
}
]
}
oc = client['Virtual_Guest'].generateOrderTemplate(vmorderparmers)
after i have check qunatity
oc['quantity']
give one how i change that
suppose i change quantity like this
oc['quantity']=2
result=client['Product_Order'].placeOrder(oc)
i got error invalid order contianer

There are many ways to order several virtual servers, in your case the method SoftLayer_Virtual_Guest::generateOrderTemplate returns the container type SoftLayer_Container_Product_Order_Virtual_Guest that can be send to the method SoftLayer_Product_Order::verifyOrder or SoftLayer_Product_Order::placeOrder.
If you look the example for Virtual Servers in the SoftLayer_Product_Order::placeOrder page. You'll notice that you need to modify the quantity AND virtualGuest parameters as following.
'virtualGuests': [
{
'domain': 'exampledomain.com',
'hostname': 'testhost1'
},
{
'domain': 'exampledomain.com',
'hostname': 'testhost2'
},
{
'domain': 'exampledomain.com',
'hostname': 'testhost3'
}
],
'quantity': 3
Then your python code should looks like this:
"""
Create a VSI using the simplified way.
The script creates an order template by using the method generateOrderTemplate(), we'll
modify the response in order to create several virtual servers.
See below for more information.
Important manual pages:
http://sldn.softlayer.com/reference/services/SoftLayer_Virtual_Guest/generateOrderTemplate/
http://sldn.softlayer.com/reference/datatypes/SoftLayer_Container_Product_Order_Virtual_Guest
http://sldn.softlayer.com/reference/datatypes/SoftLayer_Container_Product_Order
http://sldn.softlayer.com/reference/services/SoftLayer_Product_Order/verifyOrder/
http://sldn.softlayer.com/reference/services/SoftLayer_Product_Order/placeOrder/
http://sldn.softlayer.com/reference/datatypes/SoftLayer_Virtual_Guest
License: http://sldn.softlayer.com/article/License
Author: SoftLayer Technologies, Inc. <sldn#softlayer.com>
"""
import SoftLayer
# Your SoftLayer API username.
USERNAME = 'set me'
API_KEY = 'set me'
# To get the configuration options to create the server call the
# SoftLayer_Virtual_Guest::getCreateObjectOptions method.
vmorderparmers = {
'hostname': 'testhost',
'domain': 'exampledomain.com',
'datacenter': {'name': 'sjc01'},
'startCpus': 1,
'maxMemory': 1024,
'localDiskFlag': True,
'hourlyBillingFlag': True,
'operatingSystemReferenceCode': 'CENTOS_6_64',
'blockDevices': [
{
'device': '0',
'diskImage': {'capacity': 100}
}
]
}
# Declare the API client
client = SoftLayer.create_client_from_env(username=USERNAME, api_key=API_KEY)
try:
container = client['SoftLayer_Virtual_Guest'].generateOrderTemplate(vmorderparmers)
# In order to create several VSI we modify the following parameters in the response
container['quantity'] = 3
# If you set quantity greater than 1 then you need to define
# hostname/domain per virtual server you wish to order.
container['virtualGuests'] = [
{
'domain': 'exampledomain.com',
'hostname': 'testhost1'
},
{
'domain': 'exampledomain.com',
'hostname': 'testhost2'
},
{
'domain': 'exampledomain.com',
'hostname': 'testhost3'
}
]
"""
Now we are able to verify or place the order.
verifyOrder() will check your order for errors. Replace this with a call to
placeOrder() when you're ready to order. Both calls return a receipt object
that you can use for your records.
"""
receipt = client['SoftLayer_Product_Order'].verifyOrder(container)
print (receipt)
except SoftLayer.SoftLayerAPIError as e:
"""
If there was an error returned from the SoftLayer API then bomb out with the
error message.
"""
print('Unable to create the VSI. %s, %s ' % (e.faultCode, e.faultString))
I hope this help you.

google adwords python api - how to get ad group bid

I am using the adwords python api. I need to get the bid amount and type. E.g. bid=4 ad type = cpc.
I am given the adgroup id.
Below is an example on to create and ad group. Once created...how do I retrieve the settings? How do I get e.g. the bid I set?
ad_group_service = client.GetService('AdGroupService', version='v201402')
operations = [{
'operator': 'ADD',
'operand': {
'campaignId': campaign_id,
'name': 'Earth to Mars Cruises #%s' % uuid.uuid4(),
'status': 'ENABLED',
'biddingStrategyConfiguration': {
'bids': [
{
'xsi_type': 'CpcBid',
'bid': {
'microAmount': '1000000'
},
}
]
}
}
}]
ad_groups = ad_group_service.mutate(operations)

Have a look at the corresponding example on googlads's github page.
Basically you'll user the AdGroupService's get method with a selector containing the right fields and predicates to retrieve an AdGroupPage containing the AdGroup objects you're interested in:
selector = {
'fields': ['Id', 'Name', 'CpcBid'],
'predicates': [
{
'field': 'Id',
'operator': 'EQUALS',
'values': [given_adgroup_id]
}
]
}
page = adgroup_service.get(selector)
adgroup = page.entries[0]
print('Adgroup "%s" (%s) has CPC %s' % (adgroup.name, adgroup.id,
adgroup.biddingStrategyConfiguration.bids.bid))
The available fields' names and the attributes they populate in the returned objects can be found at the selector reference page.
The AdGroupService's reference page might also be of interest.