Passing arguments to a Docker container running a Python script can be done like so
docker run my_script:0.1 --arg1 val --arg2 val ...
I can't seem to figure out how to pass those arguments when running the container on AWS Fargate (perhaps it doesn't work?)
In ecs, you will run your containers as tasks. So you will first register the task that includes your container definition and then you can run the task passing your arguments as environment variables.
Here is an example task definition:
myscript-task.json: (a sample task definition)
{
"containerDefinitions": [
{
"name": "myscript",
"image": "12345123456.dkr.ecr.us-west-2.amazonaws.com/myscript:0.1",
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group" : "/ecs/fargate",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "myscript"
}
}
}
],
"family": "myscript",
"networkMode": "awsvpc",
"executionRoleArn": "arn:aws:iam::12345123456:role/ecsTaskExecutionRole",
"cpu": "256",
"memory": "512",
"requiresCompatibilities": [
"FARGATE"
]
}
You will register the task in the console or with the register-task-definition command:
aws ecs register-task-definition --cli-input-json file://myscript-task.json
You can now run the task with ecs run-task command. Using overrides parameter you will be able to run the same task with different values.
aws ecs run-task --cluster testCluster --launch-type FARGATE --task-definition myscript:1 --network-configuration 'awsvpcConfiguration={subnets=[subnet-0abcdec237054abc],assignPublicIp=ENABLED}' --overrides file://overrides.json
A sample Overrides.json:
{
"containerOverrides": [{
"name": "myscript",
"environment": [{
"name": "VAR1",
"value": "valueOfVar1"
}]
}]
}
Now you can access the variable in your python script.
Python script(sample) printing the passed environment variable.
import os
print(os.environ['VAR1'])
With log driver configured, you will be able to see the output in cloudwatch logs.
You can use container definitions parameters in ECS task definition to pass runtime arguments.
Command parameter maps to COMMAND parameter in docker run.
"command": [
"--arg1",
"val",
"--arg2",
"val"
],
It is also possible to pass parameters as environment variables.
"environment": [
{
"name": "LOG_LEVEL",
"value": "debug"
}
],
Related
I've got a python pipeline which takes a file from Cloud Storage, removes some columns, then uploads the result to BigQuery. If I run this locally using the Dataflow runner, everything works as expected, but whenever I try to set it up with the Dataflow UI so I can schedule it etc. no job gets created and this
INVALID_ARGUMENT error gets thrown in the logs:
(Ids etc. removed)
{
"insertId": "...",
"jsonPayload": {
"url": "https://datapipelines.googleapis.com/v1/projects/.../locations/europe-west1/pipelines/cloudpipeline:run",
"jobName": "projects/.../locations/europe-west1/jobs/...",
"#type": "type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished",
"targetType": "HTTP",
"status": "INVALID_ARGUMENT"
},
"httpRequest": {
"status": 400
},
"resource": {
"type": "cloud_scheduler_job",
"labels": {
"location": "europe-west1",
"project_id": "...",
"job_id": "..."
}
},
"timestamp": "2022-11-01T10:45:00.657145402Z",
"severity": "ERROR",
"logName": "projects/.../logs/cloudscheduler.googleapis.com%2Fexecutions",
"receiveTimestamp": "2022-11-01T10:45:00.657145402Z"
}
I can't find out anything about this error and gcp doesn't seem to provide any additional info.
I've tried to use gcp logging in the python code but the gcp error seems to get thrown before any code is executed. I've also removed all of my optional parameters so there shouldn't be anything you're required to enter in the gcp set-up that isn't default.
I am pretty new to azure, and struggling with the python function triggers from eventGrid.
I am using ready templates created from azure for python and getting errors.
I will share the files.
(init.py)
import json
import logging
import azure.functions as func
def main(event: func.EventGridEvent):
result = json.dumps(
{
"id": event.id,
"data": event.get_json(),
"topic": event.topic,
"subject": event.subject,
"event_type": event.event_type,
}
)
logging.info("Python EventGrid trigger processed an event: %s", result)
function.json
{
"bindings": [
{
"type": "eventGridTrigger",
"name": "event",
"direction": "in"
}
],
"disabled": false,
"scriptFile": "__init__.py"
}
host.json
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[2.*, 3.0.0)"
}
}
and the that's the dataset that i sent to event grid
{
"topic": "/subscriptions/{subscriptionID}/resourcegroups/{resourceGroupName}/providers/Microsoft.EventHub/namespaces/event",
"subject": "eventhubs/test",
"eventType": "captureFileCreated",
"eventTime": "2017-07-14T23:10:27.7689666Z",
"id": "{id}",
"data": {
"fileUrl": "https://test.blob.core.windows.net/debugging/testblob.txt",
"fileType": "AzureBlockBlob",
"partitionId": "1",
"sizeInBytes": 0,
"eventCount": 0,
"firstSequenceNumber": -1,
"lastSequenceNumber": -1,
"firstEnqueueTime": "0001-01-01T00:00:00",
"lastEnqueueTime": "0001-01-01T00:00:00"
},
"dataVersion": "",
"metadataVersion": "1"
}
and the error that i am getting is
fail: Function.{functionName}[3]
Executed 'Functions.{functionName}' (Failed, Id={someID}, Duration=121ms)
Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.{functionName}
System.InvalidOperationException: Exception binding parameter 'event'
System.NotImplementedException: The method or operation is not implemented.
probably it is super easy mistake somewhere above but i couldn't find it..
Thanks in advance!
python fail: Function.{functionName}[3] Executed 'Functions.{functionName}' (Failed, Id={someID}, Duration=121ms) Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.{functionName} System.InvalidOperationException: Exception binding parameter 'event' System.NotImplementedException: The method or operation is not implemented.
The above error is quite a common generic error message. It appears when your application hits some limitations of the Event grid.
If It is reaching the maximum hits of limitations of Event Grid refer to this
Note: Event Grid triggers aren't natively supported in an internal load balancer App Service Environment. The trigger uses an HTTP request that can't reach the function app without a gateway into the virtual network.
If you are using a consumption plan try to use the dedicated plan to avoid this issue.
Refer here to reliable event processing to relate this error.
Check the limitations of Event Grid and choose the appropriate plan to for your requirement.
Reference:
azure function fails with function invocation exception
I encountered multiple issues regarding the publishing / deployment of functions when using Python Azure Functions running on Linux and Premium Plan. Following are options what can be done in cases where it fails or it is successful but the function (on Azure) does not reflect what should have been published / deployed.
The following options may also work for non-Linux / non-Python / non-Premium Plan Function (Apps).
Wait some minutes after the publishing so that the Function (App) reflects the update
Restart the Function App
Make sure that the following AppSettings are set under "Configuration" (please adjust to your current context)
[
{
"name": "AzureWebJobsStorage",
"value": "<KeyVault reference to storage account connection string>",
"slotSetting": false
},
{
"name": "ENABLE_ORYX_BUILD",
"value": "true",
"slotSetting": false
},
{
"name": "FUNCTIONS_EXTENSION_VERSION",
"value": "~3",
"slotSetting": false
},
{
"name": "FUNCTIONS_WORKER_RUNTIME",
"value": "python",
"slotSetting": false
},
{
"name": "SCM_DO_BUILD_DURING_DEPLOYMENT",
"value": "true",
"slotSetting": false
},
{
"name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
"value": "<storage account connection string>",
"slotSetting": false
},
{
"name": "WEBSITE_CONTENTSHARE",
"value": "<func app name>",
"slotSetting": false
}
]
When using Azure DevOps Pipelines use the standard Azure Function task (https://github.com/Microsoft/azure-pipelines-tasks/blob/master/Tasks/AzureFunctionAppV1/README.md) to publish the function respectively set the AppSettings.
This task also works for Python even if it does not explicitly provide the option under "Runtime Stack" (just leave it empty).
Make sure to publish the correct files (if you publish via ZipDeploy the zip folder should contain host.json at its root)
You can check if the correct files haven been published via checking the wwwroot folder via the Azure Portal -> Function App -> Development Tools -> SSH
cd /home/site/wwwroot
dir
Check the deployment logs
Either via the link displayed as output during the deployment
Should look like "https://func-app-name.net/api/deployments/someid/log"
Via Development Tools -> Advanced Tools
If the steps so far did not help it can help to SSH to the host via the portal (Development Tools -> SSH) and to delete
# The deployments folder (and then republish)
cd /home/site
rm -r deployments
# The wwwroot folder (and then republish)
cd /home/site
rm -r wwwroot
Delete the Function App resource and redeploy it
tl;dr
Environment variables set in a zappa_settings.json don't upload as environment variables to AWS Lambda. Where do they go?
ts;wm
I have a Lambda function configured, deployed and managed using the Zappa framework. In the zappa_settings.json I have set a number of environment variables. These variables are definitely present as my application successfully runs however, when trying to inspect the Lambda function environment variables in the console or AWS CLI I see no environment variables have been uploaded to the Lambda function itself.
Extract from zappa_settings.json:
{
"stage-dev": {
"app_function": "project.app",
"project_name": "my-project",
"runtime": "python3.7",
"s3_bucket": "my-project-zappa",
"slim_handler": true,
"environment_variables": {
"SECRET": "mysecretvalue"
}
}
}
Output of aws lambda get-function-configuration --function-name my-project-stage-dev:
{
"Configuration": {
"FunctionName": "my-project-stage-dev",
"FunctionArn": "arn:aws:lambda:eu-west-1:000000000000:function:my-project-stage-dev",
"Runtime": "python3.7",
"Role": "arn:aws:iam::000000000000:role/lambda-execution-role",
"Handler": "handler.lambda_handler",
"CodeSize": 12333025,
"Description": "Zappa Deployment",
"Timeout": 30,
"MemorySize": 512,
"LastModified": "...",
"CodeSha256": "...",
"Version": "$LATEST",
"TracingConfig": {
"Mode": "PassThrough"
},
"RevisionId": "..."
},
"Code": {
"RepositoryType": "S3",
"Location": "..."
}
}
Environment is absent from the output despite being included in the zappa_settings and the AWS documentation indicating it should be included if present, this is confirmed by checking in the console. I want to know where zappa is uploading the environment variables to, and if possible why it is doing so over using Lambda's in-built environment?
AWS CLI docs:
https://docs.aws.amazon.com/cli/latest/reference/lambda/get-function-configuration.html
environment_variables are saved in zappa_settings.py when creating a package for deployment (run zappa package STAGE and explore the archive) and are then dynamically set as environment variables by modifying os.environ in handler.py.
To set native AWS variables you need to use aws_environment_variables.
Is there a way to dynamically get the version tag from my __init__.py file and append it to the dockerrun.aws.json image name for example::
{
"AWSEBDockerrunVersion": "1",
"Authentication": {
"Bucket": "dockerkey",
"Key": "mydockercfg"
},
"Image": {
"Name": "comp/app:{{version}}",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "80"
}
]
}
This when when I do eb deploy it will build the correct version. At the moment I have to keep modifying the json file with each deploy.
I also stumbled upon that last year, where AWS support stated there's no such feature at hand. I ended up writing a script that receives the docker tag as parameter and composes the dockerrun.aws.json file on the fly with the correct tagname.
I've written a bash script which runs
eb deploy
Before it executes that command I change a symlink depending on if I'm running production or staging. For Example:
ln -sf ../ebs/Dockerrun.aws.$deploy_type.json ../ebs/Dockerrun.aws.json