Develop Reference

docker login --username AWS --password-stdin https://<accountNumber>.dkr.ecr.<region>.amazonaws.com exited with error code 1:

I'm simply adopting the Eventbridge ETL design pattern and it gives me this error when I deploy:
[100%] fail: docker login --username AWS --password-stdin https://315997497220.dkr.ecr.us-west-2.amazonaws.com exited with error code 1:
❌ the-eventbridge-etl failed: Error: Failed to publish one or more
assets. See the error messages above for more information. at
Object.publishAssets
(/home/mubashir/.nvm/versions/node/v16.3.0/lib/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Object.deployStack
(/home/mubashir/.nvm/versions/node/v16.3.0/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:237:3)
at CdkToolkit.deploy
(/home/mubashir/.nvm/versions/node/v16.3.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:194:24)
at initCommandLine
(/home/mubashir/.nvm/versions/node/v16.3.0/lib/node_modules/aws-cdk/bin/cdk.ts:267:9)
Failed to publish one or more assets. See the error messages above for
more information.
The steps I took. Github repo has a video I followed
npx cdkp init the-eventbridge-etl --lang=python
cd the-eventbridge-etl
python3 -m venv .env
source .env/bin/activate
pip install -r requirements.txt
cdk synth
cdk deploy
The first error I get is related to bootstrapping. So I bootstrap.
export CDK_NEW_BOOTSTRAP=1
npx cdk bootstrap aws://315997497220/us-east-2 --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess --trust 315997497220 aws://315997497220/us-east-2
I've naturally updated the cdk.json file for using the above bootstrapping technique. I've tried all bootstrap techniques, with and without a qualifier, and with its subsequent changes to cdk.json. I don't think it's a bootstrap issue.
I get the above error and I don't know what the issue is. I have not made any changes to the code.

I geuss you need to get and pipe a password first as you use the --password-stdin flag. Try:
aws ecr get-login-password | docker login --username AWS --password-stdin https://315997497220.dkr.ecr.us-west-2.amazonaws.com

AWS CDK Python docker throwing invalid bind mount error when trying to bundle code

I'm trying to deploy a python lambda function with dependencies and I'm getting an error from the docker daemon (on Centos linux) that there is an invalid bind mount spec. The error is "/path//to/my/code:/asset-input:z,delegated": invalid mode: delegated
The following is what my code looks like for the lambda function:
python_function = Function(
self,
id="PythonFunction",
runtime=Runtime.PYTHON_3_9,
handler="app.main.lambda_handler",
timeout=Duration.seconds(20),
code=Code.from_asset(
path=str(python_function_path.resolve()),
bundling=BundlingOptions(
image=Runtime.PYTHON_3_9.bundling_image,
command=[
"bash",
"-c",
"pip install -r requirements.txt -t /asset-output && cp -au . /asset-output",
],
),
),
memory_size=128,
log_retention=RetentionDays.TWO_WEEKS,
)
This works just fine on my Mac, but trying to deploy from Centos is unsuccessful.

Your docker version is out of date. You need to be running docker CE at least version 1.17.04 or higher (this was the version when support for delegated mode was added, but ideally you should install a more recent version).
As stated in comments, your current version is 1.13.1, which does not have support for this mode.
To resolve this, you should update your docker version.

pysftp library not working in AWS lambda layer

I want to upload files to EC2 instance using pysftp library (Python script). So I have created small Python script which is using below line to connect
pysftp.Connection(
host=Constants.MY_HOST_NAME,
username=Constants.MY_EC2_INSTANCE_USERNAME,
private_key="./mypemfilelocation.pem",
)
some code here .....
pysftp.put(file_to_be_upload, ec2_remote_file_path)
This script will upload files from my local Windows machine to EC2 instance using .pem file and it works correctly.
Now I want to do this action using AWS lambda with API Gateway functionality.
So I have uploaded Python script to AWS lambda. Now I am not sure how to use pysftp library in AWS lambda, so I found solution that add pysftp library Layer in AWS lambda Layer. I did it with
pip3 install pysftp -t ./library_folder
And I make zip of above folder and added in AWS lambda Layer.
But still I got so many errors like one by one :-
No module named 'pysftp'
No module named 'paramiko'
Undefined Symbol: PyInt_FromLong
cannot import name '_bcrypt' from partially initialized module 'bcrypt' (most likely due to a circular import)
cffi module not found
I just fade up of above errors I didn't find the proper solution. How can I can use pysftp library in my AWS lambda seamlessly?

I build pysftp layer and tested it on my lambda with python 3.8. Just to see import and basic print:
import json
import pysftp
def lambda_handler(event, context):
# TODO implement
print(dir(pysftp))
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
I used the following docker tool to build the pysftp layer:
https://github.com/lambci/docker-lambda
So what I did for pysftp was:
# create pysftp fresh python 3.8 environment
python -m venv pysftp
# activate it
source pysftp/bin/activate
cd pysftp
# install pysftp in the environemnt
pip3 install pysftp
# generate requirements.txt
pip freeze > requirements.txt
# use docker to construct the layer
docker run --rm -v `pwd`:/var/task:z lambci/lambda:build-python3.8 python3.8 -m pip --isolated install -t ./mylayer -r requirements.txt
zip -r pysftp-layer.zip .
And the rest is uploading the zip into s3, creating new layer in AWS console, setting Compatible runtime to python 3.8 and using it in my test lambda function.
You can also check here how to use this docker tool (the docker command I used is based on what is in that link).
Hope this helps

how to create docker-build n Docker file after pushed from dockerhub(Error response from daemon: unexpected error reading Dockerfile)

am new to docker I am unable to run the docker code which is pulled from docker hub
1.Windows 10 pro, 64bit
2.able to run hello-world
3.C:\Program Files\Docker\Docker\Dockerfile>docker version
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:22:37 2019
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:29:19 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
C:\Program Files\Docker\Docker\Dockerfile>docker push mydockerhub/shoprite:latest
pulled successfully
5.PROBLEM
C:\Program Files\Docker\newfile>docker build -t mydockerhub/shoprite:latest .
Sending build context to Docker daemon 4.096kB
Error response from daemon: unexpected error reading Dockerfile: read /var/lib/docker/tmp/docker-builder712136353/Dockerfile: is a directory
6.Error:
I also want used commands: https://docs.google.com/document/d/1aRt71Vtx13IbGLlcMzH-CyESvudLrCAnHg6fG9Zx3eU/edit
Q1.how to create Dockerfile and docker build

You can already see the problem at step 3. You created Dockerfile as a directory. It is a file. Delete the directory and make sure you have a file called Dockerfile in the directory where you run docker build. Make sure that Dockerfile is correctly edited.

If you have a dockerfile located somewhere in a file, first go to that file through commands i.e. cd ./onedrive/desktop/newfolder/docker/.
Then run command docker build . -t <your imagename : your image tag>
Here . gives you a default path where all the files in the local directory get tard and sent to the Docker daemon. -t gives you chance to give your image a name with a tag separated by :.
You then need to use the docker run command i.e. docker run <option/command> imagename.
It will give you a writeable container layer over the specified image, and then starts it using the specified command.
Use command docker ps to see if you have created a container with the image name you used. If not, there is probably a problem with how you are doing it or your pc environment.
P.S. You need to understand that dockerfile is a docker file, not a directory. So, if you see that your current directory is something/something/docker/dockerfile, you made a mistake. The dockerfile goes into docker folder by default.
Good luck

If you are using docker-compose.yml
Just specify Dockerfile name
Wrong
dockerfile: ./
Correct
dockerfile: ./Dockerfile

Serverless - Numpy - Unable to find good bind path format

I've been beating on this for over a week and been through all sorts of forum issues and posts and cannot resolve. I'm trying to package numpy in a function, individually building requirements (I have multiple functions with multiple requirements that I'd like to keep separate).
Environment:
Windows 10 Home
Docker Toolbox for Windows:
Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24302
Built: Fri Mar 23 08:31:36 2018
OS/Arch: windows/amd64
Experimental: false
Orchestrator: swarm
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:52:55 2018
OS/Arch: linux/amd64
Experimental: false
Serverless Version:
serverless version 6.4.1
serverless-python-requirements version 6.4.1
Directory Structure:
|-test
|-env.yml
|-serverless.yml
|-Dockerfile
|-functions
|-f1
|-index.py
|-requirements.txt
|-sub_function_1.py
|-sub_function_2.py
|-f2
|-index.py
|-requirements.txt
|-sub_function_3.py
|-sub_function_4.py
serverless.yml
service: test
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
zip: true
dockerFile: Dockerfile
dockerizePip: non-linux
provider:
name: aws
runtime: python3.6
stage: dev
environment: ${file(./env.yml):${opt:stage, self:provider.stage}.env}
region: ${file(./env.yml):${opt:stage, self:provider.stage}.aws.region}
profile: ${file(./env.yml):${opt:stage, self:provider.stage}.aws.profile}
package:
individually: true
functions:
f1:
handler:index.handler
module:functions/f1
f2:
handler:index.handleer
module:functions/f2
I have my project files in C:\Serverless\test. I run npm init, followed by npm i --save serverless-python-requirements, accepting all defaults. I get the following on sls deploy -v. even though I've added C:\ to Shared Folders on the running default VM in VirtualBox, and selected auto-mount and permanent.
If I comment out both dockerizePip and dockerFile I get the following as expected based on here and other SO posts:
Serverless: Invoke invoke
{
"errorMessage": "Unable to import module 'index'"
}
If I comment out dockerfile I get:
Serverless: Docker Image: lambci/lambda:build-python3.6
Error --------------------------------------------------
error during connect: Get https://XXXXXX/v1.37/version: dial tcp
XXXXXXXXXX: connectex: A connection attempt failed because the
connected party did not properly respond after a period of time, or
established connection failed because connected host has failed to
respond.
at dockerCommand (C:\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:20:11)
at getBindPath (C:\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:100:3)
With Dockerfile
# AWS Lambda execution environment is based on Amazon Linux 1
FROM amazonlinux:1
# Install Python 3.6
RUN yum -y install python36 python36-pip
# Install your dependencies
RUN curl -s https://bootstrap.pypa.io/get-pip.py | python3
RUN yum -y install python3-devel mysql-devel gcc
# Set the same WORKDIR as default image
RUN mkdir /var/task
WORKDIR /var/task
.
Serverless: Building custom docker image from Dockerfile...
Serverless: Docker Image: sls-py-reqs-custom
Error --------------------------------------------------
Unable to find good bind path format
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Stack Trace --------------------------------------------
Error: Unable to find good bind path format
at getBindPath (C:\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:142:9)
at installRequirements (C:\Serverless\test\node_modules\serverless-python-requirements\lib\pip.js:152:7)
at installRequirementsIfNeeded (C:\Serverless\test\node_modules\serverless-python-requirements\lib\pip.js:451:3)
If I move my project to C:\Users\, I get this instead:
Serverless: Docker Image: sls-py-reqs-custom
Serverless: Trying bindPath /c/Users/Serverless/test/.serverless/requirements (run,--rm,-v,/c/Users/Serverless/test/.serverless/req
uirements:/test,alpine,ls,/test/requirements.txt)
Serverless: /test/requirements.txt
Error --------------------------------------------------
docker: Error response from daemon: create "/c/Users/Serverless/test/.serverless/requirements": "\"/c/Users/Serverless/test/.serv
erless/requirements\"" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you in
tended to pass a host directory, use absolute path.
See 'docker run --help'.
For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.
Stack Trace --------------------------------------------
Error: docker: Error response from daemon: create "/c/Users/Serverless/test/.serverless/requirements": "\"/c/Users/Serverless/test/
.serverless/requirements\"" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If y
ou intended to pass a host directory, use absolute path.
See 'docker run --help'.
at dockerCommand (C:\Users\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:20:11)
at getDockerUid (C:\Users\Serverless\test\node_modules\serverless-python-requirements\lib\docker.js:162:14)
I've seen the Makefile style recommendation from #brianz here, but I'm not sure how to adapt that to this (Makefiles are not my strong suit). I'm a bit at a loss as to what to do next and advice would be greatly appreciated. TIA.

I was unable to make the plugin work but I found a better solution anyhow - Lambda Layers. This is a bonus because it reduces the size of the lambda and allows code/file reuse. There is a pre-built lambda layer for numpy and scipy that you can use, but I built my own to show myself how it all works. Here's how I made it work:
Create a layer package:
Open an EC2 instance or Ubuntu or Linux or whatever - This is needed so we can compile the runtime binaries correctly
Make a dependencies package zip - Must use the directory structure python/lib/python3.6/site-packages for python to find during runtime
mkdir -p tmpdir/python/lib/python3.6/site-packages
pip install -r requirements.txt --no-deps -t tmpdir/python/lib/python3.6/site-packages
cd tmpdir zip -r ../py_dependencies.zip .
cd ..
rm -r tmpdir
Push layer zip to AWS - requires latest awscli
sudo pip install awscli --upgrade --user
sudo aws lambda publish-layer-version \
--layer-name py_dependencies \
--description "Python 3.6 dependencies [numpy=0.15.4]" \
--license-info "MIT" \
--compatible-runtimes python3.6 \
--zip-file fileb://py_dependencies.zip \
--profile python_dev_serverless
To use in any function that requires numpy, just use the arn that is shown in the console or during the upload above
f1:
handler: index.handler_f_use_numpy
include:
- functions/f_use_numpy.py
layers:
- arn:aws:lambda:us-west-2:XXXXX:layer:py_dependencies:1
As an added bonus, you can push common files like constants to a layer as well. Here's how I did it for testing use in windows and on the lambda:
import platform
\# Set common path
COMMON_PATH = "../../layers/common/"
if platform.system() == "Linux": COMMON_PATH = "/opt/common/"
def handler_common(event, context):
# Read from a constants.json file
with open(COMMON_PATH + 'constants.json') as f:
return text = json.load(f)

when I got the same issue, I opened docker went to settings/shared drive opted to reset credentials and after applied my changes and this cleared the error

I fixed this issue by temporarily disabling Windows Firewall.