I am trying to create a container that splits files and then uploads them to S3 container. The process works as intended, but when trying to send the file to S3 it fails with the error Unable to locate credentials inside fargate container.
My dockerfile looks like this:
FROM python:3.8-slim
RUN apt-get update \
&& apt-get install -y wget \
&& rm -rf /var/lib/apt/lists/*
RUN mkdir /tmp/splitter
RUN pip install --upgrade pip && \
pip install --no-cache-dir requests boto3
RUN wget -O /tmp/init.sh (WHATEVER) \
&& chmod +x /tmp/init.sh
CMD /tmp/init.sh
I have my role set up like the ecsTaskExecutionRole that appears in Amazon's documentation.
ecsTaskExecutionRole is not for your container to access S3. It is for ECS itself to be able to, e.g. pull your docker image from ECR.
For your application permissions in the container, you need a task role, not the task execution role. It can be confusing because both are named similarly and both have same trust policy.
The task role can be specified along task execution role:
The roles can also be set at task definition level:
Related
To enable the webdriver in my google cloud function, I created a custom container using a docker file:
FROM python:3.7
COPY . /
WORKDIR /
RUN pip3 install -r requirements.txt
RUN apt-get update
RUN apt-get install -y gconf-service libasound2 libatk1.0-0 libcairo2 libcups2 libfontconfig1 libgdk-pixbuf2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libxss1 fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils
#download and install chrome
RUN wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
RUN dpkg -i google-chrome-stable_current_amd64.deb; apt-get -fy install
#install python dependencies
COPY requirements.txt requirements.txt
RUN pip install -r ./requirements.txt
# Downloading gcloud package
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz
# Installing the package
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
# Adding the package path to local
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
#some envs
ENV PORT 5000
#copy local files
COPY . .
CMD exec gunicorn --bind :${PORT} --workers 1 --threads 8 main:app
ENTRYPOINT ["webcrawler"]
I installed gcloud in this docker so that I will be able to use gcloud deploy to deploy my cloud functions. Then, I deploy my script using this cloudbuild.yaml:
steps:
- name: 'us-central1-docker.pkg.dev/$PROJECT_ID/webcrawler-repo/webcrawler:tag1'
entrypoint: 'gcloud'
args: ['functions', 'deploy', 'MY_FUN', '--trigger-topic=MY_TOPIC', '--runtime=python37', '--entry-point=main', '--region=us-central1', '--memory=512MB', '--timeout=540s']
id: 'deploying MY_FUN'
dir: 'MY_DIR'
However, I end up getting this error for my deployment:
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Build failed: invalid storage source object "MY_FUN-ba7acf95-4297-46b3-b76e-1c25ba21ba03/version-14/function-source.zip" in bucket "gcf-sources-967732204245-us-central1": failed to get storage object: Get "https://storage.googleapis.com/storage/v1/b/gcf-sources-967732204245-us-central1/o/MY_FUN-ba7acf95-4297-46b3-b76e-1c25ba21ba03%2Fversion-14%2Ffunction-source.zip?alt=json&prettyPrint=false": RPC::UNREACHABLE: gslb: no reachable backends
ERROR
ERROR: build step 0 "us-central1-docker.pkg.dev/PROJECT_ID/webcrawler-repo/webcrawler:tag1" failed: step exited with non-zero status: 1
Any idea how to resolve this issue?
Thanks!
Cloud functions allows you to deploy only your code. The packaging into a container, with buildpack, is performed automatically for you.
If you have already a container, the best solution is to deploy it on Cloud Run. If your webserver listen on the port 5000, don't forget to override this value during the deployment (use --port parameter).
To plug your PubSub topic to your Cloud Run service, you have 2 solutions
Either manually, you create a PubSub push subscription to your Cloud Run service
Or you can use EventArc to plug it to your Cloud Run service
In both cases, you need to take care of the security by using a service account with the role run.invoker on the Cloud Run service that you pass to PubSub push subscription or to EventArc
I'm trying to make a simple MS SQL Server call from Python by using Docker. The SQL connection is able to establish if I run the python execute script, but it won't work from Docker.
My code is below
Dockerfile
from python:3
WORKDIR /code
COPY requirements.txt .
RUN apt-get update \
&& apt-get install unixodbc -y \
&& apt-get install unixodbc-dev -y \
&& apt-get install freetds-dev -y \
&& apt-get install freetds-bin -y \
&& apt-get install tdsodbc -y \
&& apt-get install --reinstall build-essential -y
RUN echo "[FreeTDS]\n\
Description = FreeTDS Driver\n\
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so\n\
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so" >> /etc/odbcinst.ini
#Pip command without proxy setting
RUN pip install -r requirements.txt
COPY src/ .
CMD ["python", "./producer.py"]
producer.py
import pyodbc
connP = pyodbc.connect('driver={FreeTDS};'
'server={MYSERV01\SQLEXPRESS};'
'database=ABCD;'
'uid=****;'
'pwd=*****')
requirement.txt
kafka-python
pyodbc==4.0.28
Error message
I referred to this article and did. I searched online for resolutions and tried several steps, but nothing helped. I'm pretty new to Docker and no experience in Python, so any help would be really good. Thanks in advance!
In your pyodbc.connect try to give the server as '0.0.0.0' instead of any other value.
If you want to debug it from inside the container, then comment the last CMD line of your Dockerfile.
Build your Docker container
docker build -f Dockerfile -t achu-docker-container .
Run your Docker Container
docker run -it achu-docker-container /bin/bash
This will place you inside the container. This is like, ssh to a different machine.
Go to your WORKDIR
cd code
python ./producer.py
What do you get the above above? (If you install any editor using apt-get install vim you will be able to interactively edit the producer.py file and fix your problem from inside the running container.
Then you can move your changes to your source Dockerfile and build a new image and container with it.
I was trying to connect to the local SQL Server database. I referred many articles and figured out that the following code works:
the server should have host.docker.inter,<port_no> -- this was the catch. When it comes to dedicated database where the sql server is different from the docker image, the server name is provided directly, but when both image and database are in same server, the following code works. Please check the port number in the SQL Configuration TCP Address (IP4All)
I'm trying to configure a Kubernetes job to run a set of bash and python scripts that contains some AWS CLI commands.
Is there a good image out there for doing that? Do I need to create a custom docker image? I just want a container with these tools installed, what's the easiest way of getting there?
Easiest would be getting any image from Docker hub containing AWS CLI, for example woahbase/alpine-awscli.
You can use it in a following way kubectl run aws-cli --image=woahbase/alpine-awscli
This would create a pod names aws-cli which would contain following image.
You would need to upload your scripts to the pod or mount a storage Configure a Pod to Use a PersistentVolume for Storage.
Keep in mind this is not recommended as this image does not belong to you and you have no idea if there were no changes before checking.
I would create my own Docker hub repo and build my own image, something like the following:
FROM alpine:3.6
RUN apk -v --update add \
python \
py-pip \
groff \
less \
mailcap \
&& \
pip install --upgrade awscli==1.14.5 s3cmd==2.0.1 python-magic && \
apk -v --purge del py-pip && \
rm /var/cache/apk/*
VOLUME /root/.aws
VOLUME /project
WORKDIR /project
ENTRYPOINT ["aws"]
I am starting with docker and built an image with jupyter and some python libraries. The end user should be able to use jupyter and access specific host data directories throught the container (read/write rights), but must be a non-root user. Here is my dockerfile so far:
FROM ubuntu:latest
RUN apt-get update && apt-get install -y \
python-pip
RUN pip install --upgrade pip && pip install jupyter \
pandas \
numpy
RUN useradd -r -g users A && \
mkdir /myhome && \
chown -R A:users /myhome
EXPOSE 8888
WORKDIR /myhome
CMD ["jupyter", "notebook", "--port=8888", "--no-browser", "--ip=0.0.0.0"]
I run this by doing docker run -it -p 8888:8888 -u="A" -v /some/host/files:/myhome
But then I got a jupyter error that says OSError: [Errno 13] Permission denied: '/home/A'
Any help appreciated. Many thanks!
When you start your container with --entrypoint=bash, you will find that the home directory /home/A of your user has not been created. To do that, you need to add the -m flag to the useradd command
Some more info: You might want to take a look at the docker-stacks projects (https://github.com/jupyter/docker-stacks/tree/master/base-notebook and derived images). That seems to match with what you're trying to do and adds some other helpful stuff. E.g. when running a dockerized jupyter, you need a "PID 1 reaper"; otherwise your exited notebook kernels turn into zombies (you can google for that :-)
Also, when sharing host files with a non-root user inside the container, you will often need to set the UID of your container user to some specific value matching with the host system, so the file system permissions are properly matched. The docker-stacks containers support that too. Their Dockerfiles might at least help as a boilerplate to run your own.
I have a scrapy project run continously by cron hosted inside a docker image.
When I run and deploy this locally everything works fine. If I try to deploy the same to AWS I get the following error inside the logs:
No EXPOSE directive found in Dockerfile, abort deployment (ElasticBeanstalk::ExternalInvocationError)
The console shows that my container was build correctly but I can not use it without an EXPOSED port.
INFO: Successfully pulled python:2.7
WARN: Failed to build Docker image aws_beanstalk/staging-app, retrying...
INFO: Successfully built aws_beanstalk/staging-app
ERROR: No EXPOSE directive found in Dockerfile, abort deployment
ERROR: [Instance: i-6eebaeaf] Command failed on instance. Return code: 1 Output: No EXPOSE directive found in Dockerfile, abort deployment.
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
But why is it not possible?
My Dockerfile looks like the following:
FROM python:2.7
MAINTAINER XDF
ENV DIRECTORY /opt/the-flat
# System
##########
RUN apt-get update -y && apt-get upgrade -y && apt-get install -y ntp vim apt-utils
WORKDIR $DIRECTORY
# GIT
##########
# http://stackoverflow.com/questions/23391839/clone-private-git-repo-with-dockerfile
RUN apt-get install -y git
RUN mkdir /root/.ssh/
ADD deploy/git-deply-key /root/.ssh/id_rsa
RUN chmod 0600 /root/.ssh/id_rsa
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan -t rsa bitbucket.org >> /root/.ssh/known_hosts
RUN ssh -T -o 'ConnectionAttempts=1' git#bitbucket.org
RUN git clone --verbose git#bitbucket.org:XDF/the-flat.git .
# Install
##########
RUN pip install scrapy
RUN pip install MySQL-python
# not working
# apt-get install -y wkhtmltopdf && pip install pdfkit
# else
# https://pypi.python.org/pypi/pdfkit
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y openssl build-essential xorg libssl-dev
RUN wget http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.10.0_rc2-static-amd64.tar.bz2
RUN tar xvjf wkhtmltopdf-0.10.0_rc2-static-amd64.tar.bz2
RUN chown root:root wkhtmltopdf-amd64
RUN mv wkhtmltopdf-amd64 /usr/bin/wkhtmltopdf
RUN pip install pdfkit
# Cron
##########
# http://www.ekito.fr/people/run-a-cron-job-with-docker/
# http://www.corntab.com/pages/crontab-gui
RUN apt-get install -y cron
RUN crontab "${DIRECTORY}/deploy/crontab"
CMD ["cron", "-f"]
It's by design. You need to have an EXPOSE port directive in your Dockerfile to tell beanstalk what port your app will be listening on. Do you have a usecase where you cannot or do not want to have EXPOSE in your Dockerfile?
ElasticBeanstalk is designed for web applications, hence the EXPOSE requirement. The use case you demonstrated is that of a jobs (workers) server, which Elastic Beanstalk doesn't handle well.
For your case, either expose a dummy port number or launch an EC2 instance yourself to bypass the EB overload.