I am trying to parallelize processes using Ray in a docker container.
from ray.util.multiprocessing import Pool
with Pool(self.n_cores) as pool:
pool.starmap(func=func, iterable=list_of_parameters)
while it is perfectly working locally, when it gets run in the docker container, the following error occurs:
✗ failed cod_simulation-run
Job 'cod_simulation-run' failed:
| AttributeError: 'StreamCapture' object has no attribute 'fileno'
Write "details cod_simulation-run" to inspect the error.
Processed 1 jobs (1 failed).
I was previously performing the same thing with python multiprocessing:
import multiprocessing as mp
with mp.Pool(self.n_cores) as pool:
pool.starmap(func=func, iterable=list_of_parameters)
and this worked both locally and in the docker container. But for efficiency reasons, I would prefer to stick to Ray.
FROM python:3.9
WORKDIR /project_name
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
RUN find .
ENV DISABLE_CONTRACTS=1
RUN pipdeptree
RUN python setup.py develop --no-deps
RUN cod-demo --help
CMD ["cod-demo"]
This is my DockerFile and I am installing ray as a requirement.
Thanks in advance for any suggestion
After pulling the appropriate tag based on your needs from their repo on dockerhub. You can simply run tests with:
docker run --shm-size=<shm-size> -t -i rayproject/ray
or
docker run --shm-size=<shm-size> -t -i --gpus all rayproject/ray:<ray-version>-gpu
In case you're using GPU. If you need some additional packages which aren't pre-installed on images you must install them through the terminal inside the image or fabricate a new docker-file with the downloaded image as the base image (the FROM command argument).
According to what was mentioned here!
Related
Im completely new to Docker and I'm trying to create and run a very simple example using instructions defined in a DockerFile.
DockerFile->
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y python3 pip
COPY ./ .
RUN python3 test.py
contents of test.py ->
import pandas as pd
import numpy as np
print('test code')
command being used to create a Docker Container ->
docker build --no-cache . -t intro_to_docker -f abs/path/to/DockerFile
folder structure -> (both files are present at abs/path/to)
abs/path/to:
-DockerFile
-test.py
Error message ->
error from sender: open .Trash: operation not permitted
(using sudo su did not resolve the issue, which i believe is linked to the copy commands)
I'm using a Mac.
any help in solving this will be much appreciated!
The Dockerfile should be inside a folder. Navigate to that folder and then run docker build command. I was also facing the same issue but got resovled when moved the docker file inside a folder
Usually the error would look like:
error: failed to solve: failed to read dockerfile: error from sender: open .Trash: operation not permitted
And in my case, it's clearly saying that it is unable to find the dockerfile.
Also, in your command, I see a . after --no-cache, I think that's not required?
So better, try navigating to the specified path and then run the build command replacing the -f option with a ., which specifies the build command to consider the current folder for its build process.
In your case
cd abs/path/to/
docker build --no-cache -t intro_to_docker .
It seems the system policies are not allowing the application to execute this command. The application "Terminal" might not have approval to access the entire file system.
Enable full disk access to terminal. Change it using "System Preferences > Security & Privacy > Privacy > Full Disk Access"
I had the same error message and my Dockerfile was located in the HOME directory, I moved the Docker file to a different location and executed the docker build from that newly moved location and it successfully executed.
I am using Python with TextBlob for sentiment analysis. I want to deploy my app (build in Plotly Dash) to Google Cloud Run with Google Cloud Build (without using Docker). When using locally on my virtual environment all goes fine, but after deploying it on the cloud the corpora is not downloaded. Looking at the requriements.txt file, there was also no reference to this corpora.
I have tried to add python -m textblob.download_corpora to my requriements.txt file but it doesn't download when I deploy it. I have also tried to add
import textblob
import subprocess
cmd = ['python','-m','textblob.download_corpora']
subprocess.run(cmd)
and
import nltk
nltk.download('movie_reviews')
to my script (callbacks.py, I am using Plotly Dash to make my app), all without success.
Is there a way to add this corpus to my requirements.txt file? Or is there another workaround to download this corpus? How can I fix this?
Thanks in advance!
Vijay
Since Cloud Run creates and destroys containers as needed for your traffic levels you'll want to embed your corpora in the pre-built container to ensure a fast cold start time (instead of downloading it when the container starts)
The easiest way to do this is add another line inside of a docker file that downloads and installs the corpora at build time like so:
RUN python -m textblob.download_corpora
Here's a full docker file for your reference:
# Python image to use.
FROM python:3.8
# Set the working directory to /app
WORKDIR /app
# copy the requirements file used for dependencies
COPY requirements.txt .
# Install any needed packages specified in requirements.txt
RUN pip install --trusted-host pypi.python.org -r requirements.txt
RUN python -m textblob.download_corpora
# Copy the rest of the working directory contents into the container at /app
COPY . .
# Run app.py when the container launches
ENTRYPOINT ["python", "app.py"]
Good luck,
Josh
I am running a docker-compose file with 3 services (or containers):
django-web
nginx
spacy-web
My final goal is to create 2 nlp objects (nlp_en, nlp_fr) in spacy-web and make use of them in django-web.
Does that look possible/desirable?
Of course, the reason behind splitting django-web and spacy-web is to have a cleaner scheme of containers.
I have created a Dockerfile for spacy-web.
I install spaCy and 2 language models (English/French).
With an entrypoint (which runs a python script: spacy_entrypoint.py), I then create 2 nlp objects, 1 for each language (nlp_en, nlp_fr).
My first problem is that this container exits immediately after creating the nlp objects. My guess is that I should add something at the end of the python script, to keep the container running, but without using much CPU. How do we keep a python container alive, to access objects created into it from another container?
If that problem can be solved, then I need to import the nlp objects (nlp_en, nlp_fr) in the other container, namely django-web. I never got there because spacy-web exits immediately after creation. In the relevant django views, I would try "from spacy-web import nlp_en, nlp_fr". Has anyone tried an python import from one Docker container to another Docker container?
Dockerfile.spacy
FROM python:3.7.4
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
RUN pip3 install --upgrade pip
RUN pip3 install -U spacy
RUN python3 -m spacy download en_core_web_lg
RUN python3 -m spacy download fr_core_news_md
ENV updated_at 2019-09-06
EXPOSE 4000
RUN mkdir -p /temp
WORKDIR /temp
COPY compose_spacy/spacy_entrypoint.py /temp
ENTRYPOINT ["python3", "spacy_entrypoint.py"]
spacy_entrypoint.py
import spacy
nlp_en = spacy.load('en_core_web_lg')
nlp_fr = spacy.load('fr_core_news_md')
When I build an image and run a container based on those files, the container exits immediately. I'd be extremely grateful for any tip pointing to the right direction !
I would suggest starting simple and just having everything in one application and one container.
I have a linux server running in which I want to be able to run some python scripts. To do so, I created a docker image of python (3.6.8) with some specific dependencies to run my code.
I am new to linux command line and need help on how to write a line that would run a given python script based on my docker (python 3.6.8)
My server's directory structure looks like this :
My docker is named geomatique_python and its image is located in docker_image.
As for the structure of the code itself, I am starting from scratch and am looking for some hints and advices.
Thanks
I'm very much for a all the things in docker approach. Ref. your mention of having specific versions set in stone, the declaritive nature of docker is great for that. You can extend an official python docker image with your libraries then bind-mount the folders into your container during the run. A minimal project might look like:
.
├── app.py
└── Dockerfile
My app.py is a simple requests script:
#!/usr/bin/env python3
import requests
r = requests.get('https://api.github.com')
if r.status_code == 200:
print("HTTP {}".format(r.status_code))
My Dockerfile contains the runtime dependencies for my app:
FROM python:3.6-slim
RUN python3 -m pip install requests
Note: I'm extending the official python image in this example.
After building the docker image (i.e. docker build --rm -t so:57697538 .) you can run a container from the image bind-mounting the directory that contains the scripts inside the container and execute them: docker run --rm -it -v ${PWD}:/src --entrypoint python3 so:57697538 /src/app.py
Admittedly for python virtualenv / virtualenvwrapper can be convenient however it's very much python-only whereas docker is language agnostic.
I have a simple Python program that I want to run in IBM Cloud functions. Alas it needs two libraries (O365 and PySnow) so I have to Dockerize it and it needs to be able to accept a Json feed from STDIN. I succeeded in doing this:
FROM python:3
ADD requirements.txt ./
RUN pip install -r requirements.txt
ADD ./main ./main
WORKDIR /main
CMD ["python", "main.py"]
This runs with: cat env_var.json | docker run -i f9bf70b8fc89
I've added the Docker container to IBM Cloud Functions like this:
ibmcloud fn action create e2t-bridge --docker [username]/e2t-bridge
However when I run it, it times out.
Now I did see a possible solution route, where I dockerize it as an Openwhisk application. But for that I need to create a binary from my Python application and then load it into a rather complicated Openwhisk skeleton, I think?
But having a file you can simply run was is the whole point of my Docker, so to create a binary of an interpreted language and then adding it into a Openwhisk docker just feels awfully clunky.
What would be the best way to approach this?
It turns out you don't need to create a binary, you just need to edit the OpenWhisk skeleton like so:
# Dockerfile for example whisk docker action
FROM openwhisk/dockerskeleton
ENV FLASK_PROXY_PORT 8080
### Add source file(s)
ADD requirements.txt /action/requirements.txt
RUN cd /action; pip install -r requirements.txt
# Move the file to
ADD ./main /action
# Rename our executable Python action
ADD /main/main.py /action/exec
CMD ["/bin/bash", "-c", "cd actionProxy && python -u actionproxy.py"]
And make sure that your Python code accepts a Json feed from stdin:
json_input = json.loads(sys.argv[1])
The whole explaination is here: https://github.com/iainhouston/dockerPython