Develop Reference

Using Ray Multiprocessing on Docker

I am trying to parallelize processes using Ray in a docker container.
from ray.util.multiprocessing import Pool
with Pool(self.n_cores) as pool:
pool.starmap(func=func, iterable=list_of_parameters)
while it is perfectly working locally, when it gets run in the docker container, the following error occurs:
✗ failed cod_simulation-run
Job 'cod_simulation-run' failed:
| AttributeError: 'StreamCapture' object has no attribute 'fileno'
Write "details cod_simulation-run" to inspect the error.
Processed 1 jobs (1 failed).
I was previously performing the same thing with python multiprocessing:
import multiprocessing as mp
with mp.Pool(self.n_cores) as pool:
pool.starmap(func=func, iterable=list_of_parameters)
and this worked both locally and in the docker container. But for efficiency reasons, I would prefer to stick to Ray.
FROM python:3.9
WORKDIR /project_name
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
RUN find .
ENV DISABLE_CONTRACTS=1
RUN pipdeptree
RUN python setup.py develop --no-deps
RUN cod-demo --help
CMD ["cod-demo"]
This is my DockerFile and I am installing ray as a requirement.
Thanks in advance for any suggestion

After pulling the appropriate tag based on your needs from their repo on dockerhub. You can simply run tests with:
docker run --shm-size=<shm-size> -t -i rayproject/ray
or
docker run --shm-size=<shm-size> -t -i --gpus all rayproject/ray:<ray-version>-gpu
In case you're using GPU. If you need some additional packages which aren't pre-installed on images you must install them through the terminal inside the image or fabricate a new docker-file with the downloaded image as the base image (the FROM command argument).
According to what was mentioned here!

AWS CDK Python docker throwing invalid bind mount error when trying to bundle code

I'm trying to deploy a python lambda function with dependencies and I'm getting an error from the docker daemon (on Centos linux) that there is an invalid bind mount spec. The error is "/path//to/my/code:/asset-input:z,delegated": invalid mode: delegated
The following is what my code looks like for the lambda function:
python_function = Function(
self,
id="PythonFunction",
runtime=Runtime.PYTHON_3_9,
handler="app.main.lambda_handler",
timeout=Duration.seconds(20),
code=Code.from_asset(
path=str(python_function_path.resolve()),
bundling=BundlingOptions(
image=Runtime.PYTHON_3_9.bundling_image,
command=[
"bash",
"-c",
"pip install -r requirements.txt -t /asset-output && cp -au . /asset-output",
],
),
),
memory_size=128,
log_retention=RetentionDays.TWO_WEEKS,
)
This works just fine on my Mac, but trying to deploy from Centos is unsuccessful.

Your docker version is out of date. You need to be running docker CE at least version 1.17.04 or higher (this was the version when support for delegated mode was added, but ideally you should install a more recent version).
As stated in comments, your current version is 1.13.1, which does not have support for this mode.
To resolve this, you should update your docker version.

SyntaxError: invalid syntax to repo init in the AOSP code

I have tried to repo init the source code Ubuntu build machine and it is successfully able to clone the code.
repo init -u git#github.com:xxx/xx_manifest.git -b xxx
Now I am trying repo init the source code in VM Ubuntu machine.
In between getting the error like below:
Traceback (most recent call last):
File "/xxx/.repo/repo/main.py", line 56, in <module>
from subcmds.version import Version
File "/xxx/.repo/repo/subcmds/__init__.py", line 38, in <module>
['%s' % name])
File "/xxx/.repo/repo/subcmds/upload.py", line 27, in <module>
from hooks import RepoHook
File "/xxx/.repo/repo/hooks.py", line 472
file=sys.stderr)
^
SyntaxError: invalid syntax
python version is same in build machine and vm machine 2.7.17.

try these commands
curl https://storage.googleapis.com/git-repo-downloads/repo-1 > ~/bin/repo
chmod a+x ~/bin/repo
python3 ~/bin/repo init -u git#....

I just had the same issue and this resolved it for me :
Download last version of repo :
curl https://storage.googleapis.com/git-repo-downloads/repo-1 > repo
Change right to make it executable : chmod a+x repo
Run your repo init with python3 and the "repo" you just download : python3 repo init -u git#github.com:xxx/xx_manifest.git -b xxx

I have experienced the same issue on Ubuntu 18.04 during the installation of the OpenSTLinux Yocto layer with the following command:
repo init -u https://github.com/STMicroelectronics/oe-manifest.git -b refs/tags/openstlinux-5.4-dunfell-mp1-20-11-12
Returns:
Get https://gerrit.googlesource.com/git-repo/clone.bundle
Get https://gerrit.googlesource.com/git-repo
remote: Counting objects: 2, done
remote: Finding sources: 100% (117/117)
remote: Total 117 (delta 63), reused 117 (delta 63)
Receiving objects: 100% (117/117), 142.25 KiB | 11.85 MiB/s, done.
Resolving deltas: 100% (63/63), completed with 32 local objects.
From https://gerrit.googlesource.com/git-repo
1469c28..0588f3d main -> origin/main
* [new tag] v2.11 -> v2.11
* [new tag] v2.11.1 -> v2.11.1
Traceback (most recent call last):
File "/home/xxx/Distribution-Package/openstlinux-5.4-dunfell-mp1-20-11-12/.repo/repo/main.py", line 56, in <module>
from subcmds.version import Version
File "/home/xxx/Distribution-Package/openstlinux-5.4-dunfell-mp1-20-11-12/.repo/repo/subcmds/__init__.py", line 38, in <module>
['%s' % name])
File "/home/xxx/Distribution-Package/openstlinux-5.4-dunfell-mp1-20-11-12/.repo/repo/subcmds/upload.py", line 27, in <module>
from hooks import RepoHook
File "/home/xxx/Distribution-Package/openstlinux-5.4-dunfell-mp1-20-11-12/.repo/repo/hooks.py", line 472
file=sys.stderr)
This issue goes away with using Python3 instead of Python (2.7). You can do this:
sudo rm /usr/bin/python
sudo ln -s /usr/bin/python3 /usr/bin/python

Try below command to make it will work 100%, tried & suggested
mkdir -p ~/.bin
PATH="${HOME}/.bin:${PATH}"
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/.bin/repo
chmod a+rx ~/.bin/repo

One solution is to modify the first line of /usr/bin/repo and change it from
#!/usr/bin/python
to
#!/usr/bin/python3
This asks the system to use Python3 instead of the default Python.

If the system you are running on doesn't have python3, like in my case, and you don't have the option to install python3, or installing it breaks other parts, the option is the degrade repo to a version that uses python2.7:
- git clone https://gerrit.googlesource.com/git-repo
- cd git-repo
- git reset --hard v1.13.11
- mkdir -p ~/.bin
- PATH="${HOME}/.bin:${PATH}"
- cp repo ~/.bin/repo
- chmod a+rx ~/.bin/repo
This will use v1.13.11 of repo, which works with python2.7

As seen in a similar error in arvestad/alv issue 1, that would be consistent with running the process with Python 2.7 instead of Python3
Double-check you Python version between:
your Ubuntu build machine (where the repo init works)
your VM Ubuntu machine (where the repo init fails)
Same error here, with the error suggesting you are executing python2 with a PYTHONPATH that's only appropriate for python3.

I don't know exactly how this works, but i just had the same issue and this resolved it for me it seems.
https://source.android.com/setup/develop#installing-repo
Dont use the legacy one, use the first one to resolve it.
Edit: It seems you also need to have python 3.6 installed on you system to have this work. You can still have update-alternatives point to python 2.7, you simply need 3.6 or newer installed.

Just install python3 and the latest repo.
I encountered the problem too, but on Mac OS. The log is exactly the same as yours.
Definitely python2 caused this problem. repo is try to run python3 files in python2 environment.
I found the this from repo docs https://gerrit.googlesource.com/git-repo/+/refs/heads/master/docs/python-support.md
So I update my repo (which located in depot_tools). Since I already have python3 installed, everything is ok now.
Hope my experience may help you.

Docker and fcntl OSError Errno 22 Invalid argument

I've encountered a weird problem and I do not know how to proceed.
I have docker 18.09.2, build 6247962 on a VMware ESXi 6.5 virtual machine running Ubuntu 18.04. I have docker 19.03.3, build a872fc2f86 on a Azure virtual machine running Ubuntu 18.04. I have the following little test script that I run on both hosts and in different docker containers:
#!/usr/bin/python3
import fcntl
import struct
image_path = 'foo.img'
f_obj = open(image_path, 'rb')
binary_data = fcntl.ioctl(f_obj, 2, struct.pack('I', 0))
bsize = struct.unpack('I', binary_data)[0]
print('bsize={0}'.format(bsize))
exit(0)
I run "ps -ef >foo.img" to get the foo.img file. The output of the above script on both virtual machines is bsize=4096.
I have the following Dockerfile on both VMs:
FROM ubuntu:19.04
RUN apt-get update && \
apt-get install -y \
python \
python3 \
vim
WORKDIR /root
COPY testfcntl01.py foo.img ./
RUN chmod 755 testfcntl01.py
If I create a docker image with the above Dockerfile on the VM running docker 18.09.2, the above gives me the same results as the host.
If I create a docker image with the above Dockerfile on the VM running docker 19.03.3, the above gives me the following error:
root#d317404714a6:~# ./testfcntl01.py
Traceback (most recent call last):
File "./testfcntl01.py", line 9, in <module>
binary_data = fcntl.ioctl(f_obj, 2, struct.pack('I', 0))
OSError: [Errno 22] Invalid argument
I compared the docker directory structure, the daemon.json file, the logs, the "docker info" between the hosts. They look to be identical. I tried with a FROM ubuntu:18.04 as well as ubuntu:19.04. I've tried with python2 as well as python3. Same results.
I do not know why the fcntl fails only on a docker container on the Azure VM running docker 19.03.3. Did something change in docker between 18 and 19 that might have caused this? Is there some configuration change that I need to make to get this to work? Something else I'm missing?
Any help would be greatly appreciated.
Thank you
Lewis Muhlenkamp
UPDATE01:
I was following the steps here to prepare my own custom Ubuntu 18.04 VHD to use in Azure. I started with a generic install of Ubuntu Server 18.04 using ubuntu-18.04.3-live-server-amd.iso that I just downloaded from Ubuntu's website. The test below works just fine on that freshly intalled VM. I finish the step
sudo apt-get install linux-generic-hwe-18.04 linux-cloud-tools-generic-hwe-18.04
and then my test fails. So, I believe there is some issue with these hardware enablement packages.

I had a pretty similar error and found that if the file is in a mounted volume, at least owned by the host, it won't fail. Ie:
docker run -it -v $PWD:/these_work ubuntu:18.04 bash
Files under the /these_work directory in the container worked, however other files that were solely accessible from within the container resulted in [Errno 22] Invalid Argument.
I came here from a yocto build error from a nearly identical method of accessing the blocksize within filemap.py:
# Get the block size of the host file-system for the image file by calling
# the FIGETBSZ ioctl (number 2).
try:
binary_data = fcntl.ioctl(file_obj, 2, struct.pack('I', 0))
except OSError:
raise IOError("Unable to determine block size")

How to run docker command in Python SDK

I am using python SDK package to run docker from python.
Here is the docker command I tried to run using python package:
docker run -v /c/Users/msagovac/pdf_ocr:/home/docker jbarlow83/ocrmypdf-polyglot --skip-text 0ce9d58432bf41174dde7148486854e2.pdf output.pdf
Here is a python code:
import docker
client = docker.from_env()
client.containers.run('jbarlow83/ocrmypdf-polyglot', '--skip-text "0ce9d58432bf41174dde7148486854e2.pdf" "output.pdf"', "-v /c/Users/msagovac/pdf_ocr:/home/docker")
Error says file ot found. I am not sure where to set run options:
-v /c/Users/msagovac/pdf_ocr:/home/docker

Try with named parameters:
client.containers.run(
image='jbarlow83/ocrmypdf-polyglot',
command='--skip-text "0ce9d58432bf41174dde7148486854e2.pdf" "output.pdf"',
volumes={'/c/Users/msagovac/pdf_ocr': {'bind': '/home/docker', 'mode': 'rw'}},
)
Also it seems that the path of the volume to mount is incorrect, try with C:/Users/msagovac/pdf_ocr