Deployed a flask application and binded it to the ssl certificate to run on "https:" with the following code:
if __name__ == '__main__':
path = "/usr/local/nginx/ssl/"
context = (path + 'abc.crt' , path + 'abc.key')
app.run_server(debug=True,host='0.0.0.0',ssl_context=context)
Now when I run this script directly through python (python scriptname.py), it works fine,
However when I run in inside the docker container ,I get the following error:
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 1005, in inner
fd=fd,
File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 848, in make_server
host, port, app, request_handler, passthrough_errors, ssl_context, fd=fd
File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 766, in __init__
self.socket = ssl_context.wrap_socket(sock, server_side=True)
File "/usr/local/lib/python3.6/site-packages/werkzeug/serving.py", line 661, in wrap_socket
**kwargs
File "/usr/local/lib/python3.6/ssl.py", line 1158, in wrap_socket
ciphers=ciphers)
File "/usr/local/lib/python3.6/ssl.py", line 750, in __init__
self._context.load_cert_chain(certfile, keyfile)
FileNotFoundError: [Errno 2] No such file or directory
I guess the container is searching for the file elsewhere, this is my docker run command:
docker run -it --network="host" -p 8050:8050 -v /home/a/b/c:/app abc:1.1
What am i missing here?
Edit : Dockerfile
FROM python:3.6
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
ENTRYPOINT ["python"]
CMD ["app.py"]
Docker container will only be able to access what you copied inside it or what your mapped to it while running.
So you have 2 options. First option is to add a COPY statement to copy the certs, but looking at the current docker file, your certs were outside the app folder and hence not copied.
Other option is to use -v option to map the certs while running the container.
docker run -it --network="host" -p 8050:8050 -v /home/certs/path:/home/certs/path -v /home/a/b/c:/app abc:1.1
But in a production like environment I would suggest you don't do this. You should use a nginx and uwsgi and make sure the your terminate the SSL at nginx
See the below repo for such an option
https://github.com/tiangolo/uwsgi-nginx-flask-docker
Related
I have built a test app to better understand the concepts of FastAPI and sqlalchemy
I want to be that app able to provide a docker-compose.yml that can be used to run my solution by just
doing docker-compose up
But I am getting
Building al_test
Traceback (most recent call last):
File "docker-compose", line 3, in <module>
File "compose\cli\main.py", line 67, in main
File "compose\cli\main.py", line 126, in perform_command
File "compose\cli\main.py", line 1070, in up
File "compose\cli\main.py", line 1066, in up
File "compose\project.py", line 615, in up
File "compose\service.py", line 362, in ensure_image_exists
File "compose\service.py", line 1125, in build
File "site-packages\docker\api\build.py", line 160, in build
File "site-packages\docker\utils\build.py", line 30, in tar
File "site-packages\docker\utils\build.py", line 49, in exclude_paths
File "site-packages\docker\utils\build.py", line 214, in rec_walk
File "site-packages\docker\utils\build.py", line 214, in rec_walk
File "site-packages\docker\utils\build.py", line 184, in rec_walk
PermissionError: [WinError 5] Отказано в доступе: '\\\\?\\C:\\$Recycle.Bin\\S-1-5-18'
[4776] Failed to execute script docker-compose
My docker file
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.7
COPY ./requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
COPY . /app
WORKDIR /app
ADD . /app
ENV DATABASE_URL="postgres://superuser:superuser#localhost:5433/al-test-3"
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
and docker-compose.yml
version: '3.8'
services:
al_test:
build: /
ports:
- 5001:80
postgresql:
image: "bitnami/postgresql:latest"
volumes:
- postgresql-data:/bitnami/postgresql
ports:
- "5446:5432"
environment:
- POSTGRESQL_DATABASE=al-test-3
- POSTGRESQL_USERNAME=superuser
- POSTGRESQL_PASSWORD=superuser
- POSTGRESQL_POSTGRES_PASSWORD=superuser
volumes:
postgresql-data:
TL;DR
I think that there is a problem with:
build: /
To build with the local folder as context, you should say
build: ./
What really happened ?
That being said, how is your project structured? Where is your Dockerfile? Is it
app
- some.py
Dockerfile
requirement.txt
docker-compose.yaml
From the standpoint of your docker-compose.yml, ./ is the folder it is in. In Unix file systems / stands for the root of your file system; what you would call C:\.
At build time
I am not sure how docker is integrated in Windows but it might be possible that / is translated as some place where your docker demon does not have the right to read, raising this PermissionError during your compilation.
At run time
You might encounter similar right issues while trying to mount various folders into your container. Here is the fix (when it occurs)
OS Windows 10,
I am using Docker Engine version 18.09.2, the API version is 1.39
The website explaining the steps to run CAT is: https://libraries.io/pypi/medcat
I am building the medcat image locally. Output looks good until the end of the build process:
Step 10/11 : ENTRYPOINT ["python"]
---> Using cache
---> 66b414e2093d
Step 11/11 : CMD ["api.py"]
---> Using cache
---> db2acf6c4649
Successfully built db2acf6c4649
Successfully tagged cat:latest
SECURITY WARNING: You are building a Docker image from Windows against
a non-Windows Docker host. All files and directories added to build
context will have '-rwxr-xr-x' permissions. It is recommended to
double check and reset permissions for sensitive files and
directories.
When I am trying to start the container I just built, I get:
IT IS UMLS
* Serving Flask app "api" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a
production deployment.
Use a production WSGI server instead.
* Debug mode: on
Traceback (most recent call last):
File "api.py", line 66, in <module>
app.run(debug=True, host='0.0.0.0', port=5000)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line
944, in run
run_simple(host, port, self, **options)
File "/usr/local/lib/python3.7/site-packages/werkzeug/serving.py",
line 1007, in run_simple
run_with_reloader(inner, extra_files, reloader_interval,
reloader_type)
File "/usr/local/lib/python3.7/site-packages/werkzeug/_reloader.py",
line 332, in run_with_reloader
sys.exit(reloader.restart_with_reloader())
File "/usr/local/lib/python3.7/site-packages/werkzeug/_reloader.py",
line 176, in restart_with_reloader
exit_code = subprocess.call(args, env=new_environ,
close_fds=False)
File "/usr/local/lib/python3.7/subprocess.py", line 323, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/local/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.7/subprocess.py", line 1522, in
_execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/cat/api/api.py'
Does anyone have experience with building on Windows? Does the security warning have anything to do with this?
Update:
I added the permission for linux executable as in the received answer at this post. Then I built the image locally using the following command docker build --network=host -t cat -f Dockerfile.MedMen ., and the end of the building process gives me the same Security Warning.
Then I checked docker run --env-file=./envs/env_medann ubuntu:18.04 env, which gave me:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=3d5fd66fadbe
TYPE=UMLS
DEBUG=False
CNTX_SPAN=6
CNTX_SPAN_SHORT=2
MIN_CUI_COUNT=100
MIN_CUI_COUNT_STRICT=1
MIN_ACC=0.01
MIN_CONCEPT_LENGTH=1
NEG_PROB=0.2
LBL_STYLE=def
SPACY_MODEL=en_core_sci_md
UMLS_MODEL=/cat/models/med_ann_norm.dat
VOCAB_MODEL=/cat/models/med_ann_norm_dict.dat
MKL_NUM_THREAD=1
NUMEXPR_NUM_THREADS=1
OMP_NUM_THREADS=1
HOME=/root
This is because windows & linux has CR-LF & LF difference issue, meanwhile, permission need to be added for linux executable.
For your case, as you have got the source code, I think you have git installed on your windows. Then, you can open Git Bash, change the path to your source code directory, and execute next in it:
find . -type f | xargs dos2unix
chmod -R 777 *
Finally, rebuild it.
Update:
I try your code completely, it seems the issue is in cat/api/api.py, it misses a #!. So, into your sourcecode, edit cat/api/api.py, add next at the beginning of the sourcecode:
#!/usr/bin/env python
Then, rebuild with Dockerfile & run it again, you can see the effect from browser:
I have a mysql database running on a localhost (ubuntu 16.04). On the same host I have a docker container in which a python script is running. This script has to connect to the mysqldb on the local host. As was described in these posts (post1, post2) I set bind-address=0.0.0.0 for my local database and found the ip-address of my local host and used it in my python script to connect to the database, but it did not work. Below I show my set-up and how I run the docker container.
My python script (analysis.py) looks as follows:
import pandas as pd
import sqlalchemy as db
def find_max_age():
cnx = db.create_engine('mysql+mysqlconnector://root:password#172.17.0.1:3306/datasets')
cnx_res = db.create_engine('mysql+mysqlconnector://root:password#172.17.0.1:3306/results')
df = pd.read_sql("select * from test_table", cnx)
idx = df['age'].idxmax() == df.index
df_res = df[idx]
df_res.to_sql('max_age4', con=cnx_res, index=False)
if __name__ == '__main__':
find_max_age()
My Dockerfile looks as follwos:
FROM python:2.7-slim
EXPOSE 80 3306
WORKDIR /app
COPY requirements.txt /app
RUN pip install -r requirements.txt
COPY analysis.py /app
CMD python analysis.py
Finally, the requirements.txt looks like
mysql-connector-python
sqlalchemy
pandas
I build the docker image as follows:
docker build -t max_age_app .
Then I start the container using this image as follows:
docker run -d max_age_app:latest
The container exits with exit code 1 and when I take a look at the respective log of the container I find the following error in it:
> Traceback (most recent call last):
File "analysis.py", line 24, in <module>
find_max_age()
File "analysis.py", line 11, in find_max_age
df = pd.read_sql("select * from test_table", cnx)
File "/usr/local/lib/python2.7/site-packages/pandas/io/sql.py", line 397, in read_sql
chunksize=chunksize)
File "/usr/local/lib/python2.7/site-packages/pandas/io/sql.py", line 1063, in read_query
result = self.execute(*args)
File "/usr/local/lib/python2.7/site-packages/pandas/io/sql.py", line 954, in execute
return self.connectable.execute(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in execute
connection = self.contextual_connect(close_with_result=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
e, dialect, self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 400, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 529, in checkout
rec = pool._do_get()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1193, in _do_get
self._dec_overflow()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1190, in _do_get
return self._create_connection()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 347, in _create_connection
return _ConnectionRecord(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 474, in __init__
self.__connect(first_connect_check=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 671, in __connect
connection = pool._invoke_creator(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 412, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python2.7/site-packages/mysql/connector/__init__.py", line 172, in connect
return CMySQLConnection(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/mysql/connector/connection_cext.py", line 78, in __init__
self.connect(**kwargs)
File "/usr/local/lib/python2.7/site-packages/mysql/connector/abstracts.py", line 731, in connect
self._open_connection()
File "/usr/local/lib/python2.7/site-packages/mysql/connector/connection_cext.py", line 179, in _open_connection
sqlstate=exc.sqlstate)
sqlalchemy.exc.DatabaseError: (mysql.connector.errors.DatabaseError) 2003 (HY000): Can't connect to MySQL server on '172.17.0.1' (111) (Background on this error at: http://sqlalche.me/e/4xp6)
In order to determine the ip of the localhost I used ifconfig command, which yielded something like this:
docker0 Link encap:Ethernet HWaddr 02:42:a2:a6:d7:ff
inet addr:172.17.0.1
enp0s3 Link encap:Ethernet HWaddr 08:00:27:bb:7e:b5
inet addr:10.0.2.15
lo Link encap:Local Loopback
inet addr:127.0.0.1
So I tried 172.17.0.1 in order to connect to the local database from within the container but it did not work.
Do I have to match any ports between the container and the localhost via the -p option when I start the container?
I would appreciate any help.
You shouldn't EXPOSE the port 3306 on the container, since the MySQL server is listening outside it, on the localhost. I suspect it's a networking issue, so try to see if the localhost has the interface with the address 172.17.0.1 and if you can reach this address from inside your container (ex. try docker exec -ti _your_container_name /bin/sh and then try ping 172.17.0.1). You should also check the mysql logs to see if you have some error reported there.
In order to solve this problem one has to correctly configure the mysql database on the localhost. In addition to what was discussed above one has to do the following steps:
Find the IP-address of your docker container with the python app. You
can use docker inspect <container_name>.
Create in your local mysql database a new user (and a respective
password) for that IP-address. Info on how to do it you can find in
this post. How to set password policies in mysql is described
here.
Afterwards, simply starting the container with the command docker run -d max_age_app will suffice for the python script to write the data into the database on the localhost.
We have established pipelines scripts that work very well. Lately, we decided to deploy to elastic beanstalk automatically, with the use of bitbucket pipelines and following the tutorial which uses the command eb deploy to deploy. Apparently, this command fails on pipelines. The config files seem legit because it runs locally. It also runs from inside a container of the same image that we have specified in the pipelines file and also by using docker exec from the local to run the command inside a container of the same image. The following are the pipelines file and the error we get using eb deploy --verbose command. I am obviously missing something here. Any help or direction would be appreciated. Thanking you in advance.
feature/KKLT-1065-deploy-via-pipelines:
- step:
deployment: staging
caches:
- composer
script:
- php -r "file_exists('.env') || copy('.env.example', '.env');"
- cat .env
- composer install
- php artisan cache:clear
- php artisan migrate
- php artisan db:seed
- eb init KMLT-staging-ttl -r eu-central-1 -p "64bit Amazon Linux 2017.09 v2.6.4 running PHP 7.1"
- eb deploy --verbose
services:
- postgres
+ eb deploy --verbose
INFO: Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ebcli/core/ebrun.py", line 41, in run_app
app.run()
File "/usr/lib/python2.7/site-packages/cement/core/foundation.py", line 797, in run
return_val = self.controller._dispatch()
File "/usr/lib/python2.7/site-packages/cement/core/controller.py", line 472, in _dispatch
return func()
File "/usr/lib/python2.7/site-packages/cement/core/controller.py", line 475, in _dispatch
self._parse_args()
File "/usr/lib/python2.7/site-packages/cement/core/controller.py", line 452, in _parse_args
self.app._parse_args()
File "/usr/lib/python2.7/site-packages/cement/core/foundation.py", line 1076, in _parse_args
for res in self.hook.run('post_argument_parsing', self):
File "/usr/lib/python2.7/site-packages/cement/core/hook.py", line 150, in run
res = hook[2](*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ebcli/core/hooks.py", line 35, in pre_run_hook
set_profile(app.pargs.profile)
File "/usr/lib/python2.7/site-packages/ebcli/core/hooks.py", line 47, in set_profile
profile = commonops.get_default_profile()
File "/usr/lib/python2.7/site-packages/ebcli/operations/commonops.py", line 973, in get_default_profile
profile = get_config_setting_from_branch_or_default('profile')
File "/usr/lib/python2.7/site-packages/ebcli/operations/commonops.py", line 1008, in get_config_setting_from_branch_or_default
setting = get_setting_from_current_branch(key_name)
File "/usr/lib/python2.7/site-packages/ebcli/operations/commonops.py", line 991, in get_setting_from_current_branch
branch_name = source_control.get_current_branch()
File "/usr/lib/python2.7/site-packages/ebcli/objects/sourcecontrol.py", line 184, in get_current_branch
stdout, stderr, exitcode = self._run_cmd(revparse_command, handle_exitcode=False)
File "/usr/lib/python2.7/site-packages/ebcli/objects/sourcecontrol.py", line 480, in _run_cmd
stdout, stderr, exitcode = exec_cmd(cmd)
File "/usr/lib/python2.7/site-packages/cement/utils/shell.py", line 40, in exec_cmd
proc = Popen(cmd_args, *args, **kw)
File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
INFO: OSError - [Errno 2] No such file or directory
Try python3 version of eb instead of python2.7. Might have more success.
I set up the local file storage according to these steps
http://docs.ckan.org/en/latest/filestore.html
Create location
sudo mkdir -p /var/lib/ckan/default
I confirm the location exists and is in the right location
I uncomment the lines
ofs.impl = pairtree
ofs.storage_dir = /var/lib/ckan/default
I confirmed I have pairtree and argparse installed
I am using jetty as the web server not Apache so I do
sudo chown jetty /var/lib/ckan/default
sudo chmod u+rwx /var/lib/ckan/default
sudo service jetty restart
Then I run this command to start my site
paster serve /etc/ckan/default/development.ini
I then get this stack trace back, I am doing all of these commands inside my virtual environment
2013-12-04 17:39:46,369 WARNI [ckan.lib.uploader] Please use config option ckan.storage_path instaed of
ofs.storage_path
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/serve.py", line 284, in command
relative_to=base, global_conf=vars)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/serve.py", line 321, in loadapp
**kw)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
return context.create()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
return self.object_type.invoke(self)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
val = callable(*args, **kw)
File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware.py", line 156, in make_app
os.makedirs(path)
File "/usr/lib/ckan/default/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/lib/ckan/default/storage'
Permission denied: '/var/lib/ckan/default/storage'
It looks like you don't have permission to read and write to this directory. Try running these commands in a terminal:
sudo chown -R `whoami` /var/lib/ckan/default
sudo chmod -R u+rwx /var/lib/ckan/default
For those stumbling on this years later...
I ran into this when trying to setup datastore and run the paster --plugin=ckan datastore set-permissions -c /etc/ckan/default/production.ini command.
I had already setup FileStore as per the docs. What I ended up having to do was upload a resource in the web UI then re-run the command above and the error no longer appeared and the output worked as expected. After uploading a resource the /var/lib/ckan/default/resources and /var/lib/ckan/default/storage directories were created with the proper permissions and the paster command didn't need to try and create them (or so I'm assuming).
Here's some additional resources in case you have a slightly different issue that's causing this error:
https://github.com/ckan/ckan/issues/3676
https://github.com/ckan/ckan/issues/4548
https://lists.okfn.org/pipermail/ckan-dev/2018-November/022974.html
Sample 500 error in ckan 2.9
File "/usr/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/lib/ckan/default/webassets/.webassets-cache'
To remove the permission error;
Set the permission and ownership. The docs on filestore misses out on the ownership and recursive mode
sudo chown www-data:www-data -R /var/lib/ckan/default
sudo chmod u+rwx -R /var/lib/ckan/default
Finally restart your server:
sudo supervisorctl restart ckan-uwsgi:*