Develop Reference

getting error while using docker-compose build: __init__() got an unexpected keyword argument 'interpolate'

I am new to docker and while running docker-compose build command i am getting this error.
Traceback (most recent call last):
File "/home/ghostface/playground/bootcamp/venv/bin/docker-compose", line 11, in <module>
sys.exit(main())
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/cli/main.py", line 72, in main
command()
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/cli/main.py", line 125, in perform_command
project = project_from_options('.', options)
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/cli/command.py", line 76, in project_from_options
environment_file=environment_file
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/cli/command.py", line 135, in get_project
config_data = config.load(config_details, compatibility, interpolate)
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/config/config.py", line 411, in load
service_dicts = load_services(config_details, main_file, compatibility, interpolate=interpolate)
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/config/config.py", line 510, in load_services
return build_services(service_config)
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/config/config.py", line 491, in build_services
for name, service_dict in service_config.items()
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/config/config.py", line 491, in <listcomp>
for name, service_dict in service_config.items()
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/config/config.py", line 483, in build_service
interpolate
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/config/config.py", line 888, in finalize_service
service_dict['environment'] = resolve_environment(service_dict, environment, interpolate)
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/config/config.py", line 687, in resolve_environment
env.update(env_vars_from_file(env_file, interpolate))
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/compose/config/environment.py", line 42, in env_vars_from_file
env = dotenv.dotenv_values(dotenv_path=filename, encoding='utf-8-sig', interpolate=interpolate)
File "/home/ghostface/playground/bootcamp/venv/lib/python3.6/site-packages/dotenv/main.py", line 283, in dotenv_values
return DotEnv(f, verbose=verbose, **kwargs).dict()
TypeError: __init__() got an unexpected keyword argument 'interpolate'
Result of docker-compose version
docker-compose version 1.26.0, build unknown
docker-py version: 4.2.1
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.1.1c 28 May 2019
Requirement.txt file:
Flask==1.1.1
Flask-Login==0.4.1
Flask-Mail==0.9.1
Flask-Migrate==2.5.2
Flask-Moment==0.9.0
Flask-SQLAlchemy==2.4.0
Flask-WTF==0.14.2
pusher==2.1.4
PyJWT==1.7.1
pylint==2.3.1
python-dotenv==0.10.3
Werkzeug==0.16.1
gunicorn
docker-compose

The code in your container uses a newer version of python-dotenv than the one that is listed in the requirements.txt file, as the interpolate option was added in v0.11.0.
You will need to change the version in requirements.txt to 0.11.0 (or the most recent one) and reinstall the dependencies.
It seems your container uses a virtual environment. You may need to check that the dependencies are correctly updated in the container, either by running pip install in the Dockerfile, or by running it locally beforehand.

Paperboy throwing error: TypeError: the JSON object must be str, not 'bytes'?

I have installed paperboy using steps given at https://github.com/timkpaine/paperboy.git
and executed using
python3 -m paperboy
while uploading notebook from UI
when click on the save button UI not responding but throwing below error on console
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
[2019-12-03 15:26:01 +0530] [24991] [ERROR] Error handling request /api/v1/notebooks?action=save
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/sync.py", line 135, in handle
self.handle_request(listener, req, client, addr)
File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/sync.py", line 176, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python3.5/dist-packages/falcon/api.py", line 244, in __call__
responder(req, resp, **params)
File "/home/exa00112/xstream/paperboy/paperboy/resources/notebook.py", line 21, in on_post
resp.body = json.dumps(self.db.notebooks.store(req.context['user'], req.params, self.session))
File "/home/exa00112/xstream/paperboy/paperboy/storage/sqla/notebook.py", line 44, in store
notebook = nbformat.writes(strip_outputs(nbformat.reads(params.get('file').file.read(), 4)))
File "/usr/local/lib/python3.5/dist-packages/nbformat/__init__.py", line 74, in reads
nb = reader.reads(s, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/nbformat/reader.py", line 58, in reads
nb_dict = parse_json(s, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/nbformat/reader.py", line 14, in parse_json
nb_dict = json.loads(s, **kwargs)
File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

Use a master branch to install paperboy https://github.com/timkpaine/paperboy.git
git clone https://github.com/timkpaine/paperboy.git
cd paperboy
conda create --name python3.7 python=3.7
conda activate python3.7
npm install
npm run build
pip install -e .
then open
paperboy/server/deploy_nix.py
change line number 2 with
from six import iteritems #from gunicorn.six import iteritems
then run
python -m paperboy

Installing Channels: Python Django version and settings

I'm trying to add Channels to my app by following this tutorial:
https://blog.heroku.com/in_deep_with_django_channels_the_future_of_real_time_apps_in_django
The Django app compiles, but when I try to run it:
(virtualenv) root#computer:path# python manage.py runserver 0.0.0.0:8000
It gives error:
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/channels/routing.py", line 75, in resolve_routing
raise ImproperlyConfigured("Cannot import channel routing %r: %s" % (routing, e))
ImproperlyConfigured: Cannot import channel routing 'codebench.routing.channel_routing': cannot import name ProtocolTypeRouter
According to this
https://github.com/VincentTide/django-channels-celery-example/issues/1
It maybe related to Python version, so I did:
(virtualenv) root#computer:path# alias python=python3
(virtualenv) root#computer:path# python -V
Python 3.5.2
(virtualenv) root#computer:path# python manage.py runserver 0.0.0.0:8000
Traceback (most recent call last):
File "manage.py", line 3, in <module>
import eventlet
ImportError: No module named 'eventlet'
(virtualenv) root#computer:path# pip install eventlet
Requirement already satisfied: eventlet in /root/.virtualenvs/codebench/lib/python2.7/site-packages
Requirement already satisfied: greenlet>=0.3 in /root/.virtualenvs/codebench/lib/python2.7/site-packages (from eventlet)
Requirement already satisfied: enum-compat in /root/.virtualenvs/codebench/lib/python2.7/site-packages (from eventlet)
Requirement already satisfied: enum34 in /root/.virtualenvs/codebench/lib/python2.7/site-packages (from enum-compat->eventlet)
Now I stuck at: ImportError: No module named 'eventlet'
I have the following packages installed:
channels==1.1.6
daphne==1.4.0
Django==1.9
eventlet==0.20.0
If I follow this tutorial: http://channels.readthedocs.io/en/1.x/getting-started.html
System check identified 1 issue (0 silenced).
February 16, 2018 - 23:19:32
Django version 1.9, using settings 'codebench.settings'
Starting Channels development server at http://0.0.0.0:8000/
Channel layer default (asgiref.inmemory.ChannelLayer)
Quit the server with CONTROL-C.
2018-02-16 23:19:32,935 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2018-02-16 23:19:32,935 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2018-02-16 23:19:32,935 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
2018-02-16 23:19:32,936 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
Traceback (most recent call last):
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
timer()
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
cb(*args, **kw)
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/eventlet/semaphore.py", line 145, in _do_acquire
waiter.switch()
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/eventlet/green/thread.py", line 41, in __thread_body
func(*args, **kwargs)
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 226, in wrapper
fn(*args, **kwargs)
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/channels/management/commands/runserver.py", line 99, in inner_run
websocket_handshake_timeout=self.websocket_handshake_timeout,
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/daphne/server.py", line 101, in run
websocket_handshake_timeout=self.websocket_handshake_timeout
TypeError: __init__() got an unexpected keyword argument 'proxy_forwarded_proto_header'
If I upgrade everything, except Python 2.7.12:
asgi-redis==1.4.3
asgiref==2.1.5
channels==2.0.2
daphne==2.0.3
Django==1.11.10
runserver will give:
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/channels/apps.py", line 6, in <module>
import daphne.server # noqa
File "/root/.virtualenvs/codebench/local/lib/python2.7/site-packages/daphne/server.py", line 192
async def handle_reply(self, protocol, message):
^
SyntaxError: invalid syntax
If I try to reinstall all required packages under Python 3.5.2:
Need this for twisted:
apt install python3.6-dev
https://askubuntu.com/questions/966939/failed-building-wheel-for-twisted
Now I'm getting random errors:
2018-02-18 14:03:51,634 - ERROR - server - Exception inside application: 'NoneType' object has no attribute 'writeSequence'
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/root/.virtualenvs/python3/lib/python3.5/site-packages/channels/http.py", line 191, in __call__
await self.handle(body)
File "/root/.virtualenvs/python3/lib/python3.5/site-packages/asgiref/sync.py", line 108, in __call__
return await asyncio.wait_for(future, timeout=None)
File "/usr/lib/python3.5/asyncio/tasks.py", line 373, in wait_for
return (yield from fut)
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "/root/.virtualenvs/python3/lib/python3.5/site-packages/asgiref/sync.py", line 123, in thread_handler
return self.func(*args, **kwargs)
File "/root/.virtualenvs/python3/lib/python3.5/site-packages/channels/http.py", line 227, in handle
self.send(response_message)
File "/root/.virtualenvs/python3/lib/python3.5/site-packages/asgiref/sync.py", line 62, in __call__
return call_result.result()
File "/usr/lib/python3.5/concurrent/futures/_base.py", line 405, in result
return self.__get_result()
File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
raise self._exception
File "/root/.virtualenvs/python3/lib/python3.5/site-packages/asgiref/sync.py", line 76, in main_wrap
result = await self.awaitable(*args, **kwargs)
File "/root/.virtualenvs/python3/lib/python3.5/site-packages/daphne/server.py", line 196, in handle_reply
protocol.handle_reply(message)
File "/root/.virtualenvs/python3/lib/python3.5/site-packages/daphne/http_protocol.py", line 210, in handle_reply
http.Request.write(self, message.get("body", b""))
File "/root/.virtualenvs/python3/lib/python3.5/site-packages/twisted/web/http.py", line 1084, in write
self.channel.writeSequence(toChunk(data))
'NoneType' object has no attribute 'writeSequence'
Also:
(python3) root#computer:path# python manage.py shell
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import channels
>>> print(channels.__file__)
/root/.virtualenvs/python3/lib/python3.5/site-packages/channels/__init__.py
>>> from channels import Group
Traceback (most recent call last):
File "<console>", line 1, in <module>
ImportError: cannot import name 'Group'

It looks as if you created a Python 2 virtualenv. Setting alias python=python3 won't work. You need to create a virtualenv using Python 3, for example:
python3 -m venv newenv
Then re-install your requirements in the new virtualenv.
Once you have activated the new environment, python will be python3. You won't need to set alias python=python3.

I end up with the following versions:
asgi-redis==0.8.3
asgiref==0.9
autobahn==0.12.1
channels==0.9.3
daphne==0.9.1
dj-database-url==0.4.0
Django==1.9.3
haikunator==1.0.1
msgpack-python==0.4.7
psycopg2==2.7.4
redis==2.10.5
six==1.10.0
Twisted==15.5.0
txaio==2.2.1
zope.interface==4.1.3
Turned out Python version doesn't matter, works with both 2.7 and 3.5.
Because I was installing the https://github.com/jacobian/channels-example,
I also changed the database from MySQL to PostgreSQL. This example works like a charm.

dev_appserver.py fails on startup

When starting the local development server with dev_appserver.py i get the following error.
$ dev_appserver.py app.yaml
INFO 2017-07-03 06:51:09,662 devappserver2.py:116] Skipping SDK update check.
Traceback (most recent call last):
File "/home/daniel/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 103, in <module>
_run_file(__file__, globals())
File "/home/daniel/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 97, in _run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/home/daniel/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 381, in <module>
main()
File "/home/daniel/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 369, in main
dev_server.start(options)
File "/home/daniel/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 176, in start
configuration.modules[0].application_root, datastore_emulator_host)
File "/home/daniel/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 524, in create_api_server
appidentity_oauth_url=options.appidentity_oauth_url)
File "/home/daniel/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 699, in setup_stubs
oauth_url=appidentity_oauth_url)
File "/home/daniel/google-cloud-sdk/platform/google_appengine/google/appengine/api/app_identity/app_identity_stub.py", line 198, in Create
import six
File "/home/daniel/google-cloud-sdk/platform/google_appengine/lib/six-1.9.0/six/__init__.py", line 825, in <module>
from . import test_six
File "/home/daniel/google-cloud-sdk/platform/google_appengine/lib/six-1.9.0/six/test_six.py", line 105, in <module>
#py.test.mark.parametrize("item_name",
AttributeError: 'NoneType' object has no attribute 'parametrize'
Version:
$ gcloud --version
Google Cloud SDK 161.0.0
app-engine-python 1.9.55
beta 2017.03.24
bq 2.0.24
core 2017.06.26
gcloud
gsutil 4.26

The gcloud-sdk component app-engine-python==1.9.55 vendored six==1.9.0 library seems to be dependent on pytest.
Issue is fixed by installing pytest
$ pip install pytest
See issue: https://issuetracker.google.com/issues/62980070

This should only be an issue for users who have a pytest version < 2.2.0 installed locally, where the current version is 3.1.3. Users will not see this issue if either (a) they do not have pytest installed or (b) have pytest >= 2.2.0 installed. The #py.test.mark.parametrize() decorator at the end of the stack trace was introduced in 2.2.0.
We've submitted a patch that resolves this for users of pytest < 2.2.0 that will be available in the next App Engine SDK release. In the meantime, users can upgrade their pytest version locally and the issue should go away.

Cloning a heroku hosted django project to your local machine

I'm new to python, Django, heroku and even PostgreSQL but am working on a Django project with a team of 3 devs (they set it up), so I'm trying to set it up on my local machine so I can start developing.
Firstly I cloned the repository to my local machine, there was a requirement.txt file, so I ran pip install -r ./requirements.txt to install all the python packages the project uses. Theres a runtime.txt file which specifies python 3.5 so I need to upgrade that. I see that gunicorn is one of the packages in requiments.txt, so I'm guessing I need to use that as the web server.
Heres the packages in requirements.txt that look important to me:
Django==1.9.6
gunicorn==19.5.0
psycopg2==2.6.1
boto3==1.3.1
boto==2.40.0
whitenoise==3.2
django-extensions==1.6.7
The site also has a .buildpacks file, which seems to be a heroku specific thing:
https://github.com/a2ikm/heroku-buildpack-libjpeg62
https://github.com/heroku/heroku-buildpack-python
https://github.com/rafaelp/heroku-buildpack-wkhtmltopdf
Theres a Procfile that contains this:
web: gunicorn myproject.wsgi --log-file -
worker: celery worker --app=genome.tasks.app -l info --beat
I'm wondering what the best way to do this is. Is there a way to automatically generate a virtual environment that has all the requirements and software specified in the project? I'm already using a virtual dev environment with created with Vagrant so I could provision it with all the dependencies, but I get the feeling theres a better way to do this.
UPDATE:
Heres what what happens when I try to run the server with python manage.py runserver:
$ python manage.py runserver
Current environment: P
Current environment: P
Performing system checks...
Unhandled exception in thread started by <function wrapper at 0x7fd0d08c71b8>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/utils/autoreload.py", line 226, in wrapper
fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/runserver.py", line 116, in inner_run
self.check(display_num_errors=True)
File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 426, in check
include_deployment_checks=include_deployment_checks,
File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 75, in run_checks
new_errors = check(app_configs=app_configs)
File "/usr/local/lib/python2.7/dist-packages/django/core/checks/urls.py", line 13, in check_url_config
return check_resolver(resolver)
File "/usr/local/lib/python2.7/dist-packages/django/core/checks/urls.py", line 23, in check_resolver
for pattern in resolver.url_patterns:
File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 33, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 417, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 33, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python2.7/dist-packages/django/core/urlresolvers.py", line 410, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/var/www/sd2/selfdecode/decodify/urls.py", line 30, in <module>
url(r'', include("genome.urls")),
File "/usr/local/lib/python2.7/dist-packages/django/conf/urls/__init__.py", line 52, in include
urlconf_module = import_module(urlconf_module)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/var/www/sd2/selfdecode/genome/urls.py", line 7, in <module>
from . import views
File "/var/www/sd2/selfdecode/genome/views.py", line 20, in <module>
from decodify.aggregators import ArrayAgg, JsonAgg, ArrayaAggDelimited
File "/var/www/sd2/selfdecode/decodify/aggregators.py", line 29
def __init__(self, *expressions, distinct=False, **extra):
^
SyntaxError: invalid syntax
I'm guessing the parse error is due to using python 2.7 in a project that uses python 3.5. I installed python 3.5 with sudo apt-get install python3-all. Trying the same thing with python3:
$ python3 manage.py runserver
Traceback (most recent call last):
File "manage.py", line 8, in <module>
from django.core.management import execute_from_command_line
ImportError: No module named 'django'
> python3
The second error means the packages I installed with pip install requirements.txt, they were installed for python 2.7, not 3.5. So this is obviously where virtual environments can come in. I tried this sudo -H virtualenv -p /usr/bin/python3 py3env, both with and without sudo, and both times I got errors :
[06:08 PM]-[vagrant#machine1]-[/var/www/sd2/selfdecode]-[git master]
$ sudo -H virtualenv -p /usr/bin/python3 py3env
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /var/www/sd2/selfdecode/py3env/bin/python3
Also creating executable in /var/www/sd2/selfdecode/py3env/bin/python
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 2328, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 713, in main
symlink=options.symlink)
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 925, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/local/lib/python2.7/dist-packages/virtualenv.py", line 1370, in install_python
os.symlink(py_executable_base, full_pth)
OSError: [Errno 71] Protocol error: 'python3' -> '/var/www/sd2/myproj/py3env/bin/python'
I came across this thread:
protocol error setting up virtualenvironment through vagrant on ubuntu
Where someone was having this problem using virtualenv inside a vagrant VM, and a possible solution is setting up the venv outside of the shared vagrant folder.