Develop Reference

Python not resolving host (timed out) when running with Gunicorn inside a Docker container (weasyprint needs to do that)

I have a Django app running in a docker container, which serves the app using Gunicorn.
This app uses Weasyprint to generate a PDF, which loads a CSS dynamically.
This CSS is served by the app, and it works fine (and loads instantly):
host_pointing_to_that_same_machine.com/dynamic_css_path.css
(it's NOT a static file, it's actually a path defined in the django URLs that outputs CSS)
That's the weasyprint code:
css = weasyprint.CSS(url=request.build_absolute_uri(reverse('dynamic_css_path')))
response = HttpResponse(pdf.write_pdf(stylesheets=[css]), content_type='application/pdf')
This creates that URL correctly, and everything works good if instead of gunicorn I use the django built-in server (python manage.py runserver), both in my development machine and at the remote server inside the docker and so.
But when I serve the app via gunicorn:
gunicorn project.wsgi:application --bind 0.0.0.0:8000
Then python is unable to resolve the host name, which I proved by doing:
1) Entering the docker's app console and trying it:
docker exec -it container_name /bin/ash
# ping host_pointing_to_that_same_machine.com
# ping any_other_host_pointing_an_external_machine.com
Everything is fine.
2) Entering Django's console inside the docker container:
# python manage.py shell
>>> import urllib
>>> r = urllib.request.urlopen(url='http://host_pointing_to_that_same_machine.com', timeout=10)
>>> print('status: '+str(r.status))
Returns 200; everything OK.
3) Entering python's shell inside the docker container:
# python
>>> from urllib import request
>>> r = request.urlopen(url='http://host_pointing_to_that_same_machine.com', timeout=10)
>>> print('status: '+str(r.status))
status: 200
4) Executing these lines of code inside the script, having Gunicorn serving them:
# docker logs container_name
Internal Server Error: /admin/app_name/path_to_pdf_generating_script/
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py", line 142, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/admin/sites.py", line 223, in inner
return view(request, *args, **kwargs)
File "/srv/apps/project/admin/PdfMakingAdmin.py", line 125, in method_called_by_the_view
timeout=10)
File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/usr/local/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 1346, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/lib/python3.6/urllib/request.py", line 1321, in do_open
r = h.getresponse()
File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
But if I change the code and instead of host_pointing_to_that_same_machine.com tries to load any domain that is not pointing to that machine, like http://example.com or anything:
# docker logs container_name
status: 200
I'm going for any of the other thousand ways I have to achieve the results I want, but still, this is bugging me and I wonder if this question and potential anwers are going to be useful for other people facing that, wether they are trying to use Weasyprint or any other script that needs to make http queries.
I'm leaving here the full traceback that weasyprint throws when trying the original code:
Environment:
Request Method: GET
Request URL: http://host_pointing_to_that_same_machine.com/admin/path-to-app/
Django Version: 2.0
Python Version: 3.6.8
Traceback:
File "/usr/local/lib/python3.6/site-packages/weasyprint/urls.py" in fetch
284. result = url_fetcher(url)
File "/usr/local/lib/python3.6/site-packages/weasyprint/urls.py" in default_url_fetcher
248. timeout=timeout, context=ssl_context)
File "/usr/local/lib/python3.6/urllib/request.py" in urlopen
223. return opener.open(url, data, timeout)
File "/usr/local/lib/python3.6/urllib/request.py" in open
526. response = self._open(req, data)
File "/usr/local/lib/python3.6/urllib/request.py" in _open
544. '_open', req)
File "/usr/local/lib/python3.6/urllib/request.py" in _call_chain
504. result = func(*args)
File "/usr/local/lib/python3.6/urllib/request.py" in http_open
1346. return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/lib/python3.6/urllib/request.py" in do_open
1321. r = h.getresponse()
File "/usr/local/lib/python3.6/http/client.py" in getresponse
1331. response.begin()
File "/usr/local/lib/python3.6/http/client.py" in begin
297. version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py" in _read_status
258. line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py" in readinto
586. return self._sock.recv_into(b)
During handling of the above exception (timed out), another exception occurred:
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
35. response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
128. response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
126. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapped_view
142. response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
44. response = view_func(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/django/contrib/admin/sites.py" in inner
223. return view(request, *args, **kwargs)
File "/srv/apps/app_name/admin/PdfAdmin.py" in attendee_list
134. css = weasyprint.CSS(url=request.build_absolute_uri(reverse('view_to_dynamic_css')))
File "/usr/local/lib/python3.6/site-packages/weasyprint/__init__.py" in __init__
322. with result as (source_type, source, base_url, protocol_encoding):
File "/usr/local/lib/python3.6/contextlib.py" in __enter__
81. return next(self.gen)
File "/usr/local/lib/python3.6/site-packages/weasyprint/__init__.py" in _select_source
407. with fetch(url_fetcher, url) as result:
File "/usr/local/lib/python3.6/contextlib.py" in __enter__
81. return next(self.gen)
File "/usr/local/lib/python3.6/site-packages/weasyprint/urls.py" in fetch
286. raise URLFetchingError('%s: %s' % (type(exc).__name__, str(exc)))
Exception Type: URLFetchingError at /admin/app_name/path-to-pdf-generator/
Exception Value: timeout: timed out

In the example, you are hitting http://host_pointing_to_that_same_machine.com but in error logs, I see that you are trying to hit http://host_pointing_to_that_same_machine.com/admin/path-to-app/

In case of host pointing to the same machine you needed to resolve the host by making a host entry into your \etc\hosts file. For docker container you can do that by including below configuration into your composer file:
extra_hosts:
- "host_pointing_to_that_same_machine.com:192.168.X.X"
Replace the ip with your domain pointing ip.
Note: If there multiple container u might be needed to include the extra_hosts for others as well.

there was a same thing for me, solution was that i've runned django via gunicorn only with one worker, like that:
python manage.py migrate &&
gunicorn your_project.wsgi:application --bind 0.0.0.0:8888
the solution is to increase gunigorn workers from 1 to 2 )
with option --workers like that:
python manage.py migrate &&
gunicorn your_project.wsgi:application --workers=3 --bind 0.0.0.0:8888

python urllib2.urlopen SSL: CERTIFICATE_VERIFY_FAILED

My operating environment is: Python2.7, django1.9
My original code was：
req = urllib2.Request(url, obj, headers)
opener = urllib2.urlopen(req)
But there was an error:[SSL: CERTIFICATE_VERIFY_FAILED],I found a solution on the Internet:
First:
import ssl
import urllib2
context = ssl._create_unverified_context()
print urllib2.urlopen("https://imaojia.com/", context=context).read()
Second:
import ssl
import urllib2
ssl._create_default_https_context = ssl._create_unverified_context
print urllib2.urlopen("https://imaojia.com/").read()
After I use them,now the code becomes:
req = urllib2.Request(url, obj, headers)
import ssl
opener = urllib2.urlopen(req, context=ssl._create_unverified_context())
Now there are new errors:
HTTP Error 503: Service Unavailable
Internal Server Error: /deploy/key_list_import/
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/project/soms/deploy/views.py", line 398, in salt_key_import
minions,minions_pre = sapi.list_all_key()
File "/project/soms/deploy/saltapi.py", line 54, in list_all_key
self.token_id()
File "/project/soms/deploy/saltapi.py", line 30, in token_id
content = self.postRequest(obj,prefix='/login')
File "/project/soms/deploy/saltapi.py", line 42, in postRequest
opener = urllib2.urlopen(req, context=ssl._create_unverified_context())
File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python2.7/urllib2.py", line 475, in error
return self._call_chain(*args)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 503: Service Unavailable
Who can give me some advice? Thank you!

Thanks to Chiheb Nexus for the answer that helped me solve, so to close this question, answer it and adopt it.
The answer for this question is my request would have caused HTTP Error 503.

not able to verify ssl-certificate from client side using any of python requests, httplib/httplib2, urllib/urlib2

this question is duplicate. but i am still asking because i am not finding any solution from given answers. so guys don't make it duplicate, instead help me to solve this problem.
i am running my server as https using bottle(paste). when trying to access from java and postman , it's verifying correctly and response coming.
versions used :
python 2.7.13
requests.version
'2.11.1'
import cryptography
cryptography.version
'2.1.3'
import OpenSSL
OpenSSL.version
'17.3.0'
but unfortunately when trying to access from python modules - it's throwing issue. ?
requests.post(url, verify=_certfile, proxies=proxies) #tried #failed to verify certificate
urllib2.urlopen(url, data=data, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
cafile=None, capath=None, cadefault=False, context=context) #tried #failed to verify certificate
import httplib2
http = httplib2.Http(disable_ssl_certificate_validation=True)
headers, content = http.request(url, "POST", body=data, headers=headers)#tried #failed to verify certificate
httplib.HTTPSConnection(host='10.201.41.50', port=8018, key_file=key_file, cert_file=cert_file, strict=None, context=context) #tried #failed to verify certificate
and my server is running using paste server using bottle framework.
ssl_cxt = SSL.Context(SSL.SSLv23_METHOD)
...
...
ssl_cxt.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT, verify_cb)
paste.httpserver.serve(app, host=IP, port=PORT, server_version=" ", ssl_context=ssl_cxt)
issues i am facing (different with each lib)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "Continuum\Anaconda2\lib\site-packages\requests\api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: unknown error (_ssl.c:2947)
============
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "Continuum\Anaconda2\lib\site-packages\requests\api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "Continuum\Anaconda2\lib\site-packages\requests\adapters.py", line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:661)
File "Continuum\Anaconda2\lib\httplib.py", line 1038, in endheaders
self._send_output(message_body)
File "Continuum\Anaconda2\lib\httplib.py", line 882, in _send_output
self.send(msg)
File "Continuum\Anaconda2\lib\httplib.py", line 844, in send
self.connect()
File "Continuum\Anaconda2\lib\httplib.py", line 1262, in connect
self.sock = self._context.wrap_socket(self.sock,
AttributeError: 'Context' object has no attribute 'wrap_socket'

I tried to solve using urllib2, socket and ssl. i am able to hit the server. i tried with .pkcs12 but seems urllib2 expects only .pem files. so will advice try to create client certificate with .pem extension specifically when you try to access from python.
import ssl
import socket
import urllib2
import json
url = "https://localhost:port/hello/"
_certfile_pem = "ssl_client.pem"
key_file = "ssl_key.pem"
cert_file = "ssl_cert.pem"
headers = {"Content-Type":"application/json"}
data = {}
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
context.verify_mode = ssl.CERT_REQUIRED
context.load_cert_chain(certfile=cert_file, keyfile=key_file)
context.load_verify_locations(_certfile_pem)
context.wrap_socket(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) #
socket overwrite required.
try:
req = urllib2.Request(url,data=json.dumps(data))
response = urllib2.urlopen(req, context=context)
content = response.read()
print content
except urllib2.HTTPError as e:
content = e.read()
print "HTTPError : ", e

django-user-accounts and anymail: Invalid email address format

I have set up django-user-accounts. When I try to use the password reset function and type in a valid email address I get the following error:
AnymailInvalidAddress at /account/password_reset/
Invalid email address format '': No email found
This is the full traceback:
Request Method: POST
Request URL: http://127.0.0.1:8000/account/password_reset/
Django Version: 1.10.5
Python Version: 3.5.2
Traceback:
File "C:\Anaconda3\lib\site-packages\anymail\utils.py" in __init__
120. raise ValueError('No email found')
During handling of the above exception (No email found), another exception
occurred:
File "C:\Anaconda3\lib\site-packages\django\core\handlers\exception.py" in inner
39. response = get_response(request)
File "C:\Anaconda3\lib\site-packages\django\core\handlers\base.py" in _legacy_get_response
249. response = self._get_response(request)
File "C:\Anaconda3\lib\site-packages\django\core\handlers\base.py" in _get_response
187. response = self.process_exception_by_middleware(e, request)
File "C:\Anaconda3\lib\site-packages\django\core\handlers\base.py" in _get_response
185. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Anaconda3\lib\site-packages\django\views\generic\base.py" in view
68. return self.dispatch(request, *args, **kwargs)
File "C:\Anaconda3\lib\site-packages\django\views\generic\base.py" in dispatch
88. return handler(request, *args, **kwargs)
File "C:\Anaconda3\lib\site-packages\django\views\generic\edit.py" in post
183. return self.form_valid(form)
File "C:\Anaconda3\lib\site-packages\account\views.py" in form_valid
582. self.send_email(form.cleaned_data["email"])
File "C:\Anaconda3\lib\site-packages\account\views.py" in send_email
608. hookset.send_password_reset_email([user.email], ctx)
File "C:\Anaconda3\lib\site-packages\account\hooks.py" in send_password_reset_email
33. send_mail(subject, message, settings.DEFAULT_FROM_EMAIL, to)
File "C:\Anaconda3\lib\site-packages\django\core\mail\__init__.py" in send_mail
62. return mail.send()
File "C:\Anaconda3\lib\site-packages\django\core\mail\message.py" in send
342. return self.get_connection(fail_silently).send_messages([self])
File "C:\Anaconda3\lib\site-packages\anymail\backends\base.py" in send_messages
87. sent = self._send(message)
File "C:\Anaconda3\lib\site-packages\anymail\backends\base_requests.py" in _send
56. return super(AnymailRequestsBackend, self)._send(message)
File "C:\Anaconda3\lib\site-packages\anymail\backends\base.py" in _send
116. payload = self.build_message_payload(message, self.send_defaults)
File "C:\Anaconda3\lib\site-packages\anymail\backends\mailgun.py" in build_message_payload
28. return MailgunPayload(message, defaults, self)
File "C:\Anaconda3\lib\site-packages\anymail\backends\mailgun.py" in __init__
67. super(MailgunPayload, self).__init__(message, defaults, backend, auth=auth, *args, **kwargs)
File "C:\Anaconda3\lib\site-packages\anymail\backends\base_requests.py" in __init__
114. super(RequestsPayload, self).__init__(message, defaults, backend)
File "C:\Anaconda3\lib\site-packages\anymail\backends\base.py" in __init__
258. value = converter(value)
File "C:\Anaconda3\lib\site-packages\anymail\backends\base.py" in parsed_emails
282. for address in addresses]
File "C:\Anaconda3\lib\site-packages\anymail\backends\base.py" in <listcomp>
282. for address in addresses]
File "C:\Anaconda3\lib\site-packages\anymail\utils.py" in __init__
126. % (address, str(err)))
Exception Type: AnymailInvalidAddress at /account/password_reset/
Exception Value: Invalid email address format '': No email found
The post request contains both a valid email and CSRF token. Anymail works fine with contact forms on my site so I believe Anymail is set up correctly.
I'm using Django 1.10 and Python 3.5

send http put request in werkzeug

in my web app, I use werkzeug to listen and process requests. In one of the functionalities, I need to listen to request(say from A) and send an http put request to another server (B), then after I get response from B, I respond A an response.
I am not very familiar with werkzeug, and not sure if it has ability to send out requests, so I used httplib to send requests.
But I am getting errors.
There are a lot of moving parts, I am wondering the following:
1. does werkzeug have ability to send out requests
2. what is the cause of the error
Appreciate any help.
Code:
def complete(self, request):
if request.method == 'GET':
location_id = self.extract_param_value(request,'location_id');
status_code = self.extract_param_value(request,'status_code');
req_url = something;
jsonData = { 'data' : {'status' : status_code, 'id': location_id}};
jsonDataDump = json.dumps(jsonData);
#send request to B
connection = httplib.HTTPConnection(HOST, timeout=5);
body_content = jsonDataDump;
headers = {"Content-type": "application/json", "Accept": "text/plain"};
connection.request('PUT', req_url, body_content, headers); #error occurs here
result = connection.getresponse();
connection.close();
#return response to A
response = Response(status=200);
return response;
Error:
1**.1**.1**.** - - [30/Dec/2011 03:06:57] "GET //complete?location_id=615201308&status_code=LIVE&message=fine HTTP/1.1" 500 -
Traceback (most recent call last):
File "/home/ec2-user/y_ws.py", line 381, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/lib/python2.6/site-packages/Werkzeug-0.8.1-py2.6.egg/werkzeug/wsgi.py", line 411, in __call__
return self.app(environ, start_response)
File "/home/ec2-user/y_ws.py", line 377, in wsgi_app
response = self.dispatch_request(request);
File "/home/ec2-user/y_ws.py", line 98, in dispatch_request
return getattr(self, endpoint)(request, **values)
File "/home/ec2-user/y_ws.py", line 184, in complete
connection.request('PUT', y_req_url, body_content, headers);
File "/usr/lib64/python2.6/httplib.py", line 914, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
self.endheaders()
File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
self._send_output()
File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
self.send(msg)
File "/usr/lib64/python2.6/httplib.py", line 739, in send
self.connect()
File "/usr/lib64/python2.6/httplib.py", line 720, in connect
self.timeout)
File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
raise error, msg
error: [Errno 111] Connection refused

Werkzeug is not a library for making HTTP requests, use httplib (as in your example)
Check with curl if the request succeeds from the host machine to rule out network issues. The error you are getting is general network error.