We're using a Flask app, served by gunicorn + Nginx in a docker container. Roughly once or twice every 24 ours, our logger displays the following message:
Dec 31 00:27:36 Computer logger: [12/31/2019 08:27:36] backend ERROR: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again. Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1925, in dispatch_request self.raise_routing_exception(req) File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1907, in raise_routing_exception raise request.routing_exception File "/usr/local/lib/python3.6/site-packages/flask/ctx.py", line 350, in match_request result = self.url_adapter.match(return_rule=True) File "/usr/local/lib/python3.6/site-packages/werkzeug/routing.py", line 1799, in match import pdb werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
What this affects or what it's caused by is totally mysterious, and the traceback is not helpful in the slightest. There is no noticeable change on the user-facing side. I've tried many things, including starting the app. in debug mode. My most recent attempt was to go into /usr/local/lib/python3.6/site-packages/werkzeug/routing.py itself and change the behavior of the error. What happens, though, is that above-logged message just prints out my code changes—but doesn't seem to actually run the changes themselves. (i.e. - Everything looks to run exactly the same way, regardless of how I change that file. Instead, the changes themselves are logged [not the effects of running that changed code]).
How does one debug an unresponsive Werkzeug 404 error like this? What options/settings/methods am I missing? Why would changes to the code of routing.py (for example) be ignored, but printed out to the screen (as if what Python is doing is just printing out code blocks between line x and line y).
Related
I'm using flask as a python framework for my object detection REST API and it works just fine on MacOS. The application takes the uploaded image and it works on that. I'm sending a POST request to upload the image to the server. Just to give you an insight into my code:
#app.route('/post', methods=['GET', 'POST'])
def post():
form = PhotoForm(CombinedMultiDict((request.files, request.form)))
if request.method == 'POST' and form.validate():
with tempfile.NamedTemporaryFile() as temp:
form.input_photo.data.save(temp)
temp.flush()
print(temp.name)
result = detect_objects(temp.name)
photo_form = PhotoForm(request.form)
return render_template('upload.html',
photo_form=photo_form, result=result)
else:
return redirect(url_for('upload'))
However, when I try to run the same application on Windows 10, I get the following server error 500:
127.0.0.1 - - [08/Feb/2018 15:32:24] "GET / HTTP/1.1" 200 -
C:\Users\KUGUOG~1.KAA\AppData\Local\Temp\tmpw6y5g2f9
[2018-02-08 15:32:37,369] ERROR in app: Exception on /post [POST]
Traceback (most recent call last):
File "c:\python36\lib\site-packages\flask\app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "c:\python36\lib\site-packages\flask\app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "c:\python36\lib\site-packages\flask\app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "c:\python36\lib\site-packages\flask\_compat.py", line 33, in reraise
raise value
File "c:\python36\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "c:\python36\lib\site-packages\flask\app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\Users\kuguoglu.berk.kaan\Desktop\ObjectDetectionRestApi\app.py", line 188, in post
result = detect_objects(temp.name)
File "C:\Users\kuguoglu.berk.kaan\Desktop\ObjectDetectionRestApi\app.py", line 149, in detect_objects
image = Image.open(image_path).convert('RGB')
File "c:\python36\lib\site-packages\PIL\Image.py", line 2543, in open
fp = builtins.open(filename, "rb")
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\KUGUOG~1.KAA\\AppData\\Local\\Temp\\tmpw6y5g2f9'
127.0.0.1 - - [08/Feb/2018 15:32:37] "POST /post HTTP/1.1" 500 -
I skimmed through quite a few posts and forums and I thought this might be related to file permission in Windows OS. As it explicitly says permission error and all the related posts I've read so far mentions file permissions I do not think that it's related to the program itself. If it was the case, it wouldn't work smoothly on MacOS in the first place.
So far, I have tried listening to different ports, including 80 and 8080, and running the program as an administrator through PyCharm terminal and command prompt. However, these did not solve the issue.
I'm using Python 3.6.3, Flask 0.12.2, in case it helps you figure out the issue. Feel free to ask me for more details, if needed. I'd be happy to provide if I can.
Cheers.
Edit: I tried some of the suggested ways in the following two posts, however, all of them failed.
How to run python script with elevated privileges on Windows
Request UAC elevation from within python scripts - 1
Request UAC elevation from within python scripts - 2
PermissionError: [Errno 13] Permission denied
Further, I disabled read-only option for the directory below to see if it solves the problem but it didn't.
C:\Users\KUGUOG~1.KAA\AppData\
Are you running with pycharm?
If you are running with cmd, run the command first as administrator.
When you find the cmd.exe, right click it and click on run as administrator
Elif you are using pycharm, run it as administrator as well.
I'm on version 1.9.9 of the SDK and I'm having issues with the devserver. I have a manually scaled module with 1 instance. I created a webapp2.RequestHandler for /_ah/start. In that handler I start a background thread. When I run my app in the devserver, the _ah/start handler returns a 200, but /_ah/background will randomly return 500 errors for a while. After sometime (usually a minute or two, but sometimes more), the 500 errors stop, but will randomly occur again every few hours. It also seems that everytime I open a new browser tab (Chrome), I get the same error. Anyone know what could be causing this?
Here is the RequestHandler for /_ah/start:
class StartupHandler(webapp2.RequestHandler):
def get(self):
runtime.set_shutdown_hook(shutdown_hook)
global foo
if foo is None:
foo = Foo()
background_thread.start_new_background_thread(do_foo, [])
self.response.http_status_message(200)
Here is the 500 error:
ERROR 2014-08-18 07:39:36,256 module.py:717] Request to '/_ah/background' failed
Traceback (most recent call last):
File "\appengine\tools\devappserver2\module.py", line 694, in _handle_request
environ, wrapped_start_response)
File "\appengine\tools\devappserver2\request_rewriter.py", line 311, in _rewriter_middleware
response_body = iter(application(environ, wrapped_start_response))
File "\appengine\tools\devappserver2\module.py", line 1672, in _handle_script_request
request_type)
File "\appengine\tools\devappserver2\module.py", line 1624, in _handle_instance_request
request_id, request_type)
File "\appengine\tools\devappserver2\instance.py", line 382, in handle
request_type))
File "\appengine\tools\devappserver2\http_proxy.py", line 190, in handle
response = connection.getresponse()
File "E:\Programing\Python27\lib\httplib.py", line 1030, in getresponse
response.begin()
File "E:\Programing\Python27\lib\httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "E:\Programing\Python27\lib\httplib.py", line 365, in _read_status
line = self.fp.readline()
File "E:\Programing\Python27\lib\socket.py", line 430, in readline
data = recv(1)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
INFO 2014-08-18 07:39:36,257 module.py:1890] Waiting for instances to restart
INFO 2014-08-18 07:39:36,262 module.py:642] lease: "GET /_ah/background HTTP/1.1" 500 -
Well this might be not the answer , but how long will it take to complete a specific task to assign to a backend? Seems like an issue with concurrency
Looks like the issue (as far as I can currently tell) is that I'm using PyCharm, which synchronizes the project's files when its window is entered or exited. This rewrites the project files even if there are no changes, which causes the devserver to restart all instances, leading to the 500 errors.
More info on PyCharm Synchronization
Link to issue at PyCharm
I am trying to get my bottle server so that when one person in a game logs out, everyone can immediately see it. As I am using long polling, there is a request open with all the users.
The bit I am having trouble with is catching the exception that is thrown when the user leaves the page from the long polling that can no longer connect to the page. The error message is here.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 425, in run_application
self.process_result()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 416, in process_result
self.write(data)
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 373, in write
self.socket.sendall(msg)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 509, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 483, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
<WSGIServer fileno=3 address=0.0.0.0:8080>: Failed to handle request:
request = GET /refreshlobby/1 HTTP/1.1 from ('127.0.0.1', 53331)
application = <bottle.Bottle object at 0x7f9c05672750>
127.0.0.1 - - [2013-07-07 10:59:30] "GET /refreshlobby/1 HTTP/1.1" 200 160 6.038377
The function to handle that page is this.
#route('/refreshlobby/<id>')
def refreshlobby(id):
while True:
yield lobby.refresh()
gevent.sleep(1)
I tried catching the exception within the function, and in a decorator which I put to wrap #route, neither of which worked. I tried making an #error(500) decorator, but that didn't trigger, either. It seems that this is to do with the internals of bottle.
Edit: I know now that I need to be catching socket.error, but I don't know whereabouts in my code
The WSGI runner
Look closely at the traceback: this in not happening in your function, but in the WSGI runner.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
The way the WSGI runner works, in your case, is:
Receives a request
Gets a partial response from your code
Sends it to the client (this is where the exception is raised)
Repeats steps 2-3
You can't catch this exception
This error is not raised in your code.
It happens when you try to send a response to a client that closed the connection.
You'll therefore not be able to catch this error from within your code.
Alternate solutions
Unfortunately, it's not possible to tell from within the generator (your code) when it stops being consumed.
It's also not a good idea to rely on your generator being garbage collected.
You have a couple other solutions.
"Last seen"
Another way to know when an user disconnects would probably be to record a "last seen", after your yield statement.
You'll be able to identify clients that disconnected if their last seen is far in the past.
Other runner
Another, non-WSGI runner, will be more appropriate for a realtime application. You could give tornado a try.
After updating from 1.7.5 (where everything worked fine) I'm getting a HTTP Error 403: Forbidden when trying to open any sites via localhost. Strange thing is I have pretty much the same setup at home as here at work and everything works there... Might be an issue with proxy server we're using at work, since that's the only difference I can think of? Here's the error log I'm getting, so if anyone knows what's going on please help (;
Traceback (most recent call last):
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 1302, in communicate
req.respond()
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 831, in respond
self.server.gateway(self).respond()
File "U:\Dev\GAE\lib\cherrypy\cherrypy\wsgiserver\wsgiserver2.py", line 2115, in respond
response = self.req.server.wsgi_app(self.env, self.start_response)
File "U:\Dev\GAE\google\appengine\tools\devappserver2\wsgi_server.py", line 246, in __call__
return app(environ, start_response)
File "U:\Dev\GAE\google\appengine\tools\devappserver2\request_rewriter.py", line 311, in _rewriter_middleware
response_body = iter(application(environ, wrapped_start_response))
File "U:\Dev\GAE\google\appengine\tools\devappserver2\python\request_handler.py", line 89, in __call__
self._flush_logs(response.get('logs', []))
File "U:\Dev\GAE\google\appengine\tools\devappserver2\python\request_handler.py", line 220, in _flush_logs
apiproxy_stub_map.MakeSyncCall('logservice', 'Flush', request, response)
File "U:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 94, in MakeSyncCall
return stubmap.MakeSyncCall(service, call, request, response)
File "U:\Dev\GAE\google\appengine\api\apiproxy_stub_map.py", line 320, in MakeSyncCall
rpc.CheckSuccess()
File "U:\Dev\GAE\google\appengine\api\apiproxy_rpc.py", line 156, in _WaitImpl
self.request, self.response)
File "U:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 200, in MakeSyncCall
self._MakeRealSyncCall(service, call, request, response)
File "U:\Dev\GAE\google\appengine\ext\remote_api\remote_api_stub.py", line 226, in _MakeRealSyncCall
encoded_response = self._server.Send(self._path, encoded_request)
File "U:\Dev\GAE\google\appengine\tools\appengine_rpc.py", line 393, in Send
f = self.opener.open(req)
File "U:\Dev\Python\lib\urllib2.py", line 410, in open
response = meth(req, response)
File "U:\Dev\Python\lib\urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "U:\Dev\Python\lib\urllib2.py", line 448, in error
return self._call_chain(*args)
File "U:\Dev\Python\lib\urllib2.py", line 382, in _call_chain
result = func(*args)
File "U:\Dev\Python\lib\urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 403: Forbidden
INFO 2013-04-19 12:28:52,576 server.py:561] default: "GET / HTTP/1.1" 500 -
INFO 2013-04-19 12:28:52,619 server.py:561] default: "GET /favicon.ico HTTP/1.1" 304 -
Also, the launcher throws an error when closing:
Traceback (most recent call last):
File "launcher\mainframe.pyc", line 327, in OnStop
File "launcher\taskcontroller.pyc", line 167, in Stop
File "launcher\dev_appserver_task_thread.pyc", line 82, in stop
File "launcher\taskthread.pyc", line 107, in stop
File "launcher\platform.pyc", line 397, in KillProcess
pywintypes.error: (5, 'TerminateProcess', 'Access is denied.')
I had this very same issue with my MacOSX when using a proxy server using Google App Engine Launcher 1.8.6. Apparently there's an issue with "proxy_bypass" on "urllib2.py".
There are two possible solutions:
Downgrade to 1.7.5, but, who wants to downgrade?
Edit "[GAE Instalattion path]/google/appengine/tools/appengine_rpc.py" and look for the line that says
opener.add_handler(fancy_urllib.FancyProxyHandler())
In my computer it was line 578, and then put a hash (#) at the beginning of the line, like this:
`#opener.add_handler(fancy_urllib.FancyProxyHandler())`
Save the file, stop and then restart your application. Now dev_appserver.py shouldn't try to use any proxy server at all.
If your application uses any external resources like a SOAP Webservice or something like that and you can't reach the server without the proxy server, then you'll have to downgrade. Please keep in mind that external javascript files (like facebook SDK or similar) are loaded from your browser, not from your application.
Since I'm not using any external REST or SOAP services it worked for me!
Hopefully it will work for you as well.
Try either:
-Accessing it through a different proxy. I.E a . proxy within a proxy
-Accessing it through your local IP i.e 192.168.1.1
I faced the same issue with version 1.9.5. Seems that the API proxy is sending some RPCs to the proxy server, which are then being rejected with HTTP 403 (since proxy servers are generally configured to reject connection attempts to arbitrary ports). In my case I was using the urlfetch module in my app to access external web pages, so disabling the proxy server was not a choice for me.
This is how I worked around the issue some time back (most probably it was based on comments found under this issue, but I cannot remember the exact sources).
NOTE:
For this approach to work, you'll have to know the hostname/IP address and default port of your proxy server, and change them appropriately in the code if you happen to connect to a different proxy server.
When you are not behind the proxy server, you will have to revert the applied changes in order to return to a working state (if you want internet access inside your app).
Here it goes:
Disable proxy settings for the Python (Google App Engine Launcher) environment in some way. (In my case it was easy since I was launching the dev_appserver.py from a Terminal shell (on Linux), and the unset http_proxy and unset https_proxy commands did the trick.)
Edit {App Engine SDK root}/google/appengine/api/urlfetch_stub.py. Find the code block
if _CONNECTION_SUPPORTS_TIMEOUT:
connection = connection_class(host, timeout=deadline)
else:
connection = connection_class(host)
(lines 376-379 in my case) and replace it with:
if _CONNECTION_SUPPORTS_TIMEOUT:
if host[:9] == 'localhost' or host[:9] == '127.0.0.1':
connection = connection_class(host, timeout=deadline)
else:
connection = connection_class('your_proxy_host_goes_here', your_proxy_port_number_goes_here, timeout=deadline)
else:
if host[:9] == 'localhost' or host[:9] == '127.0.0.1':
connection = connection_class(host)
else:
connection = connection_class('your_proxy_host_goes_here', your_proxy_port_number_goes_here)
replacing the placeholders your_proxy_host_goes_here and your_proxy_port_number_goes_here with appropriate values.
(I believe this code can be written more elegantly, though... any Python geeks out there? :) )
In my case, I also had to delete the existing compiled file urlfetch_stub.pyc (located in the same directory as urlfetch_stub.py) because the SDK didn't seem to pick up the changes until I did so.
Now you can use dev_appserver to launch your app, and use urlfetch-backed services within the app, free from HTTP 403 errors.
I've got a Flask app that I'm trying to deploy using Gunicorn and nginx. However, although it works fine locally, it throws a TemplateNotFound error when I run in with Gunicorn on my remote server.
I'm not sure how to even start debugging this, let alone why it's failing...would love help on the former, if not the latter. I thought maybe it was a permissions issue, so chmod'd the templates folder to 777...no luck. Here's all the relavant details:
install script
Starting with a bare Ubuntu 10.04 install, I run this to set up the server and pull in my code: https://github.com/total-impact/total-impact-deploy/blob/master/deploy.sh. Then I put this nginx config file at /etc/nginx/sites-available/total-impact:
server {
location / {
proxy_pass http://127.0.0.1:8000;
}
}
Finally, I navigate the app directory and run gunicorn web:app, and hit the server's IP address. This generates a 500 in the browser, and this output on the command line:
stack trace:
root#jc:/home/ti/total-impact-webapp/totalimpactwebapp# gunicorn web:app2012-05-28 23:15:06 [15313] [INFO] Starting gunicorn 0.14.3
2012-05-28 23:15:06 [15313] [INFO] Listening at: http://127.0.0.1:8000 (15313)
2012-05-28 23:15:06 [15313] [INFO] Using worker: sync
2012-05-28 23:15:06 [15316] [INFO] Booting worker with pid: 15316
2012-05-28 23:15:12,274 - totalimpactwebapp.core - ERROR - Exception on / [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1292, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1062, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1060, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.6/dist-packages/flask/app.py", line 1047, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/ti/total-impact-webapp/totalimpactwebapp/web.py", line 60, in home
return render_template('index.html', commits=False)
File "/usr/local/lib/python2.6/dist-packages/flask/templating.py", line 120, in render_template
return _render(ctx.app.jinja_env.get_template(template_name),
File "/usr/local/lib/python2.6/dist-packages/jinja2/environment.py", line 719, in get_template
return self._load_template(name, self.make_globals(globals))
File "/usr/local/lib/python2.6/dist-packages/jinja2/environment.py", line 693, in _load_template
template = self.loader.load(self, name, globals)
File "/usr/local/lib/python2.6/dist-packages/jinja2/loaders.py", line 115, in load source, filename, uptodate = self.get_source(environment, name)
File "/usr/local/lib/python2.6/dist-packages/flask/templating.py", line 61, in get_source
raise TemplateNotFound(template)
TemplateNotFound: index.html
Are your templates in [app root]/templates/?
If so, check to be sure your path is correct. Put this as the first line in the view that handles your homepage:
return app.root_path
If that's what you expect to see - or if you're using Blueprints or another method that changes the default Jinja Environment somehow - it's a little more complicated.
Oddly, Jinja doesn't seem to have a jinja2.Environment.FileSystemLoader.get_search_path() method. I assumed it would have one :(
Today, I experienced identical problems after a long period of my Flask app behaving quite normally (ie not throwing TemplateNotFound exceptions). None of the approaches mentioned by others here hit the mark or seemed appropriate (eg app.debug, path manipulation).
Instead, I tracked it down to the standard Flask app initialisation line:
app = Flask(__name__)
I had changed __name__ to another value (to get access to a named logger), not expecting for all this carnage to unfold :-) Don't change this value unless you are very familiar with Flask internals.
I have just spent 2 hours in a very similar situation and thought I'd post what ended up being the solution.
I was suddenly getting TemplateNotFound errors in the Apache logs from my Flask application, in production, which had been running fine until then. This resulted in 500 errors across the site.
First issue was that the TemplateNotFound errors did not show unless I had Flask's "debug" flag on -- there was no sign of any problems at all in the Apache log despite LogLevel set to info.
Running the app "locally" (Flask listens on localhost:5000) was fine (one can test the pages via wget 127.0.0.0:5000). It turned out that a copy of the main web app python code had somehow landed in the directory above where it should have been. This was imported by wsgi first and, as a result, the relative path to templates was incorrect.