I am running a Django REST API on Google Kubernetes Engine, using Gunicorn for my WSGI server. When my application encounters a 500 server error, the Python stack trace is not showing up in the GCP Logging console (in the "GKE Container" resource). In a different Django project of mine which uses Daphne for the ASGI/WSGI server, this traceback is being properly logged. Even more strange, the Gunicorn application has properly logged errors before, just a few weeks ago. Those errors appear in the Error Reporting console as well.
To be clear, this is the type of information that I would like to see in the GCP logs:
Internal Server Error: /v1/user/errant-endpoint
Traceback (most recent call last):
File "/path/to/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
...
File "/path/to/project/file.py", line 176, in my_file
print(test)
NameError: name 'test' is not defined
For the Gunicorn project, some of the Python tracebacks are logged, like this one when Gunicorn is started:
/usr/local/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1421: RuntimeWarning: DateTimeField User.last_login received a naive datetime (2019-02-04 05:49:47.530648) while time zone support is active.
With 500 errors, however, only the HTTP info is logged:
[04/Feb/2019:06:03:58 +0000] "POST /v1/errant-endpoint HTTP/1.1" 500
I've looked up the resources for setting up Stackdriver Logging and Stackdriver Error Reporting, but neither of these seem to apply because 1) that only appears to work when you want to explicitly log your errors (as in client.report_exception()), and 2) Error Reporting has caught previous errors without my setting up, so it seems to be possible without having to install those client libraries.
There are so many variables at play here that I'm not sure where to start. I may not have provided enough information here in order to properly diagnose this (docker setup, kubernetes configuration, etc.), but I figured I may be misunderstanding something fundamental about this process, and that someone could be so kind as to enlighten me.
EDIT:
I found in the GKE documentation how to ensure that the Stackdriver logging is enabled for my cluster. Still no luck.
UPDATE (partial solution):
I set DEBUG = True in settings.py, which prompted the errors to start being logged in Stackdriver Logging and Error Reporting. I've gone ahead and done this for my canary environment, but it's less than ideal, since it exposes some of my backend code. Still not sure why this works for my other GCP project without running in debug mode.
Related
I have set up a Python flask through IIS. I have set up IIS and wfastcgi. I have verified below -
Install IIS and CGI
Install wfastcgi
Created webconfig file
Configure IIS directories and permissions
When i try to URL I am seeing below -
HTTP Error 500.0 - Internal Server Error
An unknown FastCGI error occurred
Most likely causes:
IIS received the request; however, an internal error occurred during the processing of the request. The root cause of this error depends on which module handles the request and what was happening in the worker process when this error occurred.
IIS was not able to access the web.config file for the Web site or application. This can occur if the NTFS permissions are set incorrectly.
IIS was not able to process configuration for the Web site or application.
The authenticated user does not have permission to use this DLL.
The request is mapped to a managed handler but the .NET Extensibility Feature is not installed.
Things you can try:
Ensure that the NTFS permissions for the web.config file are correct and allow access to the Web server's machine account.
Check the event logs to see if any additional information was logged.
Verify the permissions for the DLL.
Install the .NET Extensibility feature if the request is mapped to a managed handler.
Create a tracing rule to track failed requests for this HTTP status code. For more information about creating a tracing rule for failed requests, click here.
What I am missing please help
I am trying to learn how to use Google Appengine with python, using PyCharm Pro as trial. In have installed the Cloud SDK and it is active as indicated by getting access to both admin page and the Hello World! page. I had to overcome cannot bind problem by changing the port using dev_appserver.py
After that when I try to run the app in python, I get the following lines
INFO 2017-05-16 03:23:13,314 sdk_update_checker.py:231] Checking for updates to the SDK.
INFO 2017-05-16 03:23:13,458 sdk_update_checker.py:259] The SDK is up to date.
INFO 2017-05-16 03:23:13,517 api_server.py:272] Starting API server at: http://localhost:51009
INFO 2017-05-16 03:23:13,520 dispatcher.py:205] Starting module "default" running at: http://127.0.0.1:9000
INFO 2017-05-16 03:23:13,522 admin_server.py:116] Starting admin server at: http://localhost:8000
and it stalls there, nothing else happens and I get no error messages. I tried taking all the code out of the app.py file and just put a simple print('hello') and still does the same. tried restarting and reinitializing the Cloud SDK as well as the whole computer and still the same.
I am using MacOS and python venv as interpreter with webapp2 installed.
I looked in this forum and in multiple others and didn't see this problem described any where.
Also worth mentioning that the firewall is off
I had a Python Flask website that ran fine when I ran the app pool as my own account. When I tried changing this to a service account (which should have permissions), I get the following error
HTTP Error 500.0 - Internal Server Error
The page cannot be displayed because an internal server error has
occurred.
Most likely causes: •IIS received the request; however, an internal
error occurred during the processing of the request. The root cause of
this error depends on which module handles the request and what was
happening in the worker process when this error occurred. •IIS was not
able to access the web.config file for the Web site or application.
This can occur if the NTFS permissions are set incorrectly. •IIS was
not able to process configuration for the Web site or application.
•The authenticated user does not have permission to use this DLL. •The
request is mapped to a managed handler but the .NET Extensibility
Feature is not installed.
Detailed Error Information: Module FastCgiModule
Notification ExecuteRequestHandler
Handler Python Flask
Error Code 0x80070542
Requested URL http://qatools:80/
Physical Path D:\PythonWebService\QATools
Logon Method Anonymous
Logon User Anonymous
If I change it back to run as my own account, it works. Any thoughts?
FYI after some tinkering, I changed the app pool to run with Managed Pipeline mode of "Classic" instead of "Integrated" and it worked.
I have OpenStack UI running. I made some changes in the local-setting.py file and restarted the Horizon service using service httpd restart and try to hit OpenStack UI but it returns an error: "HTTP 400 Bad request".
When I revert back all changes, restart the service and try again the error is still there.
Please help me !!
Total agree with above comments, check your apache or httpd log files.
Most probably error cause due to improper info.
added (for centos) under below file.
/etc/openstack-dashboard/local_settings
ALLOWED_HOSTS = ['XX.XX.XX.XX', 'ControllerName', 'localhost']
Hopefully this will resolve issue.
if you are running on Horizon on Apache then you need to check apache logs to identify issue. Horizon logs will not contains anything if apache was not able to execute WSGI script for Horizon.
What are possible causes of "The resource could not be found" error in Pyramid?
From my limited exeperiece debugging my Pyramid app, errors with routes and Python syntax error trigger a more explicit error message.
Is there a way to get the development server to trace a request (similiar to the way running a rails server does it)? I tried running paster in verbose mode, but am not getting any more information.
So that's a HTTPNotFound error, which Pyramid has a debugging option for to help you out.
https://docs.pylonsproject.org/projects/pyramid/en/1.2-branch/narr/viewconfig.html#notfound-errors