I'm trying to debug an internal server error in my django app, running on heroku. I'm
completely new to all of this web server stuff so I really have no idea what to do.
It seems like the stdout output is sometimes getting logged in heroku logs and sometimes not. I was reasonably sure that the program was reaching a certain line but the prints at that point are simply not showing up.
I am seeing the 500 error in my heroku logs file, but there is no stack trace or anything else in there. I am trying to create a web server to respond to GET and POST requests from various applications I have running, meaning I don't know how to debug this in a web browser, if thats even applicable. The current error is on a POST request sent to the webserver. I can't replicate this locally because the Http module I am using, http://www.python-requests.org/en/latest/ seems to be unable to connect to a local ip address.
I have done some extensive googling for the last hour and I haven't found any help. Do I need to enable logging or something somewhere in heroku? I am completely new to this so please be explicit in your explanations. I have heard mention of a way to get stack traces emailed to you but I haven't seen an explanation of how to do that. is that possible?
Thanks!
I would recommend 2 things in this case:
First: use python's logging facility rather than print statements (http://docs.python.org/2/howto/logging-cookbook.html). This gives you much more control over where your statements end up, and allows you to filter them.
Second: use a logging add-on. This vastly increases the amount of logging you can store (loggly keeps all your logs for 24 hours even in the "free" size), so you don't have to worry about the relevant information falling out before you get around to looking at it.
Related
This is either a really good or really stupid question, but I find its worth asking --
I'm making a Django app that runs on a device as an interfaces. Is there any reason to think I could just use the python manage.py runserver and go no further? Or is there a better way to do this?
Installing the full web-bundle for local-network-devices seems excessive, hence my question. (Perhaps there is not a great deal of overhead using the full-web-setup -- I dunno). This is currently on the Raspberry pi, but for prototype purposes. The end-product will not necessarily be Pi.
It depends on how many users you're expecting to connect at once. The Django development server is suitable for only one connection at a time. It isn't good at handling multiple sessions and is not designed to stay up for long periods of time. This is the reason the docs clearly state
do not use this server in a production setting!
That said, running with an application server like gunicorn may be all you need to support hosting multiple users. It uses multiple workers so that if one user's request crashes, it can continue serving all the other users.
https://docs.djangoproject.com/en/1.11/howto/deployment/wsgi/gunicorn/
Finally, if you're serving up a lot of assets like images or videos, you should really have a full web server like Nginx to intercept asset URLs so they're not served through Django itself. Django should not be serving assets directly in production.
https://www.digitalocean.com/community/tutorials/how-to-serve-django-applications-with-uwsgi-and-nginx-on-ubuntu-14-04
I am running socket.io on an Apache server through Python Flask. We're integrating it into an iOS app (using the Socket.IO-Client-Swift library) and we're having a weird issue.
From the client side code in the app (written in Swift), I can view the actual connection log (client-side in XCode) and see the connection established from the client's IP and the requests being made. The client never receives the information back (or any information back; even when using a global event response handler) from the socket server.
I wrote a very simple test script in Javascript on an HTML page and sent requests that way and received the proper responses back. With that said, it seems to likely be an issue with iOS. I've found these articles (but none of them helped fix the problem):
https://github.com/nuclearace/Socket.IO-Client-Swift/issues/95
https://github.com/socketio/socket.io-client-swift/issues/359
My next thought is to extend the logging of socket.io to find out exact what data is being POSTed to the socket namespace. Is there a way to log exactly what data is coming into the server (bear in mind that the 'on' hook on the server side that I've set up is not getting any data; I've tried to log it from there but it doesn't appear to even get that far).
I found mod_dumpio for Linux to log all POST requests but I'm not sure how well it will play with multi-threading and a socket server.
Any ideas on how to get the exact data being posted so we can at least troubleshoot the syntax and make sure the data isn't being malformed when it's sent to the server?
Thanks!
Update
When testing locally, we got it working (it was a setting in the Swift code where the namespace wasn't being declared properly). This works fine now on localhost but we are having the exact same issues when emitting to the Apache server.
We are not using mod_wsgi (as far as I know; I'm relatively new to mod_wsgi, apologies for any ignorance). We used to have a .wsgi file that called the main app script to run but we had to change that because mod_wsgi is not compatible with Flask SocketIO (as stated in the uWSGI Web Server section here). The way I am running the script now is by using supervisord to run the .py file as a daemon (using that specifically so it will autostart in the event of a server crash).
Locally, it worked great once we installed the eventlet module through pip. When I ran pip freeze on my virtual environment on the server, eventlet was installed. I uninstalled and reinstalled it just to see if that cleared anything up and that did nothing. No other Python modules that are on my local copy seem to be something that would affect this.
One other thing to keep in mind is that in the function that initializes the app, we change the port to port 80:
socketio.run(app,host='0.0.0.0',port=80)
because we have other API functions that run through a domain that is pointing to the server in this app. I'm not sure if that would affect anything but it doesn't seem to matter on the local version.
I'm at a dead end again and am trying to find anything that could help. Thanks for your assistance!
Another Update
I'm not exactly sure what was happening yet but we went ahead and rewrote some of the code, making sure to pay extra special attention to the namespace declarations within each socket event on function. It's working fine now. As I get more details, I will post them here as I figure this will be something useful for other who have the same problem. This thread also has some really valuable information on how to go about debugging/logging these types of issues although we never actually fully figured out the answer to the original question.
I assume you have verified that Apache does get the POST requests. That should be your first test, if Apache does not log the POST requests coming from iOS, then you have a different kind of problem.
If you do get the POST requests, then you can add some custom code in the middleware used by Flask-SocketIO and print the request data forwarded by Apache's mod_wsgi. The this is in file flask_socketio/init.py. The relevant portion is this:
class _SocketIOMiddleware(socketio.Middleware):
# ...
def __call__(self, environ, start_response):
# log what you need from environ here
environ['flask.app'] = self.flask_app
return super(_SocketIOMiddleware, self).__call__(environ, start_response)
You can find out what's in environ in the WSGI specification. In particular, the body of the request is available in environ['wsgi.input'], which is a file-like object you read from.
Keep in mind that once you read the payload, this file will be consumed, so the WSGI server will not be able to read from it again. Seeking the file back to the position it was before the read may work on some WSGI implementations. A safer hack I've seen people do to avoid this problem is to read the whole payload into a buffer, then replace environ['wsgi.input'] with a brand new StringIO or BytesIO object.
Are you using flask-socketio on the server side? If you are, there is a lot of debugging available in the constructor.
socketio = SocketIO(app, async_mode=async_mode, logger=True, engineio_logger=True)
I'm at beginner level, I've given my 2 months at basics level in python. I'm familiar with algorithms, matrix and etc programming stuffs into python.
I want to give my whole life to django. But problem is i don't anything about web python stuffs except starting server on it.
I've started my server using:
import http.server
import socketserver
PORT = 8000
def playServer():
handler = http.server.SimpleHTTPRequestHandler
httpd = socketserver.TCPServer(("", PORT), handler)
print("Serving at port", PORT)
httpd.serve_forever()
But when i call my simple web page, It still shows the codes of python, Please take a look at SCREENSHOT.
Please help me, How to make web page using python coding? Also let me know, Am i capable to start giving time with django? I've heard django is a framework, So i should give basics time to learn some python web page stuffs? right?
HELP WOULD BE APPRECIATED!!
https://docs.djangoproject.com/en/1.8/intro/tutorial01/, you have a lot to learn my friend but this tutorial will teach you how to get setup/install Django and give you a good base to start up with. Good luck with learning my man, any questions just comment below. BTW what you just did was not even Django at all, it was just starting a local webserver in python.
Obviously, teaching yourself is the hardest way to learn any skill. If you really do want to learn django, or anything else life, you're going to have to commit yourself, be patient, and ready to fail.
If you're really serious about this, I would work through the Django Book. This covers every aspect of django to get you started.
You're going to run into problems early and often. When you do so, stop and consort the docs, never be afraid to look directly at the source code and as always, come to SO with specific questions.
To answer you're more specific question, Django is nice in the fact that it hides all of the code you have above. For instance, once you have it configured, the line below will start a local server and you'll be able to interact with your web app through a web browser.
python manage.py runserver
Looks like you're trying to execute the python script directly from browser.
First you must call the script from the command-line, then check the address on browser.
Also, you need to call the playServer() method at the end of the file to start the HTTP server.
then go the the address:port on your browser. It'll look something like this:
Now about the second question,
I'll highly recommend you to start with Flask instead of Django.
It's simpler than django and best for beginners, and does less magic.
Also, as soon as you're comfortable with flask, you can easily move to django and have more fun while playing with it thereafter.
I've used web.py to create a web service that returns results in json.
I run it on my local box as python scriptname.py 8888
However, I now want to run it on a linux box.
How can I run it as a service on the linux box?
update
After the answers it seems like the question isn't right. I am aware of the deployment process, frameworks, and the webserver. Maybe the following back story will help:
I had a small python script that takes as input a file and based on some logic splits that file up. I wanted to use this script with a web front end I already have in place (Grails). I wanted to call this from the grails application but did not want to do it by executing a command line. So I wrapped the python script as a webservice. which takes in two parameters and returns, in json, the number of split files. This webservice will ONLY be used by my grails front end and nothing else.
So, I simply wish to run this little web.py service so that it can respond to my grails front end.
Please correct me if I'm wrong, but would I still need ngix and the like after the above? This script sounds trivial but eventually i will be adding more logic to it so I wanted it as a webservice which can be consumed by a web front end.
In general, there are two parts of this.
The "remote and event-based" part: Service used remotely over network needs certain set of skills: to be able to accept (multiple) connections, read requests, process, reply, speak at least basic TCP/HTTP, handle dead connections, and if it's more than small private LAN, it needs to be robust (think DoS) and maybe also perform some kind of authentication.
If your script is willing to take care of all of this, then it's ready to open its own port and listen. I'm not sure if web.py provides all of these facilities.
Then there's the other part, "daemonization", when you want to run the server unattended: running at boot, running under the right user, not blocking your parent (ssh, init script or whatever), not having ttys open but maybe logging somewhere...
Servers like nginx and Apache are built for this, and provide interfaces like mod_python or WSGI, so that much simpler applications can give up as much of the above as possible.
So the answer would be: yes, you still need Nginx or the likes, unless:
you can implement it yourself in Python,
or you are using the script on localhost only and are willing to take some
risks of instability.
Then probably you can do on your own.
try this
python scriptname.py 8888 2>/dev/null
it will run as daemon
I am trying to configure the Python mini-framework CherryPy with FastCGI (actually fcgid) on Apache. I am on a shared host, so I don't have access to httpd.conf, just htaccess. I have followed these tutorials to no avail:
http://tools.cherrypy.org/wiki/FastCGIWSGI
http://tools.cherrypy.org/wiki/BluehostDeployment
I keep getting 500 errors w/ the Apache logs saying "Premature end of script headers". I have tried everything (permissions/shebangs/full-paths/deamonized/not-daimonized). I know Apache is correctly executing my .fcgi, because I am able to print to the error log from python, but that's it. Has anyone out there successfully installed CherryPy or any other framework on a shared host before? Your help would be greatly appreciated. Thanks.
Apache + Bluehost + fastcgi + cherrypy + wsgi is unfortunately a lot of pieces. I wish I had a year to write the Definitive Guide for you, but alas. You might gain some insight from the rather long mailing list thread which resulted in those links you posted.
An idea: make sure your .fcgi file has a reference to the correct python executable in the initial line:
#!/usr/bin/python
I had to get Django running with fcgi on Bluehost and apache using the wrong python environment was my problem (worked from the shell, but not from the web/apache).
Other than that, if you can print to the error log from your code, can you confirm that the your code is correctly executed, without any exceptions, when you access the web page? (not when running from the shell).
The Bluehost article has been the best resource, but I didn't carefully read the part about getting the latest patches (the beginning of step 3). At the time of the article, and even now with CherryPy version 3.1.2, you can't do 'dynamic mode' fcgi (when apache spawns the process). more here. Dynamic mode is basically essential if you are on a shared host.
I have checked out the trunk (3.2.0rc1), and after jumping through some hoops, got it to work. I followed step 5, method C in the bluehost article. Here was the stuff in the main of my cherryd.fcgi:
if __name__ == '__main__':
cherrypy.config.update({
'server.socket_port': None,
'server.socket_host': None,
'server.socket_file': None
})
start( daemonize=False, fastcgi=True, imports=["hello"])
Also, in cherrypy/process/servers.py, I had to change the following line:
# from this
# if not hasattr(socket.socket, 'fromfd'):
# to this
if not hasattr(socket, 'fromfd'):
So, it is possible to get it to work, but it feels kind of hacky. You should wait for the final release of version 3.2.0, or do what I did and check out Web.py. I was able to get it working with my shared host very easily (docs explain fastcgi/htaccess well).
In your webserver's log file, it should actually show what the output was that confused it. Are you sure you're looking in the error log as well as the access log?