I want to make it that one user on the site can chat request another user on my Django site. I want the requestee to get a realtime box that say: "Do you want to chat?"
How does the following client polling approach sound:
user1 clicks on users2 nickname, generating a POST request to some /message/requests, which creates a Message of type CHAT_REQUEST in the database. Meanwhile, a Javascript piece at user2's browser, repeatedly queries the server for message updates. When it receives a Message of type CHAT_REQUEST it opens a popup...
The problem with this approach seems to be the database access.
If the client is polling every 10 seconds, and 100 users leave their browser windows open, that is 10 database requests per seconds.
Would it be better to store these messages not in the database, but in Django RAM or session information? Or will this database table be cached in RAM with PostgreSQL, and the retrieval fast?
A database table for this would put a load on your server, as you said, but might be useful if you want to keep a record of these requests for whatever reason.
Using something like memcached or an AMQP server might give you better performance. If you like you could even use a higher-performance key-value-store such as Tokyo Cabinet / Tokyo Tyrant.
I suggest you look for a "COMET like" communication instead of "AJAX like" if you worry about server performance and bandwidth usage.
By the way, REDIS looks very well suited for handling that kind of in-memory data structures.
Related
I have JS running and essentially getting user entries from my HTML session storage and pushing these to a DB. I also need to use a HTTP request to pass a json object containing the entries to a python file hosted somewhere else.
Does anyone have any idea of documentation I could look at, or perhaps how to get JSON objects from JS to Python.
My client does not want me to grab the variables directly from the DB.
You have to create some sort of communication channel between the javascript and python code. This could be anything, SOAP, HTTP, RPC, any number of and flavor of message queue.
If nothing like that is in place, it's quite the long way around. A complex application might warrant you doing this, think micro services communicating across some sort of service bus. It's a sound strategy and perhaps that's why your client is asking for it.
You already have Firebase, though! Firebase is a real-time database that already has many of the characteristics of a queue. The simplest and most idiomatic thing to do would be to let the python code be notified of changes by Firebase: Firebase as service bus is a nice strategy too!
I'm looking for a way to keep track of users that are online/offline. So if I present all users in a list i could have an icon or some kind of flag to show this. Is this built in in Django's default Auth system?
My first thought was to simply have a field in my profiles called last_logout in the models and update it with the date/time each time user logged out.
With this info and the built in last_login I should be able to make some kind of function to determine if the user is loggedin/online right?
Or should I just have a boolean field called "online" that I can change when user logs in and out?
With only django it will be hard to do. For such task async frameworks are more suitable.
For example, tornado.
Users won't do logout explicitly every time they go offline. They just close their browser and that's it. You can't know it with only django auth app. It is not designed for such tasks.
Even if you will check for not expired session, it not gives you all online users, because session can be non-expired for 30 days.
So to get real online users, possible solutions are:
Every user will send some data via javascript to your server, for example every 10 seconds. You can fetch that data on server and put user into cache and set cache key to be alive for 10 seconds. So when you need to know, who are online now, you'll check your cache. But it is not a good solution, because it will need a lot of server resources.
Use async framework (tornado) at server side (you can setup separate process for exact requests). And use websockets (SockJS is a good library for that at client side). It is a more complicated solution, but it is better.
You have to consider what exactly means for the users to be "online". Since any user can close the browser window any time and without the server knowing about that action, you'd end up having lots of false "online" users.
You have two basic options:
Keep track of the user's last activity time. Every time the user loads a page you'd update the value of the timer. To get a list of all online users you'd need to select the ones with an activity before X minutes. This is what is done by some web forums.
Open a websocket, long polling connection or some heartbeat to the server. This is what Facebook chat does. You'd need more than just django, since to keep a connection open another kind of server-side resources are needed.
Suppose I need to build a web application where each client will be simulating their trading strategy using historical stock data. The data will be provided by a 3rd party vendor over the internet: for example, fetching historical data for a single stock based on stock ticker through HTTP call. Also, I am planning to use Django as a back-end framework.
Here is my question:
I would like to be able to prefetch and cache the data on the server side, so that each client's request would not need to do an HTTP call again, but get it from the shared resource. I guess, storing it in database, like SQL could be one solution. However, is there a way to use memory shared between clients in Django on the backend side? Any pointer or suggestion would be very helpful. Thanks.
This sounds like a fine thing to store in a share cache, like memcache or redis (or, yes, even an SQL database-backed cache).
You should read https://docs.djangoproject.com/en/dev/topics/cache/; this can explain how you can store the result of your HTTP call under a cache key and then retrieve it. The caching works the same regardless of what backend (memcache, redis, local memory, SQL DB) you use, so you can test this out with the local-memory cache or DB cache and, if you like it, move to a better solution like memcache.
There are many caching strategies you could use here, but a nice place to start rather then storing the data in a SQL database you could store the data in something like Memcached https://docs.djangoproject.com/en/dev/topics/cache/#memcached. Without more information I couldn't get more specific than that.
So, in order to avoid the "no one best answer" problem, I'm going to ask, not for the best way, but the standard or most common way to handle sessions when using the Tornado framework. That is, if we're not using 3rd party authentication (OAuth, etc.), but rather we have want to have our own Users table with secure cookies in the browser but most of the session info stored on the server, what is the most common way of doing this? I have seen some people using Redis, some people using their normal database (MySQL or Postgres or whatever), some people using memcached.
The application I'm working on won't have millions of users at a time, or probably even thousands. It will need to eventually get some moderately complex authorization scheme, though. What I'm looking for is to make sure we don't do something "weird" that goes down a different path than the general Tornado community, since authentication and authorization, while it is something we need, isn't something that is at the core of our product and so isn't where we should be differentiating ourselves. So, we're looking for what most people (who use Tornado) are doing in this respect, hence I think it's a question with (in theory) an objectively true answer.
The ideal answer would point to example code, of course.
Here's how it seems other micro frameworks handle sessions (CherryPy, Flask for example):
Create a table holding session_id and whatever other fields you'll want to track on a per session basis. Some frameworks will allow you to just store this info in a file on a per user basis, or will just store things directly in memory. If your application is small enough, you may consider those options as well, but a database should be simpler to implement on your own.
When a request is received (RequestHandler initialize() function I think?) and there is no session_id cookie, set a secure session-id using a random generator. I don't have much experience with Tornado, but it looks like setting a secure cookie should be useful for this. Store that session_id and associated info in your session table. Note that EVERY user will have a session, even those not logged in. When a user logs in, you'll want to attach their status as logged in (and their username/user_id, etc) to their session.
In your RequestHandler initialize function, if there is a session_id cookie, read in what ever session info you need from the DB and perhaps create your own Session object to populate and store as a member variable of that request handler.
Keep in mind sessions should expire after a certain amount of inactivity, so you'll want to check for that as well. If you want a "remember me" type log in situation, you'll have to use a secure cookie to signal that (read up on this at OWASP to make sure it's as secure as possible, thought again it looks like Tornado's secure_cookie might help with that), and upon receiving a timed out session you can re-authenticate a new user by creating a new session and transferring whatever associated info into it from the old one.
Tornado designed to be stateless and don't have session support out of the box.
Use secure cookies to store sensitive information like user_id.
Use standard cookies to store not critical information.
For storing large objects - use standard scheme - MySQL + memcache.
The key issue with sessions is not where to store them, is to how to expire them intelligently. Regardless of where sessions are stored, as long as the number of stored sessions is reasonable (i.e. only active sessions plus some surplus are stored), all this data is going to fit in RAM and be served fast. If there is a lot of old junk you may expect unpredictable delays (the need to hit the disk to load the session).
There isn't anything built directly into Tornado for this purpose. As others have commented already, Tornado is designed to be a very fast async framework. It is lean by design. However, it is possible to hook in your own session management capability. You need to add a preamble section to each handler that would create or grab a session container. You will need to store the session ID in a cookie. If you are not strictly HTTPS then you will want to use a secure cookie. The session persistence can be any technology of your choosing such as Redis, Postgres, MySQL, a file store, etc...
There is a Github project that provides session management for Tornado. Even if you decide not to use it, it can provide insight into how to structure your own session management. The Github project is called dustdevil. Full disclosure - we created this several years ago but find it very easy to use and have it in active use today.
For my final year project I plan to code a cloud in Python. The client will be written in Java by the other member of my team. The client will have a tabbed interface and it will provide a text editor, a media player, a couple of small Java based games and a maybe a few more services.
The server will work like this:
1) Validate the user.
2) Send a file, called "dump" to the user. Dump will contain all the file names and file types that the user created by himself or the files which the user can read/write. This info will be fetched from the database.
3) The tabs in the client will display the file types associated with the tab application. e.g the media tab will only select and show the media files from the dump readable by user. The text editor tab will show only the txt files from the dump readable by the user.
4) A request to open the file will send the file back to client, which the associated application will open.
5) All the changes made to the files and all the actions (overwriting, saving, deleting etc.) will be sent back to the server along with the new object. Something similar will be done to the newly created objects.
My Questions are:
What are the best approaches for the communication between the client and the server. For the dump I plan to use some sort of encrypted XML file. For the other way round, I don't have a clue :/.
For easy integration with the database, I was planning to use Django (which I started few days back). How can I send my requests from the client to the server (without Django I'd use SQL queries) and the files from the server to the client? Maybe GET and POST will work for the former problem? Any other suggestions?
Q1: how should I transfer data between client/server securely
A: HTTPS to support encryption & JSON to serialise objects between languages (Python/Java) seems to be the most natural. You could experiment with XML-RPC over SSL or TSL if you want to be creative.
Q2: How do I send queries to the server's db?
A: My first response is to say talk to the person coding the server, and see what's easiest on that end. However, I think that your client should stick to HTTP. The server developer would ensure the server supports RESTful URIs. Then your client only access a URI and have the results processed by the server.
At its most raw, this could be implemented like this:
https://www.example.com/db?q="SELECT * FROM docs"
There are smarter ways to do it, but you get the idea.
If you're going to use a web framework on the server, it makes sense to use an HTTP-based protocol. The downside is that only the client can initiate a connection (e.g., the client needs to first ask for the "dump" file), but a simple GET request will suffice (remember, the server can send anything in the HTTP response, including your XML file).
Regarding encryption, it's best to use an existing protocol like HTTPS. There are well-vetted libraries that will correctly establish a secure connection between your client and the server.
Overall, I'm advocating the highest-level protocols that are appropriate for your application. HTTP(S) goes hand-in-hand with your web-based architecture, so make use of it.
Stick to Django. It's really productive. I would use JSON instead of XML. More convenient. import json. This should help you in communicating between client-server.
Also cloud computing is just a recent word that's just thrown around for (client+server+some services). Oh by the way all that you want to do can be completely done in Django itself. No need to go to JAVA.
Django is Cool :)