I'm in the planning phase of an Android app which synchronizes to a web app. The web side will be written in Python with probably Django or Pyramid while the Android app will be straightforward java. My goal is to have the Android app work while there is no data connection, excluding the social/web aspects of the application.
This will be a run-of-the-mill app so I want to stick to something that can be installed easily through one click in the market and not require a separate download like CloudDB for Android.
I haven't found any databases that support this functionality so I will write it myself. One caveat with writing the sync logic is there will be some shared data between users that multiple users will be able to write to. This is a solo project so I thought I'd through this up here to see if I'm totally off-base.
The app will process local saves to the local sqlite database and then send messages to a service which will attempt to synchronize these changes to the remote database.
The sync service will alternate between checking for messages for the local app, i.e. changes to shared data by other users, and writing the local changes to the remote server.
All data will have a timestamp for tracking changes
When writing from the app to the server, if the server has newer information, the user will be warned about the conflict and prompted to overwrite what the server has or abandon the local changes. If the server has not been updated since the app last read the data, process the update.
When data comes from the server to the app, if the server has newer data overwrite the local data otherwise discard it as it will be handled in the next go around by the app updating the server.
Here's some questions:
1) Does this sound like overkill? Is there an easier way to handle this?
2) Where should this processing take place? On the client or the server? I'm thinking the advantage of the client is less processing on the server but if it's on the server, this makes it easier to implement other clients.
3) How should I handle the updates from the server? Incremental polling or comet/websocket? One thing to keep in mind is that I would prefer to go with a minimal installation on Webfaction to begin with as this is the startup.
Once these problems are tackled I do plan on contributing the solution to the geek community.
1) Looks like this is pretty good way to manage your local & remote changes + support offline work. I don't think this is overkill
2) I think, you should cache user's changes locally with local timestamp until synchronizing is finished. Then server should manage all processing: track current version, commit and rollback update attempts. Less processing on client = better for you! (Easier to support and implement)
3) I'd choose polling if I want to support offline-mode, because in offline you can't keep your socket open and you will have to reopen it every time when Internet connection is restored.
PS: Looks like this is VEEERYY OLD question... LOL
Related
I would like to implement an observer design pattern. My thoughts are, if this is deployed, and another system update has occurred and restarted the server, would the observers/subscribers be lost when the server has restarted?
Sorry for this newbie question.
Your question doesn't specify any tools you are using neither the deployment methods and softwares so the best I can say is that any non-persistent data will be deleted on restart. This counts for the subscribers your server has saved in a variable, for example.
In web development, you go around this problem (and that of lost of connection) by having "temporary subscriptions" and by not implementing functionalities with solutions needing consistency in connection.
However, what you could do is give the clients some sort of unique id which could be stored in a database along with data that can restore the connection.
This is either a really good or really stupid question, but I find its worth asking --
I'm making a Django app that runs on a device as an interfaces. Is there any reason to think I could just use the python manage.py runserver and go no further? Or is there a better way to do this?
Installing the full web-bundle for local-network-devices seems excessive, hence my question. (Perhaps there is not a great deal of overhead using the full-web-setup -- I dunno). This is currently on the Raspberry pi, but for prototype purposes. The end-product will not necessarily be Pi.
It depends on how many users you're expecting to connect at once. The Django development server is suitable for only one connection at a time. It isn't good at handling multiple sessions and is not designed to stay up for long periods of time. This is the reason the docs clearly state
do not use this server in a production setting!
That said, running with an application server like gunicorn may be all you need to support hosting multiple users. It uses multiple workers so that if one user's request crashes, it can continue serving all the other users.
https://docs.djangoproject.com/en/1.11/howto/deployment/wsgi/gunicorn/
Finally, if you're serving up a lot of assets like images or videos, you should really have a full web server like Nginx to intercept asset URLs so they're not served through Django itself. Django should not be serving assets directly in production.
https://www.digitalocean.com/community/tutorials/how-to-serve-django-applications-with-uwsgi-and-nginx-on-ubuntu-14-04
I am writing a client-server type application. The server side gathers constantly changing data from other hardware and then needs to pass it to multiple clients (say about 10) for display. The server data gathering program will be written in Python 3.4 and run on Debian. The clients will be built with VB Winforms on .net framework 4 running on Windows.
I had the idea to run a lightweight web server on the server-side and use system.net.webclient.downloadstring calls on the client side to receive it. This is so that all the multi-threading async stuff is done for me by the web server.
Questions:
Does this seem like a good approach?
Having my data gathering program write a text file for the web server to serve seems unnecessary. Is there a way to have the data in memory and have the server just serve that so there is no disk file intermediary? Setting up a ramdisk was one solution I thought of but this seems like overkill.
How will the web server deal with the data being frequently updated, say, once a second? Do webservers deal with this elegantly or is there a chance the file will be served whilst it is being written to?
Thanks.
1) I am not very familiar with Python, but for the .net application you will likely want to push change notifications to it, rather than pull. The system.net.webclient.downloadstring is a request (pull). As I am not a Python developer I cannot assist in that.
3) As you are requesting data, it is possible to create some errors of the read/write while updating and reading at the same time. Even if this does not happen your data may be out of date as soon as you read it. This can be an acceptable problem, this just depends of how critical your data is.
This is why I would do a push notification rather than a pull. If worked correctly this can keep data synced and avoid some timing issues.
I am planning to integrate real time notifications into a web application that I am currently working on. I have decided to go with XMPP for this and selected openfire server which i thought to be suitable for my needs.
The front end uses strophe library to fetch the notifications using BOSH from my openfire server. However the notices are the notifications and other messages are to be posted by my application and hence I think this code needs to reside at the backend.
Initially I thougt of going with PHP XMPP libraries like XMPHP and JAXL but then I think that this would cause much overhead as each script will have to do same steps like connection, authentication etc. and I think this would make the PHP end a little slow and unresponsive.
Now I am thinking of creating a middle-ware application acting as a web service that the PHP will call and this application will handle the stuff with XMPP service. The benefit with this is that this app(a server if you will) will have to connect just once and the it will sit there listening on a port. also I am planning to build it in a asynchronous way such that It will first take all the requests from my PHp app and then when there are no more requests; go about doing the notification publishing stuff. I am planninng to create this service in Python using SleekXMPP.
This is just what I planned. I am new to XMPP and this whole web service stuff ans would like to take your comments on this regarding issues like memory and CPU usage, advantages, disadvantages, scalability issues,security etc.
Thanks in advance.
PS:-- also if something like this already exists(although I didn't find after a lot of Googling) Please direct me there.
EDIT ---
The middle-level service should be doing the following(but not limited to):
1. Publishing notifications for different level of groups and community pages.
2. Notification for single user on some event.
3. User registration(can be done using user service plugin though).
EDIT ---
Also it should like to create pub-sub nodes and subscribe and unsubscribe users from these pub-sub nodes.
Also I want to store the notifications and messages in a database(openfire doesn't). Would that be a good choice?
It seems to me like XMPP is a bit of a heavy-weight solution for what you're doing, given that communication is just one-way and you're just sending notifications (no real-time multi-user chat etc.).
I would consider using something like Socket.io (http://socket.io) for the server<=>client channel along with Redis (http://redis.io) for persistence.
Im new to Cassandra, so bear with me.
So, I am building a search engine using Cassandra as the db. I am interacting with it through Pycassa.
Now, I want to output Cassandra's response to a webpage, having the user submitted a query.
I am aware of tools such as django, fastCGI, SCGI, etc to allow python to web interaction. However, how does one run a python script on a webserver without turning this server into a single point of failure ( i.e., if this server dies than the system is not accessible by the user) - and therefore negating one purpose of Cassandra?
I've seen this problem before - sometimes people need much more CPU power and bandwidth to generate and serve some server generated HTML and images than they do to run the actual query in Cassandra. For one customer, this was many 10's times the number of servers serving the front end of the website than in their Cassandra cluster.
You'll need to load balance between these front end servers somehow - investigate running haproxy on a few dedicated machines. Its quick and easy to configure, and similarly easy to reconfigure when your setup changes (unlike DNS, which can take days to propagate changes). I think you can also configure nginx to do the same. If you keep per-session information in your front end servers, you'll need each client to go to the same front end server for each request - this is called "session persistence", and can be achieved by hashing the client's IP to pick the front end server. Haproxy will do this for you.
However this approach will again create a SPOF in your configuration (the haproxy server) - you should run more than one, and potentially have a hot standby. Finally you will need to somehow balance load between your haproxies - we typically use round robin DNS for this, as the nodes running haproxy seldom change.
The benefit of this system is that you can easily scale up (and down) the number of front end servers without changing your DNS. You can read (a little bit) more about the setup I'm referring to at: http://www.acunu.com/blogs/andy-ormsby/using-cassandra-acunu-power-britains-got-talent/
Theo Schlossnagle's Scalable Internet Architectures covers load balancing and a lot more. Highly recommended.