Is there anything specific that can be done to help make a Django Channels server less susceptible to light or accidental DDoS attack or general load increase from websocket/HTTP clients? Since Channels is not truly asynchronous (still workers behind the scenes), I feel like it would be quite easy to take down a Channels-based website - even with fairly simple hardware. I'm currently building an application on Django Channels and will run some tests later to see how it holds up.
Is there some form of throttling built in to Daphne? Should I implement some application-level throttling? This would still be slow since a worker still handles the throttled request, but the request can be much faster. Is there anything else I can do to attempt to thwart these attacks?
One thought I had was to always ensure there are workers designated for specific channels - that way, if the websocket channel gets overloaded, HTTP will still respond.
Edit: I'm well aware that low-level DDoS protection is an ideal solution, and I understand how DDoS attacks work. What I'm looking for is a solution built in to channels that can help handle an increased load like that. Perhaps the ability for Daphne to scale up a channel and scale down another to compensate, or a throttling method that can reduce the weight per request after a certain point.
I'm looking for a daphne/channels specific answer - general answers about DDoS or general load handling are not what I'm looking for - there are lots of other questions on SO about that.
I could also control throttling based on who's logged in and who is not - a throttle for users who are not logged in could help.
Edit again: Please read the whole question! I am not looking for general DDoS mitigation advice or explanations of low-level approaches. I'm wondering if Daphne has support for something like:
Throttling
Dynamic worker assignment based on queue size
Middleware to provide priority to authenticated requests
Or something of that nature. I am also going to reach out to the Channels community directly on this as SO might not be the best place for this question.
I've received an answer from Andrew Godwin. He doesn't use StackOverflow so I'm posting it here on his behalf.
Hi Jamie,
At the moment Channels has quite limited support for throttling - it pretty much consists of an adjustable channel size for incoming connections which, when full, will cause the server to return a 503 error. Workers are load-balanced based on availability due to the channels design, so there's no risk of a worker gaining a larger queue than others.
Providing more advanced DoS or DDoS protection is probably not something we can do within the scope of Channels itself, but I'd like to make sure we provide the appropriate hooks. Were there particular things you think we could implement that would help you write some of the things you need?
(It's also worth bearing in mind that right now we're changing the worker/consumer layout substantially as part of a major rewrite, which is going to mean different considerations when scaling, so I don't want to give too precise advice just yet)
Andrew
He's also written about the 2.0 migration in his blog.
I am only answering the first question. So basically it is impossible to be 100% protected from ddos attacks, because it always comes down to a battle of resources. If the server-side resources are greater than the attacker-side resources, the server will not go down (there may be slowed performance though) but if not, the server goes down [no reference required]. Why is it not possible to be 100% protected, you may ask. So basically your server "crashes" if people cannot connect to it [https://en.wikipedia.org/wiki/Crash_(computing)#Web_server_crashes --- Web server crashes sentence 1.]. So if you try to protect your server by shutting it down for 5 mins every time 10000 connections are made in a second, the ddos succeeded. It "crashed" your server. The only ddos protection that I know of that should work is Cloudfare (https://www.cloudflare.com/lp/ddos-b/?_bt=207028074666&_bk=%2Bddos%20%2Bprotection&_bm=b&_bn=g&gclid=EAIaIQobChMIu5qv4e-Z1QIVlyW9Ch2YGQdiEAAYASAAEgJbQ_D_BwE). It absorbs the impact of the ddos attack with its 10Tbps network backbone. But even it does not offer 100% ddos protection because once its 10Tbps is down, your server will go down too. So, I hope that helped.
DDoS = Distributed Denial of Service
The 'Distributed' part is the key: you can't know you're being attacked by 'someone' in particular, because requests come from all over the place.
Your server will only accept a certain number of connections. If the attacker manages to create so many connections that nobody else can connect, you're being DDoS'ed.
So, in essence you need to be able to detect that a connection is not legit, or you need to be able to scale up fast to compensate for the limit in number of connections.
Good luck with that!
DDoS protection should really be a service from your cloud provider, at the load balancer level.
Companies like OVH use sophisticated machine learning techniques to detect illegitimate traffic and ban the IPs acting out in quasi-real time.
For you to build such a detection machinery is a huge investment that is probably not worth your time (unless your web site is so critical and will lose millions of $$$ if it's down for a bit)
Theres a lot of things you cant to do about DDOS..however there are some neat 'tricks' depending on how much resources you have at your disposal, and how much somebody wants to take you offline.
Are you offering a total public service that requires direct connection to the resource you are trying to protect?
If so, you just going to need to 'soak up' DDOS with the resources you have, by scaling up and out... or even elastic... either way it's going to cost you money!
or make it harder for the attacker to consume your resources. There are number of methods to do this.
If you service requires some kind of authentication, then separate your authentication services from the resource you are trying to protect.
Many applications, the authentication and 'service' run on the same hardware. thats a DOS waiting to happen.
Only let fully authenticated users access the resources you are trying to protect with dynamic firewall filtering rules. If your authenticated then gate to the resources opens (with a restricted QOS in place) ! If your a well known, long term trusted users, then access the resource at full bore.
Have a way of auditing users resource behaviour (network,memory,cpu) , if you see particular accounts using bizarre amounts, ban them, or impose a limit, finally leading to a firewall drop policy of their traffic.
Work with an ISP that can has systems in place that can drop traffic to your specification at the ISP border.... OVH are your best bet. an ISP that exposes filter and traffic dropping as an API, i wish they existed... basically moving you firewall filtering rules to the AS border... niiiiice! (fantasy)
It won't stop DDOS, but will give you a few tools to keep resources wasted a consumption by attackers to a manageable level. DDOS have to turn to your authentication servers... (possible), or compromise many user accounts.... at already authenticated users will still have access :-)
If your DDOS are consuming all your ISP bandwidth, thats a harder problem, move to a larger ISP! or move ISP's... :-). Hide you main resource, allow it to be move dynamically, keep on the move! :-).
Break the problem into pieces... apply DDOS controls on the smaller pieces. :-)
I've tried a most general answer, but there are a lot a of depends, each DDOS mitigation requires a bit of Skin, not tin approach.. Really you need a anti-ddos ninja on your team. ;-)
Take a look at distributed protocols.... DP's maybe the answer for DDOS.
Have fun.
Let's apply some analysis to your question. A DDoS is like a DoS but with friends. If you want to avoid DDoS explotation you need minimize DoS possibilities. Thanks capitan obvious.
First thing is to do is make a list with what happens in your system and wich resources are affected:
A tcp handshake is performed (SYN_COOKIES are affected)
A ssl handshake comes later (entropy, cpu)
A connection is made to channel layer...
Then monitorize each resource and try to implement a counter-measure:
Protect to SYN_FLOOD configuring your kernel params and firewall
Use entropy generators
Configure your firewall to limit open/closed connection in short time (easy way to minimize ssl handshakes)
...
Separate your big problem (DDoS) in many simple and easy to correct tasks. Hard part is get a detailed list of steps and resources.
Excuse my poor english.
Related
I would like to implement an observer design pattern. My thoughts are, if this is deployed, and another system update has occurred and restarted the server, would the observers/subscribers be lost when the server has restarted?
Sorry for this newbie question.
Your question doesn't specify any tools you are using neither the deployment methods and softwares so the best I can say is that any non-persistent data will be deleted on restart. This counts for the subscribers your server has saved in a variable, for example.
In web development, you go around this problem (and that of lost of connection) by having "temporary subscriptions" and by not implementing functionalities with solutions needing consistency in connection.
However, what you could do is give the clients some sort of unique id which could be stored in a database along with data that can restore the connection.
I'm writing a websocket connection between a chrome plugin and a desktop application that exclusively talk to each other locally. I know that you can't have two sockets listening on the same port (is this true for all ports?)
Port clashes won't happen for the majority of my users, but the application will run on a few thousand computers; I think I can reasonably expect there to be clashes in rare situations, where another application already occupies the port.
How do bigger applications deal with this potential issue? I'm fairly unfamiliar with networking and (web)socket programming in general.
I guess there are a few non-technical workarounds like these:
Log the issue well, and let support deal with it
Let the user configure a port
But I'd rather have a sound technical way to deal with it, if possible. I could come up with some algorithm that tries a few preconfigured ports, but I expect that this is a fairly well known problem in the industry, and am hoping to learn what the consensus is for this problem.
The question is primarily technology agnostic, but if it matters, my stack is python with this websockets library on the desktop side, and of course JavaScript on the client side.
I don't think there's a reason to overthink it. A list of seemingly unoccupied fallback ports and finally a prompt to specify a port should be fine. If both parties are capable of HDD IO (I don't know how much can a chrome plugin can do), then it's even easier for the plugin and the app to meet.
For two apps on a same host, you may also add an additional loopback interface. Example.
If to speak about large networks, there's a class of so-called service discovery protocols that may help dealing with this kind of issue. Basically they are IP- or UDP-based multicast protocols that advertise different services across the network. Usually these protocols have a standardized port for communications, so it is unlikely to get occupied by another app. Examples are DHCP, Bonjour.
However, dealing with port conflicts are not their primary function. In large networks machines should be under some sort of control and should not have lots of apps grabbing random ports. If a conflict happens with third party apps, the ideal solution is to look for a setting that allows to set ports manually.
Well I don't know how bigger applications actually do it but a way could be to try till you find a free port?
found_port = False
ports = [port1, port2, port3 ....]
for port in ports
try:
mySocket.bind ( ( ip_address, port ) )
found_port = True
except:
pass
if found_port:
break
I have a subsystem that contains sensor data and posts it to the Internet via TCP or UDP requests to a server with authorization by token. All the posted data is also saved to a local capped MongoDB database.
I want this system to be tolerant to network failures and outrages and to be able to synchronize data when the network is back.
What is the correct way to implement this without re-inventing the wheel?
I see several options:
MongoDB replication.
PROs:
Replication tools exist
CONs:
How to do that in real-time? Having two systems: to post one way when the system is online and the other way when the system is offline seems to be a bad idea.
No idea how to manage access tokens (I don't want to give direct access to the server database)
Server side scheme should match the local one (but can be PRO since then manual import is trivial)
Maintaining 'last ACKed' records and and re-transmitting once a while.
PROs:
Allows for different data scheme locally and on server side
Works
CONs:
Logic is complex (detect failures, monitor for network connectivity, etc)
Exactly opposite from 'reinventing the wheel'
Manual data backfeed is hardly possible (e.g. when the system is completely disconnected for a long time and data is restored from back-ups).
I want a simple and reliable solution (the project is in Python, but I'm also fine with JavaScript/CoffeeScript in a separate process). I prefer a Python module that is tailored for this task (I failed to find one) or a piece of advice of how to organize the system UNIX way.
I believe this is a solved problem and has a known best practices which I ceased to find.
Thank you!
Need some direction on this.
I'm writing a chat room browser-application, however there is a subtle difference.
These are collaboration chats where one person types and the other person can see live ever keystroke entered by the other person as they type.
Also the the chat space is not a single line but a textarea space, like the one here (SO) to enter a question.
All keystrokes including tabs/spaces/enter should be visible live to the other person. And only one person can type at one time (I guess locking should be trivial)
I haven't written a multiple chatroom application. A simple client/server where both are communicatiing over a port is something I've written.
So here are the questions
1.) How is a multiple chatroom application written ? Is it also port based ?
2.) Showing the other persons every keystroke as they type is I guess possible through ajax. Is there any other mechanism available ?
Note : I'm going to use a python framework (web2py) but I don't think framework would matter here.
Any suggestions are welcome, thanks !
The Wikipedia entry for Comet (programming) has a pretty good overview of different approaches you can take on the client (assuming that your client's a web browser), and those approaches suggest the proper design for the server (assuming that the server's a web server).
One thing that's not mentioned on that page, but that you're almost certainly going to want to think about, is buffering input on the client. I don't think it's premature optimization to consider that a multi-user application in which every user's keystroke hits the server is going to scale poorly. I'd consider having user keystrokes go into a client-side buffer, and only sending them to the server when the user hasn't typed anything for 500 milliseconds or so.
You absolutely don't want to use ports for this. That's putting application-layer information in the transport layer, and it pushes application-level concerns (the application's going to create a new chat room) into transport-level concerns (a new port needs to be opened on the firewall).
Besides, a port's just a 16-bit field in the packet header. You can do the same thing in the design of your application's messages: put a room ID and a user ID at the start of each message, and have the server sort it all out.
The thing that strikes me as a pain about this is figuring out, when a client requests an update, what should be sent. The naive solution is to retain a buffer for each user in a room, and maintain an index into each (other) user's buffer as part of the user state; that way, when user A requests an update, the server can send down everything that users B, C, and D have typed since A's last request. This raises all kind of issues about memory usage and persistence that don't have obvious simple solutions
The right answers to the problems I've discussed here are going to depend on your requirements. Make sure those requirements are defined in great detail. You don't want to find yourself asking questions like "should I batch together keystrokes?" while you're building this thing.
You could try doing something like IRC, where the current "room" is sent from the client to the server "before" the text (/PRIVMSG #room-name Hello World), delimited by a space. For example, you could send ROOMNAME Sample text from the browser to the server.
Using AJAX would be the most reasonable option. I've never used web2py, but I'm guessing you could just use JSON to parse the data between the browser and the server, if you wanted to be fancy.
I have been working on a huge project for work for a while now, and it is almost done. However, in an effort to prevent the program was being pirated (I already know there is pretty much no method that can't be cracked ), the software needs to be able to validate. I'm not exactly sure how to do this. Could some sort of software validation server be written in Python? How would the software communicate with the server? Would the softwre check each time it is launched to see if it is valid? The program requires internet access to run anyway, so checking for validation at each launch might not be so bad.
I am programming in Python 2.6 on Windows 7. Any help would be great!
The software, when starting, should launch an https (so it can't just be sniffed easily;-) request to your server, identifying itself (however it is that you choose to identify, e.g. a serial number or whatever), and the server's response will tell it what to do (run normally, or terminate, or ask the user to register -- whatever).
Of course, any competent hacker will find and disable the part of your code where you're sending the request and dispatching on the answer, but then you already do know that everything can easily be cracked;-).
A less-easily crackable approach would be to keep some crucial part of the functionality on your server, so that the client's basically useless (or at least less useful) if it hasn't checked in with your server and obtained a token to be used in other "functionality requests" during a session.
Hard to tell, without knowing a lot more about your app, if there are bits and pieces of functionality in your app that lend themselves well to this treatment, but for example you could delegate in this way any kind of cryptographic functionality (encrypting, decrypting, signing, ...) -- if only your server knows the secret/private keys to be used for such purposes, and only performs the functionality for application sessions that have properly registered and been authorized, suddenly it's become very hard for even a good hacker to work around your registration and authorization system.
I would really urge you not to do this. As you said, whatever you do will be broken, and you may actually cause more copies of your software to be pirated by including this barrier. Asking your users nicely not to steal may do better...
That said, implementing this in a way that discourages the most casual piracy is easy: just have the program send a serial number encrypted with the server's public key to your validation script, and have the server return a version of the number encrypted using its private key. Instant validation. Yes, this server could be written in Python easily.