I am writing a simple test script(python) to test a web server's performance(all this server does is HTTP redirect). Socket is set to non-blocking, and registered to an epoll instance.
How can I know the connect() is failed because the server can't accept more connections? I am currently using EPOLLERR as the indicator. Is this correct?
Edit:
Assumptions:
1) IP layer network unreachability will not be considered.
That catches the case of Connection Refused and other socket errors. Since I assume you are registering for read/write availability (success) upon the pending socket as well, you should also manually time-out those connections which have failed to notify you of read, write, or error availability on the associated file descriptor within an acceptable time limit.
ECONNREFUSED is generally only returned when the server's accept() queue exceeds its limit or when the server isn't even bound to a socket at the remote port. ENETDOWN, EHOSTDOWN, ENETUNREACH, and EHOSTUNREACH are only returned when a lower layer than TCP (e.g., IP) knows it cannot reach the host for some reason, and so they are not particularly helpful for stress testing a web server's performance.
Thus, you need to also bound the time taken to establish a connection with a timeout to cover the full gamut of stress test failure scenarios. Choice of timeout value is up to you.
You can't know it 'failed because the server can't accept more connections', because there is no specific protocol for that condition. You can only know it failed for the usual reasons: ECONNREFUSED, connection timeout, EUNREACH, etc.
Related
I have a python socket server that listens on a port, and accepts all incoming connections using:
(conn, address) = socket.accept()
However, I wish to accept connections only from certain ip address.
Currently, I close the connection if the address isn't registered, to accomplish this.
But is there a better way to do this, by directly rejecting connections from unregistered addresses, instead of accepting connections and then closing them?
It's not possible to indicate Connection refused to clients from some IP addresses, and to establish the connection to clients from other IP addresses. This is not a Python limitation, but a lower-level, BSD socket layer limitation. You can't do it even from C.
The closest behavior in general you can do in Python is closing the connection quickly after it has been accepted:
sock, addr = server_socket.accept()
if addr[0] != '12.34.56.78':
sock.close()
return
...
Then the client would see the connection being accepted, and very shortly after that the client would see EOF when reading from it, and it wouldn't be able to write to it.
However it's possible to limit by interface (i.e. network card) at bind time, by using one of:
server_socket.bind(('', 65432)) # Bind on any interface.
server_socket.bind(('127.0.0.1', 65432)) # Bind on loopback (localhost clients only).
server_socket.bind(('34.56.78.91', 65432))
So in the 127.0.0.1 version, telnet 127.0.0.1 65432 (as a client) would work, but telnet myhostname 65432 would yield Connection refused (and the server_socket.accept() call won't get this connection).
If you read the docs you can find the BaseServer.verify_request(request, client_address) which tells you this:
Must return a Boolean value; if the value is True, the request will be processed, and if it’s False, the request will be denied. This function can be overridden to implement access controls for a server. The default implementation always returns True.
Microsoft appears to support this functionality via the SO_CONDITIONAL_ACCEPT socket option
This appears to require usage of WSAAccept to accept connections
This constant does not appear in pythons socket module on my windows 8 machine. I don't think there is an option to use WSAAccept via python's builtin socket module.
If I understand correctly, this will allow your server to respond to SYN packets immediately with RST packets when configured to do so instead of finishing the handshake and exchanging FIN packets. Note that usage of this flag removes responsibility to handle connections from the operating system and places it on the application, so there is plenty of room for errors and performance hits to occur. If a performance boost was the goal, it might not be not worth pursuing
It is possible to do at the C level on windows. Pythons ctypes module allows interfacing with C code, so it is technically possible to do via a python interface. But it likely requires a non trivial amount of effort. If you are certain you require this feature, it may be less effort to find a C socket library that supports this out of the box, then you could make a ctypes wrapper for that.
In python, tcp connect returns success even though the connect request is in queue at server end. Is there any way to know at client whether accept happened or in queue at server?
The problem is not related to Python but is caused by the underlying socket machinery that does its best to hide low level network events from the program. The best I can imagine would be to try a higher level protocol handshake (send a hello string and set a timeout for receiving the answer) but it would make no difference between the following problem:
connection is queued on peer and still not accepted
connection has been accepted, but for any other reason the server could not process it in allocated time
(only if timeout is very short) congestion on machines (including sender) and network added a delay greater that the timeout
My advice is simply that you do not even want to worry with such low level details. As problems can arise server side after the connection has been accepted, you will have to deal with possible higher level protocol errors, timeouts or connection loss. Just say that there is no difference between a timeout after connection has been accepted and a timeout to accept the connection.
If connect returns and there is no error, the TCP 3-Way Handshake has taken place successfully.
Client: connect sends a SYN (and blocks)
Server: (blocking on accept) sends a SYN,ACK
Client: connect sends an ACK
After 3, connectgives control back to you on the client side and accept also gives control back to the caller on the server side.
Of course, if the server is fully loaded, there is no guarantee that the wake-up of accept means actual processing of the request, but the fact that connect has woken up and returned with no error is a guarantee of having successfully set-up the TCP connection.
Packets can be sent.
For a good explanation see for example:
https://www.ibm.com/developerworks/aix/library/au-tcpsystemcalls/index.html
And head to the The 3-way TCP handshake section
There're basically two issues I'd like to resolve:
Client side send query string when initializing the connection to server
Server side validate user token in handshake (not after the connection is established and then validate streaming message that contains the token) and set user session accordingly.
I read a article (https://auth0.com/blog/2014/01/15/auth-with-socket-io/) that talks about this process implemented in nodejs, just wonder if the same function can be achieved by using python. (Currently I'm doing some research on twisted but haven't found anything similar)
PS: guess it's helpful to demo the use case as well. A user may login to your server over normal http then server will issue him/her an valid accessToken. Then this user may need to establish a socket connection with the server (or some other server), then the server needs to figure out who the user is and validate before establishing the socket connection.
Query strings are part of HTTP URLs.
If you're building a TCP socket server instead of an HTTP server, you don't get URLs—or headers, or anything else out-of-band.* All you get is a stream of data. You need to come up with a protocol for your data that you can fit the token into.
This means the server can't "figure out who the user is and validate before establishing the socket connection". It has to establish the socket connection, read the first message, parse it, validate the token, and then drop or continue the connection. (You can, of course, put up a front-end server that accepts connections, validates them, and then migrates or proxies them to the real back-end server. But someone has to accept, read, and parse.)
Note that this is exactly what HTTP does—it can't see the query string until it accepts the connection and reads the first line of data.
Meanwhile, the example you're looking at appears to be using WebSockets. A WebSockets client can't talk to a socket server (well, unless you build a WebSockets server on top of your socket server, or a proxy in front of it) in the first place.
* This isn't quite true. You can cram 40 bytes of options into TCP header extensions. But then you have to go below the level people are usually talking about when they say "socket server"—and there's a good chance it won't make it through the internet. Also, TCP does have a concept of "out-of-band" data, but that isn't relevant here; you still have to accept the connection and read from it to get an OOB data.
I have a Python application which opens a simple TCP socket to communicate with another Python application on a separate host. Sometimes the program will either error or I will directly kill it, and in either case the socket may be left open for some unknown time.
The next time I go to run the program I get this error:
socket.error: [Errno 98] Address already in use
Now the program always tries to use the same port, so it appears as though it is still open. I checked and am quite sure the program isn't running in the background and yet my address is still in use.
SO, how can I manually (or otherwise) close a socket/address so that my program can immediately re-use it?
Update
Based on Mike's answer I checked out the socket(7) page and looked at SO_REUSEADDR:
SO_REUSEADDR
Indicates that the rules used in validating addresses supplied in a bind(2) call should
allow reuse of local addresses. For AF_INET sockets this means that a socket may bind,
except when there is an active listening socket bound to the address. When the listen‐
ing socket is bound to INADDR_ANY with a specific port then it is not possible to bind
to this port for any local address. Argument is an integer boolean flag.
Assume your socket is named s... you need to set socket.SO_REUSEADDR on the server's socket before binding to an interface... this will allow you to immediately restart a TCP server...
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((ADDR, PORT))
You might want to try using Twisted for your networking. Mike gave the correct low-level answer, SO_REUSEADDR, but he didn't mention that this isn't a very good option to set on Windows. This is the sort of thing that Twisted takes care of for you automatically. There are many, many other examples of this kind of boring low-level detail that you have to pay attention to when using the socket module directly but which you can forget about if you use a higher level library like Twisted.
You are confusing sockets, connections, and ports. Sockets are endpoints of connections, which in turn are 5-tuples {protocol, local-ip, local-port, remote-ip, remote-port}. The killed program's socket has been closed by the OS, and ditto the connection. The only relic of the connection is the peer's socket and the corresponding port at the peer host. So what you should really be asking about is how to reuse the local port. To which the answer is SO_REUSEADDR as per the other answers.
Sometimes in our production environment occurs situation when connection between service (which is python program that uses MySQLdb) and mysql server is flacky, some packages are lost, some black magic happens and .execute() of MySQLdb.Cursor object never ends (or take great amount of time to end).
This is very bad because it is waste of service worker threads. Sometimes it leads to exhausting of workers pool and service stops responding at all.
So the question is: Is there a way to interrupt MySQLdb.Connection.execute operation after given amount of time?
if the communication is such a problem, consider writing a 'proxy' that receives your SQL commands over the flaky connection and relays them to the MySQL server on a reliable channel (maybe running on the same box as the MySQL server). This way you have total control over failure detection and retrying.
You need to analyse exactly what the problem is. MySQL connections should eventually timeout if the server is gone; TCP keepalives are generally enabled. You may be able to tune the OS-level TCP timeouts.
If the database is "flaky", then you definitely need to investigate how. It seems unlikely that the database really is the problem, more likely that networking in between is.
If you are using (some) stateful firewalls of any kind, it's possible that they're losing some of the state, thus causing otherwise good long-lived connections to go dead.
You might want to consider changing the idle timeout parameter in MySQL; otherwise, a long-lived, unused connection may go "stale", where the server and client both think it's still alive, but some stateful network element in between has "forgotten" about the TCP connection. An application trying to use such a "stale" connection will have a long wait before receiving an error (but it should eventually).