I have established a simple TLS 1.2 session between a client and a server using Python's SSL module (running LibreSSL 2.2.7 under the hood) and am wondering if session tickets are automatically rotated.
It looks like the server is hinting at the client that the session ticket should only be valid for 300 seconds (Session Ticket Lifetime Hint: 300 seconds)
But it's been almost an hour and a new session ticket hasn't been issued like I expected. Meanwhile I exchanged some application data between the two parties but that didn't seem to trigger anything.
Per RFC 4507 I understand the 300 seconds hint is not strictly required to be followed
The ticket_lifetime_hint field contains a hint from the server
about how long the ticket should be stored. The value indicates
the lifetime in seconds as a 32-bit unsigned integer in network
byte order. A value of zero is reserved to indicate that the
lifetime of the ticket is unspecified. A client SHOULD delete the
ticket and associated state when the time expires. It MAY delete
the ticket earlier based on local policy. A server MAY treat a
ticket as valid for a shorter or longer period of time than what is
stated in the ticket_lifetime_hint.
But then how do I know if ticket rotation is happening? How do I check how long my client waits before rotating tickets?
The session ticket is given by the server during the handshake. And to initiate a handshake you must either start with a new connection with an empty ticket (by playing with the HTTP keep alives for example), or force a rehandshake in an established connection. Unfortunately, keeping a connection opened for a long time and waiting to see anything happen like a ticket update is not likely to happen.
If you want to restart with new connections, either program your client to close and reopen new connections from time to time, or try the HTTP Keep-Alive header on the server side which is supposed to inform the client on how it should behave.
Unluckily we are unsure of this header behaviour because we know this header exists in RFC 2068 , but its use is described in an RFC draft which is now expired.
An example of use :
Keep-Alive: timeout=300
The SSL rehandshake is possible if you have access to a low-level API. Then the server can send an HelloRequest forcing the client to start a rehandshake and at this moment it is supposed to ask for a new ticket if the previous is considered expired.
In both cases, you should confirm with a network capture that it is behaving as expected. There is probably no way to see anything if you're not coding with a low level language (Java for example allows to code rehandshakes, but I'm not sure coding an entire server is worth).
Related
I'm trying to get a python script running which calls an external API (to which I only have read-access) in a certain interval, the API uses cookie-based authentication: Calling the /auth endpoint initially sets session cookies which are then used for authentication in further requests.
As for my problem: Because the authentication is based on an active session, the cookies aren't valid once the connection drops, and therefore has to be restarted. From what I've read, requests is based on urllib3, which keeps the connection alive by default. Yet, after a few tests I noticed that under some circumstances, the connection will be dropped anyway.
I used a Session object from the requests module and I've tested how long it takes for the connection to be dropped as follows:
from requests import session
import logging
import time import time, sleep
logging.basicConfig(level=logging.DEBUG)
def tt(interval):
credentials = {"username":"user","password":"pass"}
s = Session()
r = s.post("https://<host>:<port>/auth", json=credentials)
ts = time()
while r.status_code is 200:
r = s.get("https://<host>:<port>/some/other/endpoint")
sleep(interval)
return time() - ts # Seconds until connection drop
Might not be the best way to find that out, but I let that function run twice, once with an interval of 1 second and then with an interval of 1 minute. Both had run for about an hour until I had to manually stop the execution.
However, when I swapped the two lines within the while loop, which meant that there was a 1-minute-delay after the initial POST /auth request, the following GET request failed with a 401 Unauthorized and this message being logged beforehand:
DEBUG:urllib3.connectionpool:Resetting dropped connection: <host>
As the interval of requests may range from a few minutes to multiple hours in my prod script, I have to know beforehand how long these sessions are kept alive and whether there are some exceptions to that rule (like dropping the connection if no request after the initial POST /auth is made for a short while).
So, how long does requests or rather urllib3 keep the connection alive, and is it possible to extend that time indefinitely?
Or is it the server instead of requests that drops the connection?
By using requests.Session, keep-alive is handled for you automatically.
In the first version of your loop that continuously polls the server after the /auth call is made, the server does not drop the connection due to the subsequent GET that happens. In the second version, it's likely that sleep interval exceeds the amount of time the server is configured to keep the connection open.
Depending on the server configuration of the API, the response headers may include a Keep-Alive header with information about how long connections are kept open at a minimum. HTTP/1.0 specifies this information is included in the timeout parameter of the Keep-Alive header. You could use this information to determine how long you have until the server will drop the connection.
In HTTP/1.1, persistent connections are used by default and the Keep-Alive header is not used unless the server explicitly implements it for backwards compatibility. Due to this difference, there isn't an immediate way for a client to determine the exact timeout for connections since it may exist solely as server side configuration.
The key to keeping the connection open would be to continue polling at regular intervals. The interval you use must be less than the server's configured connection timeout.
One other thing to point out is that artificially extending the length of the session indefinitely this way makes one more vulnerable to session fixation attacks. You may want to consider adding logic that occasionally reestablishes the session to minimize risk of these types of attacks.
I am using SolrClient for python with Solr 6.6.2. It works as expected but I cannot find anything in the documentation for closing the connection after opening it.
def getdocbyid(docidlist):
for id in docidlist:
solr = SolrClient('http://localhost:8983/solr', auth=("solradmin", "Admin098"))
doc = solr.get('Collection_Test',doc_id=id)
print(doc)
I do not know if the client closes it automatically or not. If it doesn't, wouldn't it be a problem if several connections are left open? I just want to know if it there is any way to close the connection. Here is the link to the documentation:
https://solrclient.readthedocs.io/en/latest/
The connections are not kept around indefinitely. The standard timeout for any persistent http connection in Jetty is five seconds as far as I remember, so you do not have to worry about the number of connections being kept alive exploding.
The Jetty server will also just drop the connection if required, as it's not required to keep it around as a guarantee for the client. solrclient uses a requests session internally, so it should do pipelining for subsequent queries. If you run into issues with this you can keep a set of clients available as a pool in your application instead, then request an available client instead of creating a new one each time.
I'm however pretty sure you won't run into any issues with the default settings.
I wrote a program in Python using the quickfix package which connects to a vendor via FIX. We login in the morning, but don't actually send messages through the connection until the end of the day. The issue is, we don't want to keep the program open for the entirety of the day, but would rather relogin in the afternoon when we need to send the messages.
The vendor is requesting we stay logged in for the full duration between our start and stop times specified in our configurations. This is only possible by leaving my program on for the entirety of the day, because if I close it then the messages the vendor sends aren't registered as received by me. I don't send a logout message though.
Is it common practice to write a program to connect via FIX and leave it running for the entire session time? Or is it acceptable to close the program, given I don't send a logout message, and reconnect at a later time in the day?
Any design or best practice advice would be helpful here.
Is it common practice to write a program to connect via FIX and leave it running for the entire session time? Or is it acceptable to close the program, given I don't send a logout message, and reconnect at a later time in the day?
I don't know what others have done, but I used QuickFIX with Python for years and never had any problem running my system all day, OR shutting it down periodically for whatever reason and reconnecting. In the end I wound up leaving the system connected for weeks at a time, since that allowed me to record data.
I would say that the answer to both of your questions is YES. It is common to leave it running. Also, it is acceptable to just close the program.
There can always be edge cases and idiosyncratic features of your implementation and your counterparty, so you should seek to understand more why they have asked you not to disconnect. That sounds very strange to me. Is their FIX engine not capable of something very simple and standard?
Yes it is common to keep the FIX sessions running for a long time. That should not be an issue.
You can't just shutdown your program your end, as Session-level FIX.Heartbeat(35=0) messages, sent periodically (usually 30s), as meant to keep the underlying TCP connection "open", and check that both ends are still up and running properly.
By the details you gave, if your vendor (which is likely the acceptor side) requests it, it might be because they need to send you messages, with no delay as they occur.
If you (the initiator side) are not logged in at that time, they won't be able to send those messages, as they won't be able to initiate a session with you.
The vendor might monitor sessions as well, but as an initiator, it sounds odd.
as initiators are waiting for connections.
More likely they will monitor unexpected sessions drops.
All in all, it very depends of your vendor anyway, you have to follow what they say...
I'm writing a Socket Server in Python, and also a Socket Client to connect to the Server.
The Client interacts with the Server in a way that the Client sends information when an action is invoked, and the Server processes the information.
The problem I'm having, is that I am able to connect to my Server with Telnet, and probably other things that I haven't tried yet. I want to disable connection from these other Clients, and only allow connections from Python Clients. (Preferably my custom-made client, as it sends information to communicate)
Is there a way I could set up authentication on connection to differentiate Python Clients from others?
Currently there is no code, as this is a problem I want to be able to solve before getting my hands dirty.
When a new connection is made to your server, your protocol will have to specify some way for the client to authenticate. Ultimately there is nothing that the network infrastructure can do to determine what sort of process initiated the connection, so you will have to specify some exchange that allows the server to be sure that it really is talking to a valid client process.
#holdenweb has already given a good answer with basic info.
If a (terminal) software sends the bytes that your application expects as a valid identification, your app will never know whether it talks to an original client or anything else.
A possible way to test for valid clients could be, that your server sends an encrypted and authenticated question (should be different at each test!), e.g. something like "what is 18:37:12 (current date and time) plus 2 (random) hours?"
Encryption/Authentication would be another issue then.
If you keep this algorithm secret, only your clients can answer it and validate themselves successfully. It can be hacked/reverse engineered, but it is safe against basic attackers.
The real question is if Google App Engine guarantees it would complete a HTTP request even if the connection is no longer existed (such as terminated, lost Internet connection).
Says we have a python script running on Google App Engine:
db.put(status = "Outputting")
print very_very_very_long_string_like_1GB
db.put(status = "done")
If the client decides to close the connection in the middle (too much data coming...), will status = "done" be executed? Or will the instance be killed and all following code be ignored?
If the client breaks the connect, the request will continue to execute. Unless it reaches the deadline of 60 seconds.
GAE uses Pending Queue to queue up requests. If client drops connection and request is already in the queue or being executed, then it will not be aborted. Afaik all other http servres behave the same way.
This will be a real problem when you make requests that change state (PUT, POST, DELETE) on mobile networks. On Edge networks we see about 1% of large requests (uploads, ~500kb) dropped in the middle of request executing (exec takes about 1s): e.g. server gets the data and processes it, but client does not receive response, triggering it to retry. This could produce duplicate data in the DB, breaking integrity of this data.
To alleviate this you will need to make your web methods idempotent: repeating the same method with same arguments does not change state. The easiest way to achieve this would be one of:
Hash relevant data and compare to existing hashes. In you case it would be the string you are trying to save (very_very_very_long_string_like_1GB). You can do this server side.
Client provides unique request-scoped ID, and sever checks if this ID was already used.