Consider this function that makes a simple GET request to an API endpoint:
import httpx
def check_status_without_session(url : str) -> int:
response = httpx.get(url)
return response.status_code
Running this function will open a new TCP connection every time the function check_status_without_session is called. Now, this section of HTTPX documentation recommends using the Client API while making multiple requests to the same URL. The following function does that:
import httpx
def check_status_with_session(url: str) -> int:
with httpx.Client() as client:
response = client.get(url)
return response.status_code
According to the docs using Client will ensure that:
... a Client instance uses HTTP connection pooling. This means that when you make several requests to the same host, the Client will reuse the underlying TCP connection, instead of recreating one for every single request.
My question is, in the second case, I have wrapped the Client context manager in a function. If I call check_status_with_session multiple times with the same URL, wouldn't that just create a new pool of connections each time the function is called? This implies it's not actually reusing the connections. As the function stack gets destroyed after the execution of the function, the Client object should be destroyed as well, right? Is there any advantage in doing it like this or is there a better way?
Is there any advantage in doing it like this or is there a better way?
No, there is no advantage using httpx.Client in the way you've shown. In fact the httpx.<method> API, e.g. httpx.get, does exactly the same thing!
The "pool" is a feature of the transport manager held by Client, which is HTTPTransport by default. The transport is created at Client initialisation time and stored as the instance property self._transport.
Creating a new Client instance means a new HTTPTransport instance, and transport instances have their own TCP connection pool. By creating a new Client instance each time and using it only once, you get no benefit over using e.g. httpx.get directly.
And that might be OK! Connection pooling is an optimisation over creating a new TCP connection for each request. Your application may not need that optimisation, it may be performant enough already for your needs.
If you are making many requests to the same endpoint in a tight loop, iterating within the context of the loop may net you some throughput gains, e.g.
with httpx.Client(base_url="https://example.com") as client:
results = [client.get(f"/api/resource/{idx}") for idx in range(100)]
For such I/O-heavy workloads you may do even better by executing results in parallel, e.g. using httpx.AsyncClient.
Related
Microsoft recommends you maintain a single instance of CosmosClient across your whole application, and I'm trying to achieve this in my Function App (with more than just CosmosClient). However, even when re-using both database & container proxies, I always see a warning that I have hit the maximum (10) number of connections to Cosmos and that it's discarding the connection when I send through enough requests.
For context, it's a serverless Python Function App triggered by a message queue, the connections are managed in shared code in a helper function. I have to use the Cosmos SDK because I have to both read and update Cosmos doc.
Has anyone successfully navigated this in the past? would it simply be best practice to instantiate a new connection for every single function call? I tried creating a new CosmosClients when receiving burst traffic, but proved very difficult to do efficiently.
Here's an example of the class I'm using to manage connections:
COSMOS_CLIENT = None
class Client:
def __init__(self):
self.cosmos_client: CosmosClient = self._get_global_cosmos_client()
def _get_global_cosmos_client(self) -> CosmosClient:
global COSMOS_CLIENT
if COSMOS_CLIENT is None:
logging.info('[COSMOS] NEW CLIENT CONNECTION')
COSMOS_CLIENT = CosmosClient.from_connection_string(COSMOS_DB_CONNECTION_STRING
return COSMOS_CLIENT
Conceptually, because you are creating the client based on ConnectionString (there is always 1) this code should always create 1 client.
The number connections is not the number of clients.
Do not create multiple clients, always create 1 client for each account you are interacting against. That single client can perform operations on all existing databases/containers in the account.
Creating multiple clients just creates a problem, because each client will maintain its own independent connections and not reuse them and it will create a higher number of connections than reusing the single client, eventually leading to SNAT port exhaustion.
The error message: Connection pool is full, discarding connection: is not generated by the Cosmos Client directly, rather from the underlying urllib3.connectionpool. See: https://github.com/Azure/azure-sdk-for-python/issues/12102
The CosmosClient supports passing the session through transport, https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/core/azure-core/CLIENT_LIBRARY_DEVELOPER.md#transport, -> https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/cosmos/azure-cosmos/azure/cosmos/_cosmos_client_connection.py#L198.
Reference: https://github.com/Azure/azure-sdk-for-python/issues/12102#issuecomment-645641481
I am using kubernetes-client library in python and looking at the various examples, it appears we don't need to explicitly close the client connection to the API server. Does the client connection gets terminated automatically or are the examples missing the call to close the connection? I also found the docs page for the APIs (AppsV1 for example) and the examples shown there use context manager for the calls so the connection gets disconnected automatically there but I still have questions for the scripts that don't use the context manager approach.
Kubernetes's API is HTTP-based, so you can often get away without explicitly closing a connection. If you have a short script, things should get cleaned up automatically at the end of the script and it's okay to not explicitly close things.
The specific documentation page you link to shows a safe way to do it:
with kubernetes.client.ApiClient(configuration) as api_client:
api_instance = kubernetes.client.AppsV1Api(api_client)
api_instance.create_namespaced_controller_revision(...)
The per-API-version client object is stateless if you pass in an ApiClient to its constructor, so it's safe to create these objects as needed.
The ApiClient class includes an explicit close method, so you could also do this (less safely) without the context-manager syntax:
api_client = kubernetes.client.ApiClient(configuration)
apps_client = kubernetes.client.AppsV1Api(api_client)
...
api_client.close()
The library client front-page README suggests a path that doesn't explicitly create an ApiClient. Looking at one of the generated models' code, if you don't pass an ApiClient option explicitly, a new one will be created for each API-version client object; that includes a connection pool as well. That can leak local memory and cause extra connections to the cluster, but this might not matter to you for small scripts.
Our usecase is to make a large number of requests. Each request return 1 MB of data. Right now, on client side, we create a single GRPC channel and the run the following function in a loop
content_grpc_channel = grpc.insecure_channel(content_netloc)
test_stub = test_pb2_grpc.ContentServiceInternalStub(
content_grpc_channel)
def get_feature_data_future(feature_id, span_context=()):
req_feature = test_pb2.GetFeatureRequest()
req_feature.feature_id = feature_id
resp_feature_future = test_stub.GetFeature.future(
req_feature, metadata=span_context)
return resp_feature_future
My question is in python how I can create grpc client connection pool for better throughput?
In golang I see this https://godoc.org/google.golang.org/api/option#WithGRPCConnectionPool but I have a hard time to find the doc in python.
Is there such a utility in python to create grpc connection pool? Or should I create multiple grpc channels and manage those myself? I assume each channel will have different tcp connection, correct?
gRPC uses HTTP/2 and can multiplex many requests on one connection and gRPC client connections should be re-used for the lifetime of the client app.
The Golang link you mentioned, says that WithGRPCConnectionPool would be used to balance the requests. You might search for load balancing if it is what you need but remember that load balancing only makes sense if you have multiple gRPC server instances.
If you are searching for a connection pool inspired by what is done when working with databases, I would say you don't need to worry about it as the opening connection overhead doesn't exist when working with gRPC
I have a Pyro4 distributed system with multiple clients connecting to a single server. These clients connect to a remote object, and that object may allocate some resources in the system (virtual devices, in my case).
Once a client disconnects (let's say because of a crash), I need to release those resources. What is the proper way to detect that an specific client has disconnected from an specific object?
I've tried different things:
Overriding the Daemon.clientDisconnected method. I get a connection parameter from this method. But I can't correlate that to an object, because I have no access to which remote object that connection refers to.
Using Pyro4.current_context in Daemon.clientDisconnected. This doesn't work because that is a thread-local object. That in place, if I have more clients connected than threads in my pool, I get repeated contexts.
Using Proxy._pyroAnnotations as in the "usersession" example available by the Pyro4 project, doesn't help me, because again, I get the annotation from the Pyro4.core.current_context.annotations attribute, which shows me wrong annotations when Daemon.clientDisconnected is called (I imagine due to a thread related issues).
Using instance_mode="session" and the __del__ method in the remote class (as each client would have a separate instance of the class, so the instance is supposed to be destroyed once the client disconnects). But this relies on the __del__ method, which has some problems as some Python programmers would point out.
I added my current solution as an answer, but I really would like to know if there's a more elegant way of doing this with Pyro4, as this scenario is a recurrent pattern in network programming.
Pyro 4.63 will probably have some built-in support for this to make it easier to do. You can read about it here http://pyro4.readthedocs.io/en/latest/tipstricks.html#automatically-freeing-resources-when-client-connection-gets-closed and try it out if you clone the current master from Github. Maybe you can take a look and see if that would make your use case simpler?
I use the Proxy._pyroHandshake attribute as a client ID in the client side and override the Daemon.validateHandshake and Daemon.clientDisconnected. This way, on every new connection I map the handshake data (unique per client) to a connection. But I really wanted to know if there's an elegant way to do that in Pyro4, which is pattern that happens very often in network programming.
Notice that instead of using the Proxy as an attribute of Client, Client can also extends Pyro4.Proxy and use _pyroAnnotations to send the client ID to all the remote calls.
class Client:
def __init__(self):
self._client_id = uuid.uuid4()
self._proxy = Pyro4.Proxy("PYRO:server#127.0.0.1")
self._proxy._pyroHandshake = self._client_id
self._proxy._pyroBind()
def allocate_resource(self, resource_name):
self._proxy.allocate_resource(self._client_id, resource_name)
class Server:
def __init__(self):
self._client_id_by_connection = {}
self._resources_by_client_id = {}
def client_connected(self, connection, client_id):
self._client_id_by_connection[client_id] = connection
self._resources_by_client_id[client_id] = []
def client_disconnected(self, connection):
client_id = self._client_id_by_connection[connection]
for resource in self._resources_by_client_id[client_id]
resource.free()
#Pyro4.expose
def allocate_resource(self, client_id, resource_name)
new_resource = Resource(resource_name)
self._resources_by_client_id[client_id].append(new_resource)
server = Server()
daemon.register(server, objectId="server")
daemon.clientDisconnect = server.client_disconnected
daemon.validateHandshake = server.client_connected
daemon.requestLoop()
(edit: Perhaps I am wrong in what this error means. Is this indicating that the connection pool at my CLIENT is full? or a connection pool at the SERVER is full and this is the error my client is being given?)
I am attempting to make a large number of http requests concurrently using the python threading and requests module. I am seeing this error in logs:
WARNING:requests.packages.urllib3.connectionpool:HttpConnectionPool is full, discarding connection:
What can I do to increase the size of the connection pool for requests?
This should do the trick:
import requests.adapters
session = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections=100, pool_maxsize=100)
session.mount('http://', adapter)
response = session.get("/mypage")
Note: Use this solution only if you cannot control the construction of the connection pool (as described in #Jahaja's answer).
The problem is that the urllib3 creates the pools on demand. It calls the constructor of the urllib3.connectionpool.HTTPConnectionPool class without parameters. The classes are registered in urllib3 .poolmanager.pool_classes_by_scheme. The trick is to replace the classes with your classes that have different default parameters:
def patch_http_connection_pool(**constructor_kwargs):
"""
This allows to override the default parameters of the
HTTPConnectionPool constructor.
For example, to increase the poolsize to fix problems
with "HttpConnectionPool is full, discarding connection"
call this function with maxsize=16 (or whatever size
you want to give to the connection pool)
"""
from urllib3 import connectionpool, poolmanager
class MyHTTPConnectionPool(connectionpool.HTTPConnectionPool):
def __init__(self, *args,**kwargs):
kwargs.update(constructor_kwargs)
super(MyHTTPConnectionPool, self).__init__(*args,**kwargs)
poolmanager.pool_classes_by_scheme['http'] = MyHTTPConnectionPool
Then you can call to set new default parameters. Make sure this is called before any connection is made.
patch_http_connection_pool(maxsize=16)
If you use https connections you can create a similar function:
def patch_https_connection_pool(**constructor_kwargs):
"""
This allows to override the default parameters of the
HTTPConnectionPool constructor.
For example, to increase the poolsize to fix problems
with "HttpSConnectionPool is full, discarding connection"
call this function with maxsize=16 (or whatever size
you want to give to the connection pool)
"""
from urllib3 import connectionpool, poolmanager
class MyHTTPSConnectionPool(connectionpool.HTTPSConnectionPool):
def __init__(self, *args,**kwargs):
kwargs.update(constructor_kwargs)
super(MyHTTPSConnectionPool, self).__init__(*args,**kwargs)
poolmanager.pool_classes_by_scheme['https'] = MyHTTPSConnectionPool
Jahaja's answer already gives the recommended solution to your problem, but it does not answer what is going on or, as you asked, what this error means.
Some very detailed information about this is in urllib3 official documentation, the package requests uses under the hood to actually perform its requests. Here are the relevant parts for your question, adding a few notes of my own and ommiting code examples since requests have a different API:
The PoolManager class automatically handles creating ConnectionPool instances for each host as needed. By default, it will keep a maximum of 10 ConnectionPool instances [Note: That's pool_connections in requests.adapters.HTTPAdapter(), and it has the same default value of 10]. If you’re making requests to many different hosts it might improve performance to increase this number
However, keep in mind that this does increase memory and socket consumption.
Similarly, the ConnectionPool class keeps a pool of individual HTTPConnection instances. These connections are used during an individual request and returned to the pool when the request is complete. By default only one connection will be saved for re-use [Note: That's pool_maxsize in HTTPAdapter(), and requests changes the default value from 1 to 10]. If you are making many requests to the same host simultaneously it might improve performance to increase this number
The behavior of the pooling for ConnectionPool is different from PoolManager. By default, if a new request is made and there is no free connection in the pool then a new connection will be created. However, this connection will not be saved if more than maxsize connections exist. This means that maxsize does not determine the maximum number of connections that can be open to a particular host, just the maximum number of connections to keep in the pool. However, if you specify block=True [Note: Available as pool_block in HTTPAdapter()] then there can be at most maxsize connections open to a particular host
Given that, here's what happened in your case:
All pools mentioned are CLIENT pools. You (or requests) have no control over any server connection pools
That warning is about HttpConnectionPool, i.e, the number of simultaneous connections made to the same host, so you could increase pool_maxsize to match the number of workers/threads you're using to get rid of the warning.
Note that requests is already opening as many simultaneous connections as you ask for, regardless of pool_maxsize. If you have 100 threads, it will open 100 connections. But with the default value only 10 of them will be kept in the pool for later reuse, and 90 will be discarded after completing the request.
Thus, a larger pool_maxsize increases performance to a single host by reusing connections, not by increasing concurrency.
If you're dealing with multiple hosts, then you might change pool_connections instead. The default is 10 already, so if all your requests are to the same target host, increasing it will not have any effect on performance (but it will increase the resources used, as said in above documentation)
In case anyone needs to do it with Python Zeep and wants to safe bit of time to figure out
here is a quick recipe:
from zeep import Client
from requests import adapters as request_adapters
soap = "http://example.com/BLA/sdwl.wsdl"
wsdl_path = "http://example.com/PATH/TO_WSLD?wsdl"
bind = "Binding"
client = Client(wsdl_path) # Create Client
# switch adapter
session = client.transport.session
adapter = request_adapters.HTTPAdapter(pool_connections=10, pool_maxsize=10)
# mount adapter
session.mount('https://', adapter)
binding = '{%s}%s' % (soap, bind)
# Create Service
service = client.create_service(binding, wsdl_path.split('?')[0])
Basically the connection should be created before creating the service
The answer is actualy taken from the python-zeep Repo from a closed issue,
for refence I'll add it --> here