I am working on a websocket server and am trying to use python twisted + autobahn but I believe I am hitting a memory leak. In fact I was able to reproduce it with the echo code on https://github.com/crossbario/autobahn-python/tree/master/examples/twisted/websocket/echo
The symptom I see is that on the server side the protocol instances are never freed after connection is closed.
I have tried to examine this in various ways - simplest being to add a print in del method, more complex is examining with pdb and gc. And yes - observing the memory use of the process climbing steadily as connections are made and closed over and over.
What I expect to happen is - after onClose completes the protocol instance should go away for good. In fact I have other server implementations based on twisted (but without autobahn websockets) and I have confirmed that's how it works there (Although I use connectionLost instead).
Does anyone have a clue what is happening?
I faced the issue of memory overflow with an autobahn web socket server that distributed realtime data to clients. The issue was however with clients that keep the connection open but is not able to consumer the data.
This caused the memory to keep on accumulating at the server side. I was able to address the issue by finding the variable responsible keeping the buffer data. Its the transport._tempDataBuffer variable from transport layer in twisted. By defining a maximum size limit on the buffer and clearing it when full, solved the issue for me.
Don't know if you are referring to the same issue, see if this helps.
Related
I am using SolrClient for python with Solr 6.6.2. It works as expected but I cannot find anything in the documentation for closing the connection after opening it.
def getdocbyid(docidlist):
for id in docidlist:
solr = SolrClient('http://localhost:8983/solr', auth=("solradmin", "Admin098"))
doc = solr.get('Collection_Test',doc_id=id)
print(doc)
I do not know if the client closes it automatically or not. If it doesn't, wouldn't it be a problem if several connections are left open? I just want to know if it there is any way to close the connection. Here is the link to the documentation:
https://solrclient.readthedocs.io/en/latest/
The connections are not kept around indefinitely. The standard timeout for any persistent http connection in Jetty is five seconds as far as I remember, so you do not have to worry about the number of connections being kept alive exploding.
The Jetty server will also just drop the connection if required, as it's not required to keep it around as a guarantee for the client. solrclient uses a requests session internally, so it should do pipelining for subsequent queries. If you run into issues with this you can keep a set of clients available as a pool in your application instead, then request an available client instead of creating a new one each time.
I'm however pretty sure you won't run into any issues with the default settings.
Just I want to know how to close the connection in py2neo.
graph = py2neo.Graph(password = 'xxxxx',host = 'xxxx')
I try to use
graph.close()
But I receive the next msg.
AttributeError: 'Graph' object has no attribute 'close'
Lib version : py2neo==3.1.2
Regards.
There is no close method. I was wondering the same thing, and having seen no other answer, I started using netstat and tcpdump to watch the behavior of neo4j when connecting via p2neo.
Here's what I learned...
(1) It seems that neo4j (when connecting via HTTP) makes requests very restfully (no persistent connection as with other databases--e.g. postgres). This means there is actually no need for a .close() method.
(2) The down-side is that you may end up building a list of connections in TIME_WAIT status. This is because no 'Connection':'Close' header is sent. Under low load, this should not be a problem. However, at scale, this will need some tuning at the operating system level (I'll forgo how Java programmers seem to be notorious about not cleaning up after themselves and leaving this to someone else to do. I rant about this too much on too many applications).
Hopefully this helps. Happy Hacking!
To just free up the object, I used:
del graph
so far, no issues. This was because I didn't want a graph and OGM repo connection at the same time...which doesn't appear to be an issue anyways.
I initiate a request client-side, then I change my mind and call xhr.abort().
How does Django react to this? Does it terminate the thread somehow? If not, how do I get Django to stop wasting time trying to respond to the aborted request? How do I handle it gracefully?
Due to how http works and that you usually got a frontend in front of your django gunicorn app processes (or uswgi etc), your http cancel request is buffered by nginx. The gunicorns don't get a signal, they just finish processing and then output whatever to the http socket. But if that socket is closed it will have an error (which is caught as a closed connection and move one).
So it's easy to DOS a server if you can find a way to spawn many of these requests.
But to answer your question it depends on the backend, with gunicorn it will keep going until the timeout.
Just think of the Web as a platform for building easy-to-use, distributed, loosely couple systems, with no guarantee about the availability of resources as 404 status code suggests.
I think that creating tightly coupled solutions such as your idea is going against web principles and usage of REST. xhr.abort() is client side programming, it's completely different from server side. It's a bad practice trying to tighten client side technology to server side internal behavior.
Not only this is a waste of resources, but also there is no guarantee on processing status of the request by web server. It may lead to data inconsistency too.
If your request generates no server-side side effects for which the client
can be held responsible. It is better just to ignore it, since these kind of requests does not change server state & the response is usually cached for better performance.
If your request could cause changes in server state or data, for the sake of data consistency you can check whether the changes have taken effect or not using an API. In case of affection try to rollback using another API.
I want to implement a lightweight Message Queue proxy. It's job is to receive messages from a web application (PHP) and send them to the Message Queue server asynchronously. The reason for this proxy is that the MQ isn't always avaliable and is sometimes lagging, or even down, but I want to make sure the messages are delivered, and the web application returns immediately.
So, PHP would send the message to the MQ proxy running on the same host. That proxy would save the messages to SQLite for persistence, in case of crashes. At the same time it would send the messages from SQLite to the MQ in batches when the connection is available, and delete them from SQLite.
Now, the way I understand, there are these components in this service:
message listener (listens to the messages from PHP and writes them to a Incoming Queue)
DB flusher (reads messages from the Incoming Queue and saves them to a database; due to SQLite single-threadedness)
MQ connection handler (keeps the connection to the MQ server online by reconnecting)
message sender (collects messages from SQlite db and sends them to the MQ server, then removes them from db)
I was thinking of using Twisted for #1 (TCPServer), but I'm having problem with integrating it with other points, which aren't event-driven. Intuition tells me that each of these points should be running in a separate thread, because all are IO-bound and independent of each other, but I could easily put them in a single thread. Even though, I couldn't find any good and clear (to me) examples on how to implement this worker thread aside of Twisted's main loop.
The example I've started with is the chatserver.py, which uses service.Application and internet.TCPServer objects. If I start my own thread prior to creating TCPServer service, it runs a few times, but the it stops and never runs again. I'm not sure, why this is happening, but it's probably because I don't use threads with Twisted correctly.
Any suggestions on how to implement a separate worker thread and keep Twisted? Do you have any alternative architectures in mind?
You're basically considering writing an ad-hoc extension to your messaging server, the job of which it is to provide whatever reliability guarantees you've asked of it.
Instead, perhaps you should take the hardware where you were planning to run this new proxy and run another MQ node on it. The new node should take care of persisting and relaying messages that you deliver to it while the other nodes are overloaded or offline.
Maybe it's not the best bang for your buck to use a separate thread in Twisted to get around a blocking call, but sometimes the least evil solution is the best. Here's a link that shows you how to integrate threading into Twisted:
http://twistedmatrix.com/documents/10.1.0/core/howto/threading.html
Sometimes in a pinch easy-to-implement is faster than hours/days of research which may all turn out to be for nought.
A neat solution to this problem would be to use the Key Value store Redis. Its a high speed persistent data store, with plenty of clients - it has a php and a python client (if you want to use a timed/batch process to process messages - it saves you creating a database, and also deals with your persistence stories. It runs fine on Cywin/Windows + posix environments.
PHP Redis client is here.
Python client is here.
Both have a very clean and simple API. Redis also offers a publish/subscribe mechanism, should you need it, although it sounds like it would be of limited value if you're publishing to an inconsistent queue.
Sometimes in our production environment occurs situation when connection between service (which is python program that uses MySQLdb) and mysql server is flacky, some packages are lost, some black magic happens and .execute() of MySQLdb.Cursor object never ends (or take great amount of time to end).
This is very bad because it is waste of service worker threads. Sometimes it leads to exhausting of workers pool and service stops responding at all.
So the question is: Is there a way to interrupt MySQLdb.Connection.execute operation after given amount of time?
if the communication is such a problem, consider writing a 'proxy' that receives your SQL commands over the flaky connection and relays them to the MySQL server on a reliable channel (maybe running on the same box as the MySQL server). This way you have total control over failure detection and retrying.
You need to analyse exactly what the problem is. MySQL connections should eventually timeout if the server is gone; TCP keepalives are generally enabled. You may be able to tune the OS-level TCP timeouts.
If the database is "flaky", then you definitely need to investigate how. It seems unlikely that the database really is the problem, more likely that networking in between is.
If you are using (some) stateful firewalls of any kind, it's possible that they're losing some of the state, thus causing otherwise good long-lived connections to go dead.
You might want to consider changing the idle timeout parameter in MySQL; otherwise, a long-lived, unused connection may go "stale", where the server and client both think it's still alive, but some stateful network element in between has "forgotten" about the TCP connection. An application trying to use such a "stale" connection will have a long wait before receiving an error (but it should eventually).