I'm working on a game (Risk). I've already finished set-up, so a solution that doesn't require me to completely redesign networking would be preferable. The problem is I've gotten to main gameplay, and I need to allow messages from multiple users at the same time. The general idea is that when one player is taking their turn, another player can send a forfeit message, and still exit out of the game without crashing the whole server. Is there a way to do this, possibly using threading?
There is no reason that your main thread should be blocked on one connection.
You need one listening thread, when a connection is made background threads handle communications to clients.
You do need to maintian a bullet proof state machine so that clients know the appropriate messages they can send at any given state, and the server needs to know which valid messages can be processed at any given state.
Search stackoverflow, you'll find many examples, such as this:
Sockets example
Related
Im trying to make a tcp communication, where the server sends a message every x seconds through a socket, and should stop sending those messages on a certain condition where the client isnt sending any message for 5 seconds.
To be more detailed, the client also sends constant messages which are all ignored by the server on the same socket as above, and can stop sending them at any unknown time. The messages are, for simplicity, used as alive messages to inform the server that the communication is still relevant.
The problem is that if i want to send repeated messages from the server, i cannot allow it to "get busy" and start receiving messages instead, thus i cannot detect when a new messages arrives from the other side and act accordingly.
The problem is independent of the programming language, but to be more specific im using python, and cannot access the code of the client.
Is there any option of receiving and sending messages on a single socket simultaneously?
Thanks!
Option 1
Use two threads, one will write to the socket and the second will read from it.
This works since sockets are full-duplex (allow bi-directional simultaneous access).
Option 2
Use a single thread that manages all keep alives using select.epoll. This way one thread can handle multiple clients. Remember though, that if this isn't the only thread that uses the sockets, you might need to handle thread safety on your own
As discussed in another answer, threads are one common approach. The other approach is to use an event loop and nonblocking I/O. Recent versions of Python (I think starting at 3.4) include a package called asyncio that supports this.
You can call the create_connection method on an event_loop to create an asyncio connection. See this example for a simple server that reads and writes over TCP.
In many cases an event loop can permit higher performance than threads, but it has the disadvantage of requiring most or all of your code to be aware of the event model.
I wrote a program in Python using the quickfix package which connects to a vendor via FIX. We login in the morning, but don't actually send messages through the connection until the end of the day. The issue is, we don't want to keep the program open for the entirety of the day, but would rather relogin in the afternoon when we need to send the messages.
The vendor is requesting we stay logged in for the full duration between our start and stop times specified in our configurations. This is only possible by leaving my program on for the entirety of the day, because if I close it then the messages the vendor sends aren't registered as received by me. I don't send a logout message though.
Is it common practice to write a program to connect via FIX and leave it running for the entire session time? Or is it acceptable to close the program, given I don't send a logout message, and reconnect at a later time in the day?
Any design or best practice advice would be helpful here.
Is it common practice to write a program to connect via FIX and leave it running for the entire session time? Or is it acceptable to close the program, given I don't send a logout message, and reconnect at a later time in the day?
I don't know what others have done, but I used QuickFIX with Python for years and never had any problem running my system all day, OR shutting it down periodically for whatever reason and reconnecting. In the end I wound up leaving the system connected for weeks at a time, since that allowed me to record data.
I would say that the answer to both of your questions is YES. It is common to leave it running. Also, it is acceptable to just close the program.
There can always be edge cases and idiosyncratic features of your implementation and your counterparty, so you should seek to understand more why they have asked you not to disconnect. That sounds very strange to me. Is their FIX engine not capable of something very simple and standard?
Yes it is common to keep the FIX sessions running for a long time. That should not be an issue.
You can't just shutdown your program your end, as Session-level FIX.Heartbeat(35=0) messages, sent periodically (usually 30s), as meant to keep the underlying TCP connection "open", and check that both ends are still up and running properly.
By the details you gave, if your vendor (which is likely the acceptor side) requests it, it might be because they need to send you messages, with no delay as they occur.
If you (the initiator side) are not logged in at that time, they won't be able to send those messages, as they won't be able to initiate a session with you.
The vendor might monitor sessions as well, but as an initiator, it sounds odd.
as initiators are waiting for connections.
More likely they will monitor unexpected sessions drops.
All in all, it very depends of your vendor anyway, you have to follow what they say...
I am working on a small programming game/environment in Python to help my younger brother learn to code. It needs to operate over a network, and I am new to network programming. I am going to explain the concept of the game so that someone can point me in the best direction.
The idea is a simple grid of 25x25 'diodes,' squares with fixed positions and editable color values, essentially simulating a very small screen. In addition to the grid display, there is a command window, where Python code can be entered and sent to an instance of InteractiveConsole, and a chat window. A client needs to be able to send Python commands to the host, which will run the code, and then receive the output in the form of a string representing changes to the grid. My concept for doing this involves maintaining a queue on the host side of incoming and outgoing events to handle and relay to the clients on individual threads. Any given command/chat event will be sent to the host and relayed to all clients, including the client who created the event, so that those events are visible to all clients in their command/chat windows. All changes to the grid will originate with the host as a result of processing commands originated from clients and will also be sent out to all clients.
What I primarily don't understand is how to synchronize between all clients, i.e. how to know when a given item in the queue has been successfully sent out to all clients before clearing it from the queue, since any individual thread doing so prematurely will prevent the item from being sent to other clients. This is an extremely open-ended question because I understand that I will definitely need to consume some learning materials before I'm ready to implement this. I'm not asking for a specific solution but rather for some guidance on what general type of solution could work in my situation. I'm doing this in my spare time, so I don't want to spend a month going through networking tutorials that aren't pointing me in a direction that will be applicable to this project.
My approach would be to use a udp server that can broadcast to multiple clients. So basically, all the clients would connect to this server during a game session, and the server would broadcast the game state to the clients as it is updated. Since your game is relatively simple this approach would give you real time updates.
I am using mysqldb for my database currently, and I need to integrate a messaging feature that is in real-time. The chat demo that Tornado provides does not implement a database, (whereas the blog does.)
This messaging service also will also double as an email in the future (like how Facebook's message service works. The chat platform is also email.) Regardless, I would like to make sure that my current, first chat version will be able to be expanded to function as email, and overall, I need to store messages in a database.
Is something like this as simple as: for every chat message sent, query the database and display the message on the users' screen. Or, is this method prone to suffer from high server load and poor optimization? How exactly should I structure the "infrastructure" to make this work?
(I apologize for some of the inherent subjectivity in this question; however, I prefer to "measure twice, code once.")
Input, examples, and resources appreciated.
Regards.
Tornado is a single threaded non blocking server.
What this means is that if you make any blocking calls on the main thread you will eventually kill performance. You might not notice this at first because each database call might only block for 20ms. But once you are making more than 200 database calls per seconds your application will effectively be locked up.
However that's quite a few DB calls. In your case that would be 200 people hitting send on their chat message in the same second.
What you probably want to do is use a queue with a non blocking API. So Tornado receives a chat message. You put it on the queue to be saved to the database by another process, then you send the chat message back out to the other chat members.
When someone connects to a chat session you also need to send off a request to the queue for all the previous messages, when the queue responds you send those off to the newly connected user.
That's how I would approach the problem anyway.
Also see this question and answer: Any suggestion for using non-blocking MySQL api on Tornado in Python3?
Just remember, Tornado is single threaded. It's amazing. And can handle thousands of simultaneous connections. But if code in one of those connections blocks for 1 second then NOTHING else will be done for any other connection during that second.
I want to implement a lightweight Message Queue proxy. It's job is to receive messages from a web application (PHP) and send them to the Message Queue server asynchronously. The reason for this proxy is that the MQ isn't always avaliable and is sometimes lagging, or even down, but I want to make sure the messages are delivered, and the web application returns immediately.
So, PHP would send the message to the MQ proxy running on the same host. That proxy would save the messages to SQLite for persistence, in case of crashes. At the same time it would send the messages from SQLite to the MQ in batches when the connection is available, and delete them from SQLite.
Now, the way I understand, there are these components in this service:
message listener (listens to the messages from PHP and writes them to a Incoming Queue)
DB flusher (reads messages from the Incoming Queue and saves them to a database; due to SQLite single-threadedness)
MQ connection handler (keeps the connection to the MQ server online by reconnecting)
message sender (collects messages from SQlite db and sends them to the MQ server, then removes them from db)
I was thinking of using Twisted for #1 (TCPServer), but I'm having problem with integrating it with other points, which aren't event-driven. Intuition tells me that each of these points should be running in a separate thread, because all are IO-bound and independent of each other, but I could easily put them in a single thread. Even though, I couldn't find any good and clear (to me) examples on how to implement this worker thread aside of Twisted's main loop.
The example I've started with is the chatserver.py, which uses service.Application and internet.TCPServer objects. If I start my own thread prior to creating TCPServer service, it runs a few times, but the it stops and never runs again. I'm not sure, why this is happening, but it's probably because I don't use threads with Twisted correctly.
Any suggestions on how to implement a separate worker thread and keep Twisted? Do you have any alternative architectures in mind?
You're basically considering writing an ad-hoc extension to your messaging server, the job of which it is to provide whatever reliability guarantees you've asked of it.
Instead, perhaps you should take the hardware where you were planning to run this new proxy and run another MQ node on it. The new node should take care of persisting and relaying messages that you deliver to it while the other nodes are overloaded or offline.
Maybe it's not the best bang for your buck to use a separate thread in Twisted to get around a blocking call, but sometimes the least evil solution is the best. Here's a link that shows you how to integrate threading into Twisted:
http://twistedmatrix.com/documents/10.1.0/core/howto/threading.html
Sometimes in a pinch easy-to-implement is faster than hours/days of research which may all turn out to be for nought.
A neat solution to this problem would be to use the Key Value store Redis. Its a high speed persistent data store, with plenty of clients - it has a php and a python client (if you want to use a timed/batch process to process messages - it saves you creating a database, and also deals with your persistence stories. It runs fine on Cywin/Windows + posix environments.
PHP Redis client is here.
Python client is here.
Both have a very clean and simple API. Redis also offers a publish/subscribe mechanism, should you need it, although it sounds like it would be of limited value if you're publishing to an inconsistent queue.