I am working on a text-based game in Python 3.1 that would use timing as it's major source of game play. In order to do this effectively (rather than check the time every mainloop, my current method, which can be inaccurate, and slow if multiple people are playing the game at once) I was thinking about using the Threading.Timer class. Is it a bad thing to have multiple timers going at the same time? if so, how many timers is recommended?
For example, the user inputs to start the game. every second after the game starts it decides whether or not something happens, so there's a Timer(1) for every user playing at the same time. If something happens, the player has a certain time to react to it, so a timer must be set for that. If the user reacts quickly enough, that timer needs to end and it will set a new timer depending on what's going to happen next, etc
I think its a bad idea to use Timers in your case.
Using the delayed threads in python will result in more complex code, less accuracy, and quite possible worse performance. Basically, the rule is that if you think you need threads, you don't. Very few programs benefit from the use of threads.
I don't know what you are doing for input. You make reference to multiple players and I'm not sure whether thats on a single keyboard or perhaps networked. Regardless, your current strategy of a main loop may well be the best strategy. Although without seeing how your main loop operates its hard to say for certain.
It should be perfectly safe to have multiple timers going at the same time. Beware that it may not give much of a performance boost, as the CPython interpreter (the standard Python interpreter) uses a GIL (Global Interpreter Lock) which makes threading stuff a bit.... slow.
Related
I'm working on a personal project that has been refactored a number of times. It started off using multithreading, then parts of it used asyncio, and now it is back to being mainly single threaded.
As a result of all these changes I have a number of threading.Lock()'s in the code that I would like to remove and cleanup to prevent future issues.
How can I easily work out which locks are in use and hit by more than one thread during the runtime of the application?
If I am in the situation to find that out, I would try to replace the lock with a wrapper that do the counting (or print something, raise an exception, etc.) for me when the undesired behavior happened. Python is hacky, so I can simply create a function and overwrite the original threading.Lock to get the job done. That might need some careful implementation, e.g., catch both all possible pathway to lock and unlock.
However, you have to be careful that even so, you might not exercise all possible code path and thus never know if you really remove all "bugs".
I tried creating a simple game using the pygame module in python. It's a two player game, so it involves getting simultaneous inputs or in other words, events, in the same pygame screen. I tried using multithreading, that is, by running two threads that share a pygame screen, but it didn't work due to the GIL. So, when I tried using multiprocessing instead, I faced a problem in that I couldn't share the same pygame screen. So is there a way to create a pool that has the pygame screen, like there is for sequences? If not, is there any other, perhaps better method to approach my problem? Any answers will be appreciated. Thank you.
You likely don't need either multi-threading and much less multi-processing there.
What do you mean by "it didn't work due to the GIL"? Did you get any error message when trying to do multi-threaded?
It is possible to create a naive multi-threading design that could facilitate updating objects controlled by different players, but some care would have to be taken anyway. The GIL should only impact you that you really would have no benefit like using more than 1 core in your application, but won't give you any errors.
As for using multi-processing, the secondary process would not be able to use the screen Surface anyway, and passing surfaces (images) across process is mostly not worth it (if it is feasible at all).
I meant that when I pressed a button, only one of the threads detected
key down. I did not get any errors. It is due to GIL right? I'm sorry
if I'm wrong.
No - this does not have to do with the GIL in any way -- it has to do with the pygame event API not having being designed to work with threads.
There are two ways of dealing with key-presses in Pygame: either checking for KEYDOWN events, or calling pygame.key.get_pressed. I usually prefer the second due to it not depending on key auto-repeating settings at the OS or BIOS level (unless, of course, it is an action that is to be executed just once, and not while a key is pressed).
Anyway, in any pygame related project, you should check for pygame events (key presses included) in a single place in your code, and on the same thread. Once you have the information you need from the "outside world", you can set your internal variables as you want.
But, since you don't offer any code, the most direct fix I can suggest to your multi-threaded code is to use get_pressed instead of checking for events: it will likely work from both threads. (the multi-processing code would be much more complicated than that).
I have written a monitoring program for the control system at our plant. It is basically a GUI which lets the operator see the current status of the lock of the closed loop system and aware the operator in case the lock/loop breaks.
Now, the operation is heavily dependent on the responses of the GUI. My seniors told me that they prefer just the console prints instead of using TKinter based GUI as TKinter has lags while working in real time.
Can anyone please comment on this aspect?
Can this lag be checked and corrected?
Thanks in advance.
I would say that if your program is simply accessing data and not interacting with the data, then a GUI seems to be a bit of overkill. GUI's are guided user interfaces, as you know, and are made for guiding a user through an interface. If the interface is just a status, as you indicated, then I see nothing wrong with a console program.
If, however, your program also interacts with data in a way that would be difficult without a GUI, then the GUI is likely the right choice.
Have you considered a GUI in another programming language? Python is known to be a bit slow, even in console. In my experience, C++ is faster in terms of viewing data. Best of luck!
Python / tkinter in general
In a tkinter program, your code falls in one of four categories;
Initialization code that runs before the mainloop is started.
Callbacks that are run from within the mainloop.
Code running in other threads.
Code running in different processes.
In the first case, the time the code takes only influences the startup time, which for a long-running program is probably not all that relevant.
Concerning the second case, well-written callbacks should not take that long to run. In the order of tens of milliseconds, maybe up to 100 ms. If they take longer, they will render the GUI unresponsive. So unless you notice a sluggish GUI (without threads; see below) this should not be a problem.
One pitfall here are after callbacks, that is functions that will be scheduled to run after a certain time. If you launch them too often, this will also starve the GUI of time.
Another possible problem might be the manipulation of a Canvas with lots and lots of items in it.
As of Python 3.x, tkinter is thread-safe to the best of my understanding. However, in the reference implementation of Python, only one thread at a time can be executing Python bytecode. So doing heavy calculations in a second thread would slow down the GUI.
If you GUI uses multiprocessing to run calculations in another process, that should not influence the speed of your GUI much, unless you do things wrong when communicating with that other process.
Your monitoring program
What is too slow depends on the situation. In general Python is not considered a language suitable for hard real-time programs. To do hard real-time one also needs a suitable operating system.
So the question then becomes what is the acceptable lag in your system specification? Without knowing that it is impossible to precisely answer your question.
It seems that your GUI is just displaying some system status. That should not cause too much of a load, provided that you don't read/check the data too often. As described in the callbacks paragraph above it is possible to starve your GUI of CPU cycles with callbacks that run too often. From what you've written, I gather that the GUI's task is just to inform the human operator.
That leads me to believe that the task is not hugely time critical; a system that requires millisecond intervention time should not rely on a human operator.
So based on your information I would say that a competently written GUI should probably not be too slow.
There's different ways of doing concurrent in Python, below is a simple list:
process-based: process.Popen, multiprocessing.Process, old fashioned os.system, os.popen, os.exe*
thread-based: threading.Thread
microthread-based: greenlet
I know the difference between thread-based concurrency and process-based concurrency, and I know some (but not too much) about GIL's impact in CPython's thread support.
For a beginner who want to implement some level of concurrency, how to choose between them? Or, what's the general difference between them? Are there any more ways to do concurrent in Python?
I'm not sure if I'm asking the right question, please feel free to improve this question.
The reason all three of these mechanisms exist is that they have different strengths and weaknesses.
First, if you have huge numbers of small, independent tasks, and there's no sensible way to batch them up (typically, this means you're writing a C10k server, but that's not the only possible case), microthreads win hands down. You can only run a few hundred OS threads or processes before everything either bogs down or just fails. So, either you use microthreads, or you give up on automatic concurrency and start writing explicit callbacks or coroutines. This is really the only time microthreads win; otherwise, they're just like OS threads except a few things don't work right.
Next, if your code is CPU-bound, you need processes. Microthreads are an inherently single-core solution; Threads in Python generally can't parallelize well because of the GIL; processes get as much parallelism as the OS can handle. So, processes will let your 4-core system run your code 4x as fast; nothing else will. (In fact, you might want to go farther and distribute across separate computers, but you didn't ask about that.) But if your code is I/O-bound, core-parallelism doesn't help, so threads are just as good as processes.
If you have lots of shared, mutable data, things are going to be tough. Processes require explicitly putting everything into sharable structures, like using multiprocessing.Array in place of list, which gets nightmarishly complicated. Threads share everything automatically—which means there are race conditions everywhere. Which means you need to think through your flow very carefully and use locks effectively. With processes, an experienced developers can build a system that works on all of the test data but has to be reorganized every time you give it a new set of inputs. With threads, an experienced developer can write code that runs for weeks before accidentally and silently scrambling everyone's credit card numbers.
Whichever of those two scares you more—do that one, because you understand the problem better. Or, if it's at all possible, step back and try to redesign your code to make most of the shared data independent or immutable. This may not be possible (without making things either too slow or too hard to understand), but think about it hard before deciding that.
If you have lots of independent data or shared immutable data, threads clearly win. Processes need either explicit sharing (like multiprocessing.Array again) or marshaling. multiprocessing and its third-party alternatives make marshaling pretty easy for the simple cases where everything is picklable, but it's still not as simple as just passing values around directly, and it's also a lot slower.
Unfortunately, most cases where you have lots of immutable data to pass around are the exact same cases where you need CPU parallelism, which means you have a tradeoff. And the best answer to this tradeoff may be OS threads on your current 4-core system, but processes on the 16-core system you have in 2 years. (If you organize things around, e.g., multiprocessing.ThreadPool or concurrent.futures.ThreadPoolExecutor, and trivially switch to Pool or ProcessPoolExecutor later—or even with a runtime configuration switch—that pretty much solves the problem. But this isn't always possible.)
Finally, if your application inherently requires an event loop (e.g., a GUI app or a network server), pick the framework you like first. Coding with, say, PySide vs. wx, or twisted vs. gevent, is a bigger difference than coding with microthreads vs. OS threads. And, once you've picked the framework, see how much you can take advantage of its event loop where you thought you needed real concurrency. For example, if you need some code to run every 30 seconds, don't start a thread (micro- or OS) for that, ask the framework to schedule it however it wants.
I'm building a multiplayer card game with Python, gevent and django-socketio and I'm wondering about the best way to maintain state on things, bearing in mind that there'll be multiple clients connecting at once and doing things.
I'm using Redis as a datastore for the in game bits, with light object models on top (Redisco at the mo).
I'm concerned about defending against race conditions and therefore keeping the game state safe and consistent with so many clients trying to do things at once. I'm thinking that my main options are:
(1) - Ensure that all operations are safe with more that one client doing things at once (eg, a player can only interact with certain properties of their own player model, and there's some objective game state via another thread or something which does anything else.)
(2) - Use a queue with some global lock to ensure client operations all happen in a certain guaranteed order, and one finishes before the next one starts.
I'm using Python, Django, django-socketio, gevent, but think this applies more broadly.
Is this the "threadsafe" thing that people refer to?
I guess in theory I think I prefer the idea of (1), and I think that I can ensure safe operations by just modifying a single Redis key at a time, or safe sets of atomic operations, but I guess I'd either need to throw away the Redisco models or be very careful about understanding when things get saved and written. I think that's fine for just a couple of us working on things but might be dangerous longer term with more people in the codebase.
Thanks!
You have described your options well enough. Probably you need to combine both approaches.
Ensure that you have as little shared state as possible.
Use queue for modifications to whatever shared state remains.