I'm wondering how to go about implementing a macro recorder for a python gui (probably PyQt, but ideally agnostic). Something much like in Excel but instead of getting VB macros, it would create python code. Previously I made something for Tkinter where all callbacks pass through a single class that logged actions. Unfortunately my class doing the logging was a bit ugly and I'm looking for a nicer one. While this did make a nice separation of the gui from the rest of the code, it seems to be unusual in terms of the usual signals/slots wiring. Is there a better way?
The intention is that a user can work their way through a data analysis procedure in a graphical interface, seeing the effect of their decisions. Later the recorded procedure could be applied to other data with minor modification and without needing the start up the gui.
You could apply the command design pattern: when your user executes an action, generate a command that represents the changes required. You then implement some sort of command pipeline that executes the commands themselves, most likely just calling the methods you already have. Once the commands are executed, you can serialize them or take note of them the way you want and load the series of commands when you need to re-execute the procedure.
Thinking in high level, this is what I'd do:
Develop a decorator function, with which I'd decorate every event-handling functions.
This decorator functions would take note of thee function called, and its parameters (and possibly returning values) in a unified data-structure - taking care, on this data structure, to mark Widget and Control instances as a special type of object. That is because in other runs these widgets won't be the same instances - ah, you can't even serialize a toolkit widget instances, be it Qt or otherwise.
When the time comes to play a macro, you fill-in the gaps replacing the widget-representating object with the instances of the actually running objects, and simply call the original functions with the remaining parameters.
In toolkits that have an specialized "event" parameter that is passed down to event-handling functions, you will have to take care of serializing and de-serializing this event as well.
I hope this can help. I could come up with some proof of concept code for that (although I am in a mood to use tkinter today - would have to read a lot to come up with a Qt4 example).
An example of what you're looking for is in mayavi2. For your purposes, mayavi2's "script record" functionality will generate a Python script that can then be trivially modified for other cases. I hear that it works pretty well.
Related
I am trying improve my Python and tkinter skills. At this moment I am reading a book on tkinter. I noticed that in an object oriented GUI application the author defines the button callbacks this way:
def on_fast_forward_button_clicked(self):
self.player.fast_forward()
def on_rewind_button_clicked(self):
self.player.rewind()
So, if I understand correctly, we are writing to methods for each button in the application class. One with the "indirect" callback, called but the event, and other with the intended action itself, called by the previous one. Why should we write code like this, instead of just using a single method?
There's no specific reason, and in the example you give I absolutely wouldn't do it that way.
It's really quite simple: in general terms, every button should be associated with a function. That's it. That's the rule. What that function does is totally up to you.
If all it ever does is a single thing, there's no reason it can't directly call that single thing. If it needs to do more than one thing, create a special purpose function that does everything the button should do, and have that button call that function.
That being said, there's a conceptual difference between "responding to a button" and "performing an action". For example, you might have two or three ways to do the fast-forward function: you might have a command on the menubar, a command on a right-click menu, a dedicated fast-forward button, and a keyboard accelerator.
Depending on how these are implemented, the functions they call may or may not require extra arguments (eg: a function bound to an event always gets an event object). In that case it is often easier to have a special purpose function that is designed to respond to the event, and that special purpose function call the more general-purpose fast_forward function.
At the end of the day, it's all about clarity. If you think it makes the program easier to understand when you have a single function perform two roles (fastforward, and respond to a button click), by all means do it that way.
Think about it like this: currently, all the button does is have the player fast forward. What if in the future however you also wanted the clock, or some other entity to fast forward with it? Would it make any sense to stick those fast forward calls inside player.fast_forward()? Should the player's methods really have access to information like that?
on_fast_forward_button_clicked is acting as a wrapper to encompass everything that may need to be fast forwarded. If you ever need to add any functionality, all you need to do is make a minor modification:
def on_fast_forward_button_clicked(self):
self.player.fast_forward()
self.clock.fast_forward()
I want to write a Python app that uses GTK (via gi.repository) to display a textual view of a huge amount of data. (Specifically, disassembled instructions from a program, similar to what IDA shows.)
I thought this should be fairly simple: use an ordinary GtkTextView, and a custom subclass of GtkTextBuffer which will handle the "give me some text" request, generate some text (disassemble some instructions) and some tags (for colouring, formatting, etc) and return them.
The issue is I can't find any information on how to subclass GtkTextBuffer in this way, to provide the text myself. I've tried just implementing the get_text and get_slice methods in my subclass, but they seem to never be called. It seems like the only thing I can do is use a standard GtkTextBuffer and the set_text method, and try somehow to keep track of the cursor position and number of lines to display, but this seems entirely opposite to how MVC should work. There are potentially millions of lines, so generating all text in advance is infeasible.
I'm using Python 3.4 and GTK3.
Gtk.TextBuffer is from an external library that isn't written in Python. You've run into one limitation of that situation. With most Python libraries you can subclass their classes or monkeypatch their APIs however you like. GTK's C code, on the other hand, is unaware that it's being used from Python, and as you have noticed, completely ignores your overridden get_text() and get_slice() methods.
GTK's classes additionally have the limitation that you can only override methods that have been declared "virtual". Here's how that translates into Python: you can see a list of virtual methods in the Python GI documentation (example for Gtk.TextBuffer). These methods all start with do_ and are not meant to be called from your program, only overridden. Python GI will make the GTK code aware of these overrides, so that when you override, e.g., do_insert_text() and subsequently call insert_text(), the chain of calls will look something like this:
Python insert_text()
C gtk_text_buffer_insert_text()
C GtkTextBufferClass->insert_text() (internal virtual method)
Python do_insert_text()
Unfortunately, as you can see from the documentation that I linked above, get_text() and get_slice() are not virtual, so you can't override them in a subclass.
You might be able to achieve your aim by wrapping one TextBuffer (which contains the entirety of your disassembled instructions) in another (which contains an excerpt, and is actually hooked up to the text view.) You could set marks in the first text buffer to show where the excerpt should begin or end, and connect signals such that when the text in the first text buffer changes, or the marks change, then the text between the marks is copied to the second text buffer.
I'm trying to develop a system that will allow users to update local, offline databases on their laptops and, upon reconnection to the network, synchronize their dbs with the main, master db.
I looked at MySQL replication, but that documentation focuses on unidirectional syncing. So I think I'm going to build a custom app in python for doing this (bilateral syncing), and I have a couple of questions.
I've read a couple of posts regarding this issue, and one of the items which has been passively mentioned is serialization (which I would be implementing through the pickle and cPickle modules in python). Could someone please tell me whether this is necessary, and the advantages of serializing data in the context of syncing databases?
One of the uses in wikipedia's entry on serialization states it can be used as "a method for detecting changes in time-varying data." This sounds really important, because my application will be looking at timestamps to determine which records have precedence when updating the master database. So, I guess the thing I don't really get is how pickling data in python can be used to "detect changes in time-varying data", and whether or not this would supplement using timestamps in the database to determine precedence or replace this method entirely.
Anyways, high level explanations or code examples are both welcome. I'm just trying to figure this out.
Thanks
how pickling data in python can be used to "detect changes in time-varying data."
Bundling data in an opaque format tells you absolutely nothing about time-varying data, except that it might have possibly changed (but you'd need to check that manually by unwrapping it). What the article is actually saying is...
To quote the actual relevant section (link to article at this moment in time):
Since both serializing and deserializing can be driven from common code, (for example, the Serialize function in Microsoft Foundation Classes) it is possible for the common code to do both at the same time, and thus 1) detect differences between the objects being serialized and their prior copies, and 2) provide the input for the next such detection. It is not necessary to actually build the prior copy, since differences can be detected "on the fly". This is a way to understand the technique called differential execution[a link which does not exist]. It is useful in the programming of user interfaces whose contents are time-varying — graphical objects can be created, removed, altered, or made to handle input events without necessarily having to write separate code to do those things.
The term "differential execution" seems to be a neologism coined by this person, where he described it in another StackOverflow answer: How does differential execution work?. Reading over that answer, I think I understand what he's trying to say. He seems to be using "differential execution" as a MVC-style concept, in the context where you have lots of view widgets (think a webpage) and you want to allow incremental changes to update just those elements, without forcing a global redraw of the screen. I would not call this "serialization" in the classic sense of the word (not by any stretch, in my humble opinion), but rather "keeping track of the past" or something like that. Because this basically has nothing to do with serialization, the rest of this answer (my interpretation of what he is describing) is probably not worth your time unless you are interested in the topic.
In general, avoiding a global redraw is impossible. Global redraws must sometimes happen: for example in HTML, if you increase the size of an element, you need to reflow lower elements, triggering a repaint. In 3D, you need to redraw everything behind what you update. However if you follow this technique, you can reduce (though not minimize) the number of redraws. This technique he claims will avoid the use of most events, avoid OOP, and use only imperative procedures and macros. My interpretation goes as follows:
Your drawing functions must know, somehow, how to "erase" themselves and anything they do which may affect the display of unrelated functions.
Write a sideffect-free paintEverything() script that imperatively displays everything (e.g. using functions like paintButton() and paintLabel()), using nothing but IF macros/functions. The IF macro works just like an if-statement, except...
Whenever you encounter an IF branch, keep track of both which IF statement this was, and the branch you took. "Which IF statement this was" is sort of a vague concept. For example you might decide to implement a FOR loop by combining IFs with recursion, in which case I think you'd need to keep track of the IF statement as a tree (whose nodes are either function calls or IF statements). You ensure the structure of that tree corresponds to the precedence rule "child layout choices depend on this layout choice".
Every time a user input event happens, rerun your paintEverything() script. However because we have kept track of which part of the code depends on which other parts, we can automatically skip anything which did not depend on what was updated. For example if paintLabel() did not depend on the state of the button, we can avoid rerunning that part of the paintEverything() script.
The "serialization" (not really serialization, more like naturally-serialized data structure) comes from the execution history of the if-branches. Except, serialization here is not necessary at all; all you needed was to keep track of which part of the display code depends on which others. It just so happens that if you use this technique with serially-executed "smart-if"-statements, it makes sense to use a lazily-evaluated diff of execution history to determine what you need to update.
However this technique does have useful takeaways. I'd say the main takeaway is: it is also a reasonable thing to keep track of dependencies not just in an OOP-style (e.g. not just widget A depends on widget B), but dependencies of the basic combinators in whatever DSL you are programming in. Also dependencies can be inferred from the structure of your program (e.g. like HTML does).
I have a module for a physical process simulation. Now I want to do a GUI for it. I've used PyQt to create application, which allows to control calculation, its parameters and graphical output. But there is a problem - if I want to add some new feature to simulation, then in addition to coding it in computational module, I need to create wrapper function for calling it in GUI module, GUI element for calling it, set up GUI representation and bind it to the function I need. That's annoying, because I need to write an amount of obvious code instead of working on logic. I have a few ideas on automatization of process, but first I want to ask: are there some recommended ways to reduce this boilerplate work?
Welcome to GUI programming! :)
In PyQt, you can slightly reduce the amount of boilerplate by autoconnecting slots by name.
You can also create any GUI element at runtime, without resorting to QtDesigner, so you can do things like generate a list of checkboxes in a for loop.
I'm making a wxpython app that I will compile with the various freezing utility out there to create an executable for multiple platforms.
the program will be a map editer for a tile-based game engine
in this app I want to provide a scripting system so that advanced users can modify the behavior of the program such as modifying project data, exporting the project to a different format ect.
I want the system to work like so.
the user will place the python script they wish to run into a styled textbox and then press a button to execute the script.
I'm good with this so far thats all really simple stuff.
obtain the script from the text-box as a string compile it to a cod object with the inbuilt function compile() then execute the script with an exec statment
script = textbox.text #bla bla store the string
code = compile(script, "script", "exec") #make the code object
eval(code, globals())
the thing is, I want to make sure that this feature can't cause any errors or bugs
say if there is an import statement in the script. will this cause any problems taking into account that the code has been compiled with something like py2exe or py2app?
how do I make sure that the user can't break critical part of the program like modifying part of the GUI while still allowing them to modify the project data (the data is held in global properties in it's own module)? I think that this would mean modifying the globals dict that is passed to the eval function.
how to I make sure that this eval can't cause the program to hang due to a long or infinite loop?
how do I make sure that an error raised inside the user's code can't crash the whole app?
basically, how to I avoid all those problems that can arise when allowing the user to run their own code?
EDIT: Concerning the answers given
I don't feel like any of the answers so far have really answered my questions
yes they have been in part answered but not completely. I'm well aware the it is impossible to completely stop unsafe code. people are just too clever for one man (or even a teem) to think of all the ways to get around a security system and prevent them.
in fact I don't really care if they do. I'm more worried about some one unintentional breaking something they didn't know about. if some one really wanted to they could tear the app to shreds with the scripting functionality, but I couldn't care less. it will be their instance and all the problem they create will be gone when they restart the app unless they have messed with files on the HD.
I want to prevent the problems that arise when the user dose something stupid.
things like IOError's, SystaxErrors, InfiniteLoopErrors ect.
now the part about scope has been answered. I now understand how to define what functions and globals can be accessed from the eval function
but is there a way to make sure that the execution of their code can be stopped if it is taking too long?
a green thread system perhaps? (green because it would be eval to make users worry about thread safety)
also if a users uses an import module statement to load a module from even the default library that isn't used in the rest of the class. could this cause problems with the app being frozen by Py2exe, Py2app, or Freeze? what if they call a modal out side of the standard library? would it be enough that the modal is present in the same directory as the frozen executable?
I would like to get these answers with out creating a new question but I will if I must.
Easy answer: don't.
You can forbid certain keywords (import) and operations, and accesses to certain data structures, but ultimately you're giving your power users quite a bit of power. Since this is for a rich client that runs on the user's machine, a malicious user can crash or even trash the whole app if they really feel like it. But it's their instance to crash. Document it well and tell people what not to touch.
That said, I've done this sort of thing for web apps that execute user input and yes, call eval like this:
eval(code, {"__builtins__":None}, {safe_functions})
where safe_functions is a dictionary containing {"name": func} type pairs of functions you want your users to be able to access. If there's some essential data structure that you're positive your users will never want to poke at, just pop it out of globals before passing them in.
Incidentally, Guido addressed this issue on his blog a while ago. I'll see if I can find it.
Edit: found.
Short Answer: No
Is using eval in Python a bad practice?
Other related posts:
Safety of Python 'eval' For List Deserialization
It is not easy to create a safety net. The details too many and clever hacks are around:
Python: make eval safe
On your design goals:
It seems you are trying to build an extensible system by providing user to modify a lot of behavior and logic.
Easiest option is to ask them to write a script which you can evaluate (eval) during the program run.
How ever, a good design describes , scopes the flexibility and provides scripting mechanism through various design schemes ranging from configuration, plugin to scripting capabilities etc. The scripting apis if well defined can provide more meaningful extensibility. It is safer too.
I'd suggest providing some kind of plug-in API and allowing users to provide plug-ins in the form of text files. You can then import them as modules into their own namespace, catching syntax errors in the process, and call the various functions defined in the plug-in module, again checking for errors. You can provide an API module that defines the functions/classes from your program that the plug-in module has access to. That gives you the freedom to make changes to your application's architecture without breaking plug-ins, since you can just adapt the API module to expose the functionality in the same way.
If you have the option to switch to Tkinter you can use the bundled tcl interpreter to process your script. For that matter you can probably do that with a wxpython app if you don't start the tk event loop; just use the tcl interpreter without creating any windows.
Since the tcl interpreter is a separate thing it should be nearly impossible to crash the python interpreter if you are careful about what commands you expose to tcl. Plus, tcl makes creating DSLs very easy.
Python - the only scripting language with a built-in scripting engine :-).