Developing a project of mine I realize I have a need for some level of persistence across sessions, for example when a user executes the application, changes some preferences and then closes the app. The next time the user executes the app, be it after a reboot or 15 minutes, I would like to be able to retain the preferences that had been changed.
My question relates to this persistence. Whether programming an application using the win32 API or the MFC Framework .. or using the newer tools for higher level languages such as wxPython or wxRuby, how does one maintain the type of persistence I refer to? Is it done as a temporary file written to the disk? Is it saved into some registry setting? Is there some other layer it is stored in that I am unaware of?
I would advice to do it in two steps.
First step is to save your prefs. as
string, for that you can
a)
Use any xml lib or output xml by
hand to output string and read
similarly from string
b) Just use pickle module to dump your prefs object as a string
c) Somehow generate a string from prefs which you can read back as prefs e.g. use yaml, config , JSON etc actually JSON is a good option when simplejson makes it so easy.
Once you have your methods to convert to and from string are ready, you just need to store it somewhere where it is persisted and you can read back next time, for that you can
a) Use wx.Config which save to registry in windows and to other places depending on platform so you don't have to worry where it saves, you can just read back values in platform independent way. But if you wish you can just use wx.Config for directly saving reading prefs.
b) Directly save prefs. string to a file in a folder assigned by OS to your app e.g. app data folder in windows.
Benefit of saving to a string and than using wx.Config to save it, is that you can easily change where data is saved in future e.g. in future if there is a need to upload prefs. you can just upload prefs. string.
There are different methods to do this that have evolved over the years.
These methods include (but not limited to):
Registry entries.
INI files.
XML Files
Simple binary/text files
Databases
Nowadays, most people do this kind of thing with XML files residing in the user specific AppData folders. It is your choice how you do it. For example, for simple things, databases can be overkill and for huge persisted objects, registry would not be appropriate. You have to see what you are doing and do it accordingly.
Here is a very good discussion on this topic
Related
I tried with SASpy but it's not working. I am able to open the SAS .egp file but not able to run the multiple scripts within in sequence.
import os, sys, subprocess
def OpenProject(sas_exe, egp_path):
sasExe = sas_exe
sasEGpath = egp_path
subprocess.call([sasExe, sasEGpath])
sas_exe = path\path\
egp_path = path\path\path\
OpenProject(sas_exe, egp_path)
This depends a bit on exactly what the workflow is. A few side notes, then the full solution.
First: EGP is not really intended to store production processes, in my opinion. EGP should really be used for development, then production is done with .sas (text) files. EGP can directly store the nodes as .sas files; ask a new question about that if you want to know more, but it's pretty easy to figure out. Best practice is to have EGP save the code modules as .sas files, then run those - SASPy will easily do that for you.
Second: If you use SAS's built-in Git connectivity, then you can do this a bit more easily I suspect. Consider doing that if you already use Git for your other processes. Again, then you end up with a .sas file, and can directly run that via SASPy.
So: how can you do this in Python, with the assumption you do have to use the .egp itself, without too many different moving parts? The key here is the .egp format. EGP is a container file, which is actually a .zip format container that has in it, among other things, all of the SAS code you want to run, as text. Text in xml format, but still, text.
You can write a python program that opens the .egp as a .zip file, using the zipfile library, and then use xml.etree.ElementTree to parse the project.xml file inside that project. Exactly what you do from there depends on your particular details, and is well out of scope for a Stack Overflow answer, but if you do better visually you can simply rename the .egp to .zip and then open in unzip program of your choice, then browse project.xml in your text editor, and find the nodes and code related to those nodes.
You can then extract the .sas code as text, and submit it directly via SASPy, or extract it to a .sas file and then submit that however you prefer (SASPy or something else).
I do something similar to this for a project - I don't actually run code from it, I'm just parsing it to verify that the correct programs were synced from the EGP to production - but it would be trivial to actually submit the code from what I've written, which is about 50 lines of code total. I may write a SGF paper this year or next year on this topic, in which case I'll try and remember to submit it here - or you can head over to my github page and see if it's there (in the future!).
I want to make a python program (with a PyQt GUI, but I don't know whether that is relevant) that has to save some information that I want to store even when the program closes. Example for information I want to store:
The user can search for a file in a file dialog window. I want to start the file dialog window in the previously used directory, even if the program is closed in between file searches.
The user can enter their own categories to sort items, building up on some of my predefined categories. These new categories should be available the next time the program starts.
Now I'm wondering what the proper way to store such information is. Should I use pickle? A proper database (I know a tiny bit of sqlite3, but would have to read up on that)? A simple text file that I parse myself? One thing for data like in example 1., another for data like in example 2.?
Also, whatever way to store it I use, where would I put that file?
I'm asking in the context that I might want to later make my program available to others as a standalone application (using py2app, py2exe or PyInstaller).
Right now I'm just saving a pickle file in the directory that my .py file is in, like this answer reconmends, but the answer also specifically mentions:
for a personal project it might be enough.
(emphasis mine)
Is using pickle also the "proper, professional" way, if I want to make the program available to other people as a standalone application?
Choice depends on your approach to data you store, which is yours?:
user should be able to alter it without usage of my program
user should be prevented from altering it with program other than my program
If first you might consider deploying JSON open-standard file format, for which Python has ready library called json. In effect you get text (which you can save to file) which is human-readable and can be edited in text editor. Also there exist JSON file viewers and editors which made viewing/editing of JSON files easier.
I think SQLite3 is the better solution in this case as Moldovan commented.
There is a problem in pickle, sometimes pickling format can be change across python versions and there are greater advantages of using sqlite3.
I created a little script in python to generate an excel compatible xml file (saved with xls extension). The file is generated from a part database so I can place an order with the extracted data.
On the website for ordering the parts, you can import the excel file so the order fills automatically. The problem here is that each time I want to make an order, I have to open excel and save the file with xls extension of type MS Excel 97-2003 to get the import working.
The excel document then looks exactly the same, but when opened with notepad, we cannot see the xml anymore, only binary dump.
Is there a way to automate this process, by running a bat file or maybe adding some line to my python script so it is converted in the proper format?
(I know that question has been asked before, but it never has been answered)
There are two basic approaches to this.
You asked about the first: Automating Excel to open and save the file. There are in fact two ways to do that. The second is to use Python tools that can create the file directly in Python without Excel's help. So:
1a: Automating Excel through its automation interface.
Excel is designed to be controlled by external apps, through COM automation. Python has a great COM-automation interface inside of pywin32. Unfortunately, the documentation on pywin32 is not that great, and all of the documentation on Excel's COM automation interface is written for JScript, VB, .NET, or raw COM in C. Fortunately, there are a number of questions on this site about using win32com to drive Excel, such as this one, so you can probably figure it out yourself. It would look something like this:
import win32com.client
excel = win32com.client.Dispatch('Excel.Application')
spreadsheet = excel.Workbooks.Open('C:/path/to/spreadsheet.xml')
spreadsheet.SaveAs('C:/path/to/spreadsheet.xls', fileformat=excel.xlExcel8)
That isn't tested in any way, because I don't have a Windows box with Excel handy. And I vaguely remember having problems getting access to the fileformat names from win32com and just punting and looking up the equivalent numbers (a quick google for "fileformat xlExcel8" shows that the numerical equivalent is 56, and confirms that's the right format for 97-2003 binary xls).
Of course if you don't need to do it in Python, MSDN is full of great examples in JScript, VBA, etc.
The documentation you need is all on MSDN (since the Office Developer Network for Excel was merged into MSDN, and then apparently became a 404 page). The top-level page for Excel is Welcome to the Excel 2013 developer reference (if you want a different version, click on "Office client development" in the navigation thingy above and pick a different version), and what you mostly care about is the Object model reference. You can also find the same documentation (often links to the exact same webpages) in Excel's built-in help. For example, that's where you find out that the Application object has a Workbooks property, which is a Workbooks object, which has Open and Add methods that return a Workbook object, which has a SaveAs method, which takes an optional FileFormat parameter of type XlFileFormat, which has a value xlExcel8 = 56.
As I implied earlier, you may not be able to access enumeration values like xlExcel8 for some reason which I no longer remember, but you can look the value up on MSDN (or just Google it) and put the number 56 instead.
The other documentation (both here and elsewhere within MSDN) is usually either stuff you can guess yourself, or stuff that isn't relevant from win32com. Unfortunately, the already-sparse win32com documentation expects you to have read that documentation—but fortunately, the examples are enough to muddle your way through almost everything but the object model.
1b: Automating Excel via its GUI.
Automating a GUI on Windows is a huge pain, but there are a number of tools that make it a whole lot easier, such as pywinauto. You may be able to just use swapy to write the pywinauto script for you.
If you don't need to do it in Python, separate scripting systems like AutoIt have an even larger user base and even more examples to make your life easier.
2: Doing it all in Python.
xlutils, part of python-excel, may be able to do what you want, without touching Excel at all.
My application has a xml based configuration. It has also a xsd file. Before my application starts, xmllint will check the configuration against the xsd file.
With the growth of my application, the configuration structure has changed a bit. Now I have to face this problem: When I provide a new version of my application to customer, I have to upgrade the existing configuration.
How to make this done easy and clever?
My idea is to build a configuration object using python, and then read configuration v1 from file and save it as v2. But if later the structure is changed again, I have to build another configuration object model.
For all configuration settings that remain the same between configurations, have your installation script copy those over from the old config file if it exists. For the rest, just have some defaults that the user can change if necessary, as usual for a config file. Unless I've misunderstood the question, it sounds like you're making a bigger deal out of this than it needs to be.
By the way, you'd really only need one "updater" script, because you could parametrize the XML tagging such that it go through your new config file/config layout file, and then just check the tags in the old file against that and copy the data from the ones that are present in the new file. I haven't worked with XSD files before, so I don't know the specifics of working with them, but I don't think it should be that difficult.
I'm working on an App Engine project that will have customizable themes. I'd like to be able to use jQuery UI themes. The problem is figuring out what the CSS file is going to be named. (Typically, "jquery-ui-1.7.2.custom.css". Version numbers will change, and people tend to rename things, but there should only be one CSS file, and I'm OK with it being an error condition if there's two or more for some reason.) Because it's a static file (static files are uploaded to App Engine separately from the rest of the application's resources), I can't just glob the directory for a CSS file. I can't just assume that it's hard-coded, and I really don't want to make it a configuration setting, because that's a bad user experience.
Guido told me to symlink it so that App Engine sees two copies and can treat one as static and the other as an application resource, but symlinks don't work on Windows, and since this will ultimately be open source, I can't control which SDK the user uses. Another suggestion was to use a deploy-time script, but Mac users have this nice "Deploy" button in their version of the SDK and I'd rather not have to tell them, "Oh hey, sorry for the inconvenience, but you can't use that for this project."
I clearly need an out-of-the-box solution to this one, but I'm at a loss. Anyone have any good suggestions for how to get a custom jQuery UI theme out of the ThemeRoller and into an App Engine app? Some post-processing is already needed, because the only files in the zip file that ThemeRoller gives you are in the "css" directory. Maybe I can write something that takes a raw theme as input and spits out something useful on the other side (the deploy-time script trick, but somehow less user-unfriendly). The trick here is presentation — I want the user to spend as little time on the command line as possible. An ideal solution assumes the person performing this task is non-technical for the most part. No part of the solution can be much harder than installing something like WordPress or Drupal, and in a perfect world, it should be way, way easier.
To accomplish what you are asking, I would use the datastore for serving the CSS files. Since this would allow easy listing, sorting and even modification and uploading.
Other than that, your next best options would be to store the CSS data inside a script (a dictionary where the filename is the key name, and the CSS code is the value). Or, as you suggested, to run a script before deploying to AppEngine.
Personally, I would go for the storing in the datastore option, since it will allow for a great deal more user customization (such as each user being able to provide their own CSS file), just be sure to use memcache to avoid needing to access the datastore when possible (which should be a very common occurrence), as well as using HTTP headers to tell the browser to cache the CSS file locally.