My application caches some data on disk. Because the cache may be large, it should not be stored on a network drive. It should persist between invocations of the application. I have a mechanism for the user to choose a location, but would like the default to be sensible and "the right thing" for the platform.
What is the appropriate location for such a cache? Is there an API for determining the appropriate location? How do I call it from Python?
There are a number a places you can put your application files in Windows. This page shows a list (this enum is .Net specific but most of the special folders are standard on Windows in general). Basically you'll need to decide if you need a cache per user, only for the local machine, per application or shared, etc.
I don't have much experience with python so I cannot specifically help with how to get these paths, but I'm sure someone more knowledgeable here can.
The standard location for Windows application to store their (permanent) application data is referenced by the %APPDATA% (current user) or %ALLUSERSPROFILE% (all users) environment variables. You can access them using, e.g. (only rudimentary and not very elegant error checking!):
import os
app_path = os.getenv("APPDATA") + "\\MyApplicationData"
try:
os.mkdir(app_path)
except WindowsError:
# already exists
Now you have your own directory for your app.
Have a look here: http://en.wikipedia.org/wiki/Environment_variable#User_management_variables. Anything that's under the users directory is good. If its for all users, then it should be: %ALLUSERSPROFILE%. If its for a specific user, make sure the permissions are right.
Check out MSDN for more info about other Windows Versions. Environment variables can vary from systems to systems.
Perhaps the tempfile module provides what you need. It uses the Windows Temp directory (which probably is not on a network drive) but you can specify a directory if you want to. Also for security reasons this module should be the right tool - if you use tempfile.mkstemp() the file is readable and writable only by the creating user ID.
Oh. I see you have just edited your question and that you need file persistence between invocations of the app. Then tempfile is not that ideal (even though you could choose not to delete your cache between invocations).
The wx.StandardPaths module contains methods that return various standard locations in the file system and transparently tries to do "the right thing" under Unix, Mac OS X, and Windows.
A handy package for this is appdirs, which support Windows and most other operating systems. For example, to create a cache directory for "MyApp" version 1.0 published by "Author" is:
import os
from appdirs import user_cache_dir
dirname = user_cache_dir("AppName", "Author", "v1.0")
# C:\Users\username\AppData\Local\Author\AppName\Cache\v1.0
# create it, if it doesn't exist
os.makedirs(dirname, exist_ok=True)
Does the app have any preferences, settings or options that the user can specify? If so, add an option where the user can specify the location of the data, with a default of the current Windows temp directory.
There's always a chance they may not have enough space on the drive with the temp directory, and would need to use a different drive/directory.
Related
I just need to refresh a folder.
A hypothetical ideal example would be:
from aModule import refreshdir # fake
refreshdir("C:\path\to\directory")
Context:
I am using Autodesk Desktop Connector, a service that sync data on the cloud with local folders. To avoid expending resources, this tool just checks for new updates when the user opens the file or refresh the directory (so manually). However, in order to automate some operations, I need to refresh the directory with Python. There is no API for this tool.
Thanks in advance! =)
Edit:
New files can be added in the cloud. That's why it is important to refresh the folder. Example:
Before refreshing:
enter image description here
After refreshing:
enter image description here
os.listdir cannot catch those highlited files before refreshing.
Refreshing a directory is not an operating system operation, but a function of the filesystem browser / explorer. A refresh is essentially just reading in the directory contents anew.
Most likely that Adobe tool is hooking into the filesystem functions that do this enumeration of a directory's contents. If this is the case, then the task should be as simple as
import os
os.listdir("C:/path/to/directory")
Keep in mind that backslashes (\) in standard string literals start an escape sequence, i.e. if you wanted to put an actual backslash there, you'd have to write "\\". However Windows will happily use forward slashes as directory separator as well, so you can just use that :-)
To solve this problem I created a script in Python using the pywinauto library to do a manually task that clicks on the file and then clicks on the Sync option.
In this case you'll need to know the name of the files you want to sync. The code was made to AutoCAD Plant 3D project, you'll need to change the path to your files.
from pywinauto import Application
raiz = "C:\\Users\\YOUR_USERNAME\\ACCDocs\\ORGANIZATION_NAME\\PROJECT_NAME\\Project Files\\PLANT3D_PROJ_NAME\\Plant 3D Models"
Application().start('explorer.exe ' + raiz, timeout=10)
explorer = Application(backend='uia').connect(path='explorer.exe', title="Plant 3D Models")
#Plant3DModels is a variable automatically created with the title of the windows opened
explorer.Plant3DModels.set_focus()
# 'Infra-Geral.dwg' is the name of the file that I will Sync
file = explorer.Plant3DModels.ItemsView.get_item('Infra-Geral.dwg')
file.right_click_input()
explorer.ContextMenu.Sync.invoke()
I need a component that's a browser-based file browser, and I expect some django app to currently provide this. Is there such a thing?
The full story:
I'm building a django app that is used for testing. I want to use it to serve files (and strings, and etc.) and attach custom headers to it.
Currently, I have a model FileSource which has a single file_path field, which is of type django.db.models.FileField.
When creating a FileSource from the admin, the user has a nice file upload dialog, and when saving, the file he chose, is saved on the server (in a really weird location, inside the directory where django is installed, or something weird like that, because i didn't customize the storage, nor will it help me in any way)
My problem: I only want to use the file dialog for the user to select a full path on the server. The file that the user chose must be only referenced, not copied (like currently), and it must reside on the server.
The server must thus be able to list the files it has, so i basically need a little browser-based file-browser.
At that point, I expect to be able to save a full path in my DB, and then I'll be able to access that file and serve it (together with whatever custom headers the user will chose from my app).
Currently, as you might know, the browsers always lie about the full path of the file. Chromium appends "C:\fakepath" to the file name, so I need support of the backend to accomplish this.
Also, I checked out django-filebrowser and django-filer and from what I understood, they weren't built for this. If I'm wrong, a little assistence in configuring them would be awesome.
You can use a FilePathField for that. It won't upload a file, but rather allow you to choose a pre-existing file. A caveat is that you can only use one directory. If you need multiple directories, then you'd need do go with something like django-filer.
Working on a python scraper/spider and encountered a URL that exceeds the char limit with the titled IOError. Using httplib2 and when I attempt to retrieve the URL I receive a file name too long error. I prefer to have all of my projects within the home directory since I am using Dropbox. Anyway around this issue or should I just setup my working directory outside of home?
You are probably hitting limitation of the encrypted file system, which allows up to 143 chars in file name.
Here is the bug:
https://bugs.launchpad.net/ecryptfs/+bug/344878
The solution for now is to use any other directory outside your encrypted home directory. To double check this:
mount | grep ecryptfs
and see if your home dir is listed.
If that's the case either use some other dir above home, or create a new home directory without using encryption.
The fact that the filename that's too long starts with '.cache/www.example.com' explains the problem.
httplib2 optionally caches requests that you make. You've enabled caching, and you've given it .cache as the cache directory.
The easy solution is to put the cache directory somewhere else.
Without seeing your code, it's impossible to tell you how to fix it. But it should be trivial. The documentation for FileCache shows that it takes a dir_name as the first parameter.
Or, alternatively, you can pass a safe function that lets you generate a filename from the URI, overriding the default. That would allow you to generate filenames that fit within the 144-character limit for Ubuntu encrypted fs.
Or, alternatively, you can create your own object with the same interface as FileCache and pass that to the Http object to use as a cache. For example, you could use tempfile to create random filenames, and store a mapping of URLs to filenames in an anydbm or sqlite3 database.
A final alternative is to just turn off caching, of course.
As you apparently have passed '.cache' to the httplib.Http constructor, you should change this to something more appropriate or disable the cache.
I'm working on a program in Python for Windows, and would like to save variables and user preferences so that I can recall them even after the program has been terminated and restarted.
Is there an ideal way to do this on Windows machines? Would _winreg and the Windows registry be suited for this task? Or do I need to create some sort of database of my own?
Python2 has ConfigParser, which is configparser, in Python3:
import ConfigParser, os
config = ConfigParser.ConfigParser()
config.readfp(open('defaults.cfg'))
config.read(['site.cfg', os.path.expanduser('~/.myapp.cfg')])
Even on windows, you should be aware that the registry is a wretched hive of scum and villainy, and that you should not be using it to store your python app configurations.
You're usually going to want to store it in a configuration folder in the "home" folder. That's easy on *nix systems, more difficult in windows, because you actually need to get the "application data" directory. This usually works for me:
import os
if os.name != "posix":
from win32com.shell import shellcon, shell
homedir = "{}\\".format(shell.SHGetFolderPath(0, shellcon.CSIDL_APPDATA, 0, 0))
else:
homedir = "{}/".format(os.path.expanduser("~"))
After you have the homedirectory, you should create a folder named after your project:
if not os.path.isdir("{0}.{1}".format(homedir,projectname)):
os.mkdir("{0}.{1}".format(homedir,projectname))
Then you can make a config file in that folder and write your options to it in the format of your choosing (my personal favorite is in an XML).
Unless I"m missing something, regex has nothing to do with the windows registry. Just store a config file somewhere, like the user's home directory or whatever. At least that's what I'd do on linux. Store a file somewhere in ~/.config/
If you want to store just key-value pairs of preferences, do it in a text config file. If you want to store python structures, do it with pickle or shelve. If you need a light, hassle free, rdbms use sqlite.
While the pickle module is an obvious choice, the files it writes will be non-human-readable. For configuration files, it would be good if your configuration file was a simple text format that your users can read or even edit. So, I recommend you use JSON and the json module.
There are several ways, devided into two main directions
Access the Windows registry with the _winreg module.
Using the registry it is nontrivial to transfer the settings to another machine or to inspect them or back them up; you'd have to use regedit for those.
The other way store the preferences in a file in the user's 'My Documents' folder. This makes it easier to move the settings to another machine or user, and doesn't mix your program's settings with those of a host of other applications. So things like backing them up and restoring them are easier; you just copy one file. If you choose a text format, it is also easier to inspect and debug the settings.
Put your settings in a list, tuple or dictionary and save them with the cPickle module. This is probably one of the easiest methods. On the downside, while pickled data uses an ASCII format by default, it is not human-readable.
Use the ConfigParser module to save and load config files in a similar structure as Windows' .ini files. These files are human-readable.
Use the json module to store settings in json format. These files are human-readable.
Save the preferences as Python expressions to a text file, load it with execfile(). N.B. This could be used to execute arbitrary code, so there are safety considerations. But on the upside, it is very easy and readable.
To save variables, I recommend using pickle
To determine where to store preferences I recommend using the appdirs package to locate the appropriate "application data", "preferences", or "home" directory for your operating system:
from appdirs import user_data_dir
appname = "SuperApp"
appauthor = "Acme"
appdata_dirpath = user_data_dir(appname, appauthor)
# 'C:\\Users\\me\\AppData\\Local\\Acme\\SuperApp' on Windows
# '/Users/me/Library/Application Support/SuperApp' on macOS
# '/home/me/.local/share/SuperApp' on Linux
import os
os.makedirs(appdata_dirpath, exist_ok=True)
Then inside that directory you can write a file, with many different options for the file format:
Key-Value Strings? Consider an INI file, which you can read/write using the built-in configparser.ConfigParser module.
Nested Dicts, Lists, Strings, and Integers? Consider JSON, which you can read/write using the built-in json module.
Have more advanced needs? Such as storing a large number of preference items, or a need to store binary data? Consider a sqlite database, using the built-in sqlite3 module.
Although it is possible to use the pickle module I don't recommend it because the config files are not human-readable.
Developing a project of mine I realize I have a need for some level of persistence across sessions, for example when a user executes the application, changes some preferences and then closes the app. The next time the user executes the app, be it after a reboot or 15 minutes, I would like to be able to retain the preferences that had been changed.
My question relates to this persistence. Whether programming an application using the win32 API or the MFC Framework .. or using the newer tools for higher level languages such as wxPython or wxRuby, how does one maintain the type of persistence I refer to? Is it done as a temporary file written to the disk? Is it saved into some registry setting? Is there some other layer it is stored in that I am unaware of?
I would advice to do it in two steps.
First step is to save your prefs. as
string, for that you can
a)
Use any xml lib or output xml by
hand to output string and read
similarly from string
b) Just use pickle module to dump your prefs object as a string
c) Somehow generate a string from prefs which you can read back as prefs e.g. use yaml, config , JSON etc actually JSON is a good option when simplejson makes it so easy.
Once you have your methods to convert to and from string are ready, you just need to store it somewhere where it is persisted and you can read back next time, for that you can
a) Use wx.Config which save to registry in windows and to other places depending on platform so you don't have to worry where it saves, you can just read back values in platform independent way. But if you wish you can just use wx.Config for directly saving reading prefs.
b) Directly save prefs. string to a file in a folder assigned by OS to your app e.g. app data folder in windows.
Benefit of saving to a string and than using wx.Config to save it, is that you can easily change where data is saved in future e.g. in future if there is a need to upload prefs. you can just upload prefs. string.
There are different methods to do this that have evolved over the years.
These methods include (but not limited to):
Registry entries.
INI files.
XML Files
Simple binary/text files
Databases
Nowadays, most people do this kind of thing with XML files residing in the user specific AppData folders. It is your choice how you do it. For example, for simple things, databases can be overkill and for huge persisted objects, registry would not be appropriate. You have to see what you are doing and do it accordingly.
Here is a very good discussion on this topic