I'd like to save my script's data to disk to load next time the script runs. For simplicity, is it a good idea to use os.path.expanduser('~') and save a directory named ".myscript_data" there? It would only need to be read by the script, and to avoid clutter for the user, I'd like it to be hidden. Is it acceptable practice to place hidden ".files" on the users computer?
On Windows, use a subfolder of os.environ['APPDATA']; on Linux, a dot-folder under $HOME is typical.
You can also consider putting your files in a subdirectory of ~/.config*, which is a sort of emerging standard for configuration file placement. See also the XDG basefiles spec.
Not entirely related, but interesting: origin of dotfiles
*(edit) More accurately, os.environ.get('XDG_CONFIG_HOME', os.path.expanduser('~/.config')) as per the XDG spec.
Yes, it is. (I'm assuming you're on linux, right?)
See also this.
Yes, this is standard practice on most Unix systems. For transparency, it's often a good idea to print an informative message like Creating directory .dir to store script state the first time you create the storage location.
If you are expecting to store significant amounts of data, it's a good idea to confirm the location with the user.
This is also the standard place for any additional configuration files for your application.
On Linux, I suggest a file or directory (no dotfile) in os.environ['XDG_CONFIG_HOME'], which is in most cases the directory $HOME/.config. A dotfile in $HOME, however, is also often being used.
Related
I know it's possible to choose which Python version to check e.g. in vim.rc as per this SO answer, but can I do it per project? Using e.g. virtualenv or a configuration file?
It all depends on how you define "this is a project".
Several of us have been providing local vimrc plugins where a project definition is "all the files in this directory and its subdirectories". See this answer for more details and alternatives solutions on the subject: Vim: apply settings on files in directory
Note that lately I've been working on a different (and more lightweight) way (in most cases) to specify what a project is: https://github.com/LucHermitte/lh-vim-lib/blob/master/doc/Project.md (This is still highly experimental).
Reading the answer you've linked to... It only speaks about a global variable that permits to tune the behaviour of the plugin. If there was no other way but tuning this global option, you'd have needed to reset this global variable unconditionally in the local vimrc, or on a BufEnter autocommand. Fortunately, syntastic is project aware thanks to buffer local variables -- #lcd047 corrected me on this topic. This means that instead of resetting a global variable, you could instead set a buffer local variable depending on the current directory (or any other heuristic you could define in an autocommand -- without these plugins, you could simply listen for BufNew and BufRead event, but this won't support migration among machines, directories, etc).
Note that my local-vimrc plugin sources the current local vimrc configuration file every time we enter a buffer matching that configuration file. This means that if you don't add an anti-reinclusion guard, b:syntastic_python_python_exec would be reset every time you enter a buffer for which it has been defined. It shouldn't be that problematic here. Note also that I don't know how alternative plugins proceed.
I made a script in python that renames all files and folders(does not recurse) in "." directory: the directory in which file is kept. It happened that I ran the script in a directory which contained no files and only one directory let's say imp with path .\imp. While program was renaming it, the electricity went off and the job was interrupted (sorry did't had UPS).
Now as the name suggests, assume imp contains important data. The renaming process also took quite good time ( compared to others ) before electricity went off even when all it was renaming was one folder. After this endeavour is some data corrupted, lost or anything?
Just make this more useful: what happens os.rename is forced to stop when it is doing its job? How is the effect different for files and folders?
Details
Python Version - 2.7.10
Operating System - Windows 10 Pro
You are using Windows, which means you are (probably) on NTFS. NTFS is a modern, journaling file system. It should not corrupt or lose any data, though it's possible that only some of the changes that constitute a rename have been applied (for instance, the filename might change without updating the modification time, or vice-versa). It is also possible that none of those changes have been applied.
Note the word "should" is not the same as "will." NTFS should not lose data in this fashion, and if it does, it's a bug. But because all software has bugs, it is important to keep backups of files you care about.
By default, pip logs errors into "~/.pip/pip.log". Pip has an option to change the log path, and I'd like to put the log file somewhere besides ~/.pip so as not to clutter up my home directory. Where should I put it and be XDG base dir compatible?
Right now I'm considering one of these:
$XDG_DATA_HOME (typically $HOME/.local/share)
$XDG_CACHE_HOME (typically $HOME/.cache)
This is, for the moment, unclear.
Different software seem to handle this in different ways (imsettings puts it in $XDG_CACHE_HOME,
profanity in $XDG_DATA_HOME).
Debian, however, has a proposal which I can get behind (emphasis mine):
This is a recurring request/complaint (see this or this) on the xdg-freedesktop mailing list to introduce another directory for state information that does not belong in any of the existing categories (see also home-dir.proposal. Examples for this information are:
history files of shells, repls, anything that uses libreadline
logfiles
state of application windows on exit
recently opened files
last time application was run
emacs: bookmarks, ido last directories, backups, auto-save files, auto-save-list
The above example information is not essential data. However it should still persist on reboots of the system unlike cache data that a user might consider putting in a TMPFS. On the other hand the data is rather volatile and does not make sense to be checked into a VCS. The files are also not the data files that an application works on.
A default folder for a future STATE category might be: $HOME/.local/state
This would effectively introduce another environment variable since $XDG_DATA_HOME usually points to $HOME/.local/share and this hypothetical environment variable ($XDG_STATE_HOME?) would point to $HOME/.local/state
If you really want to adhere to the current standard I would place my log files in $XDG_CACHE_HOME since log files aren't required to run the program.
What I want to do:
I got two directories. Each one contains about 90.000 xml and bak files.
I need the xml files to sync at both folders when a file changes (of course the newer one should be copied).
The problem is:
Because of the huge amount of files and the fact that one of the directories is a network share I can't just loop though the directory and compare os.path.getmtime(file) values.
Even watchdog and PyQt don't work (tried the solutions from here and here).
The question:
Is there any other way to get a file changed event (on windows systems) which works for those configuration without looping though all those files?
So I finally found the solution:
I changed some of my network share settings and used the FileSystemWatcher
To prevent files getting synced on syncing i use a md5 filehash.
The code i use can be found at pastebin (It's a quick and dirty code and just the parts of it mentioned in the question here).
The code in...
https://stackoverflow.com/a/12345282/976427
Seems to work for me when passed a network share.
I'm risking giving an answer which is way off here (you didn't specify requirement regarding speed, etc) but... Dropbox would do exactly that for you for free, and would required writing no code at all.
Of course it might not suit your needs if you required real-time syncing, or if you want to avoid "sharing" your files with a third party (although you can encrypt them first).
Can you use the second option on this page?
http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html
From mentioning watchdog, I assume you are running under Linux. For the local machine inotify can help, but for the network share you are out of luck.
Mercurial's inotify extension http://hgbook.red-bean.com/read/adding-functionality-with-extensions.html has the same limitation.
In a similar situation (10K+ files) I have used a cloned mercurial repository with inotify on both the server and the local machine. They automatically commit and notified each other of changes. It had a slight delay (no problem in my case) but as a benifit had a full history
of changes and easily resynced after one of the systems had been down.
I need to modify a text file at runtime but restore its original state later (even if the computer crash).
My program runs in regular sessions. Once a session ended, the original state of that file can be changed, but the original state won't change at runtime.
There are several instances of this text file with the same name in several directories. My program runs in each directory (but not in parallel), but depending on the directory content's it does different things. The order of choosing a working directory like this is completely arbitrary.
Since the file's name is the same in each directory, it seems a good idea to store the backed up file in slightly different places (ie. the parent directory name could be appended to the backup target path).
What I do now is backup and restore the file with a self-written class, and also check at startup if the previous backup for the current directory was properly restored.
But my implementation needs serious refactoring, and now I'm interested if there are libraries already implemented for this kind of task.
edit
version control seems like a good idea, but actually it's a bit overkill since it requires network connection and often a server. Other VCS need clients to be installed. I would be happier with a pure-python solution, but at least it should be cross-platform, portable and small enough (<10mb for example).
Why not just do what every unix , mac , window file has done for years -- create a lockfile/working file concept.
When a file is selected for edit:
Check to see if there is an active lock or a crashed backup.
If the file is locked or crashed, give a "recover" option
Otherwise, begin editing the file...
The editing tends to do one or more of a few things:
Copy the original file into a ".%(filename)s.backup"
Create a ".%(filename)s.lock" to prevent others from working on it
When editing is achieved, the lock goes away and the .backup is removed
Sometimes things are slightly reversed, and the original stays in place while a .backup is the active edit; on success the .backup replaces the original
If you crash vi or some other text programs on a linux box, you'll see these files created . note that they usually have a dot(.) prefix so they're normally hidden on the command line. Word/Powerpoint/etc all do similar things.
Implement Version control ... like svn (see pysvn) it should be fast as long as the repo is on the same server... and allows rollbacks to any version of the file... maybe overkill but that will make everything reversible
http://pysvn.tigris.org/docs/pysvn_prog_guide.html
You dont need a server ... you can have local version control and it should be fine...
Git, Subversion or Mercurial is your friend.