Develop Reference

Rewrite a file in-place with Python

It might depend on each OS and also be hardware-dependent, but is there a way in Python to ask a write operation on a file to happen "in-place", i.e. at the same place of the original file, i.e. if possible on the same sectors on disk?
Example: let's say sensitivedata.raw, a 4KB file has to be crypted:
with open('sensitivedata.raw', 'r+') as f: # read write mode
s = f.read()
cipher = encryption_function(s) # same exact length as input s
f.seek(0)
f.write(cipher) # how to ask this write operation to overwrite the original bytes?
Example 2: replace a file by null-byte content of the same size, to avoid an undelete tool to recover it (of course, to do it properly we need several passes, with random data and not only null-bytes, but here it's just to give an idea)
with open('sensitivedata.raw', 'r+') as f:
s = f.read()
f.seek(0)
f.write(len(s) * '\x00') # totally inefficient but just to get the idea
os.remove('sensitivedata.raw')
PS: if it really depends a lot on OS, I'm primarily interested in the Windows case
Side-quesion: if it's not possible in the case of a SSD, does this mean that if you once in your life wrote sensitive data as plaintext on a SSD (example: a password in plaintext, a crypto private key, or anything else, etc.), then there is no way to be sure that this data is really erased? i.e. the only solution is to 100% wipe the disk and fill it many passes with random bytes? Is that correct?

That's an impossible requirement to impose. While on most spinning disk drives, this will happen automatically (there's no reason to write the new data elsewhere when it could just overwrite the existing data directly), SSDs can't do this (when they claim to do so, they're lying to the OS).
SSDs can't rewrite blocks; they can only erase a block, or write to an empty block. The implementation of a "rewrite" is to write to a new block (reading from the original block to fill out the block if there isn't enough new data), then (eventually, cause it's relatively expensive) erase the old block to make it available for a future write.
Update addressing side-question: The only truly secure solution is to run your drive through a woodchipper, then crush the remains with a millstone. :-) Really, in most cases, the window of vulnerability on an SSD should be relatively short; erasing sectors is expensive, so even SSDs that don't honor TRIM typically do it in the background to ensure future (cheap) write operations aren't held up by (expensive) erase operations. This isn't really so bad when you think about it; sure, the data is visible for a period of time after you logically erased it. But it was visible for a period of time before you erased it too, so all this is doing is extending the window of vulnerability by (seconds, minutes, hours, days, depending on the drive); the mistake was in storing sensitive data to permanent storage in the first place; even with extreme (woodchipper+millstone) solutions, someone else could have snuck in and copied the data before you thought to encrypt/destroy it.

Compressing strings and appending to file on the fly

I currently have the following csv writer class:
class csvwriter():
writer = None
writehandler = None
#classmethod
def open(cls,file):
cls.writehandler = open(file,'wb')
cls.writer = csv.writer(cls.writehandler, delimiter=',',quotechar='"', quoting=csv.QUOTE_NONNUMERIC)
#classmethod
def write(cls,arr):
cls.writer.writerow(arr)
#classmethod
def close(cls):
cls.writehandler.close()
which can generate proper csv files without ever having to store the full array in memory at a single time.
However, the files created through use of this code can be quite large, so I'm looking to compress them, rather than writing them uncompressed. (In order to save on disk usage). I can't effectively store the file in memory either, as I'm expecting files of well over 20gb to be a regular occurence.
The recipients of the resulting files are generally not sysadmins of their PCs, nor do they all use linux, so I'm constrained in the types of algorithms I'm able to use for this task. Preferably, the solution would use a compression scheme that's natively readable (no executables required) in Windows, OSX and any linux distribution.
I've found gzip provides a very handy interface in Python, but reading gzipped files in windows seems like quite a hassle.. Ideally I'd put them in a zip archive, but zip archive don't allow you to append data to files already present in the archive, which then forces me to store the whole file in memory, or write the data away to several smaller files that I would be able to fit in memory.
My question: Is there a solution that would benefit from the best of both worlds? Widespread availability of tools to read the target format on the end-user's machine, and also the ability to append, rather than write the whole file in one go?
Thanks in advance for your consideration!

gzlog may provide the functionality you're looking for. It efficiently appends short strings to a gzip file, intended for applications where short messages are appended to a long log.

Optimization tips for reading/parsing large number of JSON.gz files

I have an interesting problem at hand. As someone who's a beginner when it comes to working with data at even a morderate scale, I'd love some tips from the veterans here.
I have around 6000 Json.gz files totalling around 5GB compressed and 20GB uncompressed.
I'm opening each file and reading them line by line using the gzip module; then using json.loads() loading each line and parsing the complicated JSON structure. Then I'm inserting lines from each file into a Pytable all at once before iterating to the next file.
All this is taking me around 3 hours. Bulk inserting into the Pytable didn't really help the speed at all. Much of the time is gone getting values from the parsed JSON line since they have a truly horrible structure. Some are straightforward like 'attrname':attrvalue, but some are complicated and time consuming structures like:
'attrarray':[{'name':abc, 'value':12},{'value':12},{'name':xyz, 'value':12}...]
...where I need to pick up the value of all those objects in the attr array which have some corresponding name, and ignore those that don't. So I need to iterate through the list and inspect each JSON object inside. (I'd be glad if you can point out any quicker clever way, if it exists)
So I suppose the actual parsing part of it doesn't have much scope of speedup. Where I think their might be scope of speedup is the actual reading the file part.
So I ran a few tests (I don't have the numbers with me right now) and even after removing the parsing part of my program; simply going through the files line by line itself was taking a considerable amount of time.
So I ask: Is there any part of this problem that you think I might be doing suboptimally?
for filename in filenamelist:
f = gzip.open(filename):
toInsert=[]
for line in f:
parsedline = json.loads(line)
attr1 = parsedline['attr1']
attr2 = parsedline['attr2']
.
.
.
attr10 = parsedline['attr10']
arr = parsedline['attrarray']
for el in arr:
try:
if el['name'] == 'abc':
attrABC = el['value']
elif el['name'] == 'xyz':
attrXYZ = el['value']
.
.
.
except KeyError:
pass
toInsert.append([attr1,attr2,...,attr10,attrABC,attrXYZ...])
table.append(toInsert)

One clear piece of "low-hanging fruit"
If you're going to be accessing the same compressed files over and over (it's not especially clear from your description whether this is a one-time operation), then you should decompress them once rather than decompressing them on-the-fly each time you read them.
Decompression is a CPU-intensive operation, and Python's gzip module is not that fast compared to C utilities like zcat/gunzip.
Likely the fastest approach is to gunzip all these files, save the results somewhere, and then read from the uncompressed files in your script.
Other issues
The rest of this is not really an answer, but it's too long for a comment. In order to make this faster, you need to think about a few other questions:
What are you trying to do with all this data?
Do you really need to load all of it at once?
If you can segment the data into smaller pieces, then you can reduce the latency of the program if not the overall time required. For example, you might know that you only need a few specific lines from specific files for whatever analysis you're trying to do... great! Only load those specific lines.
If you do need to access the data in arbitrary and unpredictable ways, then you should load it into another system (RDBMS?) which stores it in a format that is more amenable to the kinds of analyses you're doing with it.
If the last bullet point is true, one option is to load each JSON "document" into a PostgreSQL 9.3 database (the JSON support is awesome and fast) and then do your further analyses from there. Hopefully you can extract meaningful keys from the JSON documents as you load them.

Most efficient way to store data on drive

baseline - I have CSV data with 10,000 entries. I save this as 1 csv file and load it all at once.
alternative - I have CSV data with 10,000 entries. I save this as 10,000 CSV files and load it individually.
Approximately how much more inefficient is this computationally. I'm not hugely interested in memory concerns. The purpose of the alternative method is because I frequently need to access subsets of the data and don't want to have to read the entire array.
I'm using python.
Edit: I can other file formats if needed.
Edit1: SQLite wins. Amazingly easy and efficient compared to what I was doing before.

SQLite is ideal solution for your application.
Simply import your CSV file into SQLite database table (it is going to be single file), then add indexes as necessary.
To access your data, use python sqlite3 library. You can use this tutorial on how to use it.
Compared to many other solutions, SQLite will be the fastest way to select partial data sets locally - certainly much, much faster than access 10000 files. Also read this answer which explains why SQLite is so good.

I would write all the lines to one file. For 10,000 lines it's probably not worthwhile, but you can pad all the lines to the same length - say 1000 bytes.
Then it's easy to seek to the nth line, just multiply n by the line length

10,000 files is going to be slower to load and access than one file, if only because the files' data will likely be fragmented around your disk drive, so accessing it will require a much larger number of seeks than would accessing the contents of a single file, which will generally be stored as sequentially as possible. Seek times are a big slowdown on spinning media, since your program has to wait while the drive heads are physically repositioned, which can take milliseconds. (slow seeks times aren't an issue for SSDs, but even then there will still be the overhead of 10,000 file's worth of metadata for the operating system to deal with). Also with a single file, the OS can speed things up for you by doing read-ahead buffering (as it can reasonably assume that if you read one part of the file, you will likely want to read the next part soon). With multiple files, the OS can't do that.
My suggestion (if you don't want to go the SQLite route) would be to use a single CSV file, and (if possible) pad all of the lines of your CSV file out with spaces so that they all have the same length. For example, say you make sure when writing out the CSV file to make all lines in the file exactly 80 bytes long. Then reading the (n)th line of the file becomes relatively fast and easy:
myFileObject.seek(n*80)
theLine = myFileObject.read(80)

Python: slow read & write for millions of small files

Conclusion:
It seems that HDF5 is the way to go for my purposes. Basically "HDF5 is a data model, library, and file format for storing and managing data." and is designed to handle incredible amounts of data. It has a Python module called python-tables. (The link is in the answer below)
HDF5 does the job done 1000% better in saving tons and tons of data. Reading/modifying the data from 200 million rows is a pain though, so that's the next problem to tackle.
I am building directory tree which has tons of subdirectories and files. There are about 10 million files spread around a hundred thousand directories. Each file is under 32 subdirectories.
I have a python script that builds this filesystem and reads & writes those files. The problem is that when I reach more than a million files, the read and write methods become extremely slow.
Here's the function I have that reads the contents of a file (the file contains an integer string), adds a certain number to it, then writes it back to the original file.
def addInFile(path, scoreToAdd):
num = scoreToAdd
try:
shutil.copyfile(path, '/tmp/tmp.txt')
fp = open('/tmp/tmp.txt', 'r')
num += int(fp.readlines()[0])
fp.close()
except:
pass
fp = open('/tmp/tmp.txt', 'w')
fp.write(str(num))
fp.close()
shutil.copyfile('/tmp/tmp.txt', path)
Relational databases seem too slow for accessing these data, so I opted for a filesystem approach.
I previously tried performing linux console commands for these but it was way slower.
I copy the file to a temporary file first then access/modify it then copy it back because i found this was faster than directly accessing the file.
Putting all the files into 1 directory (in reiserfs format) caused too much slowdown when accessing the files.
I think the cause of the slowdown is because there're tons of files. Performing this function 1000 times clocked at less than a second.. but now it's reaching 1 minute.
How do you suggest I fix this? Do I change my directory tree structure?
All I need is to quickly access each file in this very huge pool of files*

I know this isn't a direct answer to your question, but it is a direct solution to your problem.
You need to research using something like HDF5. It is designed for just the type of hierarchical data with millions of individual data points.
You are REALLY in luck because there are awesome Python bindings for HDF5 called pytables.
I have used it in a very similar way and had tremendous success.

Two suggestions:
First, a structure that involves 32-deep nesting of subdirectories is inherently flawed. Assuming that you really have "about 10 million files", one level of subdirectories should absolutely be enough (assuming you use a modern filesystem).
Second: You say you have "about 10 million files" and that each file "contains an integer string". Assuming that those are 32-bit integers and you store them directly instead of as strings, that amounts to a total dataset size of 40MiB (10M files * 4 bytes per file). Assuming that each filename is 32 bytes long, add another 320MiB for "keys" to this data.
So you'll be able to easily fit the whole dataset into memory. I suggest doing just that, and operate over the data held in main memory. And unless there is any reason you need an elaborate directory structure, I further suggest storing the data in a single file.

I would suggest you rethink your approach, using lots of extremely small files is bound to give you serious performance problems. Depending on the purpose of your program some kind of database could be far more efficient.
If you're doing lots of I/O you can also just throw more hardware at the problem and use SSDs or keep all the data in RAM (explicitly or by caching). With harddrives alone you have no chance of achiving good performance in this scenario.
I've never used it, but e.g. Redis is a persistent key-value store that is supposed to be very fast. If your data fits this model I would definately try this or something similar. You'll find some performance data in this article, which should give you an idea what speeds you can achieve.

The disk is limited by amount of bytes it can read/write per second and also by amount of operations it can perform in second.
While your small files are cached, operations are significantly faster than with uncached files.
It looks like you are hitting both issues,
doing too many i/o operations
running out of cache
I'd suggest revisiting the structure you are using, and using less larger files. Keep in minf (as a rule of thumb) than I/O operation less than 128K runtime cost is more or less equal to I/O of 1byte!

Resolving all of those subdirectories takes time. You're over-taxing the file-system.
Maybe instead of using the directory tree, you could instead encode the path information into the file name, so instead of creating a file with a path like this:
/parent/00/01/02/03/04/05/06/07
/08/09/0A/0B/0C/0D/0E/0F
/10/11/12/13/14/15/16/17
/18/19/1A/1B/1C/1D/1E/1F.txt
...you could create a file with a path like this:
/parent/00_01_02_03_04_05_06_07_
08_09_0A_0B_0C_0D_0E_0F_
10_11_12_13_14_15_16_17_
18_19_1A_1B_1C_1D_1E_1F.txt
...of course, you'll still have a problem, because now all of your ten million files will be in a single directory, and in my experience (NTFS), a directory with more than a few thousand files in it still over-taxes the file-system.
You could come up with a hybrid approach:
/parent/00_01_02_03/04_05_06_07
/08_09_0A_0B/0C_0D_0E_0F
/10_11_12_13/14_15_16_17
/18_19_1A_1B/1C_1D_1E_1F.txt
But that will still give you problems if you exhaustively create all those directories. Even though most of those directories are "empty" (in that they don't contain any files), the operating system still has to create an INODE record for each directory, and that takes space on disk.
Instead, you should only create a directory when you have a file to put into it. Also, if you delete all the files in any given directory, then delete the empty directory.
How many levels deep should you create the directory hierarchy? In my little example, I transformed your 32-level hierarchy into an 8-level hierarchy, but after doing some testing, you might decide on a slightly different mapping. It really depends on your data, and how evenly those paths are distributed through the combinatorial solution space. You need to optimize a solution with two constraints:
1) Minimize the number of directories you create, knowing that each directory becomes an INODE in the underlying file-system, and creating too many of them will overwhelm the file system.
2) Minimize the number of files in each directory, knowing that having too many files per directory (in my experience, more than 1000) overwhelms the file-system.
There's one other consideration to keep in mind: Storage space on disks is addressed and allocated using "blocks". If you create a file smaller than the minimum block size, it nevertheless consumes the whole block, wasting disk space. In NTFS, those blocks are defined by their "cluster size" (which is partially determined by the overall size of the volume), and usually defaults to 4kB:
http://support.microsoft.com/kb/140365
So if you create a file with only one byte of data, it will still consume 4kB worth of disk space, wasting 4095 bytes.
In your example, you said you had about 10 million files, with about 1gB of data. If that's true, then each of your files is only about 100 bytes long. With a cluster size of 4096, you have about a 98% space-wasted ratio.
If at all possible, try to consolidate some of those files. I don't know what kind of data they contain, but if it's a text format, you might try doing something like this:
[id:01_23_45_67_89_AB_CD_EF]
lorem ipsum dolor sit amet consectetur adipiscing elit
[id:fe_dc_ba_98_76_54_32_10]
ut non lorem quis quam malesuada lacinia
[id:02_46_81_35_79_AC_DF_BE]
nulla semper nunc id ligula eleifend pulvinar
...and so on and so forth. It might look like you're wasting space with all those verbose headers, but as far as the disk is concerned, this is a much more space-efficient strategy than having separate files for all those little snippets. This little example used exactly 230 bytes (including newlines) for three records, so you might try to put about sixteen records into each file (remembering that it's much better to have slightly less than 4096 bytes-per-file than to have slightly more than 4096, wasting a whole extra disk block).
Anyhow, good luck!

You're copying a file, opening it to read, closing it, then reopening it for writing, then recopying it back. It would be faster to do it in one go.
EDIT: the previous version has a bug when the number of digits become less than the current number of digits (e.g. if you're subtracting or adding by negative number); this version fixes it, timing result is barely unaffected
def addInFile(path, scoreToAdd):
try:
fp = open(path, 'r+')
except IOError as e:
print e
else:
num = str(scoreToAdd + int(fp.read()))
fp.seek(0)
fp.write(num)
fp.truncate(len(num))
finally:
fp.close()
alternatively, if you want to avoid file loss and writes to cache, you should do the copying and the summing in one go, then do a an overwrite-dance in another step:
def addInFile(path, scoreToAdd):
try:
orig = open(path, 'r')
tmp = open('/home/lieryan/junks/tmp.txt', 'w')
except IOError as e:
print e
else:
num = int(orig.read())
tmp.write(str(scoreToAdd + num))
finally:
orig.close()
tmp.close()
try:
# make sure /tmp/ and path is in the same partition
# otherwise the fast shutil.move become a slow shutil.copy
shutil.move(path, '/home/lieryan/junks/backup.txt')
shutil.move('/home/lieryan/junks/tmp.txt', path)
os.remove('/home/lieryan/junks/backup.txt')
except (IOError, shutil.Error) as e:
print e
also, don't use bare excepts.
Alternatively, how about grouping all the 256 files in the lowest leaf into one bigger file? Then you can read multiple numbers in one go, in one cache. And if you used a fixed width file, then you can quickly use seek() to get to any entry in the file in O(1).
Some timings, writing 1000 times on the same file:
Your original approach: 1.87690401077
My first approach (open with rw+): 0.0926730632782
My second approach, copy to the same partition: 0.464048147202
(all functions untested on their error handling path)

If you under linux and got large memory(64GB+), try tmpfs, its truly works like mounted disk and you do not need to change your code or buy another SSD.