I have an interesting problem at hand. As someone who's a beginner when it comes to working with data at even a morderate scale, I'd love some tips from the veterans here.
I have around 6000 Json.gz files totalling around 5GB compressed and 20GB uncompressed.
I'm opening each file and reading them line by line using the gzip module; then using json.loads() loading each line and parsing the complicated JSON structure. Then I'm inserting lines from each file into a Pytable all at once before iterating to the next file.
All this is taking me around 3 hours. Bulk inserting into the Pytable didn't really help the speed at all. Much of the time is gone getting values from the parsed JSON line since they have a truly horrible structure. Some are straightforward like 'attrname':attrvalue, but some are complicated and time consuming structures like:
'attrarray':[{'name':abc, 'value':12},{'value':12},{'name':xyz, 'value':12}...]
...where I need to pick up the value of all those objects in the attr array which have some corresponding name, and ignore those that don't. So I need to iterate through the list and inspect each JSON object inside. (I'd be glad if you can point out any quicker clever way, if it exists)
So I suppose the actual parsing part of it doesn't have much scope of speedup. Where I think their might be scope of speedup is the actual reading the file part.
So I ran a few tests (I don't have the numbers with me right now) and even after removing the parsing part of my program; simply going through the files line by line itself was taking a considerable amount of time.
So I ask: Is there any part of this problem that you think I might be doing suboptimally?
for filename in filenamelist:
f = gzip.open(filename):
toInsert=[]
for line in f:
parsedline = json.loads(line)
attr1 = parsedline['attr1']
attr2 = parsedline['attr2']
.
.
.
attr10 = parsedline['attr10']
arr = parsedline['attrarray']
for el in arr:
try:
if el['name'] == 'abc':
attrABC = el['value']
elif el['name'] == 'xyz':
attrXYZ = el['value']
.
.
.
except KeyError:
pass
toInsert.append([attr1,attr2,...,attr10,attrABC,attrXYZ...])
table.append(toInsert)
One clear piece of "low-hanging fruit"
If you're going to be accessing the same compressed files over and over (it's not especially clear from your description whether this is a one-time operation), then you should decompress them once rather than decompressing them on-the-fly each time you read them.
Decompression is a CPU-intensive operation, and Python's gzip module is not that fast compared to C utilities like zcat/gunzip.
Likely the fastest approach is to gunzip all these files, save the results somewhere, and then read from the uncompressed files in your script.
Other issues
The rest of this is not really an answer, but it's too long for a comment. In order to make this faster, you need to think about a few other questions:
What are you trying to do with all this data?
Do you really need to load all of it at once?
If you can segment the data into smaller pieces, then you can reduce the latency of the program if not the overall time required. For example, you might know that you only need a few specific lines from specific files for whatever analysis you're trying to do... great! Only load those specific lines.
If you do need to access the data in arbitrary and unpredictable ways, then you should load it into another system (RDBMS?) which stores it in a format that is more amenable to the kinds of analyses you're doing with it.
If the last bullet point is true, one option is to load each JSON "document" into a PostgreSQL 9.3 database (the JSON support is awesome and fast) and then do your further analyses from there. Hopefully you can extract meaningful keys from the JSON documents as you load them.
Imagine you have a filesystem tree:
root/AA/aadata
root/AA/aafile
root/AA/aatext
root/AB/abinput
root/AB/aboutput
root/AC/acinput
...
Totally around 10 million files. Each file is around 10kb in size. They are mostly like a key-value storage, separated by folders just in order to improve speed (FS would die if I put 5 million files in a single folder).
Now we need to:
archive this tree into a single big file (it must be relatively fast but have a good compression ratio as well - thus, 7z is too slow)
seek the result big file very quickly - so, when I need to get the content of "root/AB/aboutput", I should be able to read it very quickly.
I won't use Redis because in the future the amount of files might increase and there would be no space for them in RAM. But on the other side, I can use SSD-powered servers to the data access will be relatively fast (compared to HDD).
Also it should not be any exotic file system, such as squashfs or similar file systems. It should work in an ordinary EXT3 or EXT4 or NTFS.
I also thought about storing the files as a simple zlib-compressed strings, remembering the file offset for each string, and then creating something like a map which will be in RAM. Each time I need a file, I will read the content offset form the map, and then - using the offsets - from the actual file. But maybe there is something easier or already done?
Assumptions (from information in contents). You might use the following strategy: use two files (one for the "index", the second for your actual content. For simplicity, make the second file a set of "blocks" (say each of 8196). To process your files, read them into an programmatic structure for file name (key) along with the block number of the second file where the content begins. Write the file content to the second file (compressing if storage space is at a premium). Save the index information.
To retrieve, read the index file into programmattic storage and store as a binary tree. If search times are a problem, you might hash the keys and store values into a table and handle collision with a simple add to next available slot. To retrieve content, get block number (and length) from your index find; read the content from the second file (expand if you compressed).
I have a 100Mb file with roughly 10million lines that I need to parse into a dictionary every time I run my code. This process is incredibly slow, and I am hunting for ways to speed it up. One thought that came to mind is to parse the file once and then use pickle to save it to disk. I'm not sure this would result in a speed up.
Any suggestions appreciated.
EDIT:
After doing some testing, I am worried that the slow down happens when I create the dictionary. Pickling does seem significantly faster, though I wouldn't mind doing better.
Lalit
MessagePack has in my experience been much faster for dumping/loading data in python then cPickle, even when using the highest protocol.
However, if you have a dictionary with 10 million entries in it, you might want to check that you're not hitting the upper limit of your computer's memory. The process will happen much slower if you run out of memory and have to use swap.
Depending on how you use the data, you could
divide it into many smaller files and load only what's needed
create an index into the file and lazy-load
store it to a database and then query the database
Can you give us a better idea just what your data looks like (structure)?
How you are using the data? Do you actually use every row on every execution? If you only use a subset on each run, could the data be pre-sorted?
I am working on a project that basically monitors a set remote directories (FTP, networked paths, and another), if the file is considered new and meets criteria we download it and process it. However i am stuck on what the best way is to keep track of the files we already downloaded. I don't want to download any duplicate files, so i need to keep track of what is already downloaded.
Orignally i was storing it as a tree:
server->directory->file_name
When the service shuts down it writes it to a file, and rereads it back when it starts up. However given that when there is around 20,000 or so files in the tree stuff starts to slow down alot.
Is there a better way to do this?
EDIT
The lookup times start to slowdown alot, my basic implementation is a dict of a dict. The storing stuff on the disk is fine, its more or less just the lookup time. I know i can optimize the tree and partition it. However that seems excessive for such a small project i was hoping python would have something like that.
I would create a set of tuples, then pickle it to a file. The tuples would be (server, directory, file_name), or even just (server, full_file_name_including_directory). There's no need for a multiple-level data structure. The tuples will hash into the set, and give you a O(1) lookup.
You mention "stuff starts to slow down alot," but you don't say if it's reading and writing time, or lookup times that are slowing down. If your lookup times are slowing down, you may be paging. Is your data structure approaching a significant fraction of your physical memory?
One way to get back some memory is to intern() the server names. This way, each server name will be stored only once in memory.
An interesting alternative is to use a Bloom filter. This will let you use far less memory, but will occasionally download a file that you didn't have to. This might be a reasonable trade-off, depending on why you didn't want to download the file twice.
Conclusion:
It seems that HDF5 is the way to go for my purposes. Basically "HDF5 is a data model, library, and file format for storing and managing data." and is designed to handle incredible amounts of data. It has a Python module called python-tables. (The link is in the answer below)
HDF5 does the job done 1000% better in saving tons and tons of data. Reading/modifying the data from 200 million rows is a pain though, so that's the next problem to tackle.
I am building directory tree which has tons of subdirectories and files. There are about 10 million files spread around a hundred thousand directories. Each file is under 32 subdirectories.
I have a python script that builds this filesystem and reads & writes those files. The problem is that when I reach more than a million files, the read and write methods become extremely slow.
Here's the function I have that reads the contents of a file (the file contains an integer string), adds a certain number to it, then writes it back to the original file.
def addInFile(path, scoreToAdd):
num = scoreToAdd
try:
shutil.copyfile(path, '/tmp/tmp.txt')
fp = open('/tmp/tmp.txt', 'r')
num += int(fp.readlines()[0])
fp.close()
except:
pass
fp = open('/tmp/tmp.txt', 'w')
fp.write(str(num))
fp.close()
shutil.copyfile('/tmp/tmp.txt', path)
Relational databases seem too slow for accessing these data, so I opted for a filesystem approach.
I previously tried performing linux console commands for these but it was way slower.
I copy the file to a temporary file first then access/modify it then copy it back because i found this was faster than directly accessing the file.
Putting all the files into 1 directory (in reiserfs format) caused too much slowdown when accessing the files.
I think the cause of the slowdown is because there're tons of files. Performing this function 1000 times clocked at less than a second.. but now it's reaching 1 minute.
How do you suggest I fix this? Do I change my directory tree structure?
All I need is to quickly access each file in this very huge pool of files*
I know this isn't a direct answer to your question, but it is a direct solution to your problem.
You need to research using something like HDF5. It is designed for just the type of hierarchical data with millions of individual data points.
You are REALLY in luck because there are awesome Python bindings for HDF5 called pytables.
I have used it in a very similar way and had tremendous success.
Two suggestions:
First, a structure that involves 32-deep nesting of subdirectories is inherently flawed. Assuming that you really have "about 10 million files", one level of subdirectories should absolutely be enough (assuming you use a modern filesystem).
Second: You say you have "about 10 million files" and that each file "contains an integer string". Assuming that those are 32-bit integers and you store them directly instead of as strings, that amounts to a total dataset size of 40MiB (10M files * 4 bytes per file). Assuming that each filename is 32 bytes long, add another 320MiB for "keys" to this data.
So you'll be able to easily fit the whole dataset into memory. I suggest doing just that, and operate over the data held in main memory. And unless there is any reason you need an elaborate directory structure, I further suggest storing the data in a single file.
I would suggest you rethink your approach, using lots of extremely small files is bound to give you serious performance problems. Depending on the purpose of your program some kind of database could be far more efficient.
If you're doing lots of I/O you can also just throw more hardware at the problem and use SSDs or keep all the data in RAM (explicitly or by caching). With harddrives alone you have no chance of achiving good performance in this scenario.
I've never used it, but e.g. Redis is a persistent key-value store that is supposed to be very fast. If your data fits this model I would definately try this or something similar. You'll find some performance data in this article, which should give you an idea what speeds you can achieve.
The disk is limited by amount of bytes it can read/write per second and also by amount of operations it can perform in second.
While your small files are cached, operations are significantly faster than with uncached files.
It looks like you are hitting both issues,
doing too many i/o operations
running out of cache
I'd suggest revisiting the structure you are using, and using less larger files. Keep in minf (as a rule of thumb) than I/O operation less than 128K runtime cost is more or less equal to I/O of 1byte!
Resolving all of those subdirectories takes time. You're over-taxing the file-system.
Maybe instead of using the directory tree, you could instead encode the path information into the file name, so instead of creating a file with a path like this:
/parent/00/01/02/03/04/05/06/07
/08/09/0A/0B/0C/0D/0E/0F
/10/11/12/13/14/15/16/17
/18/19/1A/1B/1C/1D/1E/1F.txt
...you could create a file with a path like this:
/parent/00_01_02_03_04_05_06_07_
08_09_0A_0B_0C_0D_0E_0F_
10_11_12_13_14_15_16_17_
18_19_1A_1B_1C_1D_1E_1F.txt
...of course, you'll still have a problem, because now all of your ten million files will be in a single directory, and in my experience (NTFS), a directory with more than a few thousand files in it still over-taxes the file-system.
You could come up with a hybrid approach:
/parent/00_01_02_03/04_05_06_07
/08_09_0A_0B/0C_0D_0E_0F
/10_11_12_13/14_15_16_17
/18_19_1A_1B/1C_1D_1E_1F.txt
But that will still give you problems if you exhaustively create all those directories. Even though most of those directories are "empty" (in that they don't contain any files), the operating system still has to create an INODE record for each directory, and that takes space on disk.
Instead, you should only create a directory when you have a file to put into it. Also, if you delete all the files in any given directory, then delete the empty directory.
How many levels deep should you create the directory hierarchy? In my little example, I transformed your 32-level hierarchy into an 8-level hierarchy, but after doing some testing, you might decide on a slightly different mapping. It really depends on your data, and how evenly those paths are distributed through the combinatorial solution space. You need to optimize a solution with two constraints:
1) Minimize the number of directories you create, knowing that each directory becomes an INODE in the underlying file-system, and creating too many of them will overwhelm the file system.
2) Minimize the number of files in each directory, knowing that having too many files per directory (in my experience, more than 1000) overwhelms the file-system.
There's one other consideration to keep in mind: Storage space on disks is addressed and allocated using "blocks". If you create a file smaller than the minimum block size, it nevertheless consumes the whole block, wasting disk space. In NTFS, those blocks are defined by their "cluster size" (which is partially determined by the overall size of the volume), and usually defaults to 4kB:
http://support.microsoft.com/kb/140365
So if you create a file with only one byte of data, it will still consume 4kB worth of disk space, wasting 4095 bytes.
In your example, you said you had about 10 million files, with about 1gB of data. If that's true, then each of your files is only about 100 bytes long. With a cluster size of 4096, you have about a 98% space-wasted ratio.
If at all possible, try to consolidate some of those files. I don't know what kind of data they contain, but if it's a text format, you might try doing something like this:
[id:01_23_45_67_89_AB_CD_EF]
lorem ipsum dolor sit amet consectetur adipiscing elit
[id:fe_dc_ba_98_76_54_32_10]
ut non lorem quis quam malesuada lacinia
[id:02_46_81_35_79_AC_DF_BE]
nulla semper nunc id ligula eleifend pulvinar
...and so on and so forth. It might look like you're wasting space with all those verbose headers, but as far as the disk is concerned, this is a much more space-efficient strategy than having separate files for all those little snippets. This little example used exactly 230 bytes (including newlines) for three records, so you might try to put about sixteen records into each file (remembering that it's much better to have slightly less than 4096 bytes-per-file than to have slightly more than 4096, wasting a whole extra disk block).
Anyhow, good luck!
You're copying a file, opening it to read, closing it, then reopening it for writing, then recopying it back. It would be faster to do it in one go.
EDIT: the previous version has a bug when the number of digits become less than the current number of digits (e.g. if you're subtracting or adding by negative number); this version fixes it, timing result is barely unaffected
def addInFile(path, scoreToAdd):
try:
fp = open(path, 'r+')
except IOError as e:
print e
else:
num = str(scoreToAdd + int(fp.read()))
fp.seek(0)
fp.write(num)
fp.truncate(len(num))
finally:
fp.close()
alternatively, if you want to avoid file loss and writes to cache, you should do the copying and the summing in one go, then do a an overwrite-dance in another step:
def addInFile(path, scoreToAdd):
try:
orig = open(path, 'r')
tmp = open('/home/lieryan/junks/tmp.txt', 'w')
except IOError as e:
print e
else:
num = int(orig.read())
tmp.write(str(scoreToAdd + num))
finally:
orig.close()
tmp.close()
try:
# make sure /tmp/ and path is in the same partition
# otherwise the fast shutil.move become a slow shutil.copy
shutil.move(path, '/home/lieryan/junks/backup.txt')
shutil.move('/home/lieryan/junks/tmp.txt', path)
os.remove('/home/lieryan/junks/backup.txt')
except (IOError, shutil.Error) as e:
print e
also, don't use bare excepts.
Alternatively, how about grouping all the 256 files in the lowest leaf into one bigger file? Then you can read multiple numbers in one go, in one cache. And if you used a fixed width file, then you can quickly use seek() to get to any entry in the file in O(1).
Some timings, writing 1000 times on the same file:
Your original approach: 1.87690401077
My first approach (open with rw+): 0.0926730632782
My second approach, copy to the same partition: 0.464048147202
(all functions untested on their error handling path)
If you under linux and got large memory(64GB+), try tmpfs, its truly works like mounted disk and you do not need to change your code or buy another SSD.