Which file system for enormous number of files in one directory?

[tfboy]

I'm working on creating a Digital Cinema Package for use with some professional projectors.

The creation involves having each frame of video as a single JPEG2000 file. This is then muxed into an MXF stream and compiled into a DCP package.

The issue I'm having at the moment is terrible slowdowns using NTFS on my server. I've already checked the volume's free space and fragmentation, and they're not that bad. Yet copying / moving files is horrendously slow - around one 1MB file per second.

I reckon it's due to having all the files in a single directory. Unfortunately, I can't really split it up because the muxer needs all the files in one directory which composes a video stream.

So what options can you suggest?
- Would symbolic links work? I've never tried this in Windows / NTFS and have limited experience using ext3/4
- Should I switch to another file system? If so, which one supports 200,000+ files in one directory and is usable on Windows and is free? I would use StorNext and it would work but it's hardly an option with the hardware I have (no SAN, metadata controllers, etc).

Thoughts?

[aidanjt]

With Windows, you only *really* have two choices, FAT32 or NTFS, and FAT32 is far less capable and not any faster.

Perhaps you could suggest to the software vendor to add directory traversal support.

[Kumagoro]

I have no idea if it will make any difference but there is also exfat
Would indexing the folder make any difference?

[peterb]

I think Win 7 indexes folders by default, but that is for speed of searching rather than file transfer speed.

[GoNz0]

is this just for moving files about or is the software grinding to a halt processing so many jpegs ?

if its just moving stuff back and forth ready for the muxing use richcopy

[Betty_Swallocks]

You might find Linux handles it better.

[peterb]

You might find Linux handles it better.

There are certainly more filesystems available under a linux kernel, and xfs (or jfs) would probably be better for large numbers of large files. Of those, XFS is probably the better one because although it is slightly slower in tests, it is less processor intensive, which might be important if you are doing image manipulation at the same time.

Of course that is all academic unless it is the storage that is the problem (and you can store on a separate linux machine) or the tools that you are using (or ones with similar functionality) are also available on a linux platform.

Some links you might find useful:

http://www.debian-administration.org/articles/388

(xfs is available for other distros, it isn't debian specific)

http://xfs.org/index.php/XFS_FAQ

[Savas]

What operating systems are you using?

If there's a lot of files in one directory, then maybe something to consider is directory caching, so you won't constantly be retrieving each and every single file information every time it's in the dir. Basically things like offline mode will be your interests.

http://trycatch.be/blogs/roggenk/arc...t-caching.aspx

I guess there's something for linux too.

[scaryjim]

Just a random guess, but I'd assume the issue is that you're asking the program to do a lot of random reads, and you're using a mechnical disk: might it be worth shunting the whole lot onto an SSD and seeing if it speeds up? If you're talking vast numbers of 1MB files then, even if the filesystems thinks they're all in one directory, chances are they're scattered all over the physical disk, and since you'll need to access them in a specific order you could be looking at a lot of worse-case seek times.

Obviously the ideal situation would be having a spare SSD you could test this theory on first, but a small (i.e. 30GB) SSD wouldn't be that expensive an experiment...

EDIT: Ebuyer have a 60GB Vertex Plus for £58...

[tfboy]

The storage server is a Windows Home Server (v1), with some 2TB drives - 5400rpm slow ones!.

I'll quickly explain the workflow as there are a few stages to it: I'm creating a 3D DCP, and that involves having individual left and right eye stereoscopic files in jpeg2000 format, XYZ colour space. I'm using an open source program for the j2k and xyz conversion (OpenDCP). This is cross-platform and I could have it running on Linux. However, it's the storage that's the problem.

1) The single large video file is parsed through Adobe Media Encoder to create individual TIFF stills of each frame of the video. These frames are cropped and resized on the fly by AME. Unfortunately, the TIFF export of AME is uncompressed TIFF only, so you're looking at 8MB per TIFF file. At 24 files per second, then multiply by two to have your left and right sets of files, you can see how you quickly need several terabytes

2) The resulting TIFF is then read by OpenDCP to create a the j2k/xyz version. The actual executable of OpenDCP is run on several workstations so I can have a kind of render farm - I'm moving groups of TIFF files into individual directories to farm out the load, typically 10,000 files per directory.

3) OpenDCP saves the resulting j2k file somewhere else - it's now on a different physical drive, but still inside the same server.

I now realise one of the potential problems is that I was using a disk in the storage pool which probably has an adverse affect on performance.

In terms of random read / write, I suppose it is. I have, at most, 12 simultaneous accesses from remote machines (mapped drives) to read the files, compress them and write back the Jpeg2000 versions. I've now started writing the jp2k ones on a different physical drive (still inside the server), just in case it was a read/write issue.

Since writing the original post, I've realised that there had been massive fragmentation as the result of writing the initial files. It's taken around 6 hours to half defragment! Having defragmented, it's slightly faster, but still slow, so I think it might be more an NTFS limitation than a physical fragmentation problem.

I do have an EXT3 (maybe 4) Windows driver somewhere that works fine. How the performance is though, I don't know. I guess I'd be better with a small Linux NAS storage box to hold the data, as I can't afford to wipe my existing WHS installation. Thanks for the suggestion Peter

Jim, I'd love to use SSDs, however, 2TB+ worth isn't going to be cheap

[GoNz0]

no wonder its slow, less than ideal situation m8.

[tfboy]

LOL Gonzo. Well, I'd love to have a proper DPX post prod film workstation with a zippy SAN behind it, but it's a bit beyond my budget! I suppose I ought to try and source some hardware from work to do it as it's half a work test project (compiling 3D stereoscopic DCPs).

[peterb]

It does sound as if the set up is underpowered! If your budget can stretch to it, something like a Mac Pro with multiple processors (I have seen one with 4 six core processors for example) would make the whole process a lot quicker - but you need to check t hat the software you are using can take optimum advantage of it - but I guess if you are doing s project of that magnitude, you are pretty well up on software capabilities and limitations.

[tfboy]

The processing power of recompressing into jpeg2000 isn't the problem, it's the storage subsystem. Interestingly, I just bought a new laptop with 2.8GHz i7 and it's faster than my desktop Q6600 running at 3GHz when it comes to this compression stuff... Anyway, I digress...

Open DCP is multithreaded, but it's not that smart - it only parallel processes as many simultaneous files as you have threads, so having one instance using 4 threads is no different than having 4 instances only using 1 thread. Having faster processors would allow me to compress faster, but I don't think that's the bottleneck, it's the storage and file system that starts to struggle with the hundreds of thousands of files.

I've just downloaded and installed PerfectDisk 12 defragmenter to see how well it works - it claims it reduces fragmentation by correctly allocating space before the files are written. I'll see how well it achieves what it claims very quickly.

[Funkstar]

I wonder if something like a PERC RAID card with those drives and then just sharing the drive over your network would be better than using a WHS box. I can't help feeling that real hardware RAID (even one on a budget) would be better than software sort-of-but-not-quite-RAID on Home Server.

[badass]

The storage server is a Windows Home Server (v1), with some 2TB drives - 5400rpm slow ones!.

I'll quickly explain the workflow as there are a few stages to it: I'm creating a 3D DCP, and that involves having individual left and right eye stereoscopic files in jpeg2000 format, XYZ colour space. I'm using an open source program for the j2k and xyz conversion (OpenDCP). This is cross-platform and I could have it running on Linux. However, it's the storage that's the problem.

1) The single large video file is parsed through Adobe Media Encoder to create individual TIFF stills of each frame of the video. These frames are cropped and resized on the fly by AME. Unfortunately, the TIFF export of AME is uncompressed TIFF only, so you're looking at 8MB per TIFF file. At 24 files per second, then multiply by two to have your left and right sets of files, you can see how you quickly need several terabytes

2) The resulting TIFF is then read by OpenDCP to create a the j2k/xyz version. The actual executable of OpenDCP is run on several workstations so I can have a kind of render farm - I'm moving groups of TIFF files into individual directories to farm out the load, typically 10,000 files per directory.

3) OpenDCP saves the resulting j2k file somewhere else - it's now on a different physical drive, but still inside the same server.

I now realise one of the potential problems is that I was using a disk in the storage pool which probably has an adverse affect on performance.

In terms of random read / write, I suppose it is. I have, at most, 12 simultaneous accesses from remote machines (mapped drives) to read the files, compress them and write back the Jpeg2000 versions. I've now started writing the jp2k ones on a different physical drive (still inside the server), just in case it was a read/write issue.

Since writing the original post, I've realised that there had been massive fragmentation as the result of writing the initial files. It's taken around 6 hours to half defragment! Having defragmented, it's slightly faster, but still slow, so I think it might be more an NTFS limitation than a physical fragmentation problem.

I do have an EXT3 (maybe 4) Windows driver somewhere that works fine. How the performance is though, I don't know. I guess I'd be better with a small Linux NAS storage box to hold the data, as I can't afford to wipe my existing WHS installation. Thanks for the suggestion Peter

Jim, I'd love to use SSDs, however, 2TB+ worth isn't going to be cheap

If you using Adobe media encoder to take a video, crop, resize and output it as individual frames, then another piece of software convert those frames and combine them into one file then it looks to me like you're stuck deciding between using the glass bottle or the old shoe.
http://weblogs.asp.net/alex_papadimo...25/408925.aspx

I suspect there is software that does all you are asking of it with all of these steps internally.
How are others acheiving what you are attempting?