I have been scouring the interwebs looking for an elastic filesystem that can be installed (non-S3) on my own gear.  

I have probably read about or reviewed 50 different packages, but have yet to find any with the combination of features I am looking for.  I'd love to know if anyone on the list knows of a project that fits the bill, or at least has similar requirements:

1) Master-less:  The first node is a storage node.  There are no 'master nodes' and therefore one less SPOF.  Adding nodes increases total storage in a near-linear fashion.  Most likely a Gossip protocol is used to achieve inter-node communication.

 

2) Redundant: Data is automatically replicated to achieve redundancy.  Ideally these policies would be rack/colo aware

3) Elastic: You can add and remove nodes as needed.  Nodes crashing are handled transparently.  Data is rebalanced automatically in the background to ensure #2.  libkatema or similar used here.

4) FUSE Support: A FUSE adapter is available - recognizing it may not be the most performant interface in all cases.  A windows adapter would also be ideal - a WebDAV interface might be sufficient.

5) Open Source: Ideally written in a language more prevalent than modula

The closest thing I have found so far is Cassandra:

http://incubator.apache.org/cassandra/

which is in use at Facebook, Digg, Twitter, etc.  However, this is a document store, not a filesystem/filestore, which basically means it does more than what is needed, but also there are some filesystem semantics that are missing.

If I can't find what I'm looking for, I'm tempted to begin looking into a CassandraFS implementation that would leverage much of the core of Cassandra, but fill the gaps mentioned above.

Here is a short list of some of the projects I've looked at.

HDFS - Hadoop FS

Lustre

Gluster

MogileFS

Terrastore

Dokan - FUSE for windows

XtreemFS

Cassandra

Project Voldemort

I don't know what I don't know on this one - all leads appreciated.

Thanks,

Darren

--
Please Note: If you hit "REPLY", your message will be sent to everyone on this mailing list ([address removed])
This message was sent by Darren ([address removed]) from LA Cloud Computing.
To learn more about Darren, visit his/her member profile
To unsubscribe or to update your mailing list settings, click here

Meetup Inc. PO Box 4668 #37895 New York, New York 10163-4668 | [address removed]