Comments on: Big & Small at the same time http://stuartsierra.com/2009/06/26/big-small-at-the-same-time From programming to everything else Sat, 04 Feb 2012 20:39:31 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Stuart http://stuartsierra.com/2009/06/26/big-small-at-the-same-time/comment-page-1#comment-42598 Stuart Fri, 03 Jul 2009 17:22:05 +0000 http://stuartsierra.com/?p=246#comment-42598 Philip- That's an interesting idea. But as far as I can tell from the <a href="http://dev.mysql.com/tech-resources/articles/csv-storage-engine.html" rel="nofollow">docs</a>, the CSV storage engine can't retrieve a single record without a full-table scan. A similar approach, that I want to try, is making the final Hadoop job output a <a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/io/MapFile.html" rel="nofollow">MapFile</a>, basically a SequenceFile with an index. Then just dump the MapFiles on a file system and retrieve the records from there. But I haven't found much discussion of MapFiles outside of the API docs, so I don't know if they are really suitable for this kind of use. Philip- That’s an interesting idea. But as far as I can tell from the docs, the CSV storage engine can’t retrieve a single record without a full-table scan.

A similar approach, that I want to try, is making the final Hadoop job output a MapFile, basically a SequenceFile with an index. Then just dump the MapFiles on a file system and retrieve the records from there. But I haven’t found much discussion of MapFiles outside of the API docs, so I don’t know if they are really suitable for this kind of use.

]]> By: Philip (flip) Kromer http://stuartsierra.com/2009/06/26/big-small-at-the-same-time/comment-page-1#comment-42597 Philip (flip) Kromer Fri, 03 Jul 2009 05:54:52 +0000 http://stuartsierra.com/?p=246#comment-42597 It's a bit wacky, but have you considered doing a last-pass total sort (so that the outcome is efficiently sharded) and using a CSV storage engine? MySQL has one and so do others I believe. If it's the merge that's the problem, does just sharding help -- doing a final sort so each key knows which reducer's (shard's) index to query on? (or querying against all in parallel?) It’s a bit wacky, but have you considered doing a last-pass total sort (so that the outcome is efficiently sharded) and using a CSV storage engine? MySQL has one and so do others I believe.

If it’s the merge that’s the problem, does just sharding help — doing a final sort so each key knows which reducer’s (shard’s) index to query on? (or querying against all in parallel?)

]]> By: Stuart http://stuartsierra.com/2009/06/26/big-small-at-the-same-time/comment-page-1#comment-42582 Stuart Fri, 26 Jun 2009 21:43:44 +0000 http://stuartsierra.com/?p=246#comment-42582 I still have to load the data into the key/value store. It's too big to fit in RAM, at least on hardware that I can afford. Tokyo Cabinet is an option, but I haven't had time to explore it much yet. I still have to load the data into the key/value store. It’s too big to fit in RAM, at least on hardware that I can afford. Tokyo Cabinet is an option, but I haven’t had time to explore it much yet.

]]> By: tim wee http://stuartsierra.com/2009/06/26/big-small-at-the-same-time/comment-page-1#comment-42581 tim wee Fri, 26 Jun 2009 21:36:27 +0000 http://stuartsierra.com/?p=246#comment-42581 how about using a distributed key value store like tokyo cabinet/redis/memcached? how about using a distributed key value store like tokyo cabinet/redis/memcached?

]]>