Comments on: A Million Little Files

By: wenxiu

wenxiu — Thu, 02 Sep 2010 08:06:42 +0000

oh. sorry. ignore my last post, it is just a warning. it works! thanks~
BTW, do you happen to have a tool to convert multiple files from a directory to a single sequence file directly? I mean, to save the outside tar compress and un-compress cost.

By: wenxiu

wenxiu — Thu, 02 Sep 2010 07:48:21 +0000

Sorry, how to make it work? I got an error:

prod2@ot-9h30d06:~/QSense/hadoop/hadoop-0.20.2/tar-to-seq$ java -jar tar-to-seq.jar ../change.tar.gz seq_file
log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.

By: Jan

Jan — Fri, 04 Jun 2010 15:19:51 +0000

Hi,

I tried to create a seq file with text files inside, using your tool but unfortunately when I later open it in HADOOP, I get different content:
…6b 20 74 68 65 20 61 72 65 61 73 20 77 68 65 72 65 20 77 65 20 65 73 74 69 6d 61 74 65 64 20 74 68 65 20 74 72 61 69 6c 20 75 70 70 65 72 2d 6c 69 6d 69 74 20 66 72 6f 6d 20 74 68 65 20 73 74 61 6e 64 61 72 64 20 64 65 76 69 61 74 69 6f 6e 20 6f 66 20 61 6e 20 31 31 20 c3 97 20 31 31 20 70 69 78 65 6c 20 61 70 65 72 74 75 7…

What could be the reason?

Thank you,
Jan

By: The Case for Babar: A Tool for Creating Hadoop Sequence Files « Ryan Balfanz

Mon, 22 Feb 2010 08:38:15 +0000

[...] are they large, but we have a lot of them. We are using Hadoop after all. Stuart Sierra’s Tar-to-Seq utility was working quite nicely until this new input set, as those files were much [...]

By: Ryan

Ryan — Tue, 10 Nov 2009 10:52:25 +0000

Stuart,

Thanks for posting this. It will help me a lot! It would be nice to see some options to set SequenceFile.CompressionType, or perhaps guess based on the incoming extension like you do in openInputFile().

Thanks,
Ryan

By: Stuart

Stuart — Tue, 10 Feb 2009 16:11:48 +0000

Mark,

I’m afraid I don’t know. From my understanding of HDFS, it depends on a lot of factors — the size of the files, the bandwidth to and within the cluster, and the hardware itself. Try the Hadoop mailing list. One thing I do know is that copying lots of small files to HDFS will be slower than copying a few big files.

-Stuart

By: Mark Kerzner

Mark Kerzner — Tue, 10 Feb 2009 02:46:09 +0000

Stuart,

I have written to code to write from the PC file system to HDFS, and I also noticed that it is very slow. Instead of 40M/sec, as promised by the Tom White’s book, it seems to be 40 sec/Meg. Your tars would work about 5 times faster. But still, why is it so slow? Is there a way to speed this up?

Thanks!

By: Cloudera Hadoop & Big Data Blog » Blog Archive » The Small Files Problem

Mon, 02 Feb 2009 16:11:12 +0000

[...] to create a collection of SequenceFiles in parallel. (Stuart Sierra has written a very useful post about converting a tar file into a SequenceFile — tools like this are very useful, and it [...]

By: Stuart

Stuart — Thu, 29 Jan 2009 15:23:41 +0000

You don't need to do anything special. The code I posted here produces Hadoop SequenceFiles. You can use the built-in Hadoop class SequenceFile.Reader to read them. Normally, all you need to do is:

yourJobConf.setInputFormat(SequenceFileInputFormat.class);

By: Rasit

Rasit — Thu, 29 Jan 2009 08:38:41 +0000

Stuart, I mean, which InputReader should I use (InputReader which sends key-value pairs to Mapper class.).
Do hadoop offer some? or should I extend one of existing?