Comments on: A Million Little Files

By: net_ma

net_ma — Wed, 18 Jan 2012 15:32:43 +0000

Hi Stuart,

I downloaded your tool. But when I tried to convert a tar file, I got the following error.

The tar file I tried to convert is about 4.5GB in size and contains about 200 files.

Could you tell me what I should do?

Thank you.

java -jar tar-to-seq.jar /home/hduser/sample-archive/2011/a-250.tar a-250.seq
log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.
Exception in thread “main” java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:78)
at org.apache.hadoop.io.compress.CompressorStream.write(CompressorStream.java:71)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.writeBuffer(SequenceFile.java:1224)
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.sync(SequenceFile.java:1247)
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.append(SequenceFile.java:1297)
at org.altlaw.hadoop.TarToSeqFile.execute(TarToSeqFile.java:95)
at org.altlaw.hadoop.TarToSeqFile.main(TarToSeqFile.java:165)

By: Stuart

Stuart — Fri, 05 Aug 2011 21:28:44 +0000

Chris: the source code is included in the .tar.gz download. Apache license.

By: Chris

Chris — Fri, 05 Aug 2011 19:39:06 +0000

Would you be willing to share the source code? It would be very helpful if i could rewrite it to have your Seq file as Key: Filename Value: File Text instead of a BytesWritable for the Value.

By: Stuart

Stuart — Wed, 09 Mar 2011 22:25:29 +0000

Sorry, I don’t know anything else about Hadoop Streaming.

By: Bowen

Bowen — Wed, 09 Mar 2011 21:41:08 +0000

Thanks for the prompt reply.

So is there anyway for Hadoop Streaming to take binary files as input? Currently, I have to first download those files from HDFS to the local machine, and then process. It’s super slow…

Thanks,
Bowen

By: Stuart

Stuart — Wed, 09 Mar 2011 13:52:38 +0000

The file contents are stored as a normal Hadoop BytesWritable object. You would access it as you would any other Hadoop Writable datatype. But if I recall correctly, Hadoop Streaming only supports text.

By: Bowen

Bowen — Wed, 09 Mar 2011 05:26:01 +0000

Stuart,

I’m using Hadoop Streaming (C code) to process binary input files. After using your code to get the sequence file, how to read the data in my C code? Should I read it byte by byte?

Thanks,
Bowen

By: Hadoop binary files processing entroduced by image duplicates finder « eldad levy's playground

Sat, 05 Feb 2011 09:26:08 +0000

[...] all the images as a tar file and using the tool written by Stuart Sierra to convert it to a sequence [...]

By: wenxiu

wenxiu — Thu, 02 Sep 2010 08:06:42 +0000

oh. sorry. ignore my last post, it is just a warning. it works! thanks~
BTW, do you happen to have a tool to convert multiple files from a directory to a single sequence file directly? I mean, to save the outside tar compress and un-compress cost.

By: wenxiu

wenxiu — Thu, 02 Sep 2010 07:48:21 +0000

Sorry, how to make it work? I got an error:

prod2@ot-9h30d06:~/QSense/hadoop/hadoop-0.20.2/tar-to-seq$ java -jar tar-to-seq.jar ../change.tar.gz seq_file
log4j:WARN No appenders could be found for logger (org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.