Comments on: When You Have a Hammer, Everything Looks Like Rails

By: Joshua Paine

Joshua Paine — Wed, 14 Nov 2007 03:08:16 +0000

For the data store, have you considered couchdb? I’ve not used it, but I’m dying to. It’s pretty niche and new at this point, but maybe it’s worth a look for you.

By: Nick

Nick — Tue, 13 Nov 2007 23:35:06 +0000

There has been quite a bit of buzz on CouchDB… sounds right up your ally.

http://couchdb.com/CouchDB/CouchDBWeb.nsf/Home?OpenForm

By: Pat

Pat — Tue, 13 Nov 2007 23:12:57 +0000

Try couchdb http://couchdb.org

By: Justin Rich

Justin Rich — Tue, 13 Nov 2007 23:11:46 +0000

I have similar issues with one of my core products.

I first starting using ferret too, but one morning the index files it builds started to grow tremendously without rhyme or reason, maxing out hd space. I switched everything over to solr that day and it’s been a solid performer for over a year now. Very fast when you know how to talk to the solr server.

The acts_as_solr plugin (at least early versions of it) were naive implementations. I’ve branched my own version where I can delay commits to the solr db in order to make it fast for large updates as well. For searching, I query the solr server to return strictly ids, then query the db with those. That way not a terrible amount of data is being traded. Also, you have to make sure that documents deleted from the db also get deleted from the solr index.

For batch updates/uploads to the db, check out http://agilewebdevelopment.com/plugins/ar_extensions . It has a batch import function that you can also get to work to update records with unique keys. I had huge huge huge performance gains when using this plugin.

Sorry I can’t help you with the best way to store all of the text in general. I keep newspaper articles just fine in mysql. I’m sure there’s a way. Lexis Nexis does it.

By: Cliff Moon

Cliff Moon — Tue, 13 Nov 2007 23:04:09 +0000

Try looking at couchDB, it’s designed for semi-structured data like this.

By: Stephen De Gabrielle

Stephen De Gabrielle — Tue, 13 Nov 2007 19:11:07 +0000

Have you looked at Greenstone Digital Library system http://www.greenstone.org/ ?
Active, helpful developers and friendly community, I recommend it.

Stephen

By: Leonard Richardson

Leonard Richardson — Tue, 13 Nov 2007 03:20:37 +0000

Sorry, I didn’t read this entry closely enough. I hadn’t heard of Solr but that sounds like the best choice for indexing. For storage I’d keep the data in whatever format it fits in naturally (probably flat files, but maybe database tables) and keep all the standardized metadata in the Solr index.

Not sure if this will help either, but there you go.

By: DHH

DHH — Tue, 13 Nov 2007 03:10:41 +0000

We’re using Solr at 37signals to provide search for all our applications dealing with gigabytes of data. The data is based off web documents and text, though. Not PDFs and so on. It works really well.

By: Leonard Richardson

Leonard Richardson — Mon, 12 Nov 2007 21:24:44 +0000

You’ve basically described Lucene, the library of which Ferret is a port. I don’t know of any other library for searching structured text, though I think PyLucene is more mature than Ferret. However the last time I seriously looked into this was around 2005.