Data problems

Not a new observation, or anything, but there's been an explosion in storage capacity, with revolutions of change still coming.... but our ability to organize and manage that data has only incrementally improved over the last two decades. For example, at the high end, although every medium to large business drowns in an excess of data they have trouble managing, core rDB tech has only evolved incrementally, at least commercially - yet remains the tool of choice.

Cutting edge is considered selecting MySQL instead of Oracle - yeah, that's bold :P

Companies like Google, whose mission statement is about data organization, of course, don't just fall back on tried and true patterns - they created tools (genuine internal infrastructure) like BigTable to help address these issues.

In a similiar vein (courtesy of Slashdot), you should check out the Database Column - a "multi-author blog on database technology and innovation" ("Column", get it? :D). Clearly, there's some interesting thinking going in the space that will change how data management happens - column-based dB tech vs. row-based is really only the tip of the iceberg, but provides a nice visual metaphor for how sideways things will get.

Interestingly enough, the "middeware" thing I referred to previously was in this space. If I manage to get out off my keister with that project some weekend, I'll post a sample application... but don't hold your breath :)