Heh, I believe I am a little bit (but not much) younger than you Mr. Raberg, but your history with storage reminds me of my own! I spent so much of my scant pocket money on 100-packs of 1.44MB 3.5" disks when I was in junior high! And that never changed, today I've got a 110TB Unraid server in the basement. I still have some of the things I wrote in high school, and photos I took with my first digital camera, etc. My main annoyance is that there is no simple and easy object storage tightly integrated into the OS the way hierarchical filesystems are. That software distribution is not fundamentally managed through a distributed global immutable object storage system just seems... incorrect. Your thinking on the different "levels" of storage is also very similar to mine. I've thought for awhile now, that it should all just be seen as a cascading set of caches, from CPU cache to RAM to local SSD to LAN to Internet and it would be great if it was a unified interface across all of those. You just request a GUID (or, for human-friendliness a friendly name resolved to a GUID by a nameservice) and it gets provided to the CPU by the kernel sourced from whichever cache is closest. It would be beautiful. Your choice of column store intrigues me... as you point out, it embraces sequential array access. To me, at least, that seems counter-intuitive. Sequential access makes sense for old mechanical disks, but since solid state storage has 0 "seek" time, I would think you'd want something that benefits greatly from abandoning any sense of sequential access as preferred. I think the best place for your technology around indexes is actually in hardware itself. There are storage controller devices being developed (some are available) now that build some embedded dedicated compute to storage modules themselves, like a box which has no actual "OS" to speak of, no general-purpose CPU, but just a dedicated chip to provide an S3-compatible API over a network (I assume they bundle in some management niceties also but not sure, I've seen videos about them but never gotten to play with one directly, they're large enterprise products). I look forward to reading many of your references, this was a fantastic talk!
