Staring into the Void

[bitheerani93]

First, let’s get one myth out of the way: The Internet Archive has not been up, rock-steady and with no loss of service or connection, for twenty-eight years.

Starting out as a project to archive online materials, with a lot of speculative ideas of how to handle data at scale, the archive.org website was hosted at a shifting set of locations across its early years. It e-commerce photo editing at razor-thin margins while rubbing hardware and software elbows with all sorts of then-famous sites; it directed its staff towards nebulous and aspirational goals while trying not to burn through its resources.


Stand back, we’re not sure how big this Archive is going to get.

A lot changed in October of 2001, when the Wayback Machine was introduced to the world at a ceremony at the Bancroft Library in Berkeley, and the Web spontaneously developed something it hadn’t really had before: a memory.

That Memory went from a feature to a core utility for the internet.

Collections such as the Prelinger Library and the Live Music Archive were also coming along for the ride, providing a way for people to just get to the good stuff and not face down web banners and pop-up ads just to listen and watch culture from a growing set of sources and reaching back farther in time, to before the web itself.

Serving a massively-enlarging set of data to a massively-increasing audience became an engineering and cost problem, and ultimately the problem – how do you retrieve and provide terabytes, then hundreds of terabytes, then petabytes, then dozens of petabytes of data to your patrons without, again, falling to a thousand potential problems?