The Internet Archive is arguably the most well-known archive of the development and history of the World Wide Web. Brewster Kahle founded it 20 years ago after realizing that, despite the web’s crucial role in changing how society accessed and interacted with the outside world, the web was inherently ephemeral and disappearing with each passing day. In order to preserve the early Internet during its formative years, he established the Archive. Serving as the online equivalent of a library archive, it collected donations of crawling data, conducted its own crawls, and combined all of this information into a unified digital catalog. Due to the Archive’s success, numerous web archiving projects have been launched worldwide, ranging from national
culture to data from science.
But in the short 24 years that it has existed, the modern web has developed into a rich multimedia and increasingly intelligent network that aims to connect every individual on the planet, from a simple (mostly textual) platform for sharing scientific knowledge. Because of this ongoing state of change, the Internet has essentially kept up with the majority of the web archiving community, which means that although the Internet powers an increasing amount of the world around us, our archives are saving less and less of the Internet.
An internet archiving crawler constructed twenty years ago could, for the most part, still perform most of its functions today. It would download a webpage, extract its links, crawl each link, and extract its linkages in
The issue is that the basic idea of a collection of tiny, static HTML and picture files served up with a basic tag structure and easily parsed with a few lines of code is no longer the foundation upon which the web is constructed. Today’s online is multimodal, richly dynamic, and becoming more and more divided into parallel webs tailored to individual devices and walled gardens.Specifically, the web has evolved along four main avenues that have proven especially difficult to maintain for the community of archivists: social media, dynamic content, multimedia, and the mobile web.
The internet in 1995, the year I started my first web startup, was very different from the internet of today. Back then, web sites were just book pages with random images thrown in for effect. Streaming music and video is the main focus of the internet today. The quantity of 4K videos on YouTube and other streaming websites is starting to rise. Not only is it easy to acquire many petabytes of HD video without much effort, but most streaming video websites make it difficult to download the actual source files, which makes multimedia tough to archive. While there are several tools available to reverse the streaming
The challenge of archiving streaming video has not been solved, and popular technologies such as the Internet Archive’s Archive-IT system actually come with built-in support for sites like YouTube. However, among the many web archiving initiatives currently in progress, support for streaming video is really somewhat unusual; the great majority of them are unable to effectively download and archive a YouTube video and preserve it for posterity as part of their main crawling activity.
Because social media platforms have created walled gardens, they present possibly the most formidable obstacle to web archiving. Facebook and many other platforms do not offer commercial data firehoses that archivers can simply plug into, however Twitter has long provided a firehose of all of its public tweets, which is in fact preserved by the Library of Congress. almost all of the main social media sites, save Twitter, are heading toward default settings that promote sharing content solely with friends and substantial privacy controls. Sharing private thoughts with friends and family is more popular these days than broadcasting one’s every thought to the entire world. This implies that even if businesses like Facebook chose to
The biggest social networking networks are basically unarchiveable from a web archiving perspective. Although there are tools available to help with bulk exporting posts from Facebook, the social media giant is always updating its technological defenses and has threatened legal action in the past to deter mass user data downloads and distribution. Regardless of technological or legal barriers, people are walling off their data and preventing the public access required to archive it due to shifting social norms around privacy. Stated differently, the new private parallel Internets created by social media platforms cannot be maintained since they wall off the Internet, despite the fact that society is becoming more and more dependent on these walled gardens to conduct daily activities.
The dynamic web presents distinct difficulties for the straightforward
Leave a Reply