You are viewing a single comment's thread from:

RE: Save the Internet with SteemSafe!

in #palnet5 years ago

Great question. This is a bit different than the wayback machine, as it's not a webcrawler. A webcrawler trawls the internet and archives every site that allows it. This is more similar to https://archive.is/ where a user submits a single URL and has it archived, and it will archive any site submitted and ignores the robots.txt.