The "dark" net

Joe Anderson

New member
Sorry, I used the data field incorrectly. How does the P.C. connect to non- indexed web pages--- the "dark" net? What prevents "spider" programs from their sites?


This depends. Not being indexed by search engines makes you part of the "dark web" by some definitions but not by others.

Getting a search engine to skip indexing your site is pretty easy (at least for well-behaving search engines). Look up robots.txt for details. If you exclude a search engine via robots.txt, there is no technical reason the search engine couldn't index your site, they're just honoring your request not to.

Tor onion (nee hidden) services is another level, providing some anonymity using a more complicated scheme than just http type connections. However, Google *could* index these sites too--they just don't.

In short, if something can be accessed by any random individual, it COULD in theory be part of a search engine (making it "less dark") but there are various reasons why they DON'T index certain things.


Non-indexed pages were always referred to as "deep web" back when I was doing website work on a regular basis, as you could reach them normally if you knew where to look. Typically these were pages that served material, login pages, data frames, etc. As Zorlof said, most of this is set by the robots.txt files, or by the parameters of the search engine bots themselves.