The "dark" net

Joe Anderson

New member
Sorry, I used the data field incorrectly. How does the P.C. connect to non- indexed web pages--- the "dark" net? What prevents "spider" programs from their sites?
 

zorlof

Member
This depends. Not being indexed by search engines makes you part of the "dark web" by some definitions but not by others.

Getting a search engine to skip indexing your site is pretty easy (at least for well-behaving search engines). Look up robots.txt for details. If you exclude a search engine via robots.txt, there is no technical reason the search engine couldn't index your site, they're just honoring your request not to.

Tor onion (nee hidden) services is another level, providing some anonymity using a more complicated scheme than just http type connections. However, Google *could* index these sites too--they just don't.

In short, if something can be accessed by any random individual, it COULD in theory be part of a search engine (making it "less dark") but there are various reasons why they DON'T index certain things.
 

Performa

Member
Non-indexed pages were always referred to as "deep web" back when I was doing website work on a regular basis, as you could reach them normally if you knew where to look. Typically these were pages that served material, login pages, data frames, etc. As Zorlof said, most of this is set by the robots.txt files, or by the parameters of the search engine bots themselves.
 

Joe Anderson

New member
Knowledge truly equals power. Zorlof, a magician could' nt explain any better, and Performa's lucid light continues shining bright.
 
Top