The internet is a huge database with a variety of information, and different search engine platforms are used to find the results required by the users. However, the visible web we usually surf only feeds a small percentage of information to us and leaves over 96% of the web content hidden down in the deep web.
To access deep web contents, users are required to use dedicated browsers that allow them to use peer to peer networking and access the needed information. The information is encrypted, making it hard to fetch from most of the search engines.
Web Crawlers are software programs used by search engines to find new links from website. Unlike most of the common search engines, the use of crawlers allows the user to find, request, download and obtain information, then further redelivered. Once web crawlers found a new website link that’s visible to the search engine, it will start to request copies of the website and download its content into their own server. This also means if the information isn’t visible, the search engine will not be able to download a copy of the website, therefore this website will not show up in the search engine list.
Then search engines will use Search Engine Optimization (SEO) to analyze content’s uniqueness, then index and fetch if the new content matches the existing ones on the website. This means the website has to be relevant to the keywords in which the users have inserted to the search engine, in order to bring its SEO to the most efficient. SEO is a complex algorithm including Onsite SEO, Offsite SEO, meta-tags, content uniqueness, codes, and links. The makes information we search on the web more specific and precise.
Searching academic information will require digging into the deep web or some other specific search engines. For example, Musée du Louvre has its own search engine researching information of the artwork in the Mesee du luvre which Google could not access and download a copy. In modern days, Google has stood at the place of the largest search database, however, it could only index what is visible and searchable, which could be seen as the tip of an iceberg.
Louvre.fr. (n.d.). Les incontournables | Musée du Louvre | Paris. [online] Available at: http://www.louvre.fr/selections [Accessed 19 Apr. 2017].
Surface Web. (n.d.). [image] Available at: https://s3.amazonaws.com/libapps/customers/55/images/DeepWeb.png [Accessed 19 Apr. 2017].