Process of search engines

Before you start doing SEO of your or client’s website, you should understand the science of search engines. What exactly happens as a user goes to a search engine, types a search query and search engines returns a number of results relevant to the query of the visitor?

A few basic questions come into mind…

• How a search engine saves so many (billions) of pages?
• While there are thousands to millions of pages for a search query, how the search engine decides to serve the URLs on top? Is that random or what?
• I have a site of the same category, how I can be in the list of served results or if already in served results how I can reach to the top of the results?
• What are crawling, bots and indexing in search engines?
• And the list goes on.

Currently, there are a lot of search engines, however, only a few are popular or get the most share in the search market. Among those popular search engines, the Google is on top of the list. It almost occupies 80% share of the search market. Then comes Yahoo, Bing, Yandex etc.

As Google is a market leader in search sector we will explain how it works in this tutorial.

Crawling – How the process starts in Google?

The first step where search engine saves information into its index is “Crawling”. In this step, Google’s huge set of computer system ‘crawls’ newly added web pages. It also crawls updated pages in websites and adds it into its index. This process is accomplished by a program which is called Robot, spider or bot. Google calls it as the Googlebot.

As mentioned, the Googlebot will discover new sites, new web pages or updates in existing pages and then add those to its index.

So if you have created a new website, made it live in web server so it is available publically then you do not need to perform any process to be indexed in Google or other search engines. Their Robot program will automatically detect and index your new site pages.

Once a web page is discovered, the crawler will look into the hyperlinks in that page and will access those pages as well to its index. So it is really important that you should link different pages of your site and those links should not be broken.

Google and other search engines also recommend adding a sitemap of your website by using webmaster tools. This will facilitate Googlebot or other robots in finding your website or updates on existing website to let search engine index relatively quickly.

Google provides the Search Console formerly known as Web Master Tools (WMT) to allow webmaster, site owners or site administrators to upload sitemaps as shown below:

To sign up for GSC, go to this link: www.google.com/webmasters

Similarly, you can sign up for Bing webmaster tools and upload sitemaps there: http://www.bing.com/toolbox/webmaster/

How Google serve results

After crawling and indexing web pages, google determines relevance, importance and ranking of those pages based at over 200 factors. While many of those factors can be minor, a few are really important. Although Google would not reveal all the factors publicly, however, they keep on giving hints and instructions.

Backlinks

One of the important aspects, for example, are backlinks. Backlinks are the links of those pages that are linking to the specific webpage. Google takes this like the voting system. More the links to a page in certain niche more chances that this page will rank higher for the related search query.

However, backlinks quality has become really important over the period of time. And google Algo is quite intelligent in understanding what types of links these are. We are explaining low or high-quality backlinks in its own chapter.

Content

The other important factor is the content of web page. The content includes text information, images, headings, videos etc. in a web page.

Some experts say ‘Content is king’ as for as search engine (especially Google) SERPs are concerned. The topic about content is also covered in its own chapter in detail.

Other

Apart from that, some basic factors like meta tags (title, keywords, description) etc. plays a minor role but should be taken care properly. Especially, the title tag is really important.

Mobile friendliness

Lately, Google also introduced mobile-factor as well. So if you intend to get traffic from mobile devices then your website should be mobile friendly. It should allow visitors to view your website without horizontal scroll bars. Text size should be appropriate along with link and distance between the links.

The need to include these factors became necessary as more users are searching through mobile devices than desktop (including tablets).

So how Google serves search query

As mentioned, Google says it uses around 200 factors or signals to decide which sites to be shown in the result page, the organic part of google search page. Based on those factors, websites are ranked higher or lower in SERPs and shown to their visitors.

In the next chapter, we will go through a few important factors that Google uses to rank websites in their index like content, important tags, backlinks, website structure, user engagement etc.

Also see – Importance of backlinks | Content is king