Mind blowing SEO improving and Google rank increasing Signature Rotator SEO Tool.


Why do we need to add sitemap link on our robots.txt file?

totompro Offline referral

Posts: 46
Joined: Jun 2016
Reputation: 0

Junior Member
On this article I’d like to try to explain the need of adding sitemap link on the robots.txt file. I do it on a hope it could be useful for this article’s readers so they aren’t loose too much time to search on for a useful reading.
What’s sitemap?
Sitemap is a file contain list of URLs on our website where the URLs are allows to be crawled and indexed by search engine i.e. Google Search. On the most cases it’s intended to only consumed by search engine crawled so it isn’t need to be human friendly; but just naked link list. Usually a sitemap file has an extension of .xml but it’s also allowed to have other extension like .html or .txt by Google.
For optimization purposes sitemap file usually saved on the root directory for example on hxxp://www.example.com/sitemap.xml. It’s seems an un-important thing but we could think what’s the faster way within crawl sitemap file on the root directory (i.e. hxxp://www.example.com/sitemap.xml) and on a specific folder (i.e. hxxp://www.example.com/folder/../sitemap.xml). It looks likely the faster way to craw is when we save our sitemap file on the root directory of our website.
What’s robots.txt?
Robots.txt is a text file otherwise noted is exists on the root directory of a website. On the most case it’s the first destination to be crawled by search engine robot i.e. googlebot. Robots.txt contains information about what folder and/or what file allowed and don’t allowed to be crawled by search engine crawler (i.e. googlebot).
Benefits of adding sitemap file link to the robots.text file
On the common practice, submitting sitemap to search engine is a simple task. But on uncommon condition when we aren’t able to submit our sitemap to a search engine due to a special reason for examples because of the search engine native language is differ from ours and they don’t offers a translator or different language version this will help the search engine crawler to find the sitemap file on our website by read it on the robots.txt file. Another reason why we need to add our sitemap file link on the robots.txt file is for giving a direction to un-known crawler (if any) for the existence of our sitemap file so it could indexing our sitemap file by itself without the need of manual sitemap submission.
So, this is the end of this article, I hope it’s useful for you to increase up your website visibility on the eyes of the prospective visitor, and I don’t forget to ask you a forgiveness if there are many mistakes exists on this article made by me. Good bye and wish you success with your website!

This article is proudly presented by Home Ideas Made Simple.
Visit my blog on http://www.totoms.com
Victor Dub Offline referral

Posts: 708
Joined: Oct 2013
Reputation: 3

Senior Member
#SiteMap is one of the SEO elements that in my mind is a bit controversial. For a small websites that don't really have that much URLs it helps indexing process but for a huge ones it actually slows the indexing process down why let me explain.

If you have site map index with 1.000.000 URLs Google will never index them all case when the sitemap gets resubmitted that cases Google to go back and re-crawl already indexed URLs and by the time it gets to the end of the already indexed URLs the sitemap is resubmitted again. It's a loop that prevents Google from discovering or grabbing new URLs.

User(s) browsing this thread: 1 Guest(s)