Mind blowing SEO improving and Google rank increasing Signature Rotator SEO Tool.

Why do we need to add sitemap link on our robots.txt file?

make money online
totompro Offline referral

Posts: 49
Joined: Jun 2016
Reputation: 0

Junior Member
On this article I’d like to try to explain the need of adding sitemap link on the robots.txt file. I do it on a hope it could be useful for this article’s readers so they aren’t loose too much time to search on for a useful reading.
What’s sitemap?
Sitemap is a file contain list of URLs on our website where the URLs are allows to be crawled and indexed by search engine i.e. Google Search. On the most cases it’s intended to only consumed by search engine crawled so it isn’t need to be human friendly; but just naked link list. Usually a sitemap file has an extension of .xml but it’s also allowed to have other extension like .html or .txt by Google.
For optimization purposes sitemap file usually saved on the root directory for example on hxxp://www.example.com/sitemap.xml. It’s seems an un-important thing but we could think what’s the faster way within crawl sitemap file on the root directory (i.e. hxxp://www.example.com/sitemap.xml) and on a specific folder (i.e. hxxp://www.example.com/folder/../sitemap.xml). It looks likely the faster way to craw is when we save our sitemap file on the root directory of our website.
What’s robots.txt?
Robots.txt is a text file otherwise noted is exists on the root directory of a website. On the most case it’s the first destination to be crawled by search engine robot i.e. googlebot. Robots.txt contains information about what folder and/or what file allowed and don’t allowed to be crawled by search engine crawler (i.e. googlebot).
Benefits of adding sitemap file link to the robots.text file
On the common practice, submitting sitemap to search engine is a simple task. But on uncommon condition when we aren’t able to submit our sitemap to a search engine due to a special reason for examples because of the search engine native language is differ from ours and they don’t offers a translator or different language version this will help the search engine crawler to find the sitemap file on our website by read it on the robots.txt file. Another reason why we need to add our sitemap file link on the robots.txt file is for giving a direction to un-known crawler (if any) for the existence of our sitemap file so it could indexing our sitemap file by itself without the need of manual sitemap submission.
So, this is the end of this article, I hope it’s useful for you to increase up your website visibility on the eyes of the prospective visitor, and I don’t forget to ask you a forgiveness if there are many mistakes exists on this article made by me. Good bye and wish you success with your website!

This article is proudly presented by Home Ideas Made Simple.
Victor Dub Offline referral

Posts: 946
Joined: Oct 2013
Reputation: 5

Posting Freak
#SiteMap is one of the SEO elements that in my mind is a bit controversial. For a small websites that don't really have that much URLs it helps indexing process but for a huge ones it actually slows the indexing process down why let me explain.

If you have site map index with 1.000.000 URLs Google will never index them all case when the sitemap gets resubmitted that cases Google to go back and re-crawl already indexed URLs and by the time it gets to the end of the already indexed URLs the sitemap is resubmitted again. It's a loop that prevents Google from discovering or grabbing new URLs.
sankalppatil Offline referral

Posts: 465
Joined: Jan 2018
Reputation: 1

Senior Member
It is not required to have a sitemap reference in a robots.txt file. Instead you can create an XML sitemap index file, and include references to all your sitemap files from there. This would allow you to group multiple sitemap files. Here's a link to sitemps.org which describes the correct method and syntax to do so:

After creating a sitemap index file you can submit this to Google / Bing as long as you upload and save all your sitemaps to the same location on your host server.

Google this topic

User(s) browsing this thread: 1 Guest(s)