Sitemap and robots.txt: your website's map and house rules for Google

Google finds pages by following links, like going door to door. It works, but it is slow and sometimes it misses one. To make it easy, there are two small files every website should have: the sitemap and robots.txt.

The sitemap: a list of your pages

It is a file, usually at `yourwebsite.com/sitemap.xml`, listing every page you want Google to know about. For each one it gives:

• The exact address.
• When it last changed, so Google knows what to look at again.
• If the website has several languages, which version is which language.

You never need to understand it or even open it. What matters is that it exists, is up to date and Google knows where it is.

Why it helps

• A new website: Google has no links from other sites to reach you. The sitemap gives it the full list from day one.
• Hidden pages: a service page only reachable from a drop-down menu can take a while to be discovered.
• Changes: when you add or change pages, the sitemap reflects it and Google hears sooner.

Note: being in the sitemap does not force Google to show a page. It is an invitation, not an order.

Robots.txt: the house rules

This text file at `yourwebsite.com/robots.txt` tells search engines which parts of the website they may visit. It is usually very short: it allows everything public, closes anything private (such as an admin panel) and says where the sitemap is.

Anyone can see it: type your domain followed by `/robots.txt` in a browser.

Sitemap and robots.txt mistakes that keep a site off Google

• A robots.txt that blocks everything. It happens more than you would think: the website was built with "do not index" while in testing and nobody removed it at launch. The website works, but Google is forbidden from entering.
• A sitemap listing pages that no longer exist, or test addresses, or missing the new pages.
• A sitemap nobody submitted. Google finds it eventually, but later.
• Blocking files Google needs to render the page properly, such as stylesheets.

If your website is not on Google, this is one of the first things to check. More causes in why my website is not on Google.

How Google and Bing are told

The sitemap is submitted through Google Search Console and Bing Webmaster Tools. Bing matters more than it seems: ChatGPT and Copilot draw many of their results from it. More in Bing Webmaster Tools and IndexNow.

How I set up the sitemap and robots.txt

On Nueva Página websites both files come ready and update themselves when pages are added. At launch I register the website in Google Search Console and Bing and submit the sitemap. It is part of the basic SEO in every plan.

How to check your sitemap and robots.txt

You can do it yourself in two minutes. Type your domain followed by `/robots.txt` into a browser and check that it does not say a bare "Disallow: /", because that blocks the whole website. Also check that it ends by saying where the sitemap is. Then open your domain followed by `/sitemap.xml` and make sure your important pages appear with their full address.

If something looks wrong, Search Console will also flag it in the sitemaps section. Google explains how each should look in its guides to sitemaps and the robots.txt file. And if your website still does not appear, go through why my website is not on Google. A good sitemap and robots.txt go unnoticed; a bad one does not.

Frequently asked questions

Do I need to do anything when I add a page?

If the sitemap is generated automatically, no. If you maintain it by hand, add the page and resubmit it.

Can I hide a page from Google with robots.txt?

It is not the best way: it blocks the visit, but the page can still appear if others link to it. To keep it out, the page itself uses a "noindex" tag.

Does a one-page website need a sitemap?

Yes, even a short one. It costs nothing and helps Google find it sooner.

Will my website include a sitemap and robots.txt?

Yes. At Nueva Página the sitemap and robots.txt come ready, update themselves and go to Google and Bing at launch. See how I work.