Course: Digital Marketing
Subject: Search Engine Optimization (SEO)
Topic: XML Sitemaps and Robots.txt
Course Link: JustAcademy Digital Marketing Course
Demo Link: Register for Digital Marketing Course Demo
XML sitemaps and robots.txt are two important components of technical SEO. They help website owners communicate with search engine crawlers and manage how website URLs are discovered and accessed.
An XML sitemap provides a list of important URLs that a website wants search engines to discover and consider for crawling. A robots.txt file provides rules that tell compliant search engine crawlers which paths they may or may not request.
Although both are related to crawling, they serve different purposes. An XML sitemap helps with URL discovery, while robots.txt manages crawler access. Neither file guarantees that a page will be indexed or appear in search results.
Imagine a digital marketing website that contains 200 pages, including course pages, blog articles, service pages, and contact information. A sitemap can help search engines discover important URLs, while robots.txt can discourage crawling of selected administrative or unnecessary paths.
Correct configuration helps search engines access the intended content efficiently and reduces avoidable technical SEO problems.
After studying this topic, learners will be able to:
An XML sitemap is a structured XML file that lists URLs a website owner wants search engines to discover and consider for crawling. It may also provide additional information, such as the date when a page was significantly updated.
XML stands for Extensible Markup Language. It uses structured tags to organize information in a format that machines can process.
https://www.example.com/sitemap.xmlThis is a common sitemap location. A website may use a different location depending on its platform and configuration.
Important: Submitting a sitemap is a hint to search engines. It does not guarantee that every listed URL will be crawled, indexed, or ranked.
An XML sitemap uses a defined structure. A standard sitemap generally begins with an XML declaration and contains a root element named urlset. Each page is represented by a url element containing a required loc element.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/</loc>
</url>
<url>
<loc>https://www.example.com/courses/</loc>
</url>
<url>
<loc>https://www.example.com/blog/</loc>
</url>
</urlset>| Element | Purpose | Example |
|---|---|---|
<?xml version="1.0"?> |
Declares the XML version. | XML declaration |
encoding="UTF-8" |
Specifies the character encoding. | UTF-8 |
<urlset> |
Root element for a standard URL sitemap. | Contains URL entries. |
xmlns |
Declares the sitemap XML namespace. | Sitemap protocol namespace |
<url> |
Groups information for one URL. | One webpage entry |
<loc> |
Specifies the absolute URL of a page. | https://www.example.com/ |
<lastmod> |
Optionally identifies the last significant modification date. | 2026-10-10 |
The loc element specifies the complete URL of a page. It is required for each URL entry in a standard XML sitemap.
<loc>https://www.example.com/digital-marketing/</loc>Use absolute URLs, including the protocol and hostname, rather than relative paths.
The lastmod element indicates when a page was last significantly modified. Use it only when the date or timestamp is accurate and can be maintained reliably.
<lastmod>2026-10-10</lastmod>Significant changes can include meaningful updates to the main content, structured data, or links. A minor change, such as updating a copyright year, should not automatically be treated as a significant content update.
The changefreq element was historically used to suggest how frequently a page might change.
<changefreq>weekly</changefreq>Google ignores the changefreq value. It should not be relied on to control Google's crawling schedule.
The priority element was historically used to express the relative importance of a URL within a website.
<priority>0.8</priority>Google ignores the priority value. Adding priority values does not guarantee better rankings or more frequent crawling.
| Tag | Status | SEO Use |
|---|---|---|
loc |
Required for each URL entry | Identifies the page URL. |
lastmod |
Optional | Provides accurate modification information. |
changefreq |
Optional in the protocol | Ignored by Google. |
priority |
Optional in the protocol | Ignored by Google. |
An XML sitemap is a machine-readable file containing URLs and optional metadata. It is commonly used for general website pages.
Example: sitemap.xml
An HTML sitemap is a webpage containing links to important sections of a website. It is primarily intended to help visitors navigate the website.
Example: https://www.example.com/sitemap/
An image sitemap or image sitemap extension can provide additional information about images associated with website pages. It can be useful for websites where image discovery is important.
A video sitemap can provide information about video content, such as its location and relevant metadata, where supported and appropriate.
A news sitemap is designed for eligible news content and can provide information about recently published articles. It must follow the relevant search engine requirements.
A sitemap index is a file that lists multiple sitemap files. It is useful for large websites that divide URLs into separate sitemaps.
| Type | Main Purpose | Typical Use |
|---|---|---|
| XML Sitemap | Communicate important URLs to search engines. | Blogs, business websites, e-commerce websites. |
| HTML Sitemap | Help visitors navigate the site. | Websites with many sections. |
| Image Sitemap | Provide image-related discovery information. | Photography and image-heavy websites. |
| Video Sitemap | Provide video-related metadata. | Video publishers and educational websites. |
| News Sitemap | Support discovery of eligible news content. | Qualifying news publishers. |
| Sitemap Index | List multiple sitemap files. | Large websites with many URLs. |
Under the sitemap protocol, a single sitemap file is limited to 50,000 URLs or 50 MB when uncompressed, whichever limit is reached first.
If a website exceeds these limits, it should split its URLs into multiple sitemap files and can create a sitemap index to list them.
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://www.example.com/sitemap-pages.xml</loc>
</sitemap>
<sitemap>
<loc>https://www.example.com/sitemap-posts.xml</loc>
</sitemap>
<sitemap>
<loc>https://www.example.com/sitemap-products.xml</loc>
</sitemap>
</sitemapindex>In this example, the sitemap index references three separate sitemap files. Each referenced sitemap must be accessible and correctly formatted.
An XML sitemap should primarily contain the preferred URLs that a website wants search engines to discover and consider for indexing.
Keep sitemap entries consistent with the website's canonical tags, internal links, and redirect configuration. A sitemap should not be used as a list of every URL that exists on a website.
For a small website, a basic XML sitemap can be created with a text editor.
urlset element.url element for each intended page.loc tag to enter the complete URL.lastmod information when available.sitemap.xml.For websites with many URLs, automatic generation is usually easier to maintain.
Some SEO crawling tools can discover website URLs and assist with sitemap creation. Before publishing the output, check that it contains only appropriate, canonical URLs and that the generated file follows the sitemap protocol.
Google Search Console helps website owners monitor search visibility and review sitemap processing information for a verified website property.
Sitemap URL:
https://www.example.com/sitemap.xmlSubmitting a sitemap does not force Google to index every listed page. Use the URL Inspection tool to investigate individual URLs when necessary.
The robots.txt file is a plain-text file placed at the root of a website hostname. It contains rules that tell compliant crawlers which URLs or paths they may request.
It is part of the Robots Exclusion Protocol and is primarily used to manage crawler access and avoid unnecessary crawling.
https://www.example.com/robots.txtFor a website using the www hostname, the robots.txt file is normally located at the root of that hostname. Different hostnames, such as a subdomain, may have their own robots.txt files.
A robots.txt file consists of groups of rules. Each group identifies a crawler using User-agent and may include one or more Disallow or Allow directives.
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /
Sitemap: https://www.example.com/sitemap.xmlImportant: Robots.txt is public and is not a security mechanism. Do not place passwords, confidential paths, or sensitive information in the file.
The User-agent directive identifies the crawler for which the group of rules applies.
User-agent: *The asterisk represents a general group for crawlers that match the rule.
A group can also target a specific crawler:
User-agent: Googlebot
Disallow: /temporary/This rule asks Googlebot not to crawl paths under /temporary/. Always check the official documentation for the crawler being targeted.
The Disallow directive identifies a path that the matching crawler should not request.
User-agent: *
Disallow: /admin/This is useful when the specified paths should not be crawled by compliant crawlers. It does not provide access control and does not reliably prevent URLs from appearing in search results.
The Allow directive permits crawling of a path within a group of rules, including situations where a more general disallow rule would otherwise apply. Specificity and rule handling depend on the crawler.
User-agent: *
Disallow: /resources/
Allow: /resources/public-guide.htmlThis example requests that crawlers avoid the general resources directory while allowing the specified guide URL where supported.
The Sitemap directive can identify an XML sitemap for search engine discovery.
Sitemap: https://www.example.com/sitemap.xmlIt can be placed in robots.txt outside the user-agent rule groups. An absolute sitemap URL should be used.
Some crawlers support a Crawl-delay directive, but support is not universal and Google does not support it in Googlebot robots.txt rules. Do not assume that adding this directive will control Google's crawling rate.
User-agent: *
Disallow:This group contains an empty disallow value, meaning that no paths are disallowed by this group.
User-agent: *
Disallow: /admin/Use this type of rule only when crawling the directory should be restricted. If it contains sensitive information, use authentication and access controls as well.
User-agent: *
Disallow: /search/This can help reduce crawling of internal search-result URLs if that is appropriate for the website's technical strategy.
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xmlThis allows crawling under the general group and provides a sitemap location.
User-agent: *
Disallow: /*.pdf$This pattern asks matching crawlers that support the syntax not to crawl URLs ending in .pdf. Test such rules carefully because blocking a file type may interfere with the website's search visibility goals.
User-agent: *
Disallow: /temporary/This is an example of a path-specific crawling restriction. Remove the restriction if the content should be crawled and discovered.
| Basis | XML Sitemap | Robots.txt |
|---|---|---|
| File format | XML | Plain text |
| Primary purpose | Suggests URLs for discovery and crawling. | Provides crawling rules for compliant crawlers. |
| Common location | /sitemap.xml | /robots.txt |
| Can list URLs | Yes, using sitemap URL entries. | Contains rules and can reference sitemap files. |
| Can control crawling | No | Yes, for compliant crawlers that follow the rules. |
| Guarantees indexing | No | No |
| Prevents indexing by itself | No | No |
| Common use | Help search engines discover important pages. | Restrict crawling of selected paths. |
One of the most important technical SEO concepts is understanding the difference between blocking crawling and preventing indexing.
Robots.txt tells compliant crawlers which paths they should not request. It does not reliably prevent a blocked URL from appearing in search results.
A noindex meta tag tells supported search engines not to index a page when they can crawl it and process the directive.
<meta name="robots" content="noindex">A website can also send an X-Robots-Tag HTTP response header to communicate indexing instructions, including for certain non-HTML resources.
X-Robots-Tag: noindex| Feature | Robots.txt | Noindex |
|---|---|---|
| Main function | Controls crawling requests. | Requests exclusion from indexing. |
| Requires page access to read HTML directive | No | Yes, for a meta tag. |
| Can a blocked URL still appear in search? | Yes, in some cases. | It can be excluded after the directive is crawled and processed. |
| Best use | Managing crawler access. | Preventing an accessible page from being indexed. |
Important: If a page is blocked in robots.txt, search engines may not be able to crawl it to see a noindex directive. When the goal is to remove a page from search results, ensure the crawler can access the directive or protect the page using authentication, depending on the requirement.
Robots.txt should normally be located in the root directory of the relevant hostname.
https://www.example.com/robots.txthttps://www.example.com/blog/robots.txtThe second URL is not the root robots.txt location for the hostname. A file in a subdirectory does not replace the root robots.txt file.
Different hostnames may have separate robots.txt files. For example:
https://example.com/robots.txthttps://www.example.com/robots.txthttps://shop.example.com/robots.txtConfigure the appropriate file for each hostname that requires crawler rules.
Missing closing tags, incorrect nesting, or malformed XML can prevent a sitemap from being processed correctly.
Solution: Validate the XML syntax and correct malformed elements.
A sitemap may contain URLs that do not exist or use the wrong hostname or protocol.
Solution: Verify every URL and use the intended absolute URL.
Including URLs that redirect instead of their preferred final destinations can create unnecessary inconsistencies.
Solution: Update the sitemap to list the intended canonical destination URLs.
A sitemap may contain pages that the website intentionally excludes from indexing.
Solution: Remove those pages from the sitemap unless the indexing strategy has changed.
Old URLs may remain after pages have been removed or moved.
Solution: Update the sitemap generation process and remove obsolete URLs.
Unreliable modification dates can reduce the usefulness of sitemap metadata.
Solution: Use accurate timestamps based on significant content changes.
A sitemap may exceed the supported size or URL limits.
Solution: Split it into multiple sitemap files and create a sitemap index if needed.
User-agent: *
Disallow: /This rule requests that compliant crawlers avoid crawling the entire site. It can seriously interfere with search visibility if used on a production website unintentionally.
Solution: Remove or revise the rule if the website should be crawlable.
Search engines may need access to important resources to render and understand pages properly.
Solution: Avoid blocking resources that are necessary for understanding important content unless there is a specific reason.
Robots.txt is publicly accessible and does not secure files or prevent all crawlers from requesting blocked paths.
Solution: Use authentication, authorization, or other appropriate access controls for private content.
A robots.txt file placed in the wrong directory may not apply to the intended hostname.
Solution: Publish it at the root of the hostname.
Overlapping rules can create unintended crawler behavior.
Solution: Review rule groups, path matching, and precedence. Test the rules before deployment.
Do not assume that every crawler interprets every rule identically. The robots.txt standard and crawler documentation should guide the implementation.
XML sitemaps and robots.txt can complement each other when configured correctly.
robots.txt:
User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Sitemap: https://www.example.com/sitemap.xmlsitemap.xml:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/</loc>
</url>
<url>
<loc>https://www.example.com/courses/digital-marketing/</loc>
</url>
<url>
<loc>https://www.example.com/blog/technical-seo/</loc>
</url>
</urlset>This hypothetical configuration lists the homepage, course page, and article in the sitemap while requesting that compliant crawlers avoid the administrative and internal-search paths. Always verify that the actual URLs and restrictions match the website's needs.
| Tool | Purpose | How It Helps |
|---|---|---|
| Google Search Console | Sitemap submission and indexing diagnostics. | Shows sitemap processing information and supports URL inspection. |
| Google robots.txt documentation | Understand robots.txt behavior. | Explains syntax, limitations, and appropriate use. |
| Google sitemap documentation | Build and submit sitemaps. | Explains sitemap formats and best practices. |
| Screaming Frog SEO Spider | Website crawling and auditing. | Helps find URL, status code, canonical, and internal-link issues. |
| XML-Sitemaps.com | Sitemap generation. | Can help generate a sitemap for eligible websites within its available limits. |
Project Objective: Prepare a basic sitemap and robots.txt configuration for a sample digital marketing training website.
Assume the website has the following pages:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/</loc>
</url>
<url>
<loc>https://www.example.com/courses/digital-marketing/</loc>
</url>
<url>
<loc>https://www.example.com/blog/seo-training-guide/</loc>
</url>
<url>
<loc>https://www.example.com/blog/content-marketing/</loc>
</url>
<url>
<loc>https://www.example.com/contact/</loc>
</url>
</urlset>These are sample URLs for learning. Replace them with real, accessible, canonical URLs before using the file on an actual website.
User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Sitemap: https://www.example.com/sitemap.xmlThis example requests that compliant crawlers avoid the selected paths and provides the sitemap location.
| Audit Item | What to Check | Status |
|---|---|---|
| XML sitemap exists | Confirm that the sitemap is published and accessible. | Pending |
| Sitemap syntax | Check XML validity and namespace. | Pending |
| URL accuracy | Verify absolute URLs and intended canonical versions. | Pending |
| Redirected URLs | Remove redirecting URLs from the sitemap where appropriate. | Pending |
| Noindex URLs | Ensure intentionally excluded pages are not listed unnecessarily. | Pending |
| Robots.txt exists | Confirm the file is available at the hostname root. | Pending |
| Important pages crawlable | Check that intended public pages are not blocked accidentally. | Pending |
| Sitemap reference | Verify the sitemap URL in robots.txt if used. | Pending |
| Search Console | Review sitemap processing and URL inspection results. | Pending |
| Ongoing maintenance | Ensure sitemap and rules remain correct after website changes. | Pending |
Answer: An XML sitemap is a structured file that lists URLs a website wants search engines to discover and consider for crawling.
Answer: Robots.txt is a plain-text file that provides crawling instructions to compliant search engine crawlers.
Answer: An XML sitemap helps communicate important URLs for discovery, whereas robots.txt specifies which paths compliant crawlers should or should not request.
Answer: A common location is the website root, such as https://www.example.com/sitemap.xml. Other locations are possible when configured appropriately.
Answer: It should normally be located at the root of the relevant hostname, such as https://www.example.com/robots.txt.
Answer: The loc tag specifies the absolute URL of a page in an XML sitemap.
Answer: It communicates when a page was last significantly modified, provided the information is accurate.
Answer: Google ignores the changefreq and priority values in XML sitemaps.
Answer: It identifies the crawler to which a group of robots.txt rules applies.
Answer: It requests that the matching compliant crawler not access the specified path.
Answer: It allows crawling of a specified path where applicable, including exceptions to broader restrictions when supported by the crawler.
Answer: Not reliably. A blocked URL can still appear in search results if discovered elsewhere. Use an appropriate noindex directive or access restriction when the objective is to prevent indexing or public access.
Answer: Open Google Search Console, select the verified website property, navigate to the Sitemaps report, submit the sitemap URL, and monitor processing results.
Answer: Search engines may encounter errors or waste resources processing unsuitable URLs. Review the sitemap and remove or correct URLs that should not be included.
Answer: A sitemap index is an XML file that lists multiple sitemap files, helping large websites manage their URLs in separate groups.
Answer: A single sitemap is limited to 50,000 URLs or 50 MB uncompressed, whichever comes first.
Answer: No. Private pages should be protected with authentication and suitable access controls. Robots.txt is publicly accessible and does not provide security.
Answer: A sitemap helps with URL discovery, but submitting one does not directly guarantee higher rankings. Search visibility depends on many factors.
Answer: Use the Sitemaps report in Google Search Console and review any reported errors or processing information.
Answer: Incorrect rules may unintentionally block important pages or resources, interfering with crawling and search visibility.
Complete the following exercises to strengthen your understanding of XML sitemaps and robots.txt.
/sitemap.xml./robots.txt at the hostname root.XML sitemaps and robots.txt are essential concepts in technical SEO. XML sitemaps help search engines discover important URLs, while robots.txt communicates which paths compliant crawlers should or should not request. Correct configuration can make website crawling more efficient and help search engines discover the content that matters.
To manage these files effectively, website owners should use valid XML syntax, include preferred URLs, maintain accurate modification dates, avoid accidental crawling restrictions, and monitor results through Google Search Console. They should also understand the distinction between crawling and indexing so that robots.txt is not misused as a substitute for noindex or website security.
Regular technical SEO audits help identify outdated sitemap entries, broken URLs, conflicting rules, and indexing problems. By combining well-maintained sitemaps, carefully tested robots.txt rules, useful internal links, and consistent canonical signals, digital marketers can support better website accessibility and search engine understanding.
Learn SEO fundamentals, technical SEO, keyword research, content optimization, website analysis, and other important digital marketing skills through structured training.
Explore the Course: JustAcademy Digital Marketing Course
Register for a Demo: Register for Digital Marketing Course Demo
