Sitemap and Robots.txt

What is a Sitemap?

A sitemap is an XML file that lists all the pages on your website to help search engines crawl and index them.

Sitemap Example

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url>
        <loc>https://example.com/</loc>
        <lastmod>2024-01-01</lastmod>
        <changefreq>daily</changefreq>
        <priority>1.0</priority>
    </url>
</urlset>

Generating Sitemap in ASP.NET Core

public class SitemapService
{
    public async Task<string> GenerateSitemapAsync()
    {
        var urls = new List<SitemapUrl>();
        urls.Add(new SitemapUrl
        {
            Url = "https://example.com",
            LastModified = DateTime.Now,
            ChangeFrequency = "daily",
            Priority = 1.0
        });
        return GenerateSitemapXml(urls);
    }
}

What is Robots.txt?

Robots.txt tells search engines which pages to crawl and which to avoid.

Robots.txt Example

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Sitemap: https://example.com/sitemap.xml