Add site map and search engine robots

An attempt to generate and write the sitemap.xml file to the root directory results in the following error:

1
Please make sure that "/" is writable by the web-server.

With Magento Commerce Cloud, you can only write to specific directories, such as var, pub/media, pub/static, or app/etc. When you generate the sitemap.xml file using the Admin panel, you must specify the /media/ path.

For version 2.1.11 and later, you do not have to generate a robots.txt because it generates on demand and stores the contents in the database. It does not create a file, but you can view the content in your browser with the url: <domain.your.project>/robots.txt

This requires ECE-Tools version 2002.0.12 and later with an updated .magento.app.yaml file. See an example of these rules in the magento-cloud repository.

To generate a sitemap.xml file in version 2.1.11 and later:

1. Access the Magento Admin panel.
2. On the Marketing menu, click Site Map in the SEO & Search section.
3. In the Site Map view, click Add Sitemap.

4. In the New Site Map view, enter the following values:

   • Filename:sitemap.xml
   • Path:/media/
5. Click Save & Generate. The new site map becomes available in the Site Map grid.
6. Click the path in the Link for Google column.

To add content to the robots.txt file:

1. Access the Magento Admin panel.
2. On the Stores menu, click Configuration in the Settings section.
3. In the General list, click Design.
4. In the Search Engine Robots view, click Reset to Defaults. This populates the Edit custom instruction of robots.txt field.
5. Click Save Config.
6. Verify the file in your browser with the url: <domain.your.project>/robots.txt

Your robots.txt file appears in the <magento_root>/app/pub/media/ folder.

For versions earlier than 2.1.10, you must use the pub/media path. When you generate the robots.txt file using the Magento Admin Panel—Stores > Settings:Configuration > General:Design—it stores the contents in the database and creates an actual file.

Rewrite using Fastly VCL snippet

If you have different domains and you need separate site maps, you can create a VCL to route to the proper sitemap. Generate the sitemap.xml file in the Magento Admin panel as described above, then create a custom Fastly VCL snippet to manage the redirect. See Custom Fastly VCL snippets.

To use a Fastly VCL snippet for redirect:

Create a custom VCL snippet to rewrite the path for sitemap.xml to /media/sitemap.xml using the type and content key-value pairs.

1
2
3
4
5
6
7
{ 
  "name": "sitemapxml_rewrite",
  "dynamic": "0",
  "type": "recv",
  "priority": "90",
  "content": "if ( req.url.path ~ \"^/?sitemap.xml$\" ) { set req.url = \"/media/sitemap.xml\"; }" 
}