How to Edit robots.txt in Magento 2 Admin?

Login to the Admin Panel.

Step 1: Go to Stores > Settings > Configurations

Step 2: Choose XML Sitemap in the Catalog tab.

Step 3: Open the Search Engine Submission Settings tab and set Yes in the Enable Submission to Robots.txt dropdown.

Step 4: Click on the Save Config button.

Robots.txt configuration in Magento 2

Step 1: Go to Content > Design > Configuration

Step 2: In the opened grid, find the line Global, click the Edit link.

Step 3: Open the Search Engine Robots tab.

Step 4: Now choose what Default Robots you need from the following dropdown:

  • INDEX, FOLLOW: Search engines will regularly index your pages and check the changes.
  • NOINDEX, FOLLOW: Search engine bot won’t index your website but it’ll still monitor changes.
  • INDEX, NOFOLLOW: Search bot will index your store once but never came back to track the changes.
  • NOINDEX, NOFOLLOW: This setting will hide your store from search engine bots.

Step 5: In the Edit custom instruction of robots.txt File line, you can write the needed custom instructions. Read on to find out more about the option.

Step 6: If you want to reset the default settings, you need to click the Reset To Default button, which will remove all your custom instructions.

Step 7: Lastly, Click on Save Configuration button.