Login to the Admin Panel.
Step 1: Go to Stores > Settings > Configurations
Step 2: Choose XML Sitemap in the Catalog tab.
Step 3: Open the Search Engine Submission Settings tab and set Yes in the Enable Submission to Robots.txt dropdown.
Step 4: Click on the Save Config button.
Robots.txt configuration in Magento 2
Step 1: Go to Content > Design > Configuration
Step 2: In the opened grid, find the line Global, click the Edit link.
Step 3: Open the Search Engine Robots tab.
Step 4: Now choose what Default Robots you need from the following dropdown:
- INDEX, FOLLOW: Search engines will regularly index your pages and check the changes.
- NOINDEX, FOLLOW: Search engine bot won’t index your website but it’ll still monitor changes.
- INDEX, NOFOLLOW: Search bot will index your store once but never came back to track the changes.
- NOINDEX, NOFOLLOW: This setting will hide your store from search engine bots.
Step 5: In the Edit custom instruction of robots.txt File line, you can write the needed custom instructions. Read on to find out more about the option.
Step 6: If you want to reset the default settings, you need to click the Reset To Default button, which will remove all your custom instructions.
Step 7: Lastly, Click on Save Configuration button.
Please contact us at manish@bay20.com or call us at +91-8800519180 for any support related to Magento 2. You can also visit the Magento2 development page to check the services we offer.
