Do I need a robots.txt file?

[lauren107]

Hi everyone! I'm a new user and learning quite a bit as each day goes by! I've managed to get my site completely set up on my own, and that feels like no small thing!

I have a few technical questions to ask of the community, and for simplicity I'll make them separate posts, so as not to confuse the answers.

Firstly, I am wondering if I need a robots.txt file. I know what they do, but I have read posts here that seem to indicate there could be problems with the site displaying for mobile and other devices if I include things in there that should not be there. Given I don't know what those things even are.... I'm wondering, do I even need one? I am pretty sure WordPress does a good job of disallowing access to critical areas.

Is the file more for what I don't want indexed and showing up in the search engines? If so, can anyone possibly point me to a good one to use?

Thank you!

[Christoph]

Hi,

you don't necessarily need to create a robots.txt file.
If you are using a SEO plugin like Yoast SEO, it can/will create a robots.txt if you don't want certain posts or pages indexed.
Here is more information about it: http://www.robotstxt.org/robotstxt.html

---

https://www.christophherr.com | Genesis Customizations | Buy me a coffee

[lauren107]

Thanks, Christoph.

I am using Yoast. I check my root, and no robots.txt file was there, so perhaps I just don't know how to "tell" Yoast to do it.

I checked the link you gave me, and also found this: https://support.google.com/webmasters/answer/6062608?rd=1.

I ended up creating my robots.txt file with Google Webmaster console, as it allowed me to text for errors.

Thanks again for the help!

[Author]

Posts