- This topic has 9 replies, 5 voices, and was last updated 3 years, 6 months ago by .
- The forum ‘General Discussion’ is closed to new topics and replies.
These forums are for general discussion on WordPress and Genesis. Official support for StudioPress themes is offered exclusively at My StudioPress. Responses in this forum are not guaranteed. Please note that this forum will require a new username, separate from the one used for My.StudioPress.
It's been a few years since I looked at my robots.txt file, but recently I got a warning from Yoast saying I didn't have on, which is strange because I recall doing that.
Nevertheless, it's an easy fix. This is what I used to have in my robots.txt file:
Is this still correct or do I need to change something?
You shouldn't have a real robots.txt file unless you need to customise the default behaviour (which if you're asking about it, you likely don't).
If you delete any real robots.txt file, then WordPress will serve the default values automagically:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
My site https://turbolangs.com has no robots.txt file. It was built on Genesis + Wellness theme. It is up and running for some time but I presumed WordPress always create such file by default, my bad.
Anyway, my two other websites (both Genesis + StudioPress themes) have their own robots.txt file in place.
Why is that?
Thanks & Regards,
WordPress doesn't create them by default - they are virtual files. WordPress captures the request and sends back output that looks like a robots.txt file. If you check your hosting via FTP, you won't find them (normally).
The URL https://turbolangs.com/robots.txt seems to be working with something at least.
i need help.. i have a blog on education.. https://www.anicow.com - i am getting robots.txt error.. how to solve..
If I have a real estate website that is being served data from the MLS or Multiple Listing System, do I need to customize my robots.txt file? The only reason I ask is I'm pretty sure that Google has a crawl quota. Thanks!
Ok, sounds good. Thanks!
© 2023 WPEngine, Inc.