What should I put in my robots.txt file?

This topic is: not resolved

Tagged: robots.txt

This topic has 9 replies, 5 voices, and was last updated 3 years, 6 months ago by dtcpower.

Viewing 10 posts - 1 through 10 (of 10 total)

Author

Posts
April 1, 2018 at 11:29 am #218529

RavenManiac
Participant

It's been a few years since I looked at my robots.txt file, but recently I got a warning from Yoast saying I didn't have on, which is strange because I recall doing that.

Nevertheless, it's an easy fix. This is what I used to have in my robots.txt file:
User-agent: *
Disallow: /wp-admin/

Is this still correct or do I need to change something?

Thanks!

April 29, 2018 at 8:28 am #219401
Gary Jones
Member
You shouldn't have a real robots.txt file unless you need to customise the default behaviour (which if you're asking about it, you likely don't).

If you delete any real robots.txt file, then WordPress will serve the default values automagically:
```
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
```
WordPress Engineer, and key contributor the Genesis Framework | @GaryJ
May 8, 2018 at 6:14 am #219636

Fabio Websmith
Participant

Hy Gary,
My site https://turbolangs.com has no robots.txt file. It was built on Genesis + Wellness theme. It is up and running for some time but I presumed WordPress always create such file by default, my bad.
Anyway, my two other websites (both Genesis + StudioPress themes) have their own robots.txt file in place.
Why is that?
Thanks & Regards,
Fabio

May 8, 2018 at 7:37 am #219638

Gary Jones
Member

WordPress doesn't create them by default - they are virtual files. WordPress captures the request and sends back output that looks like a robots.txt file. If you check your hosting via FTP, you won't find them (normally).

The URL https://turbolangs.com/robots.txt seems to be working with something at least.

WordPress Engineer, and key contributor the Genesis Framework | @GaryJ

May 8, 2018 at 8:31 am #219639

anicow247
Member

i need help.. i have a blog on education.. https://www.anicow.com - i am getting robots.txt error.. how to solve..

May 8, 2018 at 9:04 am #219641

Gary Jones
Member

https://www.anicow.com/robots.txt is working fine.

WordPress Engineer, and key contributor the Genesis Framework | @GaryJ

July 21, 2019 at 2:57 pm #492458

RavenManiac
Participant

If I have a real estate website that is being served data from the MLS or Multiple Listing System, do I need to customize my robots.txt file? The only reason I ask is I'm pretty sure that Google has a crawl quota. Thanks!

July 22, 2019 at 3:13 am #492462

Gary Jones
Member

Google does have a crawl quota, so you could suggest which areas of the site should not be crawled. Do you _need_ to? No.

WordPress Engineer, and key contributor the Genesis Framework | @GaryJ

July 22, 2019 at 11:01 am #492481

RavenManiac
Participant

Ok, sounds good. Thanks!

July 24, 2019 at 9:10 am #492538

dtcpower
Member

i have blog and is working fine.http://phtech.vn/

may phat dien doosan
Author

Posts