-
Posts
-
I am using multisite subdomain wordpress with robots.txt on root folder
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploadsBut i am still able to access paths like
https://site1.domain.com/wp-content/themes/theme/framework/images/icons/
I tried using googlebot as user agent and the directories are still accessibleA robots.txt does not prohibit access to a path. As long as google does not list your icon you shouldn’t worry about.tests simulating googlebot agents which is never the same as google.
Hi!
Thank you for using Enfold.
Directory blocking should be done in the .htaccess files, not the robot.txt file. It will only block or disallow search engine crawlers. And you should not disallow the listed directories because they contain important scripts and stylesheets.
Related thread: http://stackoverflow.com/questions/9282124/deny-direct-access-to-a-folder-and-file-by-htaccess
Best regards,
IsmaelHi!
I might have misunderstood the issue, regardless, you should not disallow those folders. Crawlers need those files (scripts and stylesheets) to render or read the site properly.
// https://yoast.com/wordpress-robots-txt-example/
// https://yoast.com/dont-block-your-css-and-js-files/Regards,
IsmaelOk whatever, I will remove those rules from robots.txt.
Can you please suggest me a solution to stop google from indexing those resources ?
- You must be logged in to reply to this topic.