Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • July 28, 2015 at 7:26 pm #479879
    kenwae

    I received a notice from Google stating that they could not access js and css files. I have had a disallow wp-includes in the robots.txt file for years, without problem; but, since the js and css files are in wp-includes, it appears now to be a problem. Apparently, Google requires access to the js and css files. I removed the disallow wp-includes, which should solve that problem, but opens up a lot of other files for Google to index which may not be so desirable. Are you aware of this issue??

    And, would it suffice to do something like this:
    User Agent*
    Disallow/ wp-includes/
    Allow/ wp-includes/js
    Allow/ wp-includes/css

    Thanks for your help…..

    July 29, 2015 at 3:33 pm #480292
    Elliott

    Hi kenwae!

    It doesn’t sound like a theme issue. Try deleting your robots.txt file and then have Google crawl your site again in webmaster tools.

    Also be sure to deactivate all plugins while testing.

    Cheers!
    Elliott

    July 31, 2015 at 1:04 am #481018
    kenwae

    Thanks for the response; no, it is not a theme issue; from what I am reading on the net, this is a fairly upcoming issue across the board will all WordPress sites; whereas disallowing /wp-content in robots.txt has been a common practice, it appears now to be considered a bad idea. Same with disallowing /wp-admin. So, I’ve removed those disallows and we’ll see what happens in terms of Googlebot’s indexing of irrelevant stuff, which tends to contribute to crawl errors. But, in a general sense, perhaps users should know that disallowing /wp-content is now considered bad practice….

    And, you can close this now….

    • This reply was modified 9 years, 4 months ago by kenwae.
  • Author
    Posts
Viewing 3 posts - 1 through 3 (of 3 total)
  • The topic ‘Problems with Googlebot and robots.txt disallow/ wp-includes’ is closed to new replies.