Tagged: 

Viewing 8 posts - 1 through 8 (of 8 total)
  • Author
    Posts
  • August 15, 2018 at 12:43 am #997386
    WebDevDept

    I’m seeing a bunch of websites (even a brand new website below in private section) that are publishing tons of files from the wp-content/themes/enfold/framework/ directory into Google’s Index.

    These should not be in Google. So what’s new that happened that allows all these pages to be indexed in Google?

    And how can I fix and prevent it from happening again?

    August 15, 2018 at 6:25 am #997437
    Ismael

    Hey WebDevDept,

    Thank you for using Enfold.

    Have you tried to disallow the directory from being crawled in the robots.txt file?

    // https://yoast.com/wordpress-robots-txt-example/

    Best regards,
    Ismael

    August 15, 2018 at 6:09 pm #997633
    WebDevDept

    Hey Ismael.

    I have not, but I could. I’m just concerned it would affect the website. What are your thoughts?

    Also, this is a new issue, what changed in the last month or two?

    August 16, 2018 at 5:23 am #997754
    Ismael

    Hi,

    Thanks for the update.

    The framework directory doesn’t contain the html, css and js files necessary to render the page so the crawlers doesn’t need to crawl it to understand the content of your site. You can block the crawlers from accessing that particular folder without affecting the site’s crawlability.

    After disallowing Google from crawling that directory, go to the Google Search Console > Crawl > Fetch as Google and check if the site looks the same with that directory blocked.

    Best regards,
    Ismael

    August 16, 2018 at 6:06 pm #998056
    WebDevDept

    Hey Ismael.

    I can add the disallow to robots, no problem. But that also publicly displays the name of our theme which I’d rather not do. But if there’s no other way, I’ll do it.

    I’m aware anyone can figure out the theme being used, but I don’t want to make it so easy, you know?

    And I’m still curious if you know why this is something that only recently started showing up the past couple months.

    What’s changed recently?

    August 16, 2018 at 9:05 pm #998155
    Dude

    Hi,

    But that also publicly displays the name of our theme which I’d rather not do.

    You could add following rule to the robots.txt:

    
wp-content/themes/.*

    It will exclude all theme folders and you don’t need to reveal the theme folder name (i.e. “enfold”).

    And I’m still curious if you know why this is something that only recently started showing up the past couple months.

    Google indexes all urls, including theme resources like css or js scripts. In fact many experts recommend “don’t block the access to theme files” (i.e. Yoast: https://yoast.com/dont-block-css-and-js-files/ ).

    Best regards,
    Dude

    August 16, 2018 at 9:13 pm #998157
    WebDevDept

    I’m still unsure as to what to do. Yoast says don’t block theme files in robots.txt and you’re saying to block theme files.

    August 16, 2018 at 9:26 pm #998165
    Dude

    Hi,

    No I didn’t – I’d also recommend to allow Google to index the theme files. However if you want to block the access you can use the rule I posted in my last post.

    Best regards,
    Dude

  • Author
    Posts
Viewing 8 posts - 1 through 8 (of 8 total)
  • You must be logged in to reply to this topic.