Crawling Errors 500

I am an SEO guy and am experiencing the same problem with one of my sites using Enfold. While the front end of the site looks gorgeous, Google Webmaster Tools is throwing up this error on the dashboard:

Last crawled: 7/17/15
First detected: 5/11/15
Googlebot couldn’t access the contents of this URL because the server had an internal error when trying to process the request. These errors tend to be with the server itself, not with the request.

As you can see this problem has been happening off and on for 2 months. I’ve marked it as fixed a number of times, assuming that it was a server problem, only to find it coming back. I have multiple sites on the same platform (WP Engine) which all use Enfold and this seems to be the only one showing this problem consistently over time. I logged a case with WP Engine (their support is as awesome as Enfold’s!) and they helped me fix the 403 on the framework folder I was having by adding this piece of code to the top of my .htaccess:

Options +Indexes
# or #
IndexIgnore *

As per: http://viralpatel.net/blogs/htaccess-directory-listing-enable-disable-allow-deny-prevent-htaccess-directory-listing/

Moving back to the 500 error – I continued to work with WP Engine and we found this error in the server log file, which points a problem with the header call on line 9…

[Fri Jul 24 18:30:26.399224 2015] [:error] [xxxxxx] [client ……………] PHP Fatal error: Call to undefined function get_header() in ………/xxxxxx/wp-content/themes/enfold/index.php on line 9

WP Engine asked me to contact my theme developer as they cannot see an issue with any other sites of mine with Enfold. Any thoughts at this point are appreciated as I don’t want my Webmaster Tools showing 500’s – and to answer the earlier question, you can no longer get away with hiding CSS files via robots.txt. Since Google upgraded their mobile friendly algorithm on April 21st, they want to be able to completely render your site, so hiding front end files with robots is now frowned on 100%.

Anything I can do to assist in figuring out this issue please let me know as I’m eager to sort this. I’m about to go thru all my Enfold sites with a fine tooth comb and see if any others are showing 500 and 403 with those 2 folders.

Thanks,
Kyle

PHP Fatal error: Call to undefined function get_header()

I will try this and post the results in a few days (maybe a week if I get distracted :P) but as I said above, Google needs access to crawl the CSS files in here so it can render the site (and test desktop and mobile versions with this tool:

https://www.google.com/webmasters/tools/mobile-friendly/

Please see the Yoast article on this subject (he knows more about SEO than me):

https://yoast.com/wordpress-robots-txt-example/

Last time I tried to Disallow this folder, I just moved the problem. No 500 error, but I started failing the mobile friendly test and getting ‘blocked resource’ issues in Webmaster Tools. I will wait and see if Google starts throwing errors at me and report back :D

Thanks for your help – I really hope this works as I just completed my inventory and almost all my Enfold sites have the same SEO issue in Google Webmaster Tools.

Kyle

Hey Josue – are you suggesting we tackle this from the cro.ma link?

First we’ll look at a coding solution for this.

in your index.php file right at the top you will find the following:

<?php get_header(); ?>
or

get_header();
You should replace it with the follwing:

<?php if (function_exists(‘get_header’)) {
get_header();
}else{
header(“Location: http://&#8221; . $_SERVER[‘HTTP_HOST’] . “”);
exit;
}; ?>

Hi
This is an issue that I’ve been going around in circles with my developer for some months now.

Has anyone managed to get consistently Zero issues on google webmaster tools using enfold (blocked resources, 500 errors on crawl, no crawl errors other than the usual fixable ones through redirects etc)?

Hope you don’t mind me asking, but is Kriesi support fully aware of google webmaster tools ? It’s just that I’ve followed every change there is from past forums and robot settings previously recommended do not work – I’ve tried the lot!

** I thought a separate article on setting robots would be very helpful – as previous references to yoast really don’t help, it’s theme- specific :-) ** All previous forum replies give different settings.

it really seems to be causing a lot of people a lot of headaches

Thanks guys

Richard

Hi Kylerollins

“Glad to hear I’m not the only one who has been hitting his head up against this… Its happening on all my Enfold sites and I’m constantly going back and forth between blocking robots.txt folders so they don’t throw up errors, then getting other errors because resources in those folders are blocked.”

Yes , exactly the same

I’ve allowed access to everything but get a 500 on
wp-content/themes/enfold/

I wonder Kriesi / Rikard if there is a generic solution that is updated from previous forums such as
https://kriesi.at/support/topic/issue-with-indexing-of-theme-on-google/
And this is definitely inaccurate advice now: https://kriesi.at/support/topic/what-can-i-exclude-in-robots-txt/

Google have changed so up to date advice on what to allow and disallow would be great – perhaps a cut and paste.

cheers

Richard