Still massive DB writes issues with merged js and css (w & w/o Object Caching)

Hi there,

after facing issues with enabled Object caching (discussed and mostly solved here: https://kriesi.at/support/topic/merged-styles-object-cache-using-redis/ ) we just received alerts some days ago, that our DB server was writing 300MB of Binary logs per Minute(!). We do have Object Caching DISABLED at the moment and were still able to reproduce the problems.

Checking the Enfold forums, there are several unsolved and open threads, mentioning issues with the merged file generation since february and massive binlog production:

https://kriesi.at/support/topic/dynamic_avia-making-way-too-many-calls/

I opened this post to re-raise attention to this topic, as we can confirm that there are still massive issues which are quite hard to detect on a regular website.
Checking Server analytics we can confirm massive database writes starting at the beginning of march when we updated Enfold from 4.6.3.1 to 4.7.3. These writes continued to grow over time leading to single options being ~30MB – 50MB for a high traffic site. And still >10MB for medium traffic sites.
Or in other numbers: The options aviaAsset_avia-footer-scripts and aviaAsset_avia-head-scripts contained arrays with around 30.000 to 50.000 entries (‘error-generating-file’). Which were growing with each page / backend call. And that particular high-traffic-site was just live for about 1 week.

Most people will not really notice these issues, as the DB grows slowly. But the loaded options will grow bigger and bigger and slow down the whole page as they are autoloaded. Leading to outages in the end when the database suddenly uses up all available space. In more advanced scenarios like ours, the binary logs of the database will rapidly fill up the disk.

We tried some of the suggested solutions in the other threads but ended up with a more radical approach as a workaround which actually disables your error handling with the file generation completely as well as the unique handling:

/*
 * Enfold Error from 4.7.0
 *
 * Enfold is writing in table options endless option_name aviaAsset_avia-[location]-scripts 'error-generating-file' ...
 * Only write the option, if there is no error.
 * Clean up existing mess in the database
 */
function custom_update_option_aviaAsset_avia_scripts($value, $old_value, $option) {

	if (is_array($value) && in_array('error-generating-file', $value, true)) {
		// Cleanup Enfolds massive entries, if existing
		return array_filter( $value,  function($arr_value) { return $arr_value !== 'error-generating-file'; });
	}
	return $value;

}
add_filter( 'pre_update_option_aviaAsset_avia-head-scripts', 'custom_update_option_aviaAsset_avia_scripts', 10,3 );
add_filter( 'pre_update_option_aviaAsset_avia-footer-scripts', 'custom_update_option_aviaAsset_avia_scripts', 10,3 );

/*
 * Do not allow Enfold to add a unique id to generated files. It does not work
 */
function custom_do_not_use_enfold_uniqid_for_generated_files($uniqid, $file_type, $data, $enqueued, $file_group_name, $conditions) {
	return "";
}
add_filter( 'avf_merged_files_unique_id', 'custom_do_not_use_enfold_uniqid_for_generated_files', 10, 6);

Instantly DB writes and binlog sizes dropped back to normal. This code also cleans up the mess in the options table and reduced DB size by several hundred MB.
On the disk, still files with the same hash, but different uniques reamains.

When using the above filters to disable the unique id as well as the storage of error information, all obvious issues are solved. But still, this also means that the error handling you intended is disabled which will lead to other problems we haven’t yet encountered.

We would suggest an approach based on file content hashes combined with some kind of caching based on the current approach using the plugin and theme names and versions.
The file content hash will allow you to only generate files when the actual content changes based on a compare to the hashes. In addition, there is no need to bust the cache anymore, as long as the file content remains unchanged, making the random unique not required anymore.

As hashing the file contents could be a performance bottleneck when used wrong and to frequently on larger files, it should run only when really required. In addition using “crc32” for hashing instead of md5 might also be an improvement to speed up things. As collisions could happen, it shouldn’t be a big issue due to the very low probability. And if they happen, just regenerate the affected files. This should be a total edge case anyway.
Regarding hashing performance: https://stackoverflow.com/a/3665527

To outline one very particular and easy to implement idea:
Enhance your current code to not use a random uniqueid, but use the file contents hash as such. In this case you have a first level caching with the hash over names and versions. Level2 cache (and uniqueid) will then be the file contents hash.
Big advantage: Both parts will be calculated and not randomly generated. This will reduce most of the issue the current implementation has. It will NOT rely on the database for the random unique and it will not tend to generate masses of files for unchanged contents. Because if names, versions and content are identical, then always the same file name will be used for the generated code.

What still needs some thoughts would be the part on what to store in the database / object cache to reduce file operations to a minimum. In an optimal world, you shouldn’t need to change anything there. After our experiences, you should absolutely drop the idea of saving “error logs” to an autoloaded option.
Write something to the PHP Error log, or write an option to display a nag to the admin and use the non merged files. This should be fine for most usecases in our oppinion.

We do hope, that this leads to an even better version of the performance features and hope you can follow up our findings and suggestions. If not, we are happy to share more details with you.

Have a great and sunny day,

Jan