-
AuthorPosts
-
November 19, 2024 at 7:06 pm #1471688
Hi guys,
I have discovered a huge problem with the way avia queries run on woocommerce sites.
We noticed that some of our smaller much lower traffic sites were suddenly causing mysql memory use to climb out of control – Where the longer the server is running the more memory is used.
We are talking going from about 8gb usage to 80gb usage and climbing as more and more bots get stuck crawling the queries
Bots are getting stuck following the queries in the dropdown like:
?avia_extended_shop_select=yes&product_order=relevance
They are crawling every single query string on every single paginated page.This is causing excessive memory usage.
Anything we can do to stop crawlers hitting these pages?
I can see rel=nofollow in the links but this doesnt seem to be stopping bots.In comparison the default woocommerce filters are not actual hyperlinks they are a form submission. Therefore bots dont crawl them.
This is a serious problem.
November 20, 2024 at 10:04 am #1471720Following on from this
I can 100% confirm this was the cause of our server issues.
Essentially ALL Enfold websites with WooCommerce installed will gradually use up the ram and see mysql memory usage rise until the server crashes.
The bots getting stuck crawling all of the ?avia_extended_shop_select=yes queries includes:
Even with the rel-nofollow:
Bing bot
Google bot
A majority of major search engines
AI crawlers – New aggressive bots that ignore all instructionsSee screenshot of the before and after. Our Ram use fell from almost 90GB of ram used to 8GB used on average.
Physical Ram usage screenshot
https://www.dropbox.com/scl/fi/tw8p399gfdr9cclorc5j9/ramuse-screenshot.PNG?rlkey=d9ty4xvui4w3m2ycb9b9ez94s&st=28aameqn&dl=0
Mysql memory usage screenshot
https://www.dropbox.com/scl/fi/iixhdd5vpe7253tbavbxf/mysql-memory-usage.PNG?rlkey=15061shf0haiu3d36cdy0be3t&st=ep9fncx5&dl=0SOLUTION
I propose that Enfold removes the custom sort by options and reinstates the default Woocommerce ones.
The defaults use form fields and JS so there are no a href links in the default woo sort by dropdown. Bots cannot follow these links because there are no urls in the HTML.In your child theme functions.php add the following to remove the enfold filters and reinstate the woo ones.
// remove the enfold sort by filters function avia_woocommerce_frontend_search_params() { return; } // reinstate the woo default sort by filters add_action( 'woocommerce_before_shop_loop', 'woocommerce_catalog_ordering', 20 );
Add the following CSS to quick CSS and tweak depending on your sidebar position:
.sort-param-count { display:none; } .product-sorting { padding-top:0px; } div .product-sorting ul, div .product-sorting li { font-size: 16px; } div .product-sorting ul { width: 200px; } .main_color .sort-param a { color: #000000; } .sort-param-sort a, ul.sort-param-order { border: 1px; border-color: #969696; border-style: solid; } @media only screen and (max-width: 767px) { .responsive #top .woocommerce-ordering { position: relative; float: left; clear: both; margin: 0; padding-bottom: 25px; padding-top: 15px; top: 0px; } } @media only screen and (min-width: 768px) { .responsive #top .woocommerce-ordering { position: relative; float: left; clear: both; margin: 0; padding-bottom: 25px; padding-top: 15px; top: 0px; } } #top.woocommerce-page .woocommerce-ordering select { width: 100%; font-size:16px; }
Install the Redirection Plugin:
Add the following RegEX expression to redirect the queries so that the URL redirects BEFORE the query runs on the DB
Source URL: ^/(.*?)/\?avia_extended_shop_select=.*
Enable: Ignore Case, Regex and Ignore Slash
Target URL: /$1/
Hit Save
This will redirect any attempt to crawl the enfold filters back to the current category -
AuthorPosts
- You must be logged in to reply to this topic.