Recount functions are incredibly poor on performance/resources and timeout
|Reported by:||_ck_||Owned by:||sambauers|
All of the recount functions need to be rewritten to work completely internal with mysql instead of external loops. There is a serious growing problem with query counts that can go into the MILLIONS.
If a forum has half a million topics, many of the recount functions first fetch ALL the topic id's and THEN loop through them, writing to meta for each one separately. Updatemeta uses at least 3 queries each.
So to use just one of the recount functions on a forum with a half million topics, you currently pull an array of the topic id's several megabytes in size into ram, then proceed to use 500,000 x 3 = 1.5 MILLION QUERIES to update the meta.
Then you repeat that for many of the recount functions. By the time you are done, you've used over 10 MILLION QUERIES and melted even the most robust of mysql servers into a puddle (but probably timed out most PHP setups well before that).
I have an internal recount function in Post Count Plus that does the recount completely in mysql and handles a half million posts in 6 seconds. The manual loop method can't even finish in the 30 second timeout on most PHP installs.
Change History (28)
- Milestone set to 1.1
- Owner set to sambauers
- Priority changed from normal to high
- Severity changed from normal to critical
- Status changed from new to assigned
- Keywords needs-patch added
- Milestone changed from 1.5 to 1.1
- Summary changed from recount functions are incredibly poor on performance/resources and timeout to Recount functions are incredibly poor on performance/resources and timeout
- Version changed from 1.0 to 1.0.2