Very long load times when posting replies if topic has many subscribers

[rossagrant]

I’m starting to have some serious load time issues posting replies when a topic has more than a few subscribers.

I have an introduction topic for example.

It has over 70 voices, but I'm not too sure about the number subscribing. (Can I see this anywhere?)

When I post a reply now, it must take 10 seconds plus to actually be assigned a post number in the URL.

I actually receive the email notification quicker than the post actually bring pushed to the forum and appearing on the front end.

This is only going to grow exponentially.

I’m not a coder so this might be hard work, but is there any way we could get the reply to post THEN queue the emails?

What do you reckon JJ? Do you see this as an issue? Perhaps we could set individual topics to not allow subscribers also. Maybe an admin option?

Would love any suggestions!

Thanks!

[johnjamesjacoby]

There are two things I can imagine that would cause this:

• Having many, many users. Enough to make consecutive calls to get_userdata() slow.
• A slow mail server, making calls to wp_mail() slow.

This might be improved with 1 query to get every subscriber email address, rather than querying each user's complete dataset. The reason it does what it does now is to take advantage of any caching that get_userdata() may provide, rather than query all rows directly.

Since you're the first person seeing this, you're kind of the expert. :) Are you able to run some tests and maybe examine some logs to see what might be causing this?

[rossagrant]

Hey JJ,

Okay, I've looked into this a bit further to try to test what could be causing this.

From my email logs I can see that out of the 70 voices in the forum topic, 20 of them are subscribed to the topic for update notifications, so 20 emails are being generated each time someone posts.

I too thought I may just have a very slow mail server, or some server settings slowing this down.

I have just moved my entire mail setup to Google Apps and now run all mail through SMTP to take away any mail from my actual server.

I still get the same slow load times and again the notification email is hitting my inbox before the post is showing as posted in the front end of the site.

I'm not sure if this could be a further server limitation somewhere that is causing a bottleneck.

Any suggestions on what I could ask my host to check?

[rossagrant]

Okay, I've had a lengthy chat with my hosting company and their lead analyst.

He's taken a look at things and whilst there are things that he suggested could be done here in terms of the plugin itself (queuing the mail operation to decouple it from live script runtime - although he suggested this might not be possible on all servers so wouldn't be that viable)

It looks like a large proportion of the issue can be explained through my current server setup.

I'm moving onto a dedicated server now with PHP opcode caching to address the compilation of PHP during runtime. DB query caching to speed up the DB access. And we'll fine tune DB parameters to run OK given the concurrency available (as CPU cores).

I think a lot of this issue has been due to running the site on a semi-dedicated server using some shared resources.

This will be done in the next 10 days and I'll report back to tell you the difference.

JJ, if you think it's viable for future develpment of BBPress to try a version that uses 1 query to get every subscriber email address, rather than querying each user's complete dataset AND it isn't a total pain in the backside to create, I'll gladly test it all to see if it improves things for folk in my situation before I move my setup over.

I'm not sure if there can be a process here that posts the reply, then queues the mail AFTER it has been posted - obviously you guys are the experts, I'm just a user, so I'm not too hot on that kind of thing.

I'll keep you posted in another update. Cheers for your time so far fellas!

[johnjamesjacoby]

It's probably worth it, yes. My fear is that on large users tables (millions of rows) that doing an IN query on user ID's will lock the table. Would be great to do some stress testing here once you're able to write up a test case.

[rossagrant]

Hey JJ, just wanted to drop back in here.

I've duplicated my live site on a brand new dedicated server and the behaviour I was seeing on my shared server is still there.

I've been doing some tests with a server analyst from my host and his initial thoughts are that WP or BBPress, BP is waiting for a 'true' return on mail confirmation before it's actually allowing any php to continue/ refresh the page to let the user see his post.

If I open up a new browser window and refresh the activity stream for example - AS THE OTHER WINDOW IS STILL PROCESSING THE POST - I can see that it has actually ALREADY been posted.

This means the post is being written to the DB pretty quickly and mail is starting to be sent with the topic reply subscription notifications, but it seems that the page can't refresh for the poster until the mail has been sent and some kind of confirmation of that has been received.

It makes no difference whether I send the email locally or through SMTP (Google Apps), so it's not the mail server being slow.

The topic that this is worst on still only has about 30 subscribers though, so it's hardly hundreds or thousands.

I'm guessing that not many users even on BIG sites will get loads more than 30 or so for each topic, so this is why this may have gone unnoticed in the past.

Could you guys explain to me the process of what BBPress is actually going through BEFORE it is allowed to refresh the page the user who is posting is on.

Their post is being posted way before they get to see it - does the page wait for confirmation of the emails being sent.

There isn't a problem posting new topics with no replies or very few voices.

I think it has to be something along these lines. Thanks! :)

[rossagrant]

Any advice on the above post folks?

Just need to work out what BBPress is waiting for before it allows the page to refresh after a post.

I'm guessing it's a mail sent type of response.

I'm certain this is what must be causing the slowdown. (hopefully anyway)

[johnjamesjacoby]

Emails could go through wp_cron(). Probably needs more investigation. I wonder if there's a server config you can change to prevent wp_mail() from hanging on.

[rossagrant]

I'm moving to my dedicated server this weekend and am going to be playing with the local email setup to see if it is a setting that may be delaying email.

Part of me still feels like it could be server setup - so I want to rule it out once and for all.

Ideally we'd queue email, so that the php script gets an instant 'mail sent' response, allowing it to continue.

Will keep you posted here on what we find. :)

[rossagrant]

Quick update on this folks.

I've now moved onto a new dedicated server.

I'm still having the huge waiting times when I have more than a few subscribers to a topic.

The post that is most affected has around 30 subscribers now.

Once I hit reply, the forum post is actually posted pretty much instantly - I can open the activity stream in a second browser and see it there.

The screen where I'm posting from however hangs until ALL emails have been sent.

If I look into my Google Apps 'sent' items, I see the emails going out at a rate of one or 2 a second.

For 30 subscribers, this means the page won't actually continue to load for the original poster for around 20 seconds.

At that point, it finally carries on loading and they see their reply posted.

This doesn't look like a server error as it's happened when using both the server's own mail system and Google Apps on 2 different servers.

Is there any way at all to allow the script to load continuously, whilst queuing the emails by cron or something?

I'm more than happy to try ANY suggestions. I SO want to get this fixed as it really cuts down on interaction in my forums as people just don't want to wait 30 seconds to see their post.

[netweb]

I just did a quick test here on a localhost install (XAMPP). A single topic with 50 subscribed users and posted a reply.

The timestamps between the first email and last email is 14 seconds and this is approx the same amount of time before I saw the topic page refresh.

Replying to johnjamesjacoby:

Emails could go through wp_cron(). Probably needs more investigation. I wonder if there's a server config you can change to prevent wp_mail() from hanging on.

There are quite a few 'wp_mail' and 'notifications' plugins in the WP repo and I had a quick play with ​Mailer as that states Mailer queues and throttles emails from WordPress to keep the email flow within provider limits. SMTP ready. yet nothing changed time wise, so it's out of the running.

I installed ​Mail Queues and the topic page refreshed in a couple of seconds so I was rather hopeful. Turns out all the mail vanished with this plugin active. There is a very high chance this was due to me not setting up any mail queues in the plugin though. This plugin and a few others might be quite promising and worth setting up on a 'real' server with a FQDN etc rather than localhost.

[alex-ye]

I did some old patches about this ticket! Oh! I just forget it!
I will update them and upload them here as soon as possible.

[rossagrant]

Replying to netweb:

I just did a quick test here on a localhost install (XAMPP). A single topic with 50 subscribed users and posted a reply.

The timestamps between the first email and last email is 14 seconds and this is approx the same amount of time before I saw the topic page refresh.

Thanks for testing @netweb

Glad it's not just something on my install.

My problem topic has over 40 subscribers now and people are beginning to post twice as they think replies are just hanging.

Would be awesome to find a more efficient way for BBPress to deal with email.

I'm not a dev unfortunately, but if anyone would like me to test anything at all - please just ask!

I appreciate your efforts with this SO much - it's a pretty big issue in my community and is stunting growth in the use of my forums.

[johnjamesjacoby]

Switch to BCC with noreply@ kludge

[netweb]

That works, ~2 seconds and a single email with 50 recipients BCC'd :)

[rossagrant]

Wow, that's awesome!

This will cut down so much on the number of emails I send through Google Apps too. A busy topic can send out 50 emails a time. 10 replies in a day can mean 500 emails - with this new method we save on 490 emails being sent!

Will this patch be issued in the next release folks?

[rossagrant]

Okay, just patched my production site and the performance is SO much better! The page refreshes in around 2.5 seconds - as opposed to 30 seconds+!

I send all mail through SMTP via Google Apps and when I look at my sent items, I see one email that is actually sent to the Google Apps address itself - bizarrely it shows no other recipients BCC'd in like it would in a regular email composed with a BCC added.

HOWEVER, I am subscribed to the problem topic with another account and I DID receive the subscription email, so the patch IS working just fine.

My only concern is that many 3rd party email sending apps like Google Apps have BCC sending limits of anywhere between 99 and 500 addresses.

On bigger sites this could be breached when a topic hits 100 subscribers.

Maybe a bit edge case, but with 'Introduce Yourself' type topics on sites, 100 subscribers can be hit.

Thanks so much for this though JJ - it's a MASSIVE improvement.

[rossagrant]

One other tiny little thing I'm noticing - and it might not be an issue is that the return path which is being obtained from the site URL is creating an address in the raw email code for me that looks like noreply@​http://mysite.com - as I have the ​http:// in my General settings.

It's not an issue for me, as I use Google Apps and so the return address seems to be overridden by what I setup in the SMTP plugin to connect with Google.

Everything else in the raw source of the email produced looks good to my untrained eye though.

[johnjamesjacoby]

In 5258:

Refactor subscription notifications functions to send 1 email with all subscribers BCC'd rather than 1 email per subscriber. This speeds up posting significantly in forums or topics where there are any number of subscribers, particularly on configurations with slow email servers or when using external email services.

(Subscription notifications could be improved further in a future iteration by moving the actual sending into a wp_cron process, but this improvement is easy and significant enough where it will be sufficient for the widest audience.)

Hat tip netweb, rossagrant. Fixes #2162.

[rossagrant]

Great stuff JJ, thanks for your help here and props to netweb and alex-ye too.

This would be a great improvement to the way @mention mails are sent.

If you @mention 5+ people in a forum post etc. we currently send 5 separate emails.

This takes up 5+ seconds and grows exponentially.

Just a thought! :)

[netweb]

Commit r5258 needs reverting as it was against /branch/2.5 not /trunk

OR commit 2162-2.5-branch.diff to 2.5 Branch/ 2.5.4 Milestone AND 2162-2.diff to /trunk

Also the $do_not_reply address returns blank and email is sent from default email address eg wordpress@127.0.0.1. This is only a 'relative url' and does not contain the protocol prefix or domain.
$do_not_reply = 'noreply@' . ltrim( get_home_url( null, '', 'relative' ), '/' );

Updated: (Includes surrounding brackets <>)
$do_not_reply = '<noreply@' . ltrim( get_home_url(), '^(http|https)://' ) . '>';

Also updated typo (I think) in $headers 'From:' value

From:
$headers[] = 'From: . ' . get_bloginfo( 'name' ) . ' ' . $do_not_reply;
To:
$headers[] = 'From: ' . get_bloginfo( 'name' ) . ' ' . $do_not_reply;

In ​2162-2.5-branch.diff is a patch against the 2.5 branch and in ​2162-2.diff is the full patch against /trunk

[johnjamesjacoby]

In 5259:

Subscriptions clean-up, from r5258. Props netweb. See #2162. (2.5 branch)

[johnjamesjacoby]

In 5260:

Refactor subscription notifications functions to send 1 email with all subscribers BCC'd rather than 1 email per subscriber. This speeds up posting significantly in forums or topics where there are any number of subscribers, particularly on configurations with slow email servers or when using external email services.

(Subscription notifications could be improved further in a future iteration by moving the actual sending into a wp_cron process, but this improvement is easy and significant enough where it will be sufficient for the widest audience.)

Hat tip netweb, rossagrant. Fixes #2162. (trunk)

[johnjamesjacoby]

Moving back to 2.5.4 branch for completeness sake.

[leanderbraunschweig]

Hi there,

whilst I appreciate the refactoring of the notification mails resulting in a huge speed-up, we have been running into some issues with the way you are generating the noreply-address...

Right now the mails (with the subscribers in BCC) are being sent to noreply (AT) www.our-domain.info which results in errors because of missing MX-records for the subdomain-part (www.) – since there is no way we can customize / specify the TO-address this spells bad karma and had us struggling with user requests claiming they didn't receive any notification-mails after the recent upgrade.

Could you re-work the $do_not_reply-variable to exclude the www.-part if you make use of the home_url? Or am I missing something and this is due to our setup exclusively maybe?

Another (rather stupid) question: What happens if one BCC-address is faulty, does that impact the whole sendability / fail the delivery for all the other addresses as well? Hope not...

Thanks in advance!
Henning

[netweb]

@leanderbraunschweig We are in the process of tweaking these, could you please continue this discussion on ticket #2618 instead, rather than this closed ticket please.

Specifically we are adding some filters for the to: and from: email addresses which will partly solve your issue in allowing you to override these addresses.

That said, could you also add examples of your current home_url and site_url, if you are redirecting users from ​http://www.example.com to ​http://example.com and your home_url/site_url includes the www prefix yet your DNS configuration does not I would expect you would also be having other issues with any other WordPress Core or plugin that utilises wp_mail.

If a BCC: email address fails the remaining users will still receive the notification.
(Give it a test yourself, email yourself and a non-existent (fake) email address using BCC:)

[leanderbraunschweig]

@netweb,

thanks for getting back so swiftly – I wasn't aware that there is a different ticket assigned already for this. I'll be posting everything else over there...

[r-a-y]

Just wanted to chime in and say that I'm not really fond of the whole BCC routine that bbPress has chosen to adopt as of v2.5.4 to deal with a large subscriber base.

BCC is a potential red flag for spam:
​https://support.google.com/mail/answer/69585
​https://forums.aws.amazon.com/thread.jspa?threadID=76947

A better solution would be to offshore email sending to wp-cron or the next page load like what JJJ mentioned in comment:9 and in the commit message for r5258. Or by using an existing WP mail queue plugin like what netweb mentioned in comment:12.

The latter option relies on an external plugin; while the former option could work in theory, on very large and busy sites, this could still place a strain on the server.

Might be worth looking into how large WP newsletter plugins handle this as well.

[rossagrant]

Great stuff r-a-y - This initial BCC approach came about to speed up individual sending of notification emails and did a good job of that, but I too think and overall better job could be done if we used a cron job to queue mail.

I can still make forum posts slow down to a crawl when posting by including a few @mentions as they still get sent one by one.

If we could queue mail and allow for pages to reload instantly upon submission, withot waiting for mail to be sent, that would be absolutely awesome.

I can't code to save my life but if you guys need anything testing or writing up in codex, shout up! :)

[johnjamesjacoby]

You can also do something like:

add_filter( 'bbp_subscription_to_email', function() {
	return '';
} );

...which will remove the To: completely, avoiding the bounce back notices.

[netweb]

Replying to r-a-y:

BCC is a potential red flag for spam:

​https://support.google.com/mail/answer/69585
​https://forums.aws.amazon.com/thread.jspa?threadID=76947

A better solution would be to offshore email sending to wp-cron or the next page load like what JJJ mentioned in comment:9 and in the commit message for r5258. Or by using an existing WP mail queue plugin like what netweb mentioned in comment:12.

Thanks Ray, these make sense, I've created #2773 to add email notifications to @mentions and I've made a note linking back here to this comment and we should look to doing "one" of these solutions and move away from BCC.

[Robkk]

there is this plugin to help steer away from using BBC and use WP-Cron to send subscription emails.

​https://wordpress.org/support/plugin/asyncronous-bbpress-subscriptions

[rossagrant]

I'm testing this plugin out on my staging site and I think this functionality would be a superb addition to core. If anyone wants any specific tests, please just shout up.

[mechter]

Hi guys, I'm the author of "AsynCRONous bbPress Subscriptions". Let me know if I can be of assistance.

[mechter]

I've created a patch for WP Cron usage and some other enhancements, see #2797.

[netweb]

I'll add a note here, any testing and feedback on #2797 would be greatly appreciated.