[FQuest Alert] Email Delays

[Bob]

FutureQuest is currently experiencing some delays in local mail delivery which has had the technicians working on balancing the mail queues for a number of hours already and will continue as long as the current mail loads require.

Evaluation of the current email issues indicates that a large increase in the spam email being delivered to the FutureQuest network since midnight last night along with the resulting increase in SpamAssassin processing is causing the present situation.

An additional Pop Toaster server was placed into service over the weekend and a number of domains email are already being handled on that server. Additional SpamAssassin engines are also being readied for service. All of this is to continue to be able to process mail in a reliable and timely fashion.

We are doing all we can to add the new processing backends as quickly as possible as well as to continue to balance mail handling which all contribute towards timely mail delivery.

Our apologies for any inconvenience and thanks in advance for your cooperation and understanding as we work towards bringing your mail services back up to the level of expectations.

-Bob

[Deb]

Just for kicks.....

Those who want to peak in the window and see what it looks like.....

http://www.aota.net/4F/MQ-01242005.gif

Deb
- Tips & Dips == Bruce against the world :rasberry:

[SteveYoung]

Of course this is the morning I do email testing on a new site. I only hit send 6 times before I went to the fourms, honest! :pirate2:

[hobbes]

So, what's the tick-mark scale on the left?

[Terra]

30,000

--
Terra
--hit by a freight train, and still lives to tell about it--
FutureQuest

[hobbes]

Oy, I've just gone from a 1-hour delay to 2-hours...

[kitchin]

Lots of people have said it before, but I'll say it again. Thank godness for transparency here at FQ. I'd be wigging if I didn't know what, if anything, was going wrong ( = many previous hosts and ISP's.)

[Deb]

Make that 35,000

We've peaked the mountain folks and are heading back down the other side..... Refreshments can be obtained from the trail leader. He has oranges and juice in his back pack.

Enjoy the view...

http://aota.net/4F/MQ18-01242005.gif

Deb
- In English it means email should start flowing much faster again before the day is out :P

[hobbes]

Shall we expect this to become a daily occurrence?

-- Off with his tongue! --

[Wassercrats]

I thought Terra meant 30,000 for each tick, but I wouldn't know what's a lot anyway.

[ubu]

This reminds me of what I was talking about in the other thread...

http://www.aota.net/forums/showthread.php?t=18710


We need to fight spam, not the ISP we don't like. SPAM is the problem child of many problems.......


Jamie
Email delivery slow because of SPAM not AOL...

[MPaul]

Evaluation of the current email issues indicates that a large increase in the spam email being delivered to the FutureQuest network since midnight last night The new MovableType vulnerability (http://www.jayallen.org/comment_spam/forums/index.php?showtopic=461) that allows someone to hijack mt-comments.cgi to send out spam can only make things worst :sad:

[JoeLeBlanc]

any idea when this would be resolved? Im still not receiving email, and I usually get email in my spam box. LOL

Keep up the good work :yeah:

[ubu]

Terra, would this help you with the SPAM problem?

As I was surfing around tonight, I ran across this and to a novice like myself, it seems like it would help.. It talks about greylisting?

http://slashdot.org/articles/03/06/20/168203.shtml?tid=111&tid=126

Jamie

[Deb]

Well.... I honestly thought we had climbed the last mountain yesterday and apologize for the false hopes :(

As I am sure you are aware, we are still having slow email delivery issues due to the onslaught of messages. We've surpassed the highest number of emails delivered in a day by well over a million messages.

Here's a 172 hour graph to possibly offer some more insight as to what is happening...

http://www.aota.net/4F/72hrJan252005.gif
image snapshotted at about 8am this morning

Terra finished a program a few moments ago, that he began writing in the wee hours of the morning that will assist in balancing all of the queues to offer some consistency to the delivery times and allow the technicians more time to work on solutions while needing to spend less time "fighting the fires" and "babysitting the queues". I am hoping that this will enable them to develop more solutions to combat this problem which may include some "blocking at the door" abilities for each of you to implement in the near future.

I do apologize for this email situation. Possibly we've "upset some spammers" who knows.... but I assure you there will be no rest until the magic bullet has been put into place and this problem is resolved once and for all.

Deb
- Anxious to finish up the GoodStuff when this BadStuff decides to let us.

[phppete]

Possibly we've "upset some spammers" who knows

Possibly you've upset some customers too who have to explain to clients why they can't offer prompt support. Other hosts aren't giving their customers delayed email so why are you? Its not good enough and I am not happy.

[Deb]

Possibly you've upset some customers too who have to explain to clients why they can't offer prompt support. Other hosts aren't giving their customers delayed email so why are you? Its not good enough and I am not happy. No possibly about it in this area. Your words are true, correct, and understood.... and we're working through it as fast as we can to ensure speedy email returns ...

Deb
- :grr: :hrmm: :rollpin: :fire: :unhappy: :eeww:

[TVB]

Other hosts aren't giving their customers delayed email so why are you?

or perhaps they are simply not being as forthcoming about problems and not telling you.

Thanks Deb and FQ for doing what you can to keep the parts oiled despite the bits of rust that periodically show up.

Betsy
--back to tending my grandmother's knitting site

[phppete]

or perhaps they are simply not being as forthcoming about problems and not telling you.


That isn't the case, my clients are all web designers and they host in various places, I've asked them and not one of them are having any problems. I'm sure many cheap hosts are but I came here because I require the basic services such as email to be reliable.

[Terra]

Pete, you are right, and for that we apologize...

The only thing I can say is that we got slammed with an exceptional amount of spam email - many of them dictionary attacks, and we are now working triple shifts trying to scale up our capacity to meet the demands... In short, we got hit by a massive freight train, nothing like we've ever been hit with before...

We can't simply block it, because it is **all over the map** and cannot be isolated easily...

The entire FutureQuest Team is mobilized to handle this issue, and we really are striving to maintain the level of expectations you have in us... We've raised the bar in many areas, and the only choice we have is to never let it slip down, not even for a moment...

--
Terra
--there are just sometimes that one simply can't snap their fingers and the problem goes away - some of them really do take a lot of focused effort and energy to overcome--
FutureQuest

[cindik]

The company for which I work maintains its own web and e-mail servers. The e-mail server has been shut down at least twice in the past week by unusually heavy spam attacks, many of which are dictionary attacks.

[cindik]

From our system administrator (bear in mind, we're a small - 40 employee - company):

Our mail server is now handling 14 million messages a year. only 1.6 million are for valid addresses. 1.4 million of the 1.6 is spam. So, about 99% of all email traffic we see is spam/viruses.


If you're keeping score:
14 million messages

12.4 million (88.57%) spam to invalid addresses
1.4 million (10.00%) spam to valid addresses
200,000 (1.43%) actual e-mail

[Deb]

12.4 million (88.57%) spam to invalid addresses If clients want to help.... Please disable your catch-all email accounts.

To do so: visit http://www.example.com/CNC/ (replace example.com with your own domain name and login with your package's user/pass)

Click Email Manager from the top menu

If [Catch-All Alias] under System Accounts does not show disabled.... click on it and remove the name that is in the white text box so that the "Catch-All forwards to:" box is empty.

Click Save.

NOTE: That will prevent any email that is not addressed to a real email account on your domain from being processed and delivered.

[gtc]

If clients want to help.... Please disable your catch-all email accounts.

Done. Thanks for the tip.

[evanair]

Disabled mine as well.

Might be worth a mention in your next system-wide email announcement.

-em

[Deb]

Disabled mine as well.

Might be worth a mention in your next system-wide email announcement.

-em Thanks... you shall now receive less spam ;)

Also for reference:

http://aota.net/forums/showthread.php?t=17965
and
http://aota.net/forums/showthread.php?t=17452
and
I'm sure we'll have to revisit it with more seriousness in the near future :wah:

Deb
- Features should not have to be lost -- period :eeww:

[asjawebmaster]

Thanks to FQ for attending to the onslaught.

A couple minor things I've done to try to help:

-- I've turned off spam assassin temporarily on a couple of the accounts that get a lot of email to cut down on server demand to run the scripts on each incoming email

-- I've changed to a non-FQ smtp (maybe this does not affect server load on incoming mail?) on a couple of the heavily used accounts

FYI: Last night I was seeing delays of 1.5 hours.

[gtc]

Any idea why the sudden surge? Is it related to the MT flaw:
http://www.aota.net/forums/showthread.php?t=18728

[phppete]

Deb & Terra,

Thanks for the replies, I appreciate your honesty and your hard work in trying to combat the problem. I was talking to my brother today who also hosts with you and he said he would pay $80 a month for one account to get the emails delivered on time. I know not every problem can be solved by throwing money at it but if you need more hardware and more staff to cope with this then we certainly don't mind price increases, I realise others would and that is understandable.

Since I'm not qualified to understand how you deal with this problem then maybe that wouldn't help, I don't know.

My comments aren't meant to be harsh and I'm certainly not here to kick you while you are down. Having said that though I'm currently working 14+ hours per day and the idea is to pay you guys to sort out my email and hosting so I don't have the worry. If in the future that means paying more just help yourselves to the extra funds, personally I wouldn't be bothered if you tripled my fees, reliability is the issue, not money.

Anyway, all the best in taming this creature :)

Pete

[ubu]

If clients want to help.... Please disable your catch-all email accounts.

Deb just a suggestion, maybe the default for any new/old accounts should be in the off position. If you want it, you have to go and turn it on, instead of the other way around?

Just a thought...

Jamie

[Paul Hicks]

I've disable the catch-all gizmo. Appreciate the definition. Previously, I didn't know what it did, except create more useless email.

Fyi, my email has increased about 10x in past day or so. Mostly multi-duplicate posts from (i.e., 10 posts from the same sender, which after being deleted from the server by MailWasher Pro, magically reappear again in short order) legit email, and duplicate spam. I've configured SA to delete, rather than to just Tag.

[Deb]

Deb just a suggestion, maybe the default for any new/old accounts should be in the off position. If you want it, you have to go and turn it on, instead of the other way around?

Just a thought...

Jamie Already done :P

Reference: http://aota.net/forums/showthread.php?t=17452

We implemented the OFF BY DEFAULT status in June of last year :kewl:

[Terra]

Please view http://www.aota.net/forums/showthread.php?p=126020#post126020

for a post-mortem analysis of this unfortunate situation and the solution we had to deploy to stem the tide...

--
Terra
sysAdmin
FutureQuest

[Javier Mosqueda]

About SPAM:

I have a Recipe with grep from a file to match the from header if *unknown* then mail is passed on to a custom filter of mine (recorder) just to record the from,date,subject, and to fields (could record IP address but I don't see much help), then the mail is delivered to a blackhole.

Once the recorder has accumulated 100 destroyed mails it sends me the list of headers of those mails.

I RECEIVE 4 OF THOSE MAILS EVERY DAY (good I don't get the actual 400 spam mails), plus my normal 6 to 10 e-mail msgs.

The suggestion comes from the fact that my little process saves me from receiving all those mails in my mailbox, **BUT** ALL THOSE MAILS DID ENTERED THE FQ QUEUES.

I add to the list of allowed addresses by sending an e-mail to an automated address in my domain that has a script which sees if the address (domain or any word) exists previously, verifies a password to add and adds if needed, deletes if requested, etc. Has a bunch of little commands to update the list, etc.

If somebody *not in the list* wants to contact me, there is a form iin my web page (which hides the actual e-mail address where the message goes). If someone IS NOT IN THE LIST, sorry, but either I did not add that one, or he/she did not write through my web page.

Kind of little taboo issue, but this is war.

I can not REPLY with an autoresponder to say "you are not registered", because we have agreed here not to do that ***BUT*** IF (proposal/suggestion) main door to FQ had our "allowed" words, IP's, forbidden, this, that, etc, COULD NOT THAT DAEMON TALK TO THE SENDER DAEMON AND SAY


***SORRY BUT WE CAN NOT DELIVER YOUR MAIL, use the web page /such/ to contact /who/ and correct this problem***



We could have the option in our CNC's to activate that (when one is more aware in the use of those tools), add words to the list of allowed and/or forbidden, etc, and modify a bit that message since some sites are in other languages, change /such/ and /who/.

How is that?

Hope it helps.

[ubu]

Already done :P

Reference: http://aota.net/forums/showthread.php?t=17452

We implemented the OFF BY DEFAULT status in June of last year :kewl:

I missed that one, and since my accounts have been online since before June of 04, I never noticed. :)

Jamie
You're always one step ahead of us it seems.. ;)

[cindik]

FWIW, our company's e-mail (totally not related to FutureQuest in anyway - and it's an Exchange server) is currently lagging by over two hours.

Would you like some SPAM with your SPAM SPAM eggs SPAM sausage SPAM and SPAM?