[FQuest Alert] Outgoing Email [Archive]

JRepici

10-03-2003, 02:35 PM

Deb, All,

My very first web script was a contact.pl script that let people fill in a form that would then be sent to me. This was a first script done many years ago and I never went back and reviewed it. It was "cleverly" allowing users to set the From and To address from the form (how could I BE so dumb?).

One thing I did do right was to keep a log of every contact message sent. So I have the actual text of the emails that sent out (I'll point you to them in a private mail). This will at least get us the addresses of the spammers.

It wasn't until I scrolled right over the TO field in the logs that I realized the extent of the problem. Each TO field was stuffed with hundreds of email addresses.

I've shut down the clever over-ride feature that allows the TO field to be changed, so that it should now work better.

What can I do at this point, armed with the text of these emails?

-John Repici

<edit>changed 'from' to "TO" where needed</edit>

Deb

10-03-2003, 02:45 PM

John, you're not the only one... this attack was researched, calculated, and implemented against MANY forms, placed by many site owners within their accounts. They simply chose today to implement their findings in a network wide spam send for a variety of *cough*services*cough* such as mortgage offers and grants etc etc etc...

For now, the best step is to do as you have done along with sending what details you have to the service desk.

Thanks.

Deb
- Hey! Quit hogging all the ugly!

Juan G

10-03-2003, 03:07 PM

In the last FormMail (http://www.scriptarchive.com/formmail.html) (1.92), from the Matt's Script Archive, they say:

# @referers allows forms to be located only on servers which are defined #
# in this field. This security fix from the last version which allowed #
# anyone on any server to use your FormMail script on their web site. #

@referers = ('scriptarchive.com','209.196.21.3');

# @recipients defines the e-mail addresses or domain names that e-mail can #
# be sent to. This must be filled in correctly to prevent SPAM and allow #
# valid addresses to receive e-mail. Read the documentation to find out how #
# this variable works!!! It is EXTREMELY IMPORTANT. #

@recipients = &fill_recipients(@referers);

And, from the Readme:

[@referers] is not a security check. Referer headers can EASILY be faked.
Rather, it prevents someone on xyznotyou.com from using the FormMail
on your server to process forms on their server on a regular basis.
It remains in the script as a remnant of earlier versions when it
was used for security, but the @recipients variable is now used
to specify exactly who can receive e-mail from this installation.

I don't know if the affected forms have a correct @recipients or not. Or perhaps more safety measures are necessary?

Anyway, surely it is safer to use NMS FormMail (I use it), a drop-in replacement for the old Matt Wright's FormMail script.

NMS FormMail has settings for @allow_mail_to, $max_recipients, @referers, etc. It's at:

NMS
http://nms-cgi.sourceforge.net/

They also have other GNU-licensed replacements for scripts from Matt's Script Archive.

(Edited: NMS is a project started by the London group of the international association Perl Mongers (http://www.pm.org/))

Juan G

10-03-2003, 03:10 PM

Also, of course it is better to change the name of the script file, so that it's more difficult to find by spammers.

colulus

10-03-2003, 03:17 PM

Has this been resolved yet?
I have a client who says he still can't send email.

Also, there was something odd when I first posted this message. The times on the first message in the thread was "Today 4:36PM". Now it says "Today 8:36AM", which makes a lot more sense, and makes sense why it might not be resolved yet. Does the time displayed depend upon cookies?

Thanks

JRepici

10-03-2003, 03:25 PM

Deb,

re: "...in a network wide spam send for a variety of *cough*services*cough* such as mortgage offers and grants etc etc etc... "

Are you saying you believe this may not have been spam but some kind of strange DoS attack then? That would make the text of the "spams" they sent (and any links it may contain) just as fake as the header info yes?

-jr

P.S. I've either deleted or chmod'd (to the devil's mark :-) ) all copies of this script on my accounts.

JRepici

10-03-2003, 03:36 PM

Deb, All,

Now that I've downloaded the attack spams and am looking at them in my editor it is even more insidious than I thought.

They didn't bother to use the gaping hole I left for them in my pre-perlbescent script. Instead, they inserted their own <CR> right in the subject field and began a bcc line. They stuffed a LOT of names in each bcc line too.

-jr (...still scrolling)

Terra

10-03-2003, 03:46 PM

Has this been resolved yet?
Partially yes...

This was a massive spam run that affected many servers... It is the largest distributed (zombie) spam run that we've seen in the history of FutureQuest...

1) The email queues have been turned back on, and outbound email is flowing again... Please give it some time to work through its backlog of deferred messages... No email was lost, other than us taking a huge steel mallet to the spam messages...

2) The server core shared mailform script has been taken offline to patch the NL/CR Subject stuffing attack... I hope to have this back online within the hour...

Are you saying you believe this may not have been spam but some kind of strange DoS attack then?
Well, in a way it was a DDoS attack against our mail servers, since the end result was just the same... However, it would appear that a spammer commanding hundreds of zombies crafted the attack against many site owners, and also against our core shared mailform script...

Whoever did this, took the time to profile exactly how they wanted to attack, and when they unleashed the full scale spam run - it was devastating...

Changes in our core mailform script is now inevitable, and will become much more strict on how you will be allowed to use it... We always believed in freedom and flexibility - however this spammer has now tainted and dirtied this concept... :(

All we can do now is try to do right thing by mitigating the onslaught (done) and cleaning up the nuclear fallout left behind (95% done)...

--
Terra
--I may just end up loading up the truck and taking a quick drive down to Boca Raton, FL to set forth some mallet justice--
FutureQuest

LightGuide

10-03-2003, 03:51 PM

Originally posted by Terra:

I may just end up loading up the truck and taking a quick drive down to Boca Raton, FL to set forth some mallet justice

I can be down there in a couple of hours if you need help swinging it.

etLux

JRepici

10-03-2003, 03:51 PM

www.axiummortgages.com

specifically...

"http://www.axiummortgages.com/index.php?a=morton"

(more will follow in this message)

TVB

10-03-2003, 04:12 PM

FQ, will you be notifying clients that were compromised? We use NMS Formmail (renamed, only 1 email to the one I designated, and no confirmation sent to the submitter)

Ironically, I had the setting to send a confirmation email to the submitter but a few weeks started to notice some weirdness like someone was testing it for a future comprising and immediately changed it.

I just want to make sure we didn't get snagged up in the ickiness.

Betsy

JRepici

10-03-2003, 04:59 PM

re: "...get snagged up in the ickiness"

You'd think a cat would enjoy a little ICKiness :)

.... I've scanned about two thirds of the messages sent out through my script. I've only been able to find two href links:

"http://www.axiummortgages.com/index.php?a=morton"

and

"http://www.secureagrant.com/grant1"

Here's a sample of one of the messages with axiummortgages in it:

. . . . . . .

Content-Type: text/html
Content-Transfer-Encoding: quoted-printable

=3CHTML=3E=3CP ALIGN=3DCENTER=3E=3CFONT SIZE=3D5 PTSIZE=3D18 FAMILY=3D=
=22SANSSERIF=22 FACE=3D=22MS UI Gothic=22 LANG=3D=220=22=3E=3CB=3EFree =
Mortgage Quote=3CBR=3E=3C=2FB=3E=2D=2D=2D=2D=2D=2D=2D=2D=2D=2D=2D=2D=2D=
=3C=2FFONT=3E=3CFONT COLOR=3D=22=23000000=22 BACK=3D=22=23ffffff=22 st=
yle=3D=22BACKGROUND=2DCOLOR=3A =23ffffff=22 SIZE=3D2 PTSIZE=3D10 FAMILY=
=3D=22SANSSERIF=22 FACE=3D=22Arial=22 LANG=3D=220=22=3E=3CBR=3E=3CBR=3E=
=3C=2FFONT=3E=3CFONT COLOR=3D=22=23000000=22 BACK=3D=22=23ffffff=22 st=
yle=3D=22BACKGROUND=2DCOLOR=3A =23ffffff=22 SIZE=3D3 PTSIZE=3D12 FAMILY=
=3D=22SANSSERIF=22 FACE=3D=22Dotum=22 LANG=3D=220=22=3EThere are 89=2C0=
00 mortgage companies in the U=2ES=2E Do you have time to check all of =
their rates=3F=3CBR=3E=3CBR=3EWe do=21 Let us do the work for you=2C an=
d find you the very best rates possible out of nearly 100=2C000 compani=
es=2E You simply fill out a short form=2C and as quickly as possible we=
will send you the top three offers for you to review=2E Its that simpl=
e=2C and you dont pay a cent=2E=3CBR=3E=3C=2FFONT=3E=3CFONT COLOR=3D=22=
=23000000=22 BACK=3D=22=23ffffff=22 style=3D=22BACKGROUND=2DCOLOR=3A =23=
ffffff=22 SIZE=3D2 PTSIZE=3D10 FAMILY=3D=22SANSSERIF=22 FACE=3D=22Arial=
=22 LANG=3D=220=22=3E=3CBR=3E=3C=2FFONT=3E=3CFONT COLOR=3D=22=230000ff=
=22 BACK=3D=22=23ffffff=22 style=3D=22BACKGROUND=2DCOLOR=3A =23ffffff=22=
SIZE=3D3 PTSIZE=3D12 FAMILY=3D=22SANSSERIF=22 FACE=3D=22Arial CYR=22 L=
ANG=3D=220=22=3E=3CB=3E=3CA Href =3D=22http=3A=2F=2Fwww=2Eaxiummortgage=
s=2Ecom=2Findex=2Ephp=3Fa=3Dmorton=22=3EClick here to proceed=3C=2FA=3E=
=3C=2FP=3E=3C=2FFONT=3E=3C=2FHTML=3E=3Cbr=3E=3Cbr=3E5GAbckLNQi046 qn smQ=
CeRMsQ2q q zYvH60P4 bUzdYPr bQ 7eSs3GmgTIz16HV4 7lIYYcz5BMC2bxM1TDSp i=
bWZ3I6 Hg fMVn kBnQs WF3 kz0KgaXG 5gg3RbR2rskUek5 aqnUQhqkWe m9N2aK0=
U5 UWG9o1Tn r z8efLKWS0 0Hy bi sBe0z0XD L jEs36g xpoJfd a fdBKjMMU9 P=
me lP V \0A\0A--pnHQ8ubA7Z7GgkxZjSQJqPt0fEiTQs--

.

Terra,

re: "We always believed..."

"What does not kill us makes us stronger, unless we respond to the adversity like John Ashcroft would."

:hehe:

Seriously, this was caused by bad code (mine included), not FutureQuest's or your, open stance and ideals.

Juan G

10-03-2003, 05:30 PM

If you replace the old Matt Wright's MSA FormMail (http://www.scriptarchive.com/formmail.html) with the much safer GNU drop-in replacement NMS FormMail (http://nms-cgi.sourceforge.net/), remember that, in addition to @allow_mail_to, @referers, etc., there is a very essential safety setting in NMS FormMail, that is:

$max_recipients

The maximum number of e-mail addresses that any
single form should be allowed to send copies of the
e-mail to. If none of your forms send e-mail to more
than one recipient, then we recommend that you
improve the security of FormMail by reducing this
value to 1. Setting this variable to 0 removes all
limits on the number of recipients of each e-mail.

Other small numbers are also safe but, for example, we have:

$max_recipients = 1;

Deb

10-03-2003, 06:11 PM

FQ, will you be notifying clients that were compromised? We're still weighing all of the pros and cons right now of the actions that may, or may not, need to be taken.

I've reviewed the list of the domains that were exploited and I do not believe your domains were on the list Betsy. There are some that had problem forms, such as the confessor within this thread (uh huh I'm look'n at you JR) that have received TOS notices letting them know that their forms are terminated and in need of updating or replacing.

There is another list of folks that were using the FutureQuest Mail Form for which is still shut down due to the complexity of trying to make some changes without forcing the clients to have to make too many changes within their own accounts (remember those using it may not be as savvy with html updates, the form is shared by many, and the explanation for why it needs to be updated can be confusing to them) so that notice may take a bit longer to create in a format that is clear and avoids "mass hysteria"

That last 5% Terra talked about isn't all that small but it will be done ;)

Deb
- Breath by breath, step by step.... No worries.

Deb

10-03-2003, 06:15 PM

BTW... Thanks JR for your clarity and actions throughout this. Much appreciated as we scramble on the backside... I want to clarify this problem again was not JR's fault...he just happened to be one of the ones snagged that posted his findings for all to see ;)

JRepici

10-03-2003, 06:45 PM

Deb,

re: "(uh huh I'm look'n at you JR)"

Yeah, I got the message. In the immortal words of erkle: "Did I do that?" :o I had actually already shut down that script in all of my domains by the time I got the email.

Unfortunately it is a custom script so the new script wont exactly be a "drop in".

re: "...not JR's fault"

Whew! I thought for sure we were going to have an episode of "Who sicked the Rottweiler on JR"...

In any case, there was no excuse for such a lax chunk of code to be out there. I do take that seriously, and I'm sorry.

-jr

BTW, an update on the attack: They not only made their own bcc line from the subject, but their own To: line as well. The one email limit sounds like the best solution to this. It is good as long as they're not doing the check at the field level in the TO, CC, or BCC fields (this attack would have breezed by such a test because it was an exploit in the subject field). It may also be astute to character-limit the subject field, both in length and in which characters it can contain.

Andilinks

10-03-2003, 06:50 PM

OK, I've commented out my mail form and posting that here on the assumption that this is where the all clear will be sounded and I will be subscribed. :)

Andi

Juan G

10-03-2003, 08:41 PM

Originally posted by JRepici:
(...) BTW, an update on the attack: They not only made their own bcc line from the subject, but their own To: line as well. The one email limit sounds like the best solution to this. It is good as long as they're not doing the check at the field level in the TO, CC, or BCC fields (this attack would have breezed by such a test because it was an exploit in the subject field). It may also be astute to character-limit the subject field, both in length and in which characters it can contain.
If spammers try to add To, CC or BCC fields through the Subject field, NMS FormMail limits the subject length to the first 256 characters:

$subject = substr($subject, 0, 256);

and also removes any new line, etc., characters that spammers may add to try to have more fields:

$subject =~ s#[\r\n\t]+# #g;

Spammers may also try via the other two fields available, that is From and To. But From is checked by the subroutines validate_email (it must be one real mail) and validate_realname (128 characters allowed, etc.).

And before getting To, the email addresses of the field recipient are separated by get_recipients, even with new lines, etc., and counted and compared (by check_recipients) with the $max_recipients number.

It seems a pretty safe script. ;)

TVB

10-03-2003, 09:47 PM

I'm confused :\ Maybe it was my own naivity that kept us safe but I don't have a subject line. I have it set in the script so that everything that comes in that way is properly sorted on my local machine but that is it. Is there a way to manipulate a subject line even though I don't give submitters the option to give one?

Betsy

Juan G

10-03-2003, 10:32 PM

Well, the fixed Subject is an additional good possibility. (However, not necessary in my opinion).

In this case, if spammers try to send a Subject via POST method (GET is not allowed in the default $secure = 1; mode of NMS FormMail), but you have either:

Subject: Your fixed subject here

(instead of the original Subject: $subject) in the subroutine send_main_email_header, or:

$force_config_subject = 'Your fixed subject here';

in the User Configuration Section, then there is no way for the spammers to modify the subject, unless they find a way to add a Subject field via From or To, and it seems really difficult with this script.

You may also create any force_config_*, like force_config_recipient, etc. But in my opinion the NMS script is safe with the default configuration.

Terra

10-03-2003, 11:02 PM

The server Core patched mailform is now back in operation and addresses the CR/NL injection exploit... We have also patched a couple other corner cases...

Overall this is not a permanent solution since it is a stop gap solution to squelch a few raging fires that are causing thousands and thousands and thousands of dollars worth of damage... :rolleye:

I'm digging through the NMS code to see how to convert it to Community use... The rules vastly change when using it for one domain, versus shared usage amongst thousands of domains...

Most likely this will require severely tightening down what will be allowed, and severing its ability to send email to destinations outside of our network...

--
Terra
--exactly who do you sue? Zombies have no money and can live through any length of jail time--
FutureQuest

JRepici

10-03-2003, 11:53 PM

Juan,

Thanks for those details. It does seem like a very secure script.

Betsy,

The attacker inserted a single Line-Feed in the subject line and added "bcc: address, address, address, ......" (hundreds of addresses).

This worked because the email transport protocol sets up its headers like this:

from: {one LF between each header element} ...
to: ...
cc ...
subject...
bcc...

{Two line feeds between the headers and the start of the body of the email}

I put the "bcc" header just under the subject line here to show the point.

Terra,

Deep breaths...

Hasn't something illegal happened here? Aren't these links we've pulled from the email a direct connection to the people who actually started/sponsored this attack? Isn't there someone out there to report these people too?

<sigh>

Btw: what's a "corner case"?

Also: "severing its ability to send email outside of our network..."

Does this mean only sending email to other FutureQuest domains?

-jr

Terra

10-04-2003, 01:43 AM

Hasn't something illegal happened here? Aren't these links we've pulled from the email a direct connection to the people who actually started/sponsored this attack? Isn't there someone out there to report these people too?
With the myriad of laws and non-laws, pick one - then look at the other one that counters it... :(

It could have been a Joe-Job...

Not really... Before the feds will even touch it - the dollar amount of damage must be astronomical... Last I heard, the value was around half a million...

Does this mean only sending email to other FutureQuest domains?
The script will know which domain called it, therefore it will take a passed mailbox or alias name and concatenate the calling domain to it... This way the spammer can only attack the 'username', while the script keeps the '@domain.tld' under lock-n-key with no altering possible...

--
Terra
--mail forms were never designed to be guarded like Fort Knox--
FutureQuest

Andilinks

10-04-2003, 02:02 AM

Aren't these links we've pulled from the email a direct connection to the people who actually started/sponsored this attack? A good civil suit could pull a few of these creatures out of the woodwork with subpoenas so others might sue as well.

I would think there might be some substantial damage claims among FQ clients and some research and publicity might reveal other complainants. My own case wouldn't be very strong, not being able to show monetary damages but there must be some businesses affected.

Andi

Randall

10-04-2003, 02:52 AM

I missed the whole thing, so I guess I can't sue the guy either. :(

However, I'd be happy to curse the day that his parents first set eyes on each other, if that helps any. :noddy:

Meanwhile, I think I'd better look at that request form on my non-FQ site. Lord knows how many mistakes I made when I wrote it...

Randall

phppete

10-04-2003, 08:47 AM

What form script is this relating to?.. is it one automatically installed on every account? Where is it?

Deb

10-04-2003, 08:57 AM

What form script is this relating to?.. is it one automatically installed on every account? Where is it? It relates to forms that send email, that were able to be manipulated, in general. There is no form automatically installed on every account but it did affect the form that site owners can use, if they choose to, that is preinstalled within FutureQuest as well as other forms that site owners chose to install...

Deb
- We even had to TOS ourselves...go figure

phppete

10-04-2003, 09:13 AM

Thanks for letting me know, I havent been here for a while so when I read this thread I wasn't sure if it was referring to a pre-installed script, I did look in my cgi-bins but nothing was there.

Juan G

10-04-2003, 09:58 AM

Originally posted by Deb:
- We even had to TOS ourselves...go figure
Cheer up... ;)

Spam (http://directory.google.com/Top/Computers/Internet/Abuse/Spam/) is becoming a too big problem on the Internet lately. :devil: The spammers' ISP should TOS-Terminate them. It's not very effective but, at least, they need to learn a lesson... By the way, who is their ISP?
:hammer:

Arthur

10-04-2003, 10:20 AM

The spammers' ISP should TOS-Terminate them If it only were that easy. Nowadays many spammers make use of an army of zombies.
Zombies are computers that have been compromised and are running programs that allow hackers/spammers to control them remotely (for example via IRC channels).

There are many many thousands of computers with these backdoors that can be used for things like spamming runs or DDoS attacks. More and more people are permanently connected to the internet, but many don't take the time or lack the knowledge to protect their computer by installing anti-virus and firewall software and keeping the software up-to-date.

Arthur

Juan G

10-04-2003, 10:54 AM

Yes, they usually lurk. But, in this particular case, JR has found two of their http addresses:

http://www.axiummortgages.com/index.php?a=morton
http://www.secureagrant.com/grant1

Arthur

10-04-2003, 11:40 AM

I see what you mean, I interpreted the term 'spammers' differently. Are spammers the ones that do the actual deed, or those that pay for it to be done. Most of the time these are different parties.

Those addresses are most likely from those that hired the spammers to send out their junk.

Our investigation in the matter is still ongoing...

Arthur
"{

Andilinks

10-04-2003, 12:38 PM

the ones that do the actual deed, or those that pay for it to be done. Most of the time these are different parties. If they commit a crime together they are the same, if they do damage they can be both sued. This is not legal advice or a legal opinion, just an impromptu rambling. If you have been damaged seek legal advice.

Andi

JRepici

10-05-2003, 10:32 AM

Andi,

Be not the first by whom the new are tried,
Nor yet the last to lay the old aside. Alexander Pope

Great quote. I'm adding it to my own list of quotes.

The FTC has an online complaint form you can fill out if you feel you've been unfairly dealt with in matters of commerce. Is there anything comparable for those who have been cyber-attacked by commercial concerns?

-jr

Moke

10-06-2003, 04:58 PM

Have you shut it down again? No outgoing emails again now. We have Phoenix servers

Terra

10-06-2003, 05:10 PM

Queues are online...

However, looking at PHOENIX server, it is stuffed with a bunch of (User unknown) bounce messages that were headed to: phil@maxpatch.com and some for webmaster@maxpatch.com

Looks like it was pumped to us by: mx1.maxpatch.com

So our server is now stuck with trying to send those back, on mx1.maxpatch.com behalf, and that is what is tying the queue up right now... :(

--
Terra
--one bullet one foot--
FutureQuest

Moke

10-06-2003, 05:19 PM

I'm looking in our email manager on maxpatch.com and there is nothing showing on either of those emails.
Mail washer pro was used this morning hours ago to bounce back spam emails, about 100 of them. and we have been using Mailwasher Pro for the last 5 months to manage all the crap we get.

Terra

10-06-2003, 05:40 PM

I'm looking in our email manager on maxpatch.com and there is nothing showing on either of those emails.
You wouldn't because it is not in your POP box, but rather within the central remote delivery queue...
Mail washer pro was used this morning hours ago to bounce back spam emails, about 100 of them. and we have been using Mailwasher Pro for the last 5 months to manage all the crap we get.
Yep - I would agree that crap is mostly what is in the PHOENIX queue right now and is dutifully being bounced back as requested... Enough that it took all available outbound delivery slots and caused a minor slowdown... The burden was placed on us to bounce the resulting email...

It all just depends on the type of email, and if we can reach the remote MTA to bounce the message back to...

From what I see today, the remote MTAs that we are getting jammed by seem to be using a DROP rule, instead of a REJECT style firewall block...

To explain, imaging wanting to call a friend of yours to talk:
1) REJECT
You call, silence, person answers - "Hi, I can't talk right now, bye" - click
2) DROP
You call, -silence-, wait a couple minutes then hangup because no one eventually said anything...

Every now and then, this scenario happens, and there is not much that we can do about it, other than let the TCP protocol do its job...

--
Terra
--crap happens--
FutureQuest

Moke

10-06-2003, 06:07 PM

Thanks for explaination!

lynxtrax

10-12-2003, 11:43 AM

is there anything happening with outgoing mail on Enigma? Every single outgoing email I've sent from the web interface has bounced this week, and continues to bounce. I am able to send email from my local email client, however. Here is the error from the bounced outgoing emails

Hi. This is the qmail-send program at questmail.futurequest.net.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up.
Sorry it didn't work out.
<witheld@witheld.com>:
Sorry, I wasn't able to establish an SMTP connection. (#4.4.1)I'm not going to try again; this message has been in the queue too long.

Ordinarily I'd chalk it up to a futzy server on the receiver end, but it's too consistent and each time I tested it by sending 2 emails within 30 seconds of each other, one from the web interface and 1 from an email client, as soon as I use an email client it sends just fine and the web interface one bouced.