EFM and HTML - Page 3

*sheila* · 11-10-2003, 09:03 PM

Quote:

Originally posted by RickJ:
First, let me apologize for my erroneous conclusion.

No problem. I'm sure we've all done something similar. At least, I know *I* have.

Quote:

Here's what I did for my test:

Using Thunderbird Mail, I composed a message with the phrase "would you li<!garbagetag>ke to be financially independent." Then, using the program's formatting options, I sent this message to myself as "plain text only" and as "rich text (html) only." Only the message that was sent as HTML got through EFM's filter. I did check the headers on the message and confirmed that "Content-Type: text/html; charset=us-ascii" was in it. Hence, I thought the message was HTML.

However, it turns out that Thunderbird changed the "<" and ">" symbols to "<" and ">" prior to sending it;

Well, near as I can tell, the message indeed was HTML. However, the "tags" were not tags. They were less than and greater than symbols.

Quote:

...that's why EFM didn't block the "HTML" message.

What do you mean "block"? Do you mean strip the tag out? Block means something else to someone writing email filters.

Quote:

My mistake for not checking further into the actual source code of the message. Sorry.

Important lesson to learn. When writing email filters, you should always examine the source of the email that you are trying to filter (or not filter, as the case may be). Email clients often do some kind of transformation or rendering (and especially with HTML versions) that are not strictly equivalent to the message source.

Quote:

But now I have a different question: why is EFM ignoring my garbage tags properly but not those from a spammer?

I am assuming by "garbage tags" that you mean this:

Code Sample:

 would you li<!garbagetag>ke to be financially independent.

EFM would have no idea whether you or a spammer sent the email, so that can certainly not be the distinguishing factor. Something else must be the factor.

The only way to answer a question like this is to have an exact copy of the raw email source plus an exact copy of the email filter data file from EFM at the time the message was run through it. Anything else is conjecture.

-- I'm fairly certain there is a logical explanation. I just don't know what it is...

RickJ · 11-11-2003, 12:02 PM

Quote:

Originally posted by sheila:
The only way to answer a question like this is to have an exact copy of the raw email source plus an exact copy of the email filter data file from EFM at the time the message was run through it. Anything else is conjecture.

To eliminate any mail client peculiarities, I created a short PHP program to create and send 2 messages, one in HTML, the other in Plain Text. I ran this code from my own Linux server at home to an email address on a domain hosted at FQ. Here's the code (email addresses changed to generic ones):

Code Sample:

 



	PHP Code:
	
		
			
<?

# testmail.php



$mailto = "person@domain1.com";



$mailbody = '

<html>

<head>

 <title>test message</title>

</head>

<body>

<p>would you li<!embeddedTag>ke to be financially</p>

</body>

</html>

';

$mailsubject = "Test of EFM Filter - html";

$headers  = "MIME-Version: 1.0\r\n";

$headers .= "Content-type: text/html; charset=iso-8859-1\r\n";

$headers .= "From: \"Test mail system\" <test@domain2.com>\r\n";

mail($mailto,$mailsubject,$mailbody,$headers);



$mailbody = '

would you li<!embeddedTag>ke to be financially

';

$mailsubject = "Test of EFM Filter - plain";

$headers = "Mime-Version: 1.0\r\n";

$headers .= "Content-Type: text/plain; charset=\"us-ascii\"; format=flowed\r\n";

$headers .= "From: \"Test mail system\" <test@domain2.com>\r\n";

mail($mailto,$mailsubject,$mailbody,$headers);



?>

Here's the EFM debug file that resulted from these two messages:

Code Sample:

 

*** Tue Nov 11 11:29:06 2003 ***

An email just received.





*** Tue Nov 11 11:29:06 2003 ***

dumping header data: for MssgID:

<200311111628.hABGSV2W012095@localhost.localdomain>

Date: "Tue, 11 Nov 2003 10:28:31 -0600"

Subject: Test of EFM Filter - plain

from addys: "Test mail system" <test@domain2.com>

to addys: person@domain1.com





*** Tue Nov 11 11:29:06 2003 ***

Passed banned headers check





*** Tue Nov 11 11:29:06 2003 ***

Checking for banned words.





*** Tue Nov 11 11:29:06 2003 ***

decoded/stripped contents:

would you like to be financially









*** Tue Nov 11 11:29:06 2003 ***

exiting--(Reason: "would you like to be financially" found in message.)





*** Tue Nov 11 11:29:10 2003 ***

An email just received.





*** Tue Nov 11 11:29:10 2003 ***

dumping header data: for MssgID:

<200311111628.hABGSV2Y012090@localhost.localdomain>

Date: "Tue, 11 Nov 2003 10:28:31 -0600"

Subject: Test of EFM Filter - html

from addys: "Test mail system" <test@domain2.com>

to addys: person@domain1.com





*** Tue Nov 11 11:29:10 2003 ***

Passed banned headers check





*** Tue Nov 11 11:29:10 2003 ***

Checking for banned words.





*** Tue Nov 11 11:29:10 2003 ***

decoded/stripped contents:

test message would you li ke to be financially









*** Tue Nov 11 11:29:10 2003 ***

passed banned-word check





*** Tue Nov 11 11:29:10 2003 ***

system exit--0

As the debug log shows, EFM blocked the "plain text" message because it contained the phrase "would you like to be financially" in it after the <!embeddedTag> was stripped out or ignored by EFM. The "HTML" message did not get blocked because the embedded tag was replaced by a space; hence, no match was found in the banned words filter list.

I hope the foregoing explains the results I've been getting with the EFM text filters.

*sheila* · 11-11-2003, 01:42 PM

That makes it a lot clearer, thanks. I will have to look into it further, because off the top of my head, I cannot think of why it should be handling the messages differently, but it is probably related somehow to the fact that the message bodies are not identical.

Will get back to you on this when I have a chance...