Enterprise Network Consulting logo

Spam filters


    I decided to build this page because spam continues to be a growing problem. If you include your real email address when you post to public newsgroups or social networking sites like FaceBook, if you send more than a minimum number of emails or give out your address to more than just a few select people, or if you have your email address published on people-finder services or yellow pages or have your email address listed on your web site or other web pages you know what I mean.

    Programs available:


    Services available:

    What I did before on my Netscape mail program was just manually build filters. I was always adding to this because spammers are always coming up with new wording for their spam partly intentionally to avoid filters and partly to find new ways to say what they're saying in an effort to hook new people. The Thunderbird crew says this is redundant when using their automatic Baeysian filters, however I achieve a higher accuracy in dealing with spam in my own filters than the automatic filters in Thunderbird achieve, and since I received an average of over 13,000 emails total per month in 2013, and one month hit nearly 30,000 emails, I need all the accuracy I can get just to keep from being overwhelmed. ThunderBird's junk filtering is about 90% accurate. If I had no manual filters that would still leave perhaps 1,000 junk messages in my inbox each month. I currently see one or two junk messages in my inbox each month. That means my manual filters are better than 99% efficient. My manual spam filters also mark the message as read and move it directly to my trash folder so I don't have to do a thing to it, as opposed to always having to scroll through the spam folder, mark them all as read, select them all and move them all to the trash folder with my delete key. Every time I scroll through the messages in my spam folder I'm looking for false positives (messages that aren't spam) and also often for patterns that I can manually filter on. I often will add at least a few new entries to my existing spam filters each week.
    I discovered how well my manual filters work recently when I was waiting for an email. My Thunderbird automatically checks for and downloads messages every 10 minutes, but I was impatient. I kept clicking my Get Mail button every 30 seconds or so. The interesting thing is, every time I did that I would see Thunderbird say it was downloading 10 or more messages. When it was done, there were no new messages in my inbox or some other folder, nor were there any in my spam folder. They were all spam being filtered by my manual filters!
 

    Unfortunately, I am always having to add more filters. It takes time away from other things I'd rather be doing. It's really a matter of choice, to spend your time to make your own filters rather than use Mozilla or spend your money paying someone else to write a program or do the filters for me. It's your choice how you deal with spam you receive. I don't have much hope things will get better in the future with spam. Spamming has a very low response  rate, but it's so incredibly cheap to send out literally millions of emails they will keep doing it. Even a response rate of .1% on a million emails sent is still 1,000 responses, for example. There are efforts in the works to make it illegal to spam, and indeed there are some laws on the books in various places for just that. Unfortunately, it's costly to pursue spammers as they intentionally make it difficult to find them because of these laws and because they don't want to receive masses of emails from irate people. I expect there will be progress in tracking down and prosecuting spammers but they will never be eliminated.
    An idea I had was that more incentive is needed for the law enforcement agencies to pursue this more vigorously. My idea was that  if spammers were required to pay for the spam they send out, that would mean they'd have to be registered and thus easily found if they weren't playing by the rules. If they had to pay to send, those who didn't pay at all would be pursued for taxes not paid. I expect this would be an effective incentive: it's all about money, as they say. The spammers would pay for their own enforcement. The down side is, as has been the trend in taxing of other forms, it would spread. I don't think it would take long for taxing of other non-spamming to be created, such as regular emails ordinary people like you and I send, and that would mean I'd be paying money for spam control, this time without a choice.

Update, November, 2007

    The number of spam emails have continued to rise since I last updated this page. Pretty much everyone with an email address is now receiving far more spam than legitimate email, and the increase appears to have no end in sight. Most email servers filter spam before the user even downloads it, and even Outlook Express has built-in filtering. Some places in the world have anti-spam laws in place, and there has actually been some court action. Unfortunately, though, that seems to have at best only incrementally reduced the meteroic rise in the number of spam messages.
    I continue to be amazed by how many spam messages are so obviously misleading. Before I do business with a company, I check them out to convice myself they are legitmate. I would never do business with a company online or otherwise that doesn't at least appear to be completely honest and operating in good faith. I can't imagine anyone else being any different. I will also never deal with a company that mangles their subject line so badly I have a hard time deciphering what it means, or a company that sends out messages that are composed of a paragraph or two of random words strung together and a link at the bottom. I'll never do business with a company who uses my email address as their from address. I think I'm far from unique on these issues either.  The question then is, how can the spammers ever hope to have anyone at all respond? Could it be that they are so focused on getting past the filters arrayed against them that they lose sight of the fact that no one will ever respond unless they appear legitimate? In spite of my inablilty to find the logic in spam there does appear to contintue to be people who respond, people who send in their money only to receive junk products at best, people who keep the spammers rolling in money and rolling out yet more spam.
    I still have my manual filters and update them on occasion to augment the Mozilla (now Thunderbird) Baeysian filtering. Baeysian filtering is supposed to give accuracy at least as good as 95% and I'm inclined to agree it's around there. That still means I have to manually scan through the messages Thunderbird has marked as possible spam in case something was put in there that isn't spam. My process for this is to scroll through, deal with any that aren't spam, then mark all read and delete them from the folder. That way each time I go to check what's in the folder I know it's all new. Tbird does have an option to automaically purge messages after a set period of time, and if your spam volume is low enough that may be fine. For me, though, I get probably 95% spam and 5% legit email: not small volumes so this works best for me right now. Here is an example of my email. The other day my Tbird stopped automatically retreiving emails for a period of perhaps 4 hours. (Tbird had an automatic update window pop up and it stopped everything until I came back to the computer and answered it.) Tbird then started downloading emails. 683 of them. The accumulated backlog of just 4 hours. When it was finished, I had 118 messages in my spam folder (all correctly spam), 8 messages in my inbox and 4 or 5 scattered through various other folders (I have a lot of filters besides my spam filters, that keep me organized by moving new emails from various people or about certain subjecta into various folders). The other 80% or so of the emails were all caught by my manual spam filter rules, marked as read and deleted. I didn't have to do a thing with them.
    Before this incident I didn't realize how well my manual filters still were working and wasn't doing a lot to keep them up. I've since watched a little more closely as I go through my spam folder. I usually look for patterns, something that is obviously never going to be in a legitimate email, and add probably one new manual filter rule per week. Yes, Tbird is already catching them, but fewer messages I have to scroll through each day in the spam folder is still time not spent manually dealing with this electronic plague called spam. Actually, even if I'm breaking even - even if the time spent updating my manual filters equals the time saved by having them - I still feel like I'm Doing Something.


Update March, 2011.
    I've been noticing an increase in the number of bounce-error messages I am receiving lately, stating that a message could not be delivered because the email address doesn't exist or there is some problem with the account (like, perhaps, it's been filled to the limit with spam). Almost always these error messages are not to someone I emailed, and the content of the message is spam, meaning spammers somewhere used my email address as the reply to address. I don't expect I'm unique in having my email address used routinely by spammers so it doesn't make me concerned that my email address will be blacklisted or anything, but still it's a very minor form of identity theft so I don't like it.
    The US Can-SPAM Act of 2003 seems to be the defacto standard around the world, at least by legitimate marketing and such businesses, but of course spammers continue to ignore all such legalities. One change I have noticed, though, is many more unsolicited emails I receive these days have an unsubscribe link included. I used the phrase 'unsolicited email' rather than 'spam email' because some of the emails I receive are not spam, they are from legitimate businesses. This is a result of Can-SPAM requirements, however spammers have them too, not because they are trying to comply, but perhaps partly as a way to get through spam filtering which looks for such things and are more likely to mark the message as spam if it's not there. I say partly because, as an experiment, I clicked on these unsubscribe links in messages that were clearly spam for 3 or 4 days. The result was a marked increase in spam. What I think happened is the spammers used my unsubscribe action not to unsubscribe me, but to verify that my email address was valid, and started sending spam to me even more. They likely also added my email address to lists which they then sold to other spammers, lists of verified active addresses, the kind that sell for the most money. So, in a nutshell, clicking on those unsubscribe links actually didn't unsubscribe me, it in fact caused me to be 'subscribed' to many more spam lists! So this is the other edge of the sword of requiring email to have an unsubscribe link. Spammers use them too, but not for the purpose for which they are intended. I recommend you only use an unsubscribe link in messages that do not appear to be spam, and appear to be from legitimate businesses.
    My manual filters don't get much attention these days, mostly because what I have seems to be working quite well. I have changed one aspect of my manual spam filters though, and that is the action taken. I used to have the filter mark the message as read then delete it but I suspect on occasion a legitimate email would be deleted by the filter. I now mark the message as read and move it to my trash folder. This way I can look through the trash folder if I suspect a spam filter grabbed a legitimate email. The way it was before, once it was deleted it was simply gone.
    My spam filters are all at the top of the list of filters in ThunderBird, meaning they get acted on first, then I have a bunch of filters to move various legitimate incoming emails to various folders; those are all sorted alphabetically. It doesn't really matter what order they're in as far as filter action is concerned since generally these will all be acting on unique filter conditions, but it makes it so I can find a filter when I want to update it or delete it.
    I have reduced the total number of emails I receive now, at least temporarily, mostly due to my no longer being involved in a local volunteer effort, a radio station. I was receiving emails from artists and their promoters plus various other organizations involved in radio, plus a multitude of companies promoting themselves through news items. Although those emails used to be part of my responsibilities, and they were still from sincere non-spammer-type people, my role change suddenly caused these to be in effect spam. One thing I would change if I were to do that kind of thing again in the future is to not use forwarders, that is, configure the station email accounts I was responsible for to forward all email to my personal email address. Although this meant I dealt with all emails in one place, it meant that when I replied to those emails those people had my personal email address, and then used that directly in their communications. A better choice is to have a separate email account for such efforts, and have it configured in my ThunderBird as a separate email address, so that when I reply to emails sent to that address the return address is separate. Then, when someone else takes on those responsibilities all that is required is for the new person to configure that email account on their computer. Nothing has to change for the hundreds of artists, etc. at all. And all I have to do is delete the email account from my ThunderBird and those emails immediately stop coming to me. Since I didn't do this, I spent several months replying to various emails that no longer concerned me asking people to remove me from their mailing lists. (Yes, hardly any of them had an unsubscribe link.) I still get an occasional email that I have to do this on, but it's probably one or two per month now, and I reply to them then tell TBird it's spam so it will treat them as such if they continue to email me.
    I have put a bit of programming on my business web site which alerts me with an email when someone clicks on a link which doesn't work on my site. This is supposed to help me keep my links current, especially important when I have links to other web sites' pages. I say 'supposed to' because it also sends me an alert when 'bots running on the net try to find pages to exploit on my site. It means that the amount of legitimate alerts I get these days are completely subsumed by the 'bot errors. Although it doesn't help me keep my links current any more, it does however give me some interesting data to sift through, by looking at the pages and folders these 'bots are looking for. All this is beside the point of this page, which is spam, but it is another indicator of the problems associated with an unregulated communication medium such as the Internet. I personally would prefer to keep the Internet unregulated and so as free as possible from fees, however it does mean it's a bit like the Wild West out there. It just means we have to protect ourselves, be smart and careful, in order to keep from getting blown away or stomped to death by the galloping horses.

Since the total amount of email I receive continues to rise I've added a table here.

Email received stats
  2010 2011 2012 2013

January

118

6,660

5,279

2,004

February

84

6,951

8,953

2,205

March

64

2,272

14,184

8,423

April

60

2,275

8,015

17,087

May

729

2,535

5,794

23,767

June

1,691

2,357

3,389

29,555

July

3,548

4,507

1,849

24,241

August

3,081

5,773

1,383

14,593

September

1,362

4,454

1,257

13,234

October

529

3,682

1,703

7,002

November

703

4,747

1,600

6,583

December

808

3,480

2,704

7,736

yearly total

12,777

49,693

56,110

156,430

average per month

1,064.75

4,141.08

4,675.83

13,035.83



If you would like more information on how to configure mail filters, here is a basic set of instructions I found on Verizon's web site:
 http://www.gte.net/announcements/spam.asp