Monday, May 3, 2010

revision 0.30

E-mail Benchmarking and Testing Tools
(in alphabetical order)
These are UNIX / Linux tools and not webpages

Dumbster
new SMTP sink tool:
http://quintanasoft.com/dumbster/


Mstone
Complete mail benchmarking system, used by sendmail Inc,

SMTP source: yes
SMTP sink: *unknown*
POP3 sink: yes
IMAP client: yes
Generates Reports: yes, HTML format with graphs
IPv6: *unknown*
version reviewed:
License: Mozilla ...
Home Page: http://mstone.sourceforge.net/

Pros and Cons and Other Notes:

Example Run:



Postal
A clever and easy to use testing tool, written Russell Coker and included in many Linux distros. It was designed to be run from a high end laptop so it creates the message body from random characters and uses no disk I/O.

SMTP source: yes (postal)
SMTP sink: yes (BHM)
POP3 sink: yes (rabid)
IMAP client: no
Generates Reports: no; just gives status every minute, no totals
IPv6:
version reviewed: 0.70 released on 05/26/2008
License: GPL v3
Home Page: http://doc.coker.com.au/projects/postal/

Pros and Cons and Other Notes:
- Does not generate e-mail with MIME attachments, so no AV load
+ no I/O load sending since e-mail body random char
+ BHM (smtp-sink) configurable to defer, reject, bounce, or grey-list
+ postal and rabid work so that message body is checksummed and verified
+ shows realtime stats
- does not produce totals (gives running total) or test report

Examples:

Postal (smtp source):
example of "tame" run: 1 thread, 10 K msg size max, 9 message/connection
postal -t 1 -m 10 -c 9 -f sender-addresses.txt mta.net rcpt-addresses.txt

A slightly harder run: 10 thread, 10 K msg size max, 9 message/connection
postal postal -t 10 -m 10 -c 9 -f sender-addresses.txt mta.net rcpt-addresses.txt
time,messages,data(K),errors,connections,SSL connections
10:32,601,3276,0,124,0
10:33,653,3555,0,126,0
...

BHM:
Here is BHM (smtp sink) that is idle accept for two SMTP monitor connections in the last second shown.
[root@vmware1 postal]# ./bhm -t 100 vmware1-addresses.txt
time,messages,data(K),errors,connections
13:14,0,0,0,0
13:15,0,0,0,0
13:16,0,0,0,0
13:17,0,0,0,2
...

Rabid:
rabid -u postfix1.lars.ctqa.net postfix1.lars-addresses.txt
time,messages,data(K),errors,connections,SSL connections,IMAP connections
16:52,0,371,0,2506,0,0
16:53,0,373,0,2512,0,0

When the postal tool creates a message body, it will add the md5sum of that body as the last line. The rabid tool will read in the message body and verify the checksum. Here is an example of md5sum error reported by rabid:
MD5 mis-match, calculated:aee8d02d99b27fd3626f287315889a85, expected
60d2a8fc3ebe586bce52e4db37cd37e0!


Postfix smtp-source/smtp-sink
Testing tools from Postfix MTA project

SMTP source: yes
SMTP sink: yes
POP3 sink: no
IMAP client: no
Generates Reports: *unknown*
IPv6:
version reviewed:
License: (open source)
Home Page: http://www.postfix.org/

Pros and Cons and Other Notes:
* smtp-source functions like postal and sends lots of e-mail
* smtp-sink is a server that accepts SMTP connections but accepts and discards all e-mail
- only a part of Postfix project and only secondary to postfix system



( Removed SMTP-Benchmark since webpage/host is down )
( http://www.etc.msys.ch/software/smtp-benchmark/ )



SPECmail (2009)
SMTP/POP3 software test suite

SMTP source: yes
SMTP sink: *unknown*
POP3 sink: no?
IMAP client: yes
Generates Reports: yes
IPv6: *unknown*
version reviewed: NONE, no plans to purchase
License: psuedo-commercial product ($1800.00 as of 2007)
Home Page: http://www.spec.org/mail2009/

Pros and Cons and Other Notes:


Example Run: Not Available, no plans to purchase


Performance Testing Services
Mail Channels
Here is a serious load testing service (just uses automation and Amazon's EC2)
http://blog.mailchannels.com/2008/06/mailchannels-offers-load-testing-tool.html


Other Honorable mentions
Network Playback Tools

Spirent Avalanche
http://www.spirent.com/analysis/technology.cfm?media=7&ws=325&ss=109&stype=15&a=1

Agilent NetworkTester
http://advanced.comms.agilent.com/networktester/

Radview WebLOAD Professional
http://www.radview.com/product/Authoring-Environment.aspx




SMTP MX Surveys and Survey Tools

Michael Still's MX survey
http://www.stillhq.com/research/ (latest info)
another page: http://smtpsurvey.stillhq.com/
(also has links to other surveys)
Tools used: http://www.stillhq.com/research/remoteworker-v70.html


Mail Channel's PingedIn

http://mailchannels.com/blog/?p=24


SMTPscan from BSD ports
http://www.freshports.org/security/smtpscan/



Mail Testing Services
*TODO* fill this out more

MX Toolbox
http://www.mxtoolbox.com/SuperTool.aspx

Tuesday, September 9, 2008

revision 0.29

E-mail Benchmarking and Testing Tools
(in alphabetical order)

Dumbster
new SMTP sink tool:
http://quintanasoft.com/dumbster/


Mstone
Complete mail benchmarking system, used by sendmail Inc,

SMTP source: yes
SMTP sink: *unknown*
POP3 sink: yes
IMAP client: yes
Generates Reports: yes, HTML format with graphs
IPv6: *unknown*
version reviewed:
License: Mozilla ...
Home Page: http://mstone.sourceforge.net/

Pros and Cons and Other Notes:

Example Run:



Postal
A clever and easy to use testing tool, written Russell Coker and included in many Linux distros. It was designed to be run from a high end laptop so it creates the message body from random characters and uses no disk I/O.

SMTP source: yes (postal)
SMTP sink: yes (BHM)
POP3 sink: yes (rabid)
IMAP client: no
Generates Reports: no; just gives status every minute, no totals
IPv6:
version reviewed: 0.70 released on 05/26/2008
License: GPL v3
Home Page: http://doc.coker.com.au/projects/postal/

Pros and Cons and Other Notes:
- Does not generate e-mail with MIME attachments, so no AV & BM load
+ no I/O load sending since e-mail body random char
+ BHM (smtp-sink) configurable to defer, reject, bounce, or grey-list
+ postal and rabid work so that message body is checksummed and verified
+ shows realtime stats
- does not produce totals or test report

Examples:

Postal (smtp source):
example of "tame" run -> 1 thread, 10 K msg size max, 9 message/connection
postal -t 1 -m 10 -c 9 -f sender-addresses.txt mta.net rcpt-addresses.txt

A slightly harder run -> 10 thread, 10 K msg size max, 9 message/connection
postal postal -t 10 -m 10 -c 9 -f sender-addresses.txt mta.net rcpt-addresses.txt
time,messages,data(K),errors,connections,SSL connections
10:32,601,3276,0,124,0
10:33,653,3555,0,126,0
...

BHM:
Here is BHM (smtp sink) that is idle accept for two SMTP monitor connections in the last second shown.
[root@vmware1 postal]# ./bhm -t 100 vmware1-addresses.txt
time,messages,data(K),errors,connections
13:14,0,0,0,0
13:15,0,0,0,0
13:16,0,0,0,0
13:17,0,0,0,2
...

Rabid:
rabid -u postfix1.lars.ctqa.net postfix1.lars-addresses.txt
time,messages,data(K),errors,connections,SSL connections,IMAP connections
16:52,0,371,0,2506,0,0
16:53,0,373,0,2512,0,0

When the postal tool creates a message body, it will add the md5sum of that body as the last line. The rabid tool will read in the message body and verify the checksum. Here is an example of md5sum error reported by rabid:
MD5 mis-match, calculated:aee8d02d99b27fd3626f287315889a85, expected
60d2a8fc3ebe586bce52e4db37cd37e0!


Postfix smtp-source/smtp-sink
Testing tools from Postfix MTA project

SMTP source: yes
SMTP sink: yes
POP3 sink: no
IMAP client: no
Generates Reports: *unknown*
IPv6:
version reviewed:
License: (open source)
Home Page: http://??

Pros and Cons and Other Notes:
= smtp-source functions like postal and sends lots of e-mail
= smtp-sink is a server that accepts SMTP connections but accepts and discards all e-mail
- only a part of Postfix project and only secondary to postfix system


SMTP-Benchmark
Written by Marc Balmer, who packages it for OpenBSD.
The smtp-benchmark packgae consists of two programs, smtpsend and smtpsink.

SMTP source: yes (smtpsend)
SMTP sink: yes (smtpsink)
POP3 sink: no
IMAP client: no
Generates Reports: *unknown*
IPv6: *unknown*
version reviewed: 1.04 released on 10/29/2006
License: smtp-benchmark (psuedo-open source)
homepage: http://www.etc.msys.ch/software/smtp-benchmark/
( alt ftp://ftp.etc.msys.ch/pub/sources/smtp-benchmark/ )

Pros and Cons and Other Notes:
Did not end up testing, since the author did not compile the code due to time limitations. The code most likely will easily compile
- Easily compiles for OpenBSD but not necessarily for other UNIXes

Example Run:



SPECmail2001
SMTP/POP3 software test suite

SMTP source: yes
SMTP sink: *unknown*
POP3 sink: yes
IMAP client: no
Generates Reports: can assume yes
IPv6: *unknown*
version reviewed: NONE, no plans to purchase
License: psuedo-commercial product ($1800.00 as of 2007)
Home Page: http://www.spec.org/osg/mail2001/

Pros and Cons and Other Notes:
- Does not simulate IMAP

Example Run: Not Available, no plans to purchase


Mail Testing Service
Mail Channels
Here is a serious load testing service (just uses automation and Amazon's EC2)
http://blog.mailchannels.com/2008/06/mailchannels-offers-load-testing-tool.html

Other Honorable mentions
Network Playback Tools

Spirent Avalanche
http://www.spirent.com/analysis/technology.cfm?media=7&ws=325&ss=109&stype=15&a=1

Agilent NetworkTester
http://advanced.comms.agilent.com/networktester/

Radview WebLOAD Professional
http://www.radview.com/product/Authoring-Environment.aspx