I am sure you are familiar with Internet Password Lockout which was added in version 8. It's a nice feature everybody likes to use having a Domino Server reachable from all the Internet. Unfortunately this feature works for HTTP and HTTPS only. If you use authenticated SMTP, for example allow some mobile users to relay emails, there is no protection....
On a Domino Server I was seeing lots of break in attempts:

01/23/2009 19:50:37 SMTP Server: Authentication failed for user abc ; connecting host 125.120.107.140
01/23/2009 19:50:38 SMTP Server: Authentication failed for user abc ; connecting host 125.120.107.140
01/23/2009 19:50:39 SMTP Server: Authentication failed for user abc ; connecting host 125.120.107.140
01/23/2009 19:50:40 SMTP Server: Authentication failed for user abc ; connecting host 125.120.107.140
01/23/2009 19:50:41 SMTP Server: Authentication failed for user abc ; connecting host 125.120.107.140

As the operting system is Linux, the weapon of chice was fail2ban, a python based script that adds IPs for a given time to the iptables firewall rules to lock out the attacker.

To get fail2ban working with Domino I had to patch the file datedetector.py as the date format used by Domino was not detected automatically. Add the following lines to the list of date formats:

# Domino format [01/23/2006 09:22:55]
template = DateStrptime()
template.setName("Month/Day/Year Hour:Minute:Second")
template.setRegex("\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}")
template.setPattern("%m/%d/%Y %H:%M:%S")
self.__templates.append(template)

In the directory /etc/fail2ban/filter.d create a file domino-smtp.conf:

# Fail2Ban configuration file
#
# Author: Christian A. Brandlehner
#
# $Revision: 001 $
#

[Definition]

# Option: failregex
# Notes.: regex to match the password failure messages in the logfile. The
# host must be matched by a group named "host". The tag "<HOST>" can
# be used for standard IP/hostname matching and is only an alias for
# (?:::f{4,6}:)?(?P\S+)
# Values: TEXT
#
# Sample log entry
# 01/23/2009 19:54:51 SMTP Server: Authentication failed for user bd ; connecting host 125.120.107.140
#
failregex = .* SMTP Server: Authentication failed for user .* connecting host <HOST>


# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

To activate the jail edit /etc/fail2ban/jail.conf and edit these line:

[domino-smtp]
# Note: make sure you have console logging enabled. If not say "start consolelog" on Dominos console.
enabled = true
port = smtp,465
protocol = tcp
filter = domino-smtp
logpath = /home/domino01/data/IBM_TECHNICAL_SUPPORT/console.log

Start fail2ban and monitor /var/log/fail2ban.log for messages that looks like:

2009-01-30 09:56:14,216 fail2ban.actions: WARNING [domino-smtp] Ban 90.152.223.221

This still needs some tuning to set how long an IP will be blocked, how many retries are allowed before blocking, etc.