How to write a FICS bot - part IV

Fourth article of the FICS bot writing tutorial. In this chapter, I am to reimplement the simple registration bot described in part II using techniques introduced in part III (Python, Twisted and FICS block mode). This version makes far better foundation for complicated multitask bot, would somebody want to write one.

In fact, you are to see some parts of WatchBot core here.

Just like in part II our bot will handle single command:

tell BotTutorial join

After obtaining it, the bot will check the standard rating of a given player, and save the name and the rating to the local file. General design is also very similar to the version from the part II. We are to use different language (Python instead of Perl) and framework (Twisted instead of Net::Telnet). We are to make heavy use of a block mode and some Twisted idioms (both introduced in part III).

Commented code

Let's go to the code.

Standard header:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
############################################################
# (c) Marcin Kasperski, 2008
############################################################
# Example FICS bot code
############################################################

############################################################
# Configuration
############################################################

Some configuration variables. Most of them are obvious, so just a few remarks.

FICS_HOST = 'freechess.org'
FICS_PORT = 5000

FICS_USERNAME = 'BotTutorial'
FICS_PASSWORD = ''

Again, give non-empty password for true account. Otherwise the bot is logging as guest.

# Automatically stop after that many minutes (don't leave
# the testing code running forever)
STOP_AFTER = 15

CONTACT_USER = '<enter-your-name-here>'

Replace with your nick. Let FICS staff know who is running the code.

FINGER_TEXT = """
Simple example of a FICS bot. Somebody is learning. See
http://blog.mekk.waw.pl/archives/19-How-to-write-a-FICS-bot-part-IV.html

This bot is run by %(CONTACT_USER)s

Usage:
    tell %(FICS_USERNAME)s join
""" % locals()

The % locals() trick resolves all variables written as %(name)s against the local variables list. Here CONTACT_USER and FICS_USERNAME are to be substituted with the values defined above.

PROTECT_LOGOUT_FREQ = 45

Every that many minutes we are to issue dummy command to avoid automatical disconnection.

SAVE_FILE = "registered.players"

The file we save to.

IGNORED_PLAYERS = ['RoboAdmin', 'Mamer', 'Relay']

We ignore any tells from those players (in particular, RoboAdmin is welcoming any guest, so let's not start from informing him that he issued a bad command).

LOG_COMMUNICATION = False

Switch to True to have all incoming data dumped (just two if-s below).

############################################################
# Imports
############################################################

from twisted.internet import reactor, defer, protocol, task
from twisted.protocols import basic
from twisted.python import log as twisted_log
import re, sys, sets

Here we import different modules we are to use.

############################################################
# Logging setup
############################################################

# Initialise Twisted logging. Here - to the console
twisted_log.startLogging(sys.stdout)

############################################################
# Small helper functions
############################################################

Logging initialization. Twisted logging is always worth turning on, so error messages and warnings show up. It can be directed to a file (even to the auto-rotated one), here we just dump it to the console. See Twisted logging docs for more details.

For the sake of brevity we will also use Twisted log object to save our own messages. In bigger app I'd recommend Python logging module which gives a bit more control.

Trivial helper.

re_empty = re.compile("^[\s\r\n]*$")

In Python it makes sense to compile every regular expression used and save it as a variable - so it does not need to be reparsed on every match. Perl uses some dirty but effective optimization techniques which avoid this, but in Python we need to be explicit.

def is_empty(text):
    """
    Returns true if given text consists of whitespace only.
    """
    return re_empty.match(text) and True or False

The test and value1 or value2 trick is equivalent to the test ? value1 : value2 expression known from C/C++. It returns value1 if the test is true, and value2 otherwise. The function above returns true if the parameter is all whitespace.

IGNORED_PLAYERS_SET = sets.Set([ x.lower() for x in IGNORED_PLAYERS])

def is_player_ignored(who):
    return who.lower() in IGNORED_PLAYERS_SET

Note that we lowercase FICS nicknames. On FICS Mekk and mekk are synonyms. It is safer to always lowercase nicks before comparing, searching or sorting.

Below the oversimplified function called to actually save the registration info. In real bot we would do something smarter (maybe save to database, maybe send email...). But let's keep this example short.

############################################################
# Saving registration info
############################################################

def register_player(who, rating):
    """
    Saving info that player who of rating rating registered.
    Here - oversimplistic procedure, we just append a line
    to the file.
    """
    twisted_log.msg("Registering player %(who)s with rating %(rating)d"
                    % locals())
    f = file(SAVE_FILE, "a")
    f.write("%(who)s: %(rating)d\n" % locals())
    f.close()

Note that in multithreaded application we would need to care about race conditions (two threads writing to the file simultaneously). In Twisted script this is not a problem, the subroutine can be only interrupted and intermixed with another code if (and where) it allows to. register_player will run from the start to the end without any other code running.

############################################################
# Regexp gallery.
############################################################

# Command(s) the bot is handling

re_cmd_join = re.compile("^\s*join\s*$")

All regular expressions we use. First, the commands our bot is handling (as you remember, we handle just one - join).

Different prompts received while logging in.

# Login process support

re_login = re.compile(
    '([Uu]ser|[Ll]ogin):')
re_password = re.compile(
    '[Pp]assword:')
re_guestlogin = re.compile(
    r"Press return to enter the (?:server|FICS) as \"([^\"]*)\"")
re_normallogin = re.compile(
    r"Starting FICS session as ([a-zA-Z0-9]+)")

Detecting block mode wrappers (see part III for detailed info about block mode).

# Block mode support

BLOCK_START = chr(21)        # \U
BLOCK_SEPARATOR = chr(22)    # \V
BLOCK_END = chr(23)          # \W

re_block_reply_start = re.compile(
    BLOCK_START + r"(\d+)" + BLOCK_SEPARATOR + r"(\d+)" + BLOCK_SEPARATOR)
re_block_reply_end = re.compile(BLOCK_END)

Different out-of-bound notifications we get from FICS. Currently we care only about tells from other players.

# FICS notifications we care about

re_tell = re.compile(r"""
^
(?P<who>[^\s()]+)    # Johny
(?:\(\S+\))*         # (TD)(TM)(SR) etc
\stells\syou:\s      #  tells you:
(?P<what>.*)         # blah blah
$
""", re.VERBOSE)

Finally, regexps used to interpret the finger command reply.

# Extracting info from finger command reply

re_finger_not_played = re.compile(
    "has not played any rated games")
re_finger_std_rating = re.compile(
    "^Standard\s+(\d+)", re.MULTILINE + re.IGNORECASE)

The class below is not required by Twisted, it is just my design. I split the main code into two parts. FicsProtocol (below) handles all details of FICS connection, while MyHandler is a place where the bot-specific code is to be written. FicsProtocol (after being extended a bit) may work as a module shared between different bots. MyHandler is specific for given bot.

############################################################
# Handler (hooks for true code)
############################################################

class MyHandler(object):
    """
    The class where we put the actual bot code (our functions)
    """

def __init__(self):
        self._bad_tells = {}
        # We will set this variable in FicsFactory, while
        # making association between handler and protocol
        self.protocol = None

The _bad_tells attribute will be used to mark which players sent many incorrect tells to our bot (so we can avoid the dead loop while conversating with another bot).

    ########################################################
    # Protocol callbacks
    ########################################################

def onTell(self, who, what):
         """
         Called whenever we obtain a tell
         """
         if is_player_ignored(who):
             return
             # No, no chat with RoboAdmin.

This is the main design concept of Handler versus Protocol split. Would we need to handle seeks (for example), we would add onSeek method here (and add some code in protocol to actually call this method after spotting some seek line). As this bot cares only about tells, we use only onTell.

         if re_cmd_join.match(what):
             self.processJoin(who)
             return

The bot obtained (and recognized) proper join command. As the handling code is complicated, let's delegate it to the separate subroutine.

         # Bad command handling
         c = self._bad_tells.get(who, 0) + 1
         self._bad_tells[who] = c
         if c <= 3:
             self.protocol.runCommand(
                 "tell %(who)s I do not understand your command" % locals())
             return
         else:
             twisted_log.err("More than 3 mistakes from %(who)s, ignoring his wrong command" % locals())

And here we avoid chat loops. Alternative approach would be to use timestamps and minimal delays instead of counts.

The function used to actually handle the registration process. We are using here the extremely elegant Twisted idiom called inlineCallbacks. In short, it lets one interrupt the routine at the moments one pick.

Note that this code requires Python 2.5 (or newer) to run.

    ########################################################
    # Utility functions
    ########################################################

@defer.inlineCallbacks
    def processJoin(self, who):
        """
        Called when we receive join request. We use
        finger to grab the user rating, then save
        his or her data.
        """
        finger_data = yield self.protocol.runCommand(
            "finger %(who)s /s r" % dict(who=who),
            )

The finger player /s r command is sent to FICS, then the routine yields.

At the moment the routine yields, it's execution is suspended, and other code will be able to run. Once the bot obtains the reply (and callback is fired on deferred returned from runCommand), the routine will be resumed.

Note: this is not what yield is usually doing. The point is in smart programming trick implemented in inlineCallbacks function. More-or-less it registers whole rest of the subroutine as callback to be called when the result of asynchronous runCommand routine is available. See article about inlineCallbacks for equivalent explicit code.

You need not understand it in detail, just remember that everytime you yield, the execution is suspended and other code may run.

        rating = None
        m = re_finger_std_rating.search(finger_data)
        if m:
            rating = int(m.group(1))
        elif re_finger_not_played.search(finger_data):
            rating = 0
        else:
            twisted_log.err("Can't detect rating of %(who)s" % locals())
            rating = 0

Here we just parse the finger results. Note, that contrary to the part II, whole reply is available. Blessings of the block mode (and some code below).

        register_player(who, rating)

We have all the information, so we call the registration routine. Note that if this routine was asynchronous (say, would send email and return deferred fired once it is actually sent), we would yield it

        yield self.protocol.runCommand(
            "tell %(who)s You are registered with rating %(rating)d" % locals())

... just like we do with this notification. It is not strictly necessary as we do not need a reply, but in case we add more code to this routine, it is safer to keep this yield to suspend it until the tell is executed.

As I said above, the protocol class is to encapsulate technical details of FICS connection - and is good candidate for code sharing.

############################################################
# Protocol (technical details of FICS connection)
############################################################

class FicsProtocol(basic.LineReceiver):
    """
    Wrapper for technical details of FICS connection.
    LineReceiver handles telnet for us, in this class
    we handle login process, issue commands and grab
    their results, interpret FICS notifications.
    """

LineReceiver handles telnet-like connection for us.

    prompt = 'fics%'

def __init__(self):
        # To be set later
        self.handler = None

##################################################
    # Login and post-login initialization
    ##################################################

def connectionMade(self):
        twisted_log.msg("TCP connection made to the server")
        self.delimiter = "\n\r"
        self._inReply = None
        self._keepAlive = None
        self.setRawMode()

This function is called when the physical TCP connection to FICS is made. setRawMode puts the LineReceiver class into the state in which it does not split the information obtained into separate lines. We are to hunt for login prompt, so we don't care about individual lines. That's not really needed, we just use this mode to have different callbacks called while logging in and different while working after login. Alternatively we would need to keep some state info.

    def connectionLost(self, reason):
        twisted_log.msg("Connection lost " + str(reason))
        if self._keepAlive:
            self._keepAlive.stop()
            self._keepAlive = None

Called after we are disconnected. Cancelling the avoid disconnection task.

    def rawDataReceived(self, data):
        """
        Raw telnet data obtained. We use this mode only while
        logging in, later on we switch to the line mode.
        """

Callback called by Twisted superclass when some data appears on the socket and the object is in raw mode. As I said above, the bot is in raw mode only during login. If we weren't using this trick, we would need to detect the state in lineReceived below and move this code there.

        if LOG_COMMUNICATION:
            twisted_log.msg("Raw data received: %s" % data)
        if re_login.search(data):
            self.sendLine(FICS_USERNAME)
            return

Login prompt detected? Let's send username.

        if re_password.search(data):
            self.sendLine(FICS_PASSWORD)
            return

Password prompt? Here is a password.

        name = self.checkIfLogged(data)

Maybe we got info that we were logged in?

        if name:
            self.user = name
            self.setLineMode()
            twisted_log.msg("Logged in as %s" % name)
            self.loggedIn()

As you see we enter the line mode here, so all future communication will result in lineReceived being called, not this function.

As it is 1:15 AM, I omitted the detection of duplicate guest account case, and this bot will be disconnected by FICS in such a case (and try to reconnect). This is not a good idea, so treat adding those extra regexps as an excercise. reactor.stop() is the function to be called to stop the program.

    def checkIfLogged(self, data):
        if FICS_PASSWORD:
            m = re_normallogin.search(data)
            if m:
                return m.group(1)
        else:
            m = re_guestlogin.search(data)
            if m:
                self.sendLine("")
                return m.group(1)
        return None

Just a few lines of code moved to separate function for brevity

    def loggedIn(self):
        """
        Just logged in. Initialization
        """

Simple post-login function called to initialize FICS variables.

        # Internal list of callbacks for future commands
        self._reply_deferreds = []

self.sendLine("iset defprompt 1")
        self.sendLine("iset nowrap 1")

self.sendLine("set interface PythonCodeFollowingMekkTutorial")
        self.sendLine("set open 0")
        self.sendLine("set shout 0")
        self.sendLine("set cshout 0")
        self.sendLine("set seek 0")
        # self.sendLine("tell 0")  # guest tells
        self.sendLine("set gin 0")
        self.sendLine("set pin 0")
        self.sendLine("- channel 53")

# finger
        finger = FINGER_TEXT.split("\n")[1:]
        for no in range(0, 10):
            text = (no < len(finger)) and finger[no] or ""
            self.sendLine("set %d %s" % (no+1, text))

# Enable block mode
        self.sendLine("iset block 1")

Since this very moment the way commands have to be issued rapidly changes.

        # Setup 
        if PROTECT_LOGOUT_FREQ:
            def _keep_alive_fun():
                self.runCommand("date")
            self._keep_alive = task.LoopingCall(_keep_alive_fun)
            self._keep_alive.start(PROTECT_LOGOUT_FREQ * 60)
        else:
            self._keep_alive = None

task.LoopingCall is a Twisted idiom to call some code at regular intervals. Note: no threads, no signals.

    ##################################################
    # Normal works
    ##################################################

def lineReceived(self, line):
        """
        Called whenever we obtain a line of text from FICS
        """

The function called by parent class whenever new line of text is available from FICS.

        if LOG_COMMUNICATION:
            twisted_log.msg("Received line: %s" % line)

if self._inReply:
            n = re_block_reply_end.search(line)
            if n:
                (id, code, text) = self._inReply
                self._inReply = None
                self.handleCommandReply(id, code,
                                        text + "\n" + line[:n.start()])
            else:
                self._inReply = (self._inReply[0], self._inReply[1],
                                 self._inReply[2] + "\n" + line)
            return

The _inReply attribute is set (a few lines below) when we are inside a reply block (we saw reply start, but reply end is yet to come). It is a tuple consisting of the command identifier (this magic number we use to match reply to the command), command code (FICS identifier what kind of command was it) and all text so far.

If this line also fails to contain the block end mark, we just append it to the reply text. If it is there, we call handleCommandReply to make appropriate processing of the whole reply.

So - as a whole it is a bottom part of block reply handling.

        if line.startswith(self.prompt+' '):
            line = line[len(self.prompt)+1:]

FICS prompt occassionally drops in.

        if is_empty(line):
            return

No need to handle empty lines.

        m = re_block_reply_start.match(line)
        if m:
            id = m.group(1)
            code = m.group(2)
            text = line[m.end():]
            n = re_block_reply_end.search(text)
            if n:
                self.handleCommandReply(id, code, text[:n.start()])
            else:
                self._inReply = (id, code, text)
            return

Here we have an upper part of block reply handling. If we got the reply start mark, we extract the command id and code, and start aggregating the reply text. It may happen that the end mark is already here, in such a case we can fire reply processing. Otherwise we enter the reply aggregating state.

        self.parseNormalLine(line)

Finally, it may be a normal FICS notification. As this routine is already fairly long, it is delegated to the function below.

    def parseNormalLine(self, line):
        """
        Here we parse the normal FICS-initiated notification.
        """
        m  = re_tell.match(line)
        if m:
            return self.handler.onTell(who = m.group('who'), what = m.group('what'))
        # Good place to add other events handling

Remember how I described handler versus protocol relationship? Would we need to handle seeks, gins, game positions, or any other information, I'd add next if here and new handling method (onSeek, onGameInfo, ...) in the handler.

Our bot is simple, so we care only about tells.

    def runCommand(self, command):
        """
        Sends given command to FICS.

Returns deferred which will be fired with the command result
        (given as text) - once it is obtained.
        """

# Allocate spare command id
        l = len(self._reply_deferreds)
        id = l
        for i in range(0, l):
            if not self._reply_deferreds[i]:
                id = i
                break

Just finding the smallest number which is curretly not used.

        # Issue the command 
        self.sendLine('%d %s' % (id+1, command))

Issuing command according to the block mode (prefixed with identifier).

        # Create the resulting deferred and save it. We will
        # fire it from handleCommandReply.
        d = defer.Deferred()
        if id == l:
            self._reply_deferreds.append(d)
        else:
            self._reply_deferreds[id] = d

twisted_log.msg("CommandSend(%d, %s)" % (id+1, command))
        return d

I hope you already know what deferreds are? If not, revert back to part III for links.

We return one which will be fired once we get the reply to this command.

    def handleCommandReply(self, id, code, text):
        """
        We just obtained reply to the command identified by id.
        So we locate the callback registered to handle it, and call it.
        """
        # code is a command code, we don't use it currently
        reply_deferred = None
        pos = int(id) - 1
        if pos >= 0:
            reply_deferred = self._reply_deferreds[pos]
            self._reply_deferreds[pos]=None
        twisted_log.msg("CommandReply(%s, %s)" % (id, text))
        if not reply_deferred is None:
            return reply_deferred.callback(text)

And here we locate the deferred allocated to the command of given id, and fire it.

############################################################
# Factory (connection management)
############################################################

class FicsFactory(protocol.ReconnectingClientFactory):
    """
    By using ReconnectingClientFactory we re-connect
    every time we get disconnected from FICS.
    """
    def __init__(self, handler):
        self.handler = handler
        # ReconettingClientFactory settings
        self.maxDelay = 180
        self.noisy = True
    def buildProtocol(self, addr):
        twisted_log.msg("Connected to the server")
        self.resetDelay()  # reconnection delay
        self.protocol = FicsProtocol()
        self.protocol.handler = self.handler
        self.handler.protocol = self.protocol
        return self.protocol

Glue class required by Twisted. By using ReconnectingClientFactory we won't stop after being disconnected. The buildProtocol function is called when the connection is made, here we decide that it will be handled by our FicsProtocol, and not something else.

############################################################
# Main
############################################################

handler = MyHandler()
reactor.connectTCP(FICS_HOST, FICS_PORT, 
                   FicsFactory(handler))
if STOP_AFTER:
    reactor.callLater(STOP_AFTER * 60, reactor.stop)

reactor.run()

reactor.run() never ends (unless reactor.stop() is called somewhere). This is the function which monitor all existing network connections, and other event sources - and calls appropriate functions when something happens.

Download

You can download the code here

Summary

This example is too simple to truly show the strength of Twisted asynchronous programming. So, for a moment, you must take my word that it makes a big difference comparing to the part II. In the future I am to show some examples of more complicated code.

Some things you could spot:

• we always know that we obtained reply to some command, and we are able to restore the context in which this command was issued (imagine extending the bot from part II so it uses finger in two different contexts),

• we have no problems with handling many tasks simultaneously (try logging as 2-3 guests and issuing join at the same time),

• we don't use threads, so we avoid whole bunch of problems related to race conditions, locks and deadlocks (also, we take noticeably less memory and don't waste CPU on context switches).

In short: we layed a good foundation for possible complicated bot.

In the next part I will write more about communicating with FICS players.