Fourth article of the FICS bot writing tutorial. In this chapter, I am to reimplement the simple registration bot described in part II using techniques introduced in part III (Python, Twisted and FICS block mode). This version makes far better foundation for complicated multitask bot, would somebody want to write one.
In fact, you are to see some parts of WatchBot core here.
Just like in part II our bot will handle single command:
tell BotTutorial join
After obtaining it, the bot will check the standard rating of a given player, and save the name and the rating to the local file. General design is also very similar to the version from the part II. We are to use different language (Python instead of Perl) and framework (Twisted instead of Net::Telnet). We are to make heavy use of a block mode and some Twisted idioms (both introduced in part III).
Commented code
Let's go to the code.
Standard header:
#!/usr/bin/env python # -*- coding: utf-8 -*- ############################################################ # (c) Marcin Kasperski, 2008 ############################################################ # Example FICS bot code ############################################################ ############################################################ # Configuration ############################################################
Some configuration variables. Most of them are obvious, so just a few remarks.
FICS_HOST = 'freechess.org' FICS_PORT = 5000 FICS_USERNAME = 'BotTutorial' FICS_PASSWORD = ''
Again, give non-empty password for true account. Otherwise the bot is logging as guest.
# Automatically stop after that many minutes (don't leave # the testing code running forever) STOP_AFTER = 15 CONTACT_USER = '<enter-your-name-here>'
Replace with your nick. Let FICS staff know who is running the code.
FINGER_TEXT = """
Simple example of a FICS bot. Somebody is learning. See
http://blog.mekk.waw.pl/archives/19-How-to-write-a-FICS-bot-part-IV.html
This bot is run by %(CONTACT_USER)s
Usage:
tell %(FICS_USERNAME)s join
""" % locals()
The % locals() trick resolves all variables written as %(name)s against
the local variables list. Here CONTACT_USER and FICS_USERNAME
are to be substituted with the values defined above.
PROTECT_LOGOUT_FREQ = 45
Every that many minutes we are to issue dummy command to avoid automatical disconnection.
SAVE_FILE = "registered.players"
The file we save to.
IGNORED_PLAYERS = ['RoboAdmin', 'Mamer', 'Relay']
We ignore any tells from those players (in particular, RoboAdmin is
welcoming any guest, so let's not start from informing him that he
issued a bad command).
LOG_COMMUNICATION = False
Switch to True to have all incoming data dumped (just two
if-s below).
############################################################ # Imports ############################################################ from twisted.internet import reactor, defer, protocol, task from twisted.protocols import basic from twisted.python import log as twisted_log import re, sys, sets
Here we import different modules we are to use.
############################################################ # Logging setup ############################################################ # Initialise Twisted logging. Here - to the console twisted_log.startLogging(sys.stdout) ############################################################ # Small helper functions ############################################################
Logging initialization. Twisted logging is always worth turning on, so error messages and warnings show up. It can be directed to a file (even to the auto-rotated one), here we just dump it to the console. See Twisted logging docs for more details.
For the sake of brevity we will also use Twisted log object to save our own messages. In bigger app I'd recommend Python logging module which gives a bit more control.
Trivial helper.
re_empty = re.compile("^[\s\r\n]*$")
In Python it makes sense to compile every regular expression used and save it as a variable - so it does not need to be reparsed on every match. Perl uses some dirty but effective optimization techniques which avoid this, but in Python we need to be explicit.
def is_empty(text):
"""
Returns true if given text consists of whitespace only.
"""
return re_empty.match(text) and True or False
The test and value1 or value2 trick is equivalent to
the test ? value1 : value2 expression known from C/C++. It returns
value1 if the test is true, and value2 otherwise. The function
above returns true if the parameter is all whitespace.
IGNORED_PLAYERS_SET = sets.Set([ x.lower() for x in IGNORED_PLAYERS])
def is_player_ignored(who):
return who.lower() in IGNORED_PLAYERS_SET
Note that we lowercase FICS nicknames. On FICS Mekk and mekk
are synonyms. It is safer to always lowercase nicks before comparing,
searching or sorting.
Below the oversimplified function called to actually save the registration info. In real bot we would do something smarter (maybe save to database, maybe send email...). But let's keep this example short.
############################################################
# Saving registration info
############################################################
def register_player(who, rating):
"""
Saving info that player who of rating rating registered.
Here - oversimplistic procedure, we just append a line
to the file.
"""
twisted_log.msg("Registering player %(who)s with rating %(rating)d"
% locals())
f = file(SAVE_FILE, "a")
f.write("%(who)s: %(rating)d\n" % locals())
f.close()
Note that in multithreaded application we would need to care
about race conditions (two threads writing to the file simultaneously).
In Twisted script this is not a problem, the subroutine can be
only interrupted and intermixed with another code if (and where) it allows
to. register_player will run from the start to the end without any
other code running.
############################################################
# Regexp gallery.
############################################################
# Command(s) the bot is handling
re_cmd_join = re.compile("^\s*join\s*$")
All regular expressions we use. First, the commands our bot
is handling (as you remember, we handle just one - join).
Different prompts received while logging in.
# Login process support
re_login = re.compile(
'([Uu]ser|[Ll]ogin):')
re_password = re.compile(
'[Pp]assword:')
re_guestlogin = re.compile(
r"Press return to enter the (?:server|FICS) as \"([^\"]*)\"")
re_normallogin = re.compile(
r"Starting FICS session as ([a-zA-Z0-9]+)")
Detecting block mode wrappers (see part III for detailed info about block mode).
# Block mode support
BLOCK_START = chr(21) # \U
BLOCK_SEPARATOR = chr(22) # \V
BLOCK_END = chr(23) # \W
re_block_reply_start = re.compile(
BLOCK_START + r"(\d+)" + BLOCK_SEPARATOR + r"(\d+)" + BLOCK_SEPARATOR)
re_block_reply_end = re.compile(BLOCK_END)
Different out-of-bound notifications we get from FICS. Currently we care only about tells from other players.
# FICS notifications we care about re_tell = re.compile(r""" ^ (?P<who>[^\s()]+) # Johny (?:\(\S+\))* # (TD)(TM)(SR) etc \stells\syou:\s # tells you: (?P<what>.*) # blah blah $ """, re.VERBOSE)
Finally, regexps used to interpret the finger command reply.
# Extracting info from finger command reply
re_finger_not_played = re.compile(
"has not played any rated games")
re_finger_std_rating = re.compile(
"^Standard\s+(\d+)", re.MULTILINE + re.IGNORECASE)
The class below is not required by Twisted, it is just my design.
I split the main code into two parts. FicsProtocol (below) handles
all details of FICS connection, while MyHandler is a place where
the bot-specific code is to be written. FicsProtocol (after
being extended a bit) may work as a module shared between different
bots. MyHandler is specific for given bot.
############################################################
# Handler (hooks for true code)
############################################################
class MyHandler(object):
"""
The class where we put the actual bot code (our functions)
"""
def __init__(self):
self._bad_tells = {}
# We will set this variable in FicsFactory, while
# making association between handler and protocol
self.protocol = None
The _bad_tells attribute will be used to mark which players sent
many incorrect tells to our bot (so we can avoid the dead loop
while conversating with another bot).
########################################################
# Protocol callbacks
########################################################
def onTell(self, who, what):
"""
Called whenever we obtain a tell
"""
if is_player_ignored(who):
return
# No, no chat with RoboAdmin.
This is the main design concept of Handler versus Protocol split.
Would we need to handle seeks (for example), we would add onSeek
method here (and add some code in protocol to actually call
this method after spotting some seek line). As this bot cares
only about tells, we use only onTell.
if re_cmd_join.match(what):
self.processJoin(who)
return
The bot obtained (and recognized) proper join command. As the
handling code is complicated, let's delegate it to the separate
subroutine.
# Bad command handling
c = self._bad_tells.get(who, 0) + 1
self._bad_tells[who] = c
if c <= 3:
self.protocol.runCommand(
"tell %(who)s I do not understand your command" % locals())
return
else:
twisted_log.err("More than 3 mistakes from %(who)s, ignoring his wrong command" % locals())
And here we avoid chat loops. Alternative approach would be to use timestamps and minimal delays instead of counts.
The function used to actually handle the registration process. We are using here the extremely elegant Twisted idiom called inlineCallbacks. In short, it lets one interrupt the routine at the moments one pick.
Note that this code requires Python 2.5 (or newer) to run.
########################################################
# Utility functions
########################################################
@defer.inlineCallbacks
def processJoin(self, who):
"""
Called when we receive join request. We use
finger to grab the user rating, then save
his or her data.
"""
finger_data = yield self.protocol.runCommand(
"finger %(who)s /s r" % dict(who=who),
)
The finger player /s r command is sent to FICS, then the routine
yields.
At the moment the routine yields, it's execution is suspended,
and other code will be able to run. Once the bot obtains the
reply (and callback is fired on deferred returned from runCommand),
the routine will be resumed.
Note: this is not what
yieldis usually doing. The point is in smart programming trick implemented ininlineCallbacksfunction. More-or-less it registers whole rest of the subroutine as callback to be called when the result of asynchronousrunCommandroutine is available. See article about inlineCallbacks for equivalent explicit code.You need not understand it in detail, just remember that everytime you
yield, the execution is suspended and other code may run.
rating = None
m = re_finger_std_rating.search(finger_data)
if m:
rating = int(m.group(1))
elif re_finger_not_played.search(finger_data):
rating = 0
else:
twisted_log.err("Can't detect rating of %(who)s" % locals())
rating = 0
Here we just parse the finger results. Note, that contrary to the part II, whole reply is available. Blessings of the block mode (and some code below).
register_player(who, rating)
We have all the information, so we call the registration routine. Note that if this routine was asynchronous (say, would send email and return deferred fired once it is actually sent), we would yield it
yield self.protocol.runCommand(
"tell %(who)s You are registered with rating %(rating)d" % locals())
... just like we do with this notification. It is not strictly necessary as we do not need a reply, but in case we add more code to this routine, it is safer to keep this yield to suspend it until the tell is executed.
As I said above, the protocol class is to encapsulate technical details of FICS connection - and is good candidate for code sharing.
############################################################
# Protocol (technical details of FICS connection)
############################################################
class FicsProtocol(basic.LineReceiver):
"""
Wrapper for technical details of FICS connection.
LineReceiver handles telnet for us, in this class
we handle login process, issue commands and grab
their results, interpret FICS notifications.
"""
LineReceiver handles telnet-like connection for us.
prompt = 'fics%'
def __init__(self):
# To be set later
self.handler = None
##################################################
# Login and post-login initialization
##################################################
def connectionMade(self):
twisted_log.msg("TCP connection made to the server")
self.delimiter = "\n\r"
self._inReply = None
self._keepAlive = None
self.setRawMode()
This function is called when the physical TCP connection to FICS is made.
setRawMode puts the LineReceiver class into the state in which it does not split
the information obtained into separate lines. We are to hunt for login prompt,
so we don't care about individual lines. That's not really needed, we just
use this mode to have different callbacks called while logging in and different
while working after login. Alternatively we would need to keep some state info.
def connectionLost(self, reason):
twisted_log.msg("Connection lost " + str(reason))
if self._keepAlive:
self._keepAlive.stop()
self._keepAlive = None
Called after we are disconnected. Cancelling the avoid disconnection task.
def rawDataReceived(self, data):
"""
Raw telnet data obtained. We use this mode only while
logging in, later on we switch to the line mode.
"""
Callback called by Twisted superclass when some data appears on the socket
and the object is in raw mode. As I said above, the bot is in raw mode only
during login. If we weren't using this trick, we would need to detect the state
in lineReceived below and move this code there.
if LOG_COMMUNICATION:
twisted_log.msg("Raw data received: %s" % data)
if re_login.search(data):
self.sendLine(FICS_USERNAME)
return
Login prompt detected? Let's send username.
if re_password.search(data):
self.sendLine(FICS_PASSWORD)
return
Password prompt? Here is a password.
name = self.checkIfLogged(data)
Maybe we got info that we were logged in?
if name:
self.user = name
self.setLineMode()
twisted_log.msg("Logged in as %s" % name)
self.loggedIn()
As you see we enter the line mode here, so all future communication
will result in lineReceived being called, not this function.
As it is 1:15 AM, I omitted the detection of duplicate guest account case,
and this bot will be disconnected by FICS in such a case (and try to reconnect).
This is not a good idea, so treat adding those extra regexps as an
excercise. reactor.stop() is the function to be called to stop
the program.
def checkIfLogged(self, data):
if FICS_PASSWORD:
m = re_normallogin.search(data)
if m:
return m.group(1)
else:
m = re_guestlogin.search(data)
if m:
self.sendLine("")
return m.group(1)
return None
Just a few lines of code moved to separate function for brevity
def loggedIn(self):
"""
Just logged in. Initialization
"""
Simple post-login function called to initialize FICS variables.
# Internal list of callbacks for future commands
self._reply_deferreds = []
self.sendLine("iset defprompt 1")
self.sendLine("iset nowrap 1")
self.sendLine("set interface PythonCodeFollowingMekkTutorial")
self.sendLine("set open 0")
self.sendLine("set shout 0")
self.sendLine("set cshout 0")
self.sendLine("set seek 0")
# self.sendLine("tell 0") # guest tells
self.sendLine("set gin 0")
self.sendLine("set pin 0")
self.sendLine("- channel 53")
# finger
finger = FINGER_TEXT.split("\n")[1:]
for no in range(0, 10):
text = (no < len(finger)) and finger[no] or ""
self.sendLine("set %d %s" % (no+1, text))
# Enable block mode
self.sendLine("iset block 1")
Since this very moment the way commands have to be issued rapidly changes.
# Setup
if PROTECT_LOGOUT_FREQ:
def _keep_alive_fun():
self.runCommand("date")
self._keep_alive = task.LoopingCall(_keep_alive_fun)
self._keep_alive.start(PROTECT_LOGOUT_FREQ * 60)
else:
self._keep_alive = None
task.LoopingCall is a Twisted idiom to call some code at regular
intervals. Note: no threads, no signals.
##################################################
# Normal works
##################################################
def lineReceived(self, line):
"""
Called whenever we obtain a line of text from FICS
"""
The function called by parent class whenever new line of text is available from FICS.
if LOG_COMMUNICATION:
twisted_log.msg("Received line: %s" % line)
if self._inReply:
n = re_block_reply_end.search(line)
if n:
(id, code, text) = self._inReply
self._inReply = None
self.handleCommandReply(id, code,
text + "\n" + line[:n.start()])
else:
self._inReply = (self._inReply[0], self._inReply[1],
self._inReply[2] + "\n" + line)
return
The _inReply attribute is set (a few lines below) when we are inside
a reply block (we saw reply start, but reply end is yet to come). It is
a tuple consisting of the command identifier (this magic number we use
to match reply to the command), command code (FICS identifier what kind
of command was it) and all text so far.
If this line also fails to contain the block end mark, we just append
it to the reply text. If it is there, we call handleCommandReply to
make appropriate processing of the whole reply.
So - as a whole it is a bottom part of block reply handling.
if line.startswith(self.prompt+' '):
line = line[len(self.prompt)+1:]
FICS prompt occassionally drops in.
if is_empty(line):
return
No need to handle empty lines.
m = re_block_reply_start.match(line)
if m:
id = m.group(1)
code = m.group(2)
text = line[m.end():]
n = re_block_reply_end.search(text)
if n:
self.handleCommandReply(id, code, text[:n.start()])
else:
self._inReply = (id, code, text)
return
Here we have an upper part of block reply handling. If we got the reply start mark, we extract the command id and code, and start aggregating the reply text. It may happen that the end mark is already here, in such a case we can fire reply processing. Otherwise we enter the reply aggregating state.
self.parseNormalLine(line)
Finally, it may be a normal FICS notification. As this routine is already fairly long, it is delegated to the function below.
def parseNormalLine(self, line):
"""
Here we parse the normal FICS-initiated notification.
"""
m = re_tell.match(line)
if m:
return self.handler.onTell(who = m.group('who'), what = m.group('what'))
# Good place to add other events handling
Remember how I described handler versus protocol relationship? Would we need to
handle seeks, gins, game positions, or any other information, I'd add next if here and
new handling method (onSeek, onGameInfo, ...) in the handler.
Our bot is simple, so we care only about tells.
def runCommand(self, command):
"""
Sends given command to FICS.
Returns deferred which will be fired with the command result
(given as text) - once it is obtained.
"""
# Allocate spare command id
l = len(self._reply_deferreds)
id = l
for i in range(0, l):
if not self._reply_deferreds[i]:
id = i
break
Just finding the smallest number which is curretly not used.
# Issue the command
self.sendLine('%d %s' % (id+1, command))
Issuing command according to the block mode (prefixed with identifier).
# Create the resulting deferred and save it. We will
# fire it from handleCommandReply.
d = defer.Deferred()
if id == l:
self._reply_deferreds.append(d)
else:
self._reply_deferreds[id] = d
twisted_log.msg("CommandSend(%d, %s)" % (id+1, command))
return d
I hope you already know what deferreds are? If not, revert back to part III for links.
We return one which will be fired once we get the reply to this command.
def handleCommandReply(self, id, code, text):
"""
We just obtained reply to the command identified by id.
So we locate the callback registered to handle it, and call it.
"""
# code is a command code, we don't use it currently
reply_deferred = None
pos = int(id) - 1
if pos >= 0:
reply_deferred = self._reply_deferreds[pos]
self._reply_deferreds[pos]=None
twisted_log.msg("CommandReply(%s, %s)" % (id, text))
if not reply_deferred is None:
return reply_deferred.callback(text)
And here we locate the deferred allocated to the command of given id, and fire it.
############################################################
# Factory (connection management)
############################################################
class FicsFactory(protocol.ReconnectingClientFactory):
"""
By using ReconnectingClientFactory we re-connect
every time we get disconnected from FICS.
"""
def __init__(self, handler):
self.handler = handler
# ReconettingClientFactory settings
self.maxDelay = 180
self.noisy = True
def buildProtocol(self, addr):
twisted_log.msg("Connected to the server")
self.resetDelay() # reconnection delay
self.protocol = FicsProtocol()
self.protocol.handler = self.handler
self.handler.protocol = self.protocol
return self.protocol
Glue class required by Twisted. By using ReconnectingClientFactory
we won't stop after being disconnected. The buildProtocol function
is called when the connection is made, here we decide that it will be
handled by our FicsProtocol, and not something else.
############################################################
# Main
############################################################
handler = MyHandler()
reactor.connectTCP(FICS_HOST, FICS_PORT,
FicsFactory(handler))
if STOP_AFTER:
reactor.callLater(STOP_AFTER * 60, reactor.stop)
reactor.run()
reactor.run() never ends (unless reactor.stop() is called somewhere).
This is the function which monitor all existing network connections, and other event
sources - and calls appropriate functions when something happens.
Download
You can download the code here
Summary
This example is too simple to truly show the strength of Twisted asynchronous programming. So, for a moment, you must take my word that it makes a big difference comparing to the part II. In the future I am to show some examples of more complicated code.
Some things you could spot:
-
we always know that we obtained reply to some command, and we are able to restore the context in which this command was issued (imagine extending the bot from part II so it uses
fingerin two different contexts), -
we have no problems with handling many tasks simultaneously (try logging as 2-3 guests and issuing
joinat the same time), -
we don't use threads, so we avoid whole bunch of problems related to race conditions, locks and deadlocks (also, we take noticeably less memory and don't waste CPU on context switches).
In short: we layed a good foundation for possible complicated bot.
In the next part I will write more about communicating with FICS players.