Gopherbot
DevOps Chatbot

By David Parsley, parsley@linuxjedi.org

Gophers + Robot by Renee French (cropped) licensed under Creative Commons License 3.0

Status of the Gopherbot Software

The project as a whole is going on 7 years old, and the core v1 functionality is fairly robust. On the other hand, some of the original plugins (like duo two factor) have suffered from bit-rot and may not function correctly; good news, though - the "knock knock joke" plugin definitely works.

The new job, task, pipeline and cron functionality for v2 is pretty heavily used and I have a good deal of confidence in it. The CI/CD functionality is unmaintained and not very functional. Similarly, the logging functionality and commands are not 100%.

I'm a little sorry that the only currently supported team chat connector is Slack, because that's what my team uses. This is definitely a "scratching my own itch" kind of project, and it's hard to motivate myself to write connectors for protocols I don't use. I'd be happy to work with others on this, however.

Status of this Manual

December '22 / January '23 - The manual is being actively revised for the new Gopherbot IDE.

This manual is a work in progress; currently incomplete and partly outdated. This section will be removed when the manual is considered complete. For now, to help you make the most of the manual, this is the current state of the different sections:

Front Matter

The foreword, introduction and terminology pages should be up to date.

Build and Installation

These sections are frequently updated to track with current development.

Configuration Reference

The section on environment variables is updated fairly often, as well as the section on configuration file loading. The sections dealing with actual contents of configuration files was woefully outdated and need to be replaced. The best place to look for examples of legal configuration is the defaults in conf/.

Extension Development and APIs

These sections fairly out of date and in need of TLC.

Foreword

My work with ChatOps began with Hubot around 2012 when I was working as an systems engineer for a small hosting provider. The owner had sent me a link to Hubot, asking me to have a look. I took to the concept immediately, and started automating all kinds of tasks that our support team could easily access from our team chat. Soon they were using our robot to troubleshoot email routing and DNS issues, migrate mailboxes, and build new cPanel server instances.

Not being a very talented (or motivated) Javascript/NodeJS programmer, my Hubot commands invariably followed the same pattern: write it in Javascript if it was trivially easy to do so, otherwise shell out to bash and return the results. This was productive and gave results, but it was ugly and limited in functionality.

When I began teaching myself Go, I needed a good project to learn with. After my experience with Hubot, I decided to write a robot that was more approachable for Systems and DevOps engineers like myself - tasked with providing functionality most easily accessible from e.g. bash or python scripts. Towards that end, Gopherbot's design:

• Is CGI-like in operation: the compiled server process spawns scripts which can then use a simple API for interacting with the user / chat service
• Supports any number of scripting languages by using a simple json-over-http localhost interface
• Uses a multi-process design with method calls that block

Ultimately, Gopherbot gives me a strong alternative to writing Yet Another Web Application to deliver some kind of reporting, security, or management functionality to managers and technical users. It's a good meet-in-the-middle solution that's nearly as easy to use as a web application, with some added benefits:

• The chat application gives you a single pane of glass for access to a wide range of functionality
• The shared-view nature of channels gives an added measure of security thanks to visibility, and also a simple means of training users to interact with a given application
• Like a CGI, applications can focus on functionality, with security and access control being configured in the server process

It is my hope that this design will appeal to other engineers like myself, and that somewhere, somebody will exclaim "Wait, what? I can write chat bot plugins in BASH?!?"

David Parsley, March 2017 / September 2019

#!/bin/bash

# echo.sh - trivial shell plugin example for Gopherbot

# START Boilerplate
[ -z "$GOPHER_INSTALLDIR" ] && { echo "GOPHER_INSTALLDIR not set" >&2; exit 1; }
source $GOPHER_INSTALLDIR/lib/gopherbot_v1.sh

command=$1
shift
# END Boilerplate

configure(){
	cat <<"EOF"
---
Help:
- Keywords: [ "repeat" ]
  Helptext: [ "(bot), repeat (me) - prompt for and trivially repeat a phrase" ]
CommandMatchers:
- Command: "repeat"
  Regex: '(?i:repeat( me)?)'
EOF
}

case "$command" in
# NOTE: only "configure" should print anything to stdout
	"configure")
		configure
		;;
	"repeat")
		REPEAT=$(PromptForReply SimpleString "What do you want me to repeat?")
		RETVAL=$?
		if [ $RETVAL -ne $GBRET_Ok ]
		then
			Reply "Sorry, I had a problem getting your reply: $RETVAL"
		else
			Reply "$REPEAT"
		fi
		;;
esac

Introduction

Gopherbot DevOps Chatbot is a tool for teams of developers, operators, infrastructure engineers and support personnel - primarily for those that are already using Slack or another team chat platform for day-to-day communication. It belongs to and integrates well with a larger family of tools including Ansible, git and ssh, and is able to perform many tasks similar to Jenkins or TravisCI; all of this functionality is made available to your team via the chat platform you're already using.

To help give you an idea of the kinds of tasks you can accomplish, here are a few of the things my teams have done with Gopherbot over the years:

• Generating and destroying AWS instances on demand
• Running software build, test and deploy pipelines, triggered by git service integration with team chat
• Updating service status on the department website
• Allowing support personnel to search and query user attributes
• Running scheduled backups to gather artifacts over ssh and publish them to an artifact service
• Occasionally - generating silly memes

The primary strengths of Gopherbot stem from its simplicity and flexibility. It installs and bootstraps readily on a VM or in a container with just a few environment variables, and can be run behind a firewall where it can perform tasks like rebooting server hardware over IPMI. Simple command plugins can be written in bash, python or ruby, with easy to use encrypted secrets for accomplishing privileged tasks. Like any user, the robot can also have its own (encrypted, naturally) ssh key for performing remote work and interfacing with git services.

The philosophy underlying Gopherbot is the idea of solving the most problems with the smallest set of general purpose tools, accomplishing a wide variety of tasks reasonably well. The interface is much closer to a CLI then a Web GUI, but it's remarkable what can be accomplished with a shared CLI for your team's infrastructure.

The major design goals for Gopherbot are reliability and portability, leaning heavily on "configuration as code". Ideally, custom add-on plugins and jobs that work for a robot instance in Slack should work just as well if your team moves, say, to Rocket.Chat. This goal ends up being a trade-off with supporting specialized features of a given platform, though the Gopherbot API enables platform-specific customizations if desired.

Secondary but important design goals are configurability and security. Individual commands can be constrained to a subset of channels and/or users, require external authorization or elevation plugins, and administrators can customize help and command matching patterns for stock plugins. Gopherbot has been built with security considerations in mind from the start; employing strong encryption, privilege separation, and a host of other measures to make your robot a difficult target for potential attackers.

Version 2 for the most part assumes that your robot will employ encryption and get its configuration from a git repository. Other deployments are possible, but not well documented. This manual will focus on working with Gopherbot instances whose configuration is stored on GitHub, but other git services are easy to use, as well.

That's it for the "marketing" portion of this manual - by now you should have an idea whether Gopherbot would be a good addition to your DevOps tool set.

Terminology

Up-to-date with v2.6

This section is most important for referring back to as you read the documentation, to disambiguate terms.

(It's also for the author, to help maintain some consistency)

• Gopherbot - The installed software archive that comprises the core Gopherbot DevOps Chatbot service daemon and included extensions
• robot - you'll see the term robot in several different contexts in the documentation with these several meanings:
  • robot - Far and away the most common use - a configured instance of a running Gopherbot daemon, available in your team chat; normally associated with a git repository that holds all the configuration and extensions for the robot (normally in the context of your robot)
  • Robot - The Go object passed to user plugins, jobs and tasks
  • robot - the Go library for loadable modules, i.e. import github.com/lnxjedi/robot
• default (or base) robot - If you run Gopherbot with no custom configuration, you get Floyd, the default robot; this configuration is typically found under /opt/gopherbot, and defines how a robot behaves in the absence of other configuration, and the configuration of your specific robot is merged with the base robot
• standard robot - A standard robot is what you get from using robot.skel or running the autosetup plugin from the default robot; more generally, any robot that has the standard robot.skel configuration as its base is still a standard robot
• GOPHER_HOME - The top-level directory for a given robot's repository; the Gopherbot binary (/opt/gopherbot/gopherbot) is run from this directory to start or interact with the robot
• bootstrapping - When you start the gopherbot daemon in a container or host with a few environment variables, or in an empty directory with a suitable .env environment file, the bootstrap plugin included with the default robot will use a deploy key to clone your robot from a git repository and start it up; this process is called bootstrapping your robot, allows deploying your robot to new environments quickly, and supports the devops mantras of disposable environments / infrastructure-as-code
• plugin (or command plugin) - A piece of code that provides new interactive commands; plugins may also provide code for authorization and/or elevation, which may also interact with users
• authorizer - special plugin command used to determine whether a given user is authorized for a given command, normally checking some kind of group membership
• elevator - special plugin command providing additional verification of user identity; this can be as simple as a totp token or Duo two-factor, or as complex as prompting another user before allowing a command to proceed
• job - jobs are pieces of code that typically use the pipeline API for creating pipelines to perform complex scheduled tasks such as backups and monitoring, or for software builds that may be triggered by detected updates to git repositories; see the chapter on jobs and pipelines
• task - tasks are small pieces of code that generally form the parts of a pipeline, such as initializing (and tearing down) the ssh-agent, running pipeline scripts, or sending notifications; the task is also the base object for jobs and plugins, so "task" may refer to any entry in ExternalPlugins, ExternalTasks, ExternalJobs, etc.
• parameter - a name/value setting configurable for tasks, plugins, jobs and repositories, presented to external scripts as environment variables

In addition, this manual may periodically reference four important robots:

• Floyd - Floyd is a production robot in the linuxjedi Slack team, currently running on a t2.nano in AWS. Floyd was formerly responsible for building, testing and publishing the Gopherbot artifacts, but is now relegated to looking up recipes and telling jokes. The name is taken from Infocom's Planetfall, circa 1983. Also, as above, the Gopherbot default robot is named Floyd, but every other mention of Floyd refers to my production robot, whose repository can always be found on Github.
• Data - Data was a production robot in my home Kubernetes cluster (both now retired), responsible for building and publishing Gopherbot containers and documentation, and is probably the best publicly-available example of a production robot. You can find his configuration on Github. The name is taken from Star Trek TNG.
• Clu - Clu is "the best program that's ever been written ... dogged and relentless"; also, the development robot that changes frequently as I develop Gopherbot. Clu runs on workstations, chromebooks, containers, or wherever I happen to be doing development. If you've seen Tron: Legacy - it's not THAT Clu, but rather the short-lived Clu from the original TRON, circa 1982. Clu's repository is also always available on Github, and generally has a pretty up-to-date README (not so much Floyd).
• Bishop - Bishop is used mainly for writing this documentation, and has his configuration repository wiped periodically when I test the autosetup plugin. You'll see lots of Bishop in the sections on managing your own robot. The repository may or may not be available at any given moment, and even if it is, it won't be very exciting. His name comes from Aliens, the Alien sequel, circa 1986.

The Gopherbot IDE

Starting with version 2.6, the Gopherbot CI/CD pipelines are configured to create a pre-built development container with a daily snapshot of the most current code. This container is very large, >2G, but the base is built weekly to pick up the latest security updates, and includes the following bundled software:

• A Debian base image with Ruby and Python3 (and other requirements)
• The most recent release of Go
• The most recent release of OpenVSCode Server
• Pre-installed OpenVSCode language extensions for python, ruby and Go, including required modules

The Gopherbot IDE was developed mainly for me - the Gopherbot developer - and for lazy¹ DevOps engineers and programmers that just want to quickly get up and running. All new development and documentation after Dec '22 will be created with the IDE, but the Gopherbot software will remain flexible enough to support a variety of workflows and development environments. This is just my effort at providing something that definitely works.

Requirements

To work with the Gopherbot IDE, you'll need one of:

• A Linux host with docker installed (note that Podman may work but is untested)
• A MacOS host with Docker Desktop installled
• A Windows host with the Windows Subsystem for Linux (WSL/WSL2) installed, and docker installed there

Note that the common theme here is docker and bash - the primary script for managing the Gopherbot IDE (cbot.sh) is written in bash.

The Gopherbot IDE is designed for use with ssh git credentials, though it also includes the gh utility. It would be helpful in following the tutorials to have an ssh keypair to use for development; if you're only using the IDE for managing a single robot, this could be a special-purpose keypair that you configure as a read-write deployment key for your robot git repository. Generating and configuring read-write deployment keys is outside the scope of this manual.

Previewing Gopherbot

If you're just previewing the Gopherbot software, you can just download the cbot.sh wrapper script and run the preview, otherwise you should skip down to Getting Started.

1. Download the latest cbot.sh wrapper script for docker:

curl -o cbot.sh https://raw.githubusercontent.com/lnxjedi/gopherbot/main/cbot.sh

2. Make the script executable:

chmod +x cbot.sh

3. Run the preview container:

./cbot.sh preview

4. Connect to the URL, then:

   • open a terminal window in the home (bot) directory (or type cd to change to the home directory)
   • run gopherbot to start Floyd, the default robot

5. To clean up:

./cbot.sh preview -r
docker image rm ghcr.io/lnxjedi/gopherbot-dev:latest

Getting Started

This section will describe setting up a simple directory called botwork for working with you robot(s); feel free to adjust specific paths and commands according to your preferences and comfort with the command line.

For this walk-through, the commands you'll use, along with sample generated output, are shown in text boxes. You'll need to copy/paste (or type) the commands shown, modifying for your particular setup/robot. Also note that some of the commands shown may be wider than the text box, so you'll need to slide the scrollbar to see the full command if you're typing them yourself.

1. In a terminal window create a new botwork directory, then change to that directory:

~$ mkdir botwork
~$ cd botwork/
~/botwork$

2. Download the latest cbot.sh script, mark it executable, then run it without any arguments for help/usage:

$ curl -o cbot.sh https://raw.githubusercontent.com/lnxjedi/gopherbot/main/cbot.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4787  100  4787    0     0  13599      0 --:--:-- --:--:-- --:--:-- 13560
~/botwork$ chmod +x cbot.sh
~/botwork$ ./cbot.sh
Usage: ./botc.sh profile|dev|start|stop|remove (options...) (arguments...)
...

3. Generate a new profile for working on your robot; in the example command shown, you'll need to change the robot name, full name and email address (used for attributing git pushes):

$ ./cbot.sh profile -k ~/.ssh/id_rsa dolores "David Parsley" parsley@linuxjedi.org | tee dolores.env
## Lines starting with #| are used by the cbot.sh script
GIT_AUTHOR_NAME="David Parsley"
GIT_AUTHOR_EMAIL=parsley@linuxjedi.org
GIT_COMMITTER_NAME="David Parsley"
GIT_COMMITTER_EMAIL=parsley@linuxjedi.org
#|CONTAINERNAME=dolores
#|SSH_KEY_PATH=/home/david/.ssh/id_rsa

Note that the ssh-key argument is optional. OpenVSCode includes a GitHub extension that can be used for publishing to GitHub, but that workflow isn't documented here.

4. Start a development container:

$ ./cbot.sh start dolores.env
Starting 'dolores-dev':
dolores
Copying /home/david/.ssh/id_rsa to dolores:/home/bot/.ssh/id_ssh ...
Access your dev environment at: http://localhost:7777/?workspace=/home/bot/gopherbot.code-workspace&tkn=XXXX

5. Connect to the IDE Copy the URL provided and paste it in to your browser; note that VSCode server operates best in a separate browser tab running full-screen.

That's it! You're ready to start setting up a new robot.

Installing and Configuring a Gopherbot Robot

Up-to-date with v2.6

There are three distinct tasks involved in installing and running a Gopherbot robot:

1. The first section discusses installing the Gopherbot from the pre-built distribution archive or source code, normally in /opt/gopherbot; this provides the gopherbot binary, default configuration, and an assortment of included batteries (libraries, plugins, jobs, tasks, helper scripts and more); if you're using a Gopherbot container, installation is essentially a no-op.
2. Configuring a runnable instance of a robot for your team; the included autosetup plugin should make this an "easy button" - discussed in the chapter on Initial Configuration.
3. Deploying and running your robot on a server, VM, or in a container - covered in the chapter on Running your Robot.

Gopherbot and Robots

It's helpful to understand the relationship between Gopherbot and the individual robots you'll run. It's apt to compare Gopherbot with Ansible:

• Gopherbot is similar to Ansible - a common code base with an assortment of included batteries, but with limited functionality on it's own; several Go tasks and plugins are part of the compiled binary, but the bulk of the base robot configuration is stored in yaml and script files under /opt/gopherbot
• A Robot is comparable to a collection of playbooks and/or roles - this is your code for accomplishing work in your environment, which uses Gopherbot and it's included extensions to do it's work; in the case of yaml configuration files your robot configuration is merged with the base, while jobs and plugins can be overridden by providing replacements in your robot's repository

Similar to Ansible playbooks and roles, individual robots may periodically require updates as you upgrade the Gopherbot core.

Running in a Container

If you plan on running your Gopherbot robot in a container, you can skip ahead to Team Chat Credentials. For reference, it might be useful to peruse the contents of the Gopherbot install archive describing the contents of the Gopherbot install directory and default robot.

Installation on Linux

Up-to-date with v2.6

Note that Gopherbot is currently Linux-only. Earlier versions ran on MacOS - and even Windows (with support for PowerShell plugins!) - but these platforms have fallen out of use. Patches are welcome if you'd like to maintain one of these platforms, and I'd be happy to provide advice and assistance. If you have a Mac with Docker installed or Windows with WSL and Docker, you can still use the cbot.sh bash script for setting up and running a container-based robot on your Mac or Windows machine.

The rest of this section assumes you'll be installing Gopherbot from a pre-built archive or source on a Linux host.

Software Requirements

Up-to-date with v2.6

Since Gopherbot is primarily a Go daemon that utilizes external tools and scripts to perform most of the real work, you'll probably want to have most of the (common) listed dependencies. Note that if you deploy your robot in a container, these are included in the base container, which you can further customize by adding your own tools.

• git - with version 2, Gopherbot is tightly integrated with git for updating configuration and keeping state; Gopherbot requires fairly recent versions of git supporting git remote get-url ...
  • Note that the version of git in CentOS 7 is not supported, see the Inline with Upstream Stable site for a newer version
• ssh - robots configured from this manual require ssh for setup and deployment; additionally, most robots should have an encrypted private key / public key pair for performing git operations and running remote jobs
• bash - the majority of the batteries included scripts included with Gopherbot are written in good 'ol Bash; this is nearly universal but listed here for those that may wish to build containers from scratch, since many base containers have much less functional /bin/sh shells. (busybox for example)
• jq - required by the gopherbot/lib/gopherbot_v1.sh bash library for parsing the JSON responses from the robot
  • Note this is available from the EPEL repositories for CentOS 7
• python - (version 3) next to bash, the second most common language for extensions is python version 3, which includes several management jobs

Optional

• ruby - Ruby isn't heavily used with the default extensions, but is a supported language and included in the pre-built containers
• go - While Gopherbot is written in Go, writing plugins in Go is considered an advanced topic and not well covered in this manual

Installing Gopherbot

Up-to-date with v2.6

If you want to run Gopherbot directly on a Linux host / VM, you can download a release and skip down to installing the archive.

Building from Source

Requirements:

• A recent (1.18+) version of Go
• Standard build utilities; make, tar, gzip
• A Linux system to build on that matches your target deployment host

Steps:

1. Clone the Gopherbot repository: git clone https://github.com/lnxjedi/gopherbot.git
2. make dist in the repository root; this will compile the binary and create the gopherbot-linux-amd64.tar.gz archive

Installing the Archive

1. Extract the downloaded or built archive in /opt to create /opt/gopherbot, e.g.:

[root]# cd /opt
[opt]# tar xzvf /path/to/gopherbot/gopherbot-linux-amd64.tar.gz

2. (Optional) Also as root, make the gopherbot binary setuid nobody (see below):

[opt]# cd gopherbot
[gopherbot]# ./setuid-nobody.sh

Creating Symlinks to Executables

The trivial gb-install-links script will create a set of symlinks to executables. For instance, if $HOME/bin is in your $PATH, you could:

$ /opt/gopherbot/gb-install-links $HOME/bin

See Appendix A for a description of the contents of the installation archive.

Privilege Separation

Gopherbot need never run as root; all of it's privileges derive from the collection of encrypted secrets that a given robot collects. However, given that chat bots may use 3rd-party command plugins, Gopherbot can be installed setuid nobody. This will cause the robot to run with a umask of 0022, and external plugins will run by default as real/effective user nobody. Since Gopherbot child processes do not inherit environment from the parent daemon, this effectively prevents any potential access to the GOPHER_ENCRYPTION_KEY, and any ability to modify the robot's running environment.

NOTE! Be wary of a false sense of security! The process still retains it's primary GID and supplementary groups, so if e.g. your robot unix user belongs to the wheel group, external scripts running as nobody will still be able to sudo. Privilege separation is just a simple means of providing additional hardening for your robot's execution environment.

Team Chat Credentials

Step zero for setting up a new robot is obtaining credentials to use with your team chat platform. This sections details the steps in obtaining these credentials for the only currently supported platform - Slack.

Slack Socket Mode

Starting with v2.5.0, Gopherbot uses the Slack socket mode EventsAPI for bot credentials, which has more fine-grained permissions and settings than the old RTM-style credentials.

Generating a Slack App Manifest

To save a LOT of time, we'll use a Slack app manifest to configure the settings for your app. You'll need to create a customized yaml file in a text editor based on the template provided with Gopherbot.

If you're setting up your robot in the Gopherbot IDE, you can just press <ctrl-p> to open the file search dialog and search for appmanifest.yaml, then File | Save As... to save a copy in the /home/bot directory.

You can start also with a copy of Clu's template from the Gopherbot source, or copy and paste from here:

# See: https://api.slack.com/reference/manifests
_metadata:
  major_version: 1
  minor_version: 1
display_information:
  name: Clu Gopherbot
  description: Clu tries all the new, dangerous and/or breaking changes to Gopherbot.
features:
  app_home:
    home_tab_enabled: false
    messages_tab_enabled: true
    messages_tab_read_only_enabled: false
  bot_user:
    display_name: Clu Gopherbot
    always_online: false
  slash_commands:
    - command: /clu
      description: Provides access to Clu's hidden commands
      should_escape: false
oauth_config:
  scopes:
    # See: https://api.slack.com/scopes
    bot:
      - app_mentions:read
      - channels:history
      - channels:join
      - channels:read
      - chat:write
      - chat:write.public
      - commands
      - groups:history
      - groups:read
      - groups:write
      - im:history
      - im:read
      - im:write
      - links:read
      - mpim:history
      - mpim:read
      - mpim:write
      - users.profile:read
      - users:read
      - users:read.email
      - users:write
settings:
  event_subscriptions:
    bot_events:
      - message.channels
      - message.groups
      - message.im
      - message.mpim
  interactivity:
    is_enabled: false
  org_deploy_enabled: false
  socket_mode_enabled: true
  is_hosted: false

You should change the display information and display name, and decide if you want your robot to respond to "slash commands". Keep in mind:

• When a user sends a slash command, it doesn't echo to the channel; since one of the benefits of ChatOps is the ease of learning from other users by observation, you might lose some of this benefit (the robot will still respond to it's name, and also the 1-character alias that most users prefer)
• Slack slash commands are available in every channel, even if the robot hasn't joined or been invited to the channel; individual plugins will still adhere to channel restrictions, but plugins configured with AllChannels: true (like ping) will respond everywhere

Creating a New Slack App

1. Once you're logged in to your Slack team in the browser, visit the Slack apps page, then click Create New App
2. Select From an app manifest, choose your workspace, then click Next
3. Select the YAML tab and paste in the full contents of the app manifest you created (replacing the default contents), then click Next
4. Review the settings, then click Create to create the Slack app for your robot
5. Note that the App Credentials shown aren't the credentials needed for Gopherbot
6. Click Install to Workspace, review the requested permissions, then finally click Allow

Obtaining the App Token and Bot Token for your App

Now that you've created a Slack app for your robot, you'll need to generate and/or locate the credentials you'll need for configuration.

1. From the app configuration Basic Information page, scroll down to App-Level Tokens and click the Generate Token and Scopes button
2. Give the token a name (e.g. "Token for Gopherbot"), then add both the connections:write and authorizations:read scopes
3. Click Generate, then copy and save your app token (xapp-*) in a safe place for later
4. Click Done to close the dialog
5. Select the Install App or OAuth & Permissions section on the left, then copy and save your Bot User OAuth token (xoxb-*) in a safe place for later

Initial Robot Setup

This chapter discusses the process of setting up a new Gopherbot robot to connect to your team chat. The autosetup plugin does most of the heavy lifting.

Environment Requirements

Up-to-date with v2.6

To set up your robot you'll need:

• Access to a Linux host with the Gopherbot software installed, or a running instance of the Gopherbot IDE
• The name of a channel where your robot will run jobs by default, e.g. clu-jobs or data-jobs
• A completely empty (no README, LICENSE, etc.) public or private git repository, to store your robot; a common naming convention is botname-gopherbot. For example, you can find Clu at https://github.com/parsley42/clu-gopherbot
• If you're using the IDE and/or the autosetup plugin, you'll need to be able to configure a read-only (and optionally read/write) deploy key for the robot's repository - this is widely supported with almost all of the major git hosting services and applications, check your repository settings or consult the documentation for your particular service (the GitHub documentation can be found here)

Note on Deploy Keys: If you are unfamiliar with ssh deploy keys, you should take a few minutes to read your git provider's documentation. A standard Gopherbot robot uses two deploy keys which are dedicated for use with the robot's repository.

The other requirements listed here are mainly items for consideration before setting up your Gopherbot robot.

Git Access

Gopherbot version 2 integrates heavily with git, using ssh keys for the authentication mechanism. This guide and the setup plugin require a git repository that your robot can push to it with it's encrypted management ssh key (manage_key), which will be set up as an encrypted read-write deployment key. In addition to saving it's initial configuration to this repository, the standard robot configured with this guide will back up it's long-term memories to a separate robot-state branch.

Note: The standard robot configured with this guide will have THREE DIFFERENT SSH KEYPAIRS, with the following uses:

• A dedicated encrypted manage_key, configured as a read-write deploy key for the robot's git repository; the robot will use this for saving it's initial configuration and backing up it's long-term memories from the state/ directory; it can also be used in the development lifecycle
• An unencrypted, read-only deploy_key that can be used for deploying your robot to e.g. a container or new VM
• A default encrypted robot_key which the robot will use for all other CI/CD and remote ssh jobs; this is the key that should be associated with a git user, or machine user

Additionally, you may want to take advantage of Gopherbot's CI/CD funcationality or ability to run git-driven jobs, which can be scheduled and/or on-demand. It's worth considering how you'll set up your robot to access git repositories, whether to create a new machine/robot user, or to simply add your personal robot's key to your own ssh keys.

Machine Users

If your robot will be doing a lot of git pushing and pulling, it's a good idea to create a machine account for your robot with the git service of your choice. Both Floyd and Clu have machine accounts and belong to the lnxjedi organization on Github, though Data's was just added to Floyd (since he took over his job, hah). Having an organization and adding robots to teams makes it easy to provide flexible read/write access to repositories without having to jump through repository collaborator hoops.

Deploy Keys

Github, at least, allows you to associate unique ssh deploy keys with a single repository, and even grant read-write access. The limitation of one repository per key pair increases administration overhead, and makes your robot's life more difficult. Though not fully documented here, it's possible to do this with Gopherbot by carefully managing the KEYNAME and BOT_SSH_PHRASE parameters (environment variables). See the section on task environment variables for more information on parameter precedence.

The standard setup uses a read-write deploy key because it is the easiest means of configuring your robot initially, compatible with private repositories.

User SSH Keys

Git services also allow you to add multiple ssh keys to an individual user. It's possible to add your robot's robot_key.pub, allowing your robot read-write access to all the repositories you have access to. This is the least recommended means of providing git repository write access for your robot, but may be the most expedient, and even fairly acceptable for private robots that only run on your workstation.

Brain Storage

Gopherbot supports the notion of long-term memories, which are technically just key-blob stores. The included lists and links plugins both use long-term memory storage.

File backed brains

The standard configuration for a new robot uses the file-backed brain that's backed up to a robot-state branch in the robot's git repository, with memories stored in $GOPHER_HOME/state/brain. This brain works reasonably well for most robots.

NOTE: If you write an extension that updates memories frequently, consider using memories with a _ (ephemeral memory) prefix - this will automatically exclude the memory from being backed up to git (and thus spamming the robot's repository). If your robot has frequently updated memories that require permanent storage, the default git-backed brain probably shouldn't be used.

DynamoDB brains

If you need frequently-changing memories that are backed up, you should switch to the dynamo brain. As of this writing, the AWS free tier provides a very generous 25GB of DynamoDB storage - far more than any reasonable robot should use. See the section on configuring the DynamoDB brain.

Robot Directory Structure

Up-to-date with v2.6

Gopherbot robots run in the context of a standard directory structure. The root of this directory structure is the $HOME for a given robot, and your robot starts by running the gopherbot binary in this directory. There are no requirements on this directory except that it needs to be owned and writable by the UID running the gopherbot binary; it should not be located under /opt/gopherbot, to avoid complicating upgrades. The stock containers create a bot user and home directory in /home/bot, but the software was written to e.g. pass -c <filename> to ssh to allow multiple robots to run on a single system without real system users and home directories. In this kind of setup, the standard location is /var/lib/robots/<robotname>. You can always give your robot the info command to report on where it's running, and in what directory ($GOPHER_HOME):

parsley - 10:02 AM:
*info

Mr. Data - 10:02 AM:
Here's some information about me and my running environment:
The hostname for the server I'm running on is: data-gopherbot-7878979b4-v7fsh
...
The gopherbot install directory is: /opt/gopherbot
My home directory ($GOPHER_HOME) is: /home/robot
My git repository is: git@github.com:parsley42/data-gopherbot.git
My software version is: Gopherbot v2, commit: 1a60a8e2
The administrators for this robot are: parsley
The administrative contact for this robot is: David Parsley, <parsley@linuxjedi.org>

Note that in the standard directory structure, much of the content is automatically generated when a robot first starts. Thanks to the bootstrap plugin in the default robot, a fully configured robot can be started with just a .env file in an empty directory. In the case of container-based robots, environment variables are provided by the container engine, and all content is generated during bootstrapping.

We'll use Clu as example:

• clu/ - Top-level directory, $GOPHER_HOME; mostly empty of files and not containing a git repository (.git/)
  • .env (optional, user provided) - file containing environment variables for the robot, including it's encryption key and git clone url; on start-up the permissions will be forced to 0600, or start-up will fail - note that this file may be absent in containers, where the initial environment variables are provided by the container engine
  • gopherbot (optional/generated) - convenience symlink to /opt/gopherbot/gopherbot
  • known_hosts (generated) - used by the robot to record the hosts it connects to over ssh
  • robot.log (generated) - log file created when the robot is run in terminal mode
  • custom/ (bootstrapped) - git repository for your robot, containing custom configuration, plugins, jobs and tasks; this is populated during initial robot setup, or cloned during bootstrapping - for Clu, this is https://github.com/parsley42/clu-gopherbot
  • state/ (bootstrapped) - for the standard file-backed brain, state/ contains the robot's encrypted memories in a state/brain/ directory; this directory is normally linked to the robot-state branch of the robot's configuration repository - for Clu this is https://github.com/parsley42/clu-gopherbot/tree/robot-state
  • history/ (generated) - the standard robot keeps job / plugin logs here
  • workspace/ (generated) - default location for the robot's workspace, where repositories are cloned, etc.

Where:

• generated items are created by the gopherbot binary when a robot first starts
• bootstrapped items are cloned from git during initial bootstrapping of a robot

The custom/ directory

The custom/ directory is essentially your robot, and corresponds to your robot's git repository. There's a good deal of flexibility in how the robot's custom directory is layed out, but there a few standardized locations:

• custom/ - Top-level directory for your robot's git repository
  • conf (mandatory) - location of robot's yaml configuration files
    • robot.yaml (mandatory) - primary configuration for your robot, defines all tasks, jobs, plugins, namespaces, parameter sets, and other bits
    • slack.yaml - configuration for the slack connector, including encrypted credentials and user mapping
    • terminal.yaml - configuration for the terminal connector; normally included users and channel definitions to mirror the contents of slack.yaml for use in developing extensions
    • jobs/ (mandatory) - directory of <job name>.yaml files with extended configuration for jobs defined in robot.yaml
    • plugins/ (mandatory) - directory of <plugin name>.yaml files with extended configuration for plugins defined in robot.yaml
  • git/ (mandatory)
    • config (mandatory) - contents of your robot's git config defining the name and email used for git operations
  • jobs/ (conventional) - common location for job scripts, actual path specified in robot.yaml
  • lib/ (standard) - location of script libraries; jobs and plugins run with standard environment variables for Ruby and Python so that import and require automatically look here
  • plugins/ (conventional) - common location for plugin scripts, actual path specified in robot.yaml
  • ssh/ (mandatory) - location of robot's ssh configuration files
    • config (optional) - any robot-specific ssh configuration, e.g. host-ip mappings
    • deploy_key.pub (optional) - copy of the public key used for bootstrapping
    • manage_key (optional) - encrypted private key used by the robot to save it's configuration; can be removed after initial configuration
    • manage_key.pub (optional) - public key used as a read-write deploy key, allowing the private key to store the robot's initial configuration in git; can be removed
    • robot_key (optional, default) - the robot's "personal" encrypted ssh key for other ssh / git operations
    • robot_key.pub (optional, default) - the robot's public key corresponding to robot_key; the robot will respond to show pubkey (or just pubkey) with the contents of this file
  • tasks/ (conventional) - common location for simple task scripts, actual path specified in robot.yaml

Note that many of these directories are also under /opt/gopherbot, and make up the configuration of the default (or base) robot.

During Development

When developing jobs, tasks and plugins for your robot, you'll mostly use the terminal connector and treat state/ as disposable. A fairly standard workflow goes like this:

1. Run your robot with the ./cbot.sh start <path/to/profile> script, which uses the gopherbot-dev container, providing the path to a profile used for that robot
2. Use the terminal connector, configured to mirror your team chat environment, for developing extensions for your robot
3. In the custom/ directory, create commits as desired, creating and pushing commits as normal
4. Send an administrator update command to your production robot to pull down the latest changes and reload

For more information on developing, see the chapter on Developing Extensions.

Deployment to Production

Production robots normally clone custom/ the first time they start on a new VM or container, and are updated by an administrator update command.

Production robots can also be configured to automatically update whenever the master branch updates. Floyd, for example, does this. See the Triggers: section in Floyd's updatecfg.yaml file.

Quick Start with the Gopherbot IDE

Once you have a running Goperbot IDE and the required credentials:

1. Press <ctrl-shift-`> to open a new terminal in /home/bot

2. Run gopherbot init slack to generate /home/bot/answerfile.txt:

[~]$ gopherbot init slack
Edit 'answerfile.txt' and re-run gopherbot with no arguments to generate your robot.

3. In the left-side file explorer, locate and open answerfile.txt under bot

4. Follow the directions in the file to fill in the blanks, then save the file.

If you're not already familiar with ssh deploy keys, you should read up on the documentation for your git provider; see for example the GitHub deploy keys documentation, which also has useful information about machine users.

5. When you've finished editing and saving answerfile.txt, re-run gopherbot without any arguments; your robot will process the answerfile to generate your robot's initial configuration:

$ gopherbot 
[~]$ gopherbot 
2022/12/14 17:52:50 Info: Logging to robot.log
null connector: Initializing encryption and restarting...
2022/12/14 17:52:51 Info: Logging to robot.log
null connector: Continuing automatic setup...
...
null connector: ********************************************************


null connector: Initial configuration of your robot is complete. To finish
setting up your robot, and to add yourself as an administrator:
1) Open a second terminal window in the same directory as answerfile.txt; you'll
need this for completing setup.
...
(NOTE: Scroll back to the line of *** above and follow the directions to finish
setup)

6. Follow the instructions to get your robot connected to your team chat, add yourself as a robot administrator, and save your robot to it's git repository.

NOTE: the quickest way to open a second terminal is to click the "Split Terminal" box in the upper-right corner of your initial terminal. To access the public keys for configuring deployment keys, you can e.g. cat custom/ssh/deploy_key.pub, or open it from the file explorer

7. Once you've uploaded your robot to it's git repository, you'll only need the contents of /home/bot/.env for deploying your robot - you should browse the bot section in file explorer, locate and download the .env.

8. Once you've downloaded your robot's .env, you should append it's contents to your IDE development profile for the robot, so you can easily bootstrap and update your robot later. Here's are example commands for doing this in your host OS terminal (NOT in an IDE / OpenVSCode terminal window):

david@penguin:~/botwork$ cat ~/Downloads/.env >> bishop.env # NOTE: supply the path to _your_ download directory
david@penguin:~/botwork$ rm ~/Downloads/.env
david@penguin:~/botwork$ cat bishop.env 
## Lines starting with #| are used by the cbot.sh script
GIT_AUTHOR_NAME="David Parsley"
GIT_AUTHOR_EMAIL=parsley@linuxjedi.org
GIT_COMMITTER_NAME="David Parsley"
GIT_COMMITTER_EMAIL=parsley@linuxjedi.org
#|CONTAINERNAME=bishop
#|SSH_KEY_PATH=/home/david/.ssh/id_rsa
GOPHER_ENCRYPTION_KEY=<redacted>
GOPHER_CUSTOM_REPOSITORY=git@github.com:parsley42/bishop-gopherbot.git
## You should normally keep GOPHER_PROTOCOL commented out, except when
## used in a production container. This allows for the normal case where
## the robot starts in terminal mode for local development.
GOPHER_PROTOCOL=slack
## To use the deploy key below, add ssh/deploy_key.pub as a read-only
## deploy key for the custom configuration repository.
GOPHER_DEPLOY_KEY=-----BEGIN_OPENSSH_PRIVATE_KEY-----:<redacted>:-----END_OPENSSH_PRIVATE_KEY-----:

That's it - your robot is ready to be deployed and start doing some work. Once you've saved the robot's .env file to a safe location, you can delete the container. The rest of this manual details deploying and managing your robot.

9. To clean up:

david@penguin:~/botwork$ ./cbot.sh stop bishop.env # to just stop the container
# OR
david@penguin:~/botwork$ ./cbot.sh rm bishop.env # to remove the container

Note: After setting up your robot, you may get an e-mail from GitHub (or other hosted git provider) about having uploaded an ssh private key. Don't panic! If you download the keys and try to use them with ssh-add, you'll find they're encrypted - false alarm. You'll need your robot's GOPHER_ENCRYPTION_KEY to decrypt the passphrase.

Deploying and Running Your Robot

Gopherbot is built on a self-deploying foundation, and deploying Gopherbot generally means starting the gopherbot binary in an empty directory with four environment variables set, or a .env file in the directory defining the four environment variables. Using Clu as an example:

GOPHER_ENCRYPTION_KEY=<redacted>
GOPHER_PROTOCOL=slack
GOPHER_CUSTOM_REPOSITORY=git@github.com:parsley42/clu-gopherbot.git
GOPHER_DEPLOY_KEY=-----BEGIN_OPENSSH_PRIVATE_KEY-----:<much junk removed>:-----END_OPENSSH_PRIVATE_KEY-----:

Mostly, that's is - gopherbot will recognize there's no robot present and start bootstrapping your robot from it's git repository.

NOTE: The rest of this section is outdated.

Gopherbot is very flexible about being able to bootstrap and run in a variety of environments, and is designed to be remotely updated via git integration. This chapter discusses the two primary ways you'll run your robot:

• Using Podman, Docker, Kubernetes, or any number of other container-centric environments, you can bootstrap and run your robot in a Container
• The resources/ directory contains a template robot.service that can be used to run your robot on a Linux host using Systemd

The .env file

Regardless of your running environment, you'll need a copy of your robot's .env file generated when you configured your robot. If GOPHER_PROTOCOL is set, you might want it commented out so you can run your robot with the terminal connector, instead of having it connect to your team chat. For Clu, the .env looks like this:

GOPHER_ENCRYPTION_KEY=<redacted>
#GOPHER_PROTOCOL=slack
GOPHER_CUSTOM_REPOSITORY=git@github.com:parsley42/clu-gopherbot.git
GOPHER_DEPLOY_KEY=-----BEGIN_OPENSSH_PRIVATE_KEY-----:<much junk removed>:-----END_OPENSSH_PRIVATE_KEY-----:

Deployment Environment Variables

Running in a Container

NOTE: This section is outdated.

A major goal for Gopherbot v2 was container-native operation, and the ability to run your robot with all it's functionality and state/context, without requiring a persistent volume mount or a custom image with your robot baked-in. You can launch your robot using any of the stock images; you should only need to create a custom image if your robot requires specific extra tools or libraries to do it's work, such as e.g. an ldap client or specific python module.

This section gives an example of running your robot with docker/podman, but otherwise doesn't delve in to the specifics of running your robot in any particular container environment. Whether your robot runs as a persistent container on a Docker host, or as a managed container in a Kubernetes or Openshift cluster - or elsewhere - the same principles apply. If you're planning on deploying in a container, it is presumed that you're already running other containerized workloads, and can provide the requirements by common means for your container environment.

The contents of your robot's .env file are all that's needed - the built-in bootstrap plugin will use your robot's deploy_key key to clone it's configuration from GOPHER_CUSTOM_REPOSITORY, then use the GOPHER_ENCRYPTION_KEY to decrypt the manage_key ssh private key to restore file-backed memories if you use the default file brain for the standard robot.

Container Considerations

Backing up State and Memories

The standard robot includes a backup job in the stock robot.yaml. You can modify this to set the frequency of backups. The underlying job uses git status --porcelain to determine if a backup is needed, so there's little overhead in the case where memories don't change frequently. Since new memories tend to come in batches - for instance, adding bookmarks with the links plugin - an hourly schedule probably strikes a good balance between keeping the robot-state branch updated while not creating too many small commits.

The bootstrap plugin will trigger a restore during start-up, so launching your robot into a new, empty container should restore it's state and memories automatically. Take care that you don't run multiple production instances of your robot; not only would this run the risk of corrupting state, but this can produce strange behavior in your team chat.

Providing Environment Variables

When launching your robot in a container, you'll need to provide the environment variables defined in the robot's .env file. There are two primary ways of accomplishing this:

• Both docker and podman allow you to set environment variables with a --env-file .env argument
• Container orchestration environments or e.g. docker-compose provide other means of providing these values; take care that the value for GOPHER_ENCRYPTION_KEY doesn't get committed to a repository

Setting the GOPHER_PROTOCOL Environment Variable

When starting gopherbot at the CLI, or using systemd, the value for GOPHER_PROTOCOL should be commented out in the .env; however this value is required for launching in a container so that your robot will start and connect to your team chat.

Docker Example

NOTE: This section is outdated.

For certain environments, running your robot in a standard docker container on your server or workstation is perfectly reasonable. You can use this example to help get your robot running.

Production Container

For the example, you'll need to create an empty directory named after your robot, copy the robot's .env file to <botname>/environment, and run your robot from that directory.

A command similar to this is suitable for a Linux host OS using journald for logging. Using Clu:

$ docker container run --name "clu" --restart unless-stopped -d \
	  --log-driver journald --log-opt tag="clu" \
	  --env-file environment -e HOSTNAME="my.host.name" \
	  quay.io/lnxjedi/gopherbot:latest

Then, to verify your robot is running:

$ docker logs clu
Info: PID == 1, spawning child
Info: Starting pid 1 signal handler
Initialized logging ...
...
Info: Initializing plugin: citools
Info: Initializing plugin: ssh-admin
Info: Robot is initialized and running

Note: when running on Linux, you could use podman in place of docker.

Deploying to Kubernetes

NOTE: This section is outdated.

Eventually, as Kubernetes eats the world, this will be THE way to run your Gopherbot robot. It's the platform of choice for my own production robot, Data, who runs my CI/CD pipelines and other jobs. For now, this is somewhat experimental. If you have a k8s cluster, and helm 3, see the README.md in resources/helm-gopherbot of the Gopherbot archive.

Running with Systemd

One way of running your robot is to use a systemd unit file on a systemd-managed Linux host:

• Copy resources/robot.service to /etc/systemd/system/<botname>.service and edit with values for your system; you'll need to create a local user, and a directory for your robot that the user can write to
• Reload systemd with systemctl daemon-reload
• Enable the service with systemctl enable <botname>
• Place your robot's .env in the robot's home directory, mode 0400, owned by the robot user; you can leave GOPHER_PROTOCOL commented out, since the value should be set in the <botname>.service file
• Start the service: systemctl start <botname>

That's it! Your robot should start and connect to your team chat.

Robot Basics

Central to the design of Gopherbot is the idea that once your robot connects to your team chat and joins one or more channels, the robot "hears" every message in those channels, just as a user would. Every time the robot hears a message¹:

• It checks to see if the message was directed to it by name
• It checks for "ambient" message matches
• It checks for matching job triggers

This chapter focuses on the first case - sending commands directly to your robot.

Addressing your Robot and using Ping

Most chat platforms provide some kind of capability for "mentioning" your robot using an @... syntax; normally, prefixing a message with your robot's mention name will cause it to process the message as a command. For maximum compatibility when switching chat platforms, Gopherbot robots all have a regular 'name' like 'Floyd' which they recognize, as well as a single-character 'alias'. To verify your robot "hears" messages, you would normally use the single "ping" command, which defaults to being available in all the channels where the robot is present. Here are some examples of using "ping" with Floyd, whose alias is ;:

c:general/u:alice -> floyd, ping
general: @alice PONG
c:general/u:alice -> ;ping
general: @alice PONG
c:general/u:alice -> floyd ping
general: @alice PONG
c:general/u:alice -> floyd: ping
general: @alice PONG
c:general/u:alice -> ping, floyd
general: @alice PONG
c:general/u:alice -> ping
c:general/u:alice -> floyd
general: @alice PONG
c:general/u:alice -> ping
c:general/u:alice -> ;
general: @alice PONG
c:general/u:alice -> ping floyd
c:general/u:alice ->

The last three examples are instructive: 1) if you type a command for your robot but forget to address the robot, typing the robot's name or alias alone as your next message will cause it to process your previous message as a command if done within a short period of time. 2) While "<command>, <botname>" is considered a command, "<command> <botname>" (without a comma) is not; this allows the robot to be discussed (e.g. "try using floyd") without the robot parsing it as a command.

Command Matching and "command not found"

Like the UNIX command-line, your robot is sensitive to typos; more accurately, every robot command is checked against a set of regular expressions to see if a plugin is matched. It's not rocket science, and it's not AI - it's just good 'ol regexes. When you address your robot directly, but the message doesn't match a command regex, the robot's reply is a little more verbose than "command not found":

general: @alice Sorry, that didn't match any commands I know,
or may refer to a command that's not available in this channel;
try 'floyd, help <keyword>'

If you're sure you've typed the command correctly, your plugin may not be available in the current channel; the help system is useful for that:

c:chat/u:alice -> tell me a joke, clu
chat: @alice Sorry, that didn't match any commands I know,
or may refer to a command that's not available in this channel;
try 'Clu, help <keyword>'
c:chat/u:alice -> clu, help joke
chat: Command(s) matching keyword: joke
Clu, tell me a (knock-knock) joke (channels: general, random, botdev, clu-jobs)
c:chat/u:alice -> |cgeneral
Changed current channel to: general
c:general/u:alice -> tell me a joke, clu
general: Hang on while I Google that for you (just kidding ;-)
general: @alice Knock knock
c:general/u:alice -> Who's there?
general: @alice Weevil
c:general/u:alice -> Weevil who?
general: Weevil weevil rock you

Availability by Channel

Individual Gopherbot robots normally limit commands to certain channels, contextually; for instance, the "build" command may be limited to a job channel, where developers can all "see" each other starting builds. Generally speaking, limiting commands to select channels is fairly common for a Gopherbot robot. By adding Channels: [ "foo", "bar" ] to a plugin's *.yaml configuration file, an administrator can easily override the default channels for a given plugin.

Default Channels

Many of the robot's plugins aren't context-sensitive, and can be available anywhere. Practically, however, you might want to limit these to a small number of channels to prevent common channels from getting junked up with robot noise. The slack protocol configuration for the standard robot has:

DefaultChannels: [ "general", "random" ]

The Built-in Help System

For a quick intro to the robot, the ambient¹ command "help", by itself, will provide you with a quick overview of using the help system:

c:chat/u:alice -> help
chat: @alice I've sent you a private message introducing myself
(dm:alice): Hi, I'm Clu, a staff robot. I see you've asked for help.
...

Note that if you address the robot with it's name or alias, you instead get help for all the commands in the current channel:

c:chat/u:alice -> clu, help
chat: @alice (the help output was pretty long, so I sent you a private message)
(dm:alice): Command(s) available in channel: chat
Clu, help with robot - give general help on the help system and using the robot

Clu, help <keyword> - find help for commands matching <keyword>
...

Gopherbot ships with a simple keyword-based help system. Each plugin specifies a set of help texts linked to keywords, and these can easily be added to by providing custom AppendHelp: ... for a given plugin. Since Gopherbot commands are commonly linked to channels, keyword help will list the channels where a command is available if it's not in the current channel.

Additionally, some plugins may define custom "help with <foo>" commands that give extended help information about the plugin.

Standard Plugins and Commands

In addition to "ping", and "help", which we've already introduced, there are a few other standard commands normally available to all users:

• info - provides basic information about the robot's software version and where it's running:

c:chat/u:alice -> !info
chat: Here's some information about me and my running environment:
The hostname for the server I'm running on is: 
My name is 'Clu', alias '!', and my Terminal internal ID is '(unknown)'
This is channel 'chat', Terminal internal ID: #chat
The gopherbot install directory is: /home/davidparsley/git/gopherbot
My home directory ($GOPHER_HOME) is: /home/davidparsley/git/clu
My git repository is: git@github.com:parsley42/clu-gopherbot.git
My software version is: Gopherbot v2.0.0-beta3-snapshot, commit: dec5573
The administrators for this robot are: alice
The administrative contact for this robot is: David Parsley, <parsley@linuxjedi.org>

• whoami - gives information about how the robot "sees" you, with information useful for the UserRoster:

c:chat/u:alice -> !whoami
chat: You are 'Terminal' user 'alice/u0001', speaking in channel 'chat/#chat',
email address: alice@example.com

The "links" Plugin

To simplify sharing common bookmarks with the rest of your team, you can use the "links" plugin:

c:general/u:alice -> !link employee manual to https://example.com/hr.html
general: Link added
c:general/u:alice -> !look up manual
general: Here's what I have for "manual":
https://example.com/hr.html: employee manual
c:general/u:alice -> !help with links
general: The links plugin stores URLs and associates them with a text key that can
be words or phrases. The 'link' command stores a link and key in one command, and the
'save' command will prompt the user to enter the key. The lookup command
will return all links whose key contains the provided word or phrase,
case insensitive. Links can be deleted with the 'remove' command.

The "lists" Plugin

Slightly less useful, the "lists" plugin can be used for keeping simple lists of items:

c:general/u:alice -> !add beans to the grocery list
general: Ok, I added beans to the grocery list
c:general/u:alice -> !add milk to the list
general: Ok, I added milk to the grocery list
c:general/u:alice -> !show the list
general: Here's what I have on the grocery list:
bacon
tomatoes
bananas
wine
beer
mint cookies
salad
beans
milk

Note that the last few commands simply referred to "the list", instead of fully specifying "the grocery list". Again, it's not AI; see the next section on "context".

Context

Commands can be configured to store certain matched fields as labeled context items, e.g. "item" or "list". This feature is somewhat experimental, but could occasionally be useful. A somewhat contrived example uses the "list" and "item" contexts with the aforementioned links and lists plugins:

c:general/u:alice -> !link broiled salmon to https://cooking.com/salmon.html
general: Link added
c:general/u:alice -> !add it to the dinner meals list
general: @alice I don't have a 'dinner meals' list, do you want to create it?
c:general/u:alice -> yes
general: Ok, I created a new dinner meals list and added broiled salmon to it
c:general/u:alice -> !link tuna casserole to https://cooking.com/tuna.html
general: Link added
c:general/u:alice -> !add it to the list
general: Ok, I added tuna casserole to the dinner meals list

Managing Your Robot and Adding Extensions

Gopherbot robots are designed to be remotely administered and updated, for common cases where a robot runs behind network firewalls, in virtual cloud networks, or in a container environment. Many of the frequently desired updates - such as changing the schedule of an automated job - can be safely and easily updated by pushing a commit to your robot's repository and instructing it to update. More significant updates can be tested by modelling with the terminal connector before committing and saving, then updating your production robot.

This chapter covers:

• How to update your robot with git
• The two primary ways to set up a dev environment
• Gopherbot CLI commands

You should have a robot deployed "in production" (connected to your team chat) to work the examples in the following sections.

Updating from Git

The most trivial changes can be made by pushing updates directly to your robot's repository, and instructing your robot to update. For this exercise, we'll add your robot's job channel to the list of default channels, making many more plugins available there.

Note: In the example dialogs, Bishop's alias, -, precedes most commands - you'll need to substitute your own robot's alias.

1. You can use the normal git CLI to clone the repository, or you can use your git provider's web interface to make changes. We're going to modify conf/slack.yaml; find the DefaultChannels line:

DefaultChannels: [ "general", "random" ]

... then update the list, and add your robot's job channel; using bishop as an example:

DefaultChannels: [ "general", "random", "bishop-jobs" ]

2. Commit and push your changes

3. In your robot's job channel, verify that the lists plugin isn't available:

parsley:
-list lists

Bishop Gopherbot:
@parsley: Sorry, that didn't match any commands I know, or may refer to a command that's not available in this channel; try 'bishop, help <keyword>'

4. Now instruct your robot to update it's configuration, triggering a git pull and a reload:

parsley:
-update

Bishop Gopherbot:
Ok, I'll trigger the 'updatecfg' job to issue a git pull and reload configuration...
Custom configuration repository successfully updated
@parsley: Configuration reloaded successfully
... done

5. Check and verify that the lists plugin is now available:

parsley:
-list lists

Bishop Gopherbot:
I don't have any lists

Simple as that.

Container Dev Environment

The most straight-forward and widely available way to set up a development environment for your robot is to take advantage of the gopherbot-dev container used in Quick Setup:

1. Create a new <botname>-dev directory and copy the robot's .env file to <botname>-dev/environment

I think I've repeated this at least a dozen times, but be sure GOPHER_PROTOCOL is commented out (or missing altogether) in the environment file. Not only does this prevent connecting a second robot with the same name and credentials to your team chat, but if you examine the default distributed configuration yaml files, you'll see that it disables certain scheduled jobs and modifies the robot's behavior in other ways more suitable for a dev environment.

2. From this directory, run the gopherbot-dev container, this time supplying the robot's environment. Using "clu" as an example:

$ docker run -p 127.0.0.1:3000:3000 --name clu-dev --rm --env-file environment quay.io/lnxjedi/gopherbot-dev:latest

Note that you can also find a generic run-robot.sh script to use for this in github.com/lnxjedi/gopherbot/resources/containers/dev.

3. Now open your browser and connect to http://127.0.0.1:3000, where you'll be presented with the Theia interface.

I find a three-pane layout most convenient; the top pane where code and configuration can be edited, and two terminal windows - one for running the robot in terminal mode, and one for running gopherbot CLI commands.

4. In a terminal window, run gopherbot. The bootstrap plugin will clone your robot's configuration repository, and start the robot in terminal mode:

$ gopherbot
2020/12/01 17:38:01 Initialized logging ...
...
general: Restore finished
OUT: unset SSH_AUTH_SOCK;
OUT: unset SSH_AGENT_PID;
OUT: echo Agent pid 625 killed;
c:general/u:alice ->

When you're finished with your robot, you can press <ctrl-c> to stop and remove the dev container, or from another window: $ docker stop clu-dev. Later sections will discuss how to push changes in this environment.

Local Install Dev Environment

If you've installed Gopherbot on a Linux host or VM, you can just create an empty directory under your home directory, add your .env file (without GOPHER_PROTOCOL), and start gopherbot in terminal mode - letting the bootstrap plugin retrieve the rest of your robot:

[parse@hakuin ~]$ mkdir clu-dev
[parse@hakuin ~]$ cd clu-dev/
[parse@hakuin clu-dev]$ vim .env # paste, save
[parse@hakuin clu-dev]$ ln -s /opt/gopherbot/gopherbot .
[parse@hakuin clu-dev]$ ./gopherbot 
2020/12/02 14:28:46 Initialized logging ...
2020/12/02 14:28:46 Loaded initial private environment from '.env'
...
clu-jobs: Restore finished
OUT: unset SSH_AUTH_SOCK;
OUT: unset SSH_AGENT_PID;
OUT: echo Agent pid 11337 killed;
c:general/u:alice ->

Once you're robot has bootstrapped to the directory and created the custom/ subdirectory, you can press <ctrl-d> to exit the robot, or open another terminal window in the same directory to use Gopherbot's CLI commands, discussed in the next section. When you're finished, it's good practice to remove all but the .env file and gopherbot symlink, to avoid working with a stale repository.

CLI Operation

The gopherbot binary can run as both a daemon and a command-line interface, mainly used for generating encrypted secrets. This section discusses running Gopherbot's CLI utility commands.

Setting up For CLI Operation

You'll need to create a dev environment, using either a container or local install, then run gopherbot at least once to run the bootstrap plugin and populate the custom/ subdirectory. Once bootstrapping finishes, you can press <ctrl-d> to exit from the robot, or open a separate terminal window in the same directory.

Note that the following examples assume gopherbot in the current directory is a symlink to /opt/gopherbot/gopherbot, and commands start with ./gopherbot .... If you're using the container web dev environment, you can just use gopherbot, since it'll be in your $PATH.

CLI Commands

Help

$ ./gopherbot --help
Usage: gopherbot [options] [command [command options] [command args]]
  "command" can be one of:
	decrypt - decrypt a string or file
	encrypt - encrypt a string or file
...

Encrypting and Decrypting Strings

You'll use this later in the exercise on adding plugins:

$ ./gopherbot encrypt MyLousyPassword
+LrXWBPZrbO0aJVI/lCKHR81mcD9v0LvrHojvU/qDia2lpjFNN/t+D0e5g==
$ ./gopherbot decrypt +LrXWBPZrbO0aJVI/lCKHR81mcD9v0LvrHojvU/qDia2lpjFNN/t+D0e5g==
MyLousyPassword

Encrypting and Decrypting Files

Encrypting Secrets

Once you've set up CLI access, you can use the gopherbot binary to encrypt various secrets:

$ gopherbot encrypt foobarbaz
UK+T6/HocaR9AAD8Ty2giHdNR3r03pbffpah/rk+MZumPK4Y3A==

Using the Terminal Connector

Administrator Commands

Logging

Writing Your First Plugin

Writing Custom Extensions for Your Robot

This chapter documents writing custom script extensions for your robot.

Style Guide

This section provides guidance and suggestions for making your extensions more usable.

Help for Invalid Command Syntax

One pattern for robot commands (especially for robots managing infrastructure) is to use a CLI-like <verb> (... args) or <verb-noun> (... args) structure - e.g. build-app. These kinds of commands can have arbitrarily complex syntax, however; build-app (branch) (--skip-checks), or multiple forms of the command. To mimic a CLI environment, you can take advantage of the first-match behavior in the list of CommandMatchers for a given plugin:

CommandMatchers:
- Regex: '(?i:build-?app(?: (\w[\w-]*))?(?: (--skip-checks))?)'
  Command: build
- Regex: '(?i:build-?app\b.*)'
  Command: buildhelp

Then, add code to your plugin to handle the buildhelp command (ruby example):

when "buildhelp"
    bot_alias = bot.GetBotAttribute("alias")
    bot.Say("Usage for build-app:")
    bot.Say("#{bot_alias}build-app (branch) (--skip-checks)", "fixed")

The more complex forms of build-app should come first, if none of them match, it will fall-through to buildhelp command.

NOTE:

• If a user command matches multiple plugins, the robot will complain and do nothing; the first-match behavior only applies within a single list of CommandMatchers
• The help message should probably match the configured keyword help

Tool Integrations

To simplify using tools like ssh and ansible in your pipeline, Gopherbot ships with some predefined pipeline elements detailed in the following sections.

Integrating with SSH

The GopherCI job(s) use ssh tasks for cloning repositories with public keys, and you can also use these tasks in your own pipelines for executing remote tasks.

Configuring SSH

You start by choosing a passphrase for your robot's ssh keypair - make it something quite long; you shouldn't need to type it more than once. Use the encrypt command (normally with the terminal connector) to produce the encrypted value, and put it in a stanza like the following in your robot.yaml:

ExternalTasks:
  "ssh-init":
    Parameters:
    - Name: BOT_SSH_PHRASE
      Value: {{ decrypt "xxxxx" }}

Initializing

Once the robot knows it's passphrase, you can use the generate keypair administrator command to generate a new keypair, which will be stored in $GOPHER_CONFIGDIR/ssh/. The private key is encrypted with the robot's (also encrypted) passphrase, so this can be committed to the repository. The pubkey administrator command will display the robot's public key.

Using in Pipelines

A pipeline using ssh might look something like this:

AddTask ssh-init

AddTask ssh-scan my.remote.host

AddTask exec ssh $SSH_OPTIONS user@my.remote.host "whoami"

• The ssh-init task will:
  • Start an ssh-agent and add the robot's key
  • Use SetParameter to store SSH_AUTH_SOCK and SSH_AGENT_PID in the pipeline
  • Set SSH_OPTIONS to e.g. -F $GOPHER_CONFIGDIR/ssh/config if the robot has a custom ssh config file
  • Add a FinalTask to kill the ssh-agent when the pipeline finishes
• ssh-scan insures a host is listed in known_hosts, if desired (this may be unneeded depending on the contents of ssh/config)
• The exec task calls ssh as normal, using $SSH_OPTIONS to pick up custom configuration if it exists

Configuring Gopherbot

NOTE: This chapter has not been updated for Gopherbot version 2, but has reference material that may be of use when examining default and standard configuration.

Gopherbot has very powerful and flexible configuration capabilities based on yaml templates. The core concept is simple; Gopherbot ships with default configuration in the conf/ directory of the installation archive, and individual robots can modify and override the default configuration with environment variables and custom configuration files. This chapter examines the configuration system in detail.

Configuration File Loading

Gopherbot uses YAML for it's configuration files, and Go text templates for expanding the files it reads. Any time Gopherbot loads a configuration file, say conf/robot.yaml, it first looks for the file in the installation directory, and loads and expands that file if found. Next it looks for the same file in the custom configuration directory; if found, it loads and expands that file, then recursively merges the two data structures:

• Map values merge and override
• Array values are replaced or appended

To illustrate with an example, take the following two hypothetical excerpts from terminal.yaml:

Default from the install archive:

ProtocolConfig:
  StartChannel: general
  Channels:
  - random
  - general
  StartUser: alice
UserRoster:
- UserName: "alice"
  UserID: "u0001"

Custom local configuration:

ProtocolConfig:
  StartChannel: jobs
  Channels:
  - jobs
  - general
AppendUserRoster:
- UserName: "bob"
  UserID: "u0002"

The resulting configuration would be:

ProtocolConfig:
  StartChannel: jobs
  Channels:
  - jobs
  - general
  StartUser: alice
UserRoster:
- UserName: "alice"
  UserID: "u0001"
- UserName: "bob"
  UserID: "u0002"

Template Expansion

Gopherbot uses standard Go text templates for expanding the configuration files it reads. In addition to the stock syntactic elements, the following functions and methods are available:

The env function

{{ env "GOPHER_ALIAS" }}

This would expand to the value of the GOPHER_ALIAS environment variable.

The default function

{{ env "GOPHER_ALIAS" | default ";" }}

The default function takes two arguments, and returns the first argument if it's length is > 0, the second argument otherwise. The example would expand to the value of the GOPHER_ALIAS environment variable if set, or ; otherwise.

The decrypt function

ProtocolConfig:
  SlackToken: xoxb-18000000000-470000000000-{{ decrypt "xxxxx" }}

To make it safe to store secret values in configuration, administrators can send a direct message to the robot requesting encryption, e.g.:

c:(direct)/u:alice -> encrypt MyLousyPassword
(dm:alice): RPzxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+9rf==

The encrypted value can then be pasted in to the decrypt function. See the section on encryption for more information.

NOTE: In the example, the common, low-entropy portion of a slack token is left as-is, and only the high-entropy portion of the token is encrypted, to prevent attacks where a portion of the encrypted value is known.

The .Include method

{{ .Include "terminal.yaml" }}

.Include is a method on the configuration file object, which is either an install file or a custom file. If the above example is present in the installed conf/robot.yaml, it will include only the installed conf/terminal.yaml, if present, and ignore that file if it's also present in the custom directory.

Note that .Include'd files are also expanded as templates in the same manner.

Job and Plugin Configuration

Troubleshooting

TODO: docs on using CLI commands to dump yaml.

API for Plugins, Jobs and Tasks

NOTE: This chapter is badly outdated, mainly because it's missing a lot of information. The documenation for individual API calls, however, should be mostly accurate.

Gopherbot provides an object-oriented API for writing your own command plugins, jobs and tasks. With the exception of the bash library, API calls are accessed from methods on a robot object. The following sections detail the usage of the various methods.

Gopherbot Environment Variables

Gopherbot makes extensive use of environment variables, both for configuring the robot and plugins, and for providing parameters to external scripts. This article describes the various environment variables and their use; for the environment applicable to a given running task, see per-task environment.

Robot Execution Environment

Certain environment variables can be supplied to the running Gopherbot process to configure and/or bootstrap your robot. These environment variables can be set by:

• systemd - Not recommended; while systemd can provide environment variables to your robot, it's very insecure and will allow local users on the system to view the values
• docker, docker-compose or Kubernetes - these and other container environments provide more secure means of providing environment variables to containers
• $GOPHER_HOME/.env - the most secure means of providing environment variables to your robot is creating a .env file in $GOPHER_HOME, outside of any git repository, mode 0600

The last two options are recommended for production deployments of a Gopherbot robot.

Start-up Environment

The following values can be provided to your robot on start-up:

• GOPHER_ENCRYPTION_KEY - 32+ character encryption key used for decrypting the binary-encrypted-key
• GOPHER_CUSTOM_REPOSITORY - clone URL for the robot's custom configuration, used in bootstrapping
• GOPHER_CUSTOM_BRANCH - branch to use if other than master
• GOPHER_LOGFILE - where to write out a log file
• GOPHER_CONFIGDIR - absolute or relative path to configuration directory
• GOPHER_DEPLOY_KEY - ssh deploy key for cloning the custom repository

For the optional state and private repositories, the included jobs will use the GOPHER_CUSTOM_REPOSITORY value with s/gopherbot/state/ and s/gopherbot/private/ (same branch). If desired, the values can also be supplied:

• GOPHER_STATE_REPOSITORY - repository holding state, normally just a file-backed brain, defaults to $GOPHER_CUSTOM_REPOSITORY and robot-state branch
• GOPHER_STATE_BRANCH - if GOPHER_STATE_REPOSITORY is set, this defaults to master, otherwise robot-state
• GOPHER_PRIVATE_REPOSITORY - non-public repository with environment, for dev only
• GOPHER_PRIVATE_BRANCH - branch to use if other than master

Configuration Environment Variables

Gopherbot normally takes almost all of it's configuration from the collection of *.yaml files in the custom configuration directory, but for easy flexibility, a collection of environment variables are referenced in the default configuration. These are some of the values that are expanded; the actual configuration files are the definitive reference.

• GOPHER_PROTOCOL - used to select a non-default protocol (e.g. "terminal")
• GOPHER_LOGLEVEL - error, warn, info, debug, trace
• GOPHER_BOTNAME - the name the robot will answer to, e.g. "floyd"
• GOPHER_ALIAS - the one-character alias for the robot, e.g. ";"
• GOPHER_BOTMAIL - the robot's email address
• GOPHER_BOTFULLNAME - the robot's full name
• GOPHER_HISTORYDIR - directory for storing file-based historical job logs
• GOPHER_WORKSPACE - workspace directory where e.g. build jobs clone and run
• GOPHER_BRAIN - non-default brain provider to use
• GOPHER_STATEDIR - default dir for storing state, normally just the brain
• GOPHER_BRAIN_DIRECTORY - directory where file-based memories are stored, overrides above
• GOPHER_JOBCHANNEL - where jobs run by default if not otherwise specified
• GOPHER_TIMEZONE - UNIX tz, e.g. "America/New_York" (default)

External Script Environment

Gopherbot always scrubs the environment when executing tasks, so environment variables set on execution are not automatically passed to child processes. The only environment variables that are passed through from original execution are:

• HOME - this should rarely be used; for portable robots, use GOPHER_HOME, instead
• HOSTNAME
• LANG
• PATH - this should be used with care since it can make your robot less portable
• USER

In addition to the above passed-through environment vars, Gopherbot supplies the following environment variables to external scripts:

• GOPHER_INSTALLDIR - absolute path to the gopherbot install, normally /opt/gopherbot
• RUBYLIB - path for Ruby require 'gopherbot_v1', normally /opt/gopherbot/lib
• PYTHONPATH - path for Python import, normally /opt/gopherbot/lib

Automatic Environment Variables

During startup, Gopherbot will examine it's environment and potentially set values for a few environment variables to support the bootstrap and setup plugins, and simplify common operations.

First, Gopherbot will check for custom configuration or the presence of a GOPHER_CUSTOM_REPOSITORY environment variable. In the absence of either, the following will be automatically set:

• GOPHER_UNCONFIGURED - set true
• GOPHER_LOGFILE - set to "robot.log" if not already set
• GOPHER_PROTOCOL - set to "terminal" so the default robot will start

If no custom configuration is present but GOPHER_CUSTOM_REPOSITORY is set:

• GOPHER_PROTOCOL - set to "nullconn", the null connector, to allow the bootstrap plugin to bootstrap your robot

If the robot is configured but GOPHER_PROTOCOL isn't set:

• GOPHER_PROTOCOL - set to "terminal" for local operations
• GOPHER_LOGFILE - set to "robot.log" if not already set

Finally, if encryption is initialized on start-up, GOPHER_ENCRYPTION_INITIALIZED will be set to true, regardless of whether the robot is configured.

Pipeline Environment Variables

The following environment variable are set for all pipelines, whether started by a plugin or a job:

• GOPHER_CHANNEL - the channel where the plugin/job is providing output
• GOPHER_CHANNEL_ID - the protocol channel ID
• GOPHER_MESSAGE_ID - the opaque message id of the matching message
• GOPHER_THREAD_ID - an opaque string value identifying the conversation thread
• GOPHER_THREADED_MESSAGE - set "true" if the message was received in a thread
• GOPHER_USER - the user whose message created the pipeline (if any)
• GOPHER_PROTOCOL - the name of the protocol in use, e.g. "slack"
• GOPHER_BRAIN - the type of brain in use
• GOPHER_ENVIRONMENT - "production", unless overridden
• GOPHER_PIPE_NAME - the name of the plugin or job that started the pipeline
• GOPHER_TASK_NAME - the name of the running task
• GOPHER_PIPELINE_TYPE - the event type that started the current pipeline, one of:
  • plugCommand - direct robot command, not run job ...
  • plugMessage - ambient message matched
  • catchAll - catchall plugin ran
  • jobTrigger - triggered by a JobTrigger
  • scheduled - started by a ScheduledTask
  • jobCommand - started from run job ... command

The following are also supplied whenever a job is run:

• GOPHER_JOB_NAME - the name of the running job
• GOPHER_START_CHANNEL - the channel where the job was started
• GOPHER_START_CHANNEL_ID - the protocol ID for the channel where the job was started
• GOPHER_START_MESSAGE_ID - the opaque message id for the message that started the job
• GOPHER_START_THREAD_ID - the opaque thread id where the job was started
• GOPHER_START_THREADED_MESSAGE - whether the job was started from a message in a thread
• GOPHER_REPOSITORY - the extended namespace from repositories.yaml, if any
• GOPHER_LOG_LINK - link to job log, if non-ephemeral
• GOPHER_LOG_REF - log reference used for email log and tail log commands

The following are set at the end of the main pipeline, and can be referenced in final and fail tasks:

• GOPHER_FINAL_TASK - name of final task that ran in the pipeline
• GOPHER_FINAL_TYPE - type of last task to run, one of "task", "plugin", "job"
• GOPHER_FINAL_COMMAND - if type == "plugin", set to the plugin command
• GOPHER_FINAL_ARGS - space-separated list of arguments to final task
• GOPHER_FINAL_DESC - Description: of final task
• GOPHER_FAIL_CODE - numeric return value if final task failed
• GOPHER_FAIL_STRING - string value of robot.TaskRetVal returned

Pipelines and tasks that have Homed: true and/or Privileged: true may also get:

• GOPHER_HOME - absolute path to the startup directory for the robot, relative paths are relative to this directory; unset if cwd can't be determined
• GOPHER_WORKSPACE - the workspace directory (normally relative to GOPHER_HOME)
• GOPHER_CONFIGDIR - absolute path to custom configuration directory, normally $GOPHER_HOME/custom

GopherCI Environment Variables

In addition to the environment variables set by the Gopherbot engine, the localbuild GopherCI builder sets the following environment variables that can be used to modify pipelines:

• GOPHERCI_BRANCH - the branch being built (GOPHER_REPOSITORY is set by ExtendNamespace)
• GOPHERCI_DEPBUILD - set to "true" if the build was triggered by a dependency
• GOPHERCI_DEPREPO - the updated repository that triggered this build
• GOPHERCI_DEPBRANCH - the updated branch
• GOPHERCI_CUSTOM_PIPELINE - pipeline being run if other than "pipeline"

Gopherbot's functionality can be easily extended by writing plugins in one of several different languages. A single plugin can provide:

• One or more new commands the robot will understand
• Elevation logic for providing extra assurance of user identity
• Authorization logic for determining a user's rights to issue various commands

This article deals mainly with writing plugins in one of the scripting languages supported by Gopherbot, the most popular means for writing new command plugins. For writing native compiled-in plugins in Go, see gopherbot/main.go and the sample plugins in goplugins/. API documentation for Robot methods is available at:

https://godoc.org/github.com/lnxjedi/gopherbot/bot#Robot

Note that the script plugin API is implemented on top of the native Go API, so that document may also be of use for scripting plugin authors. The file bot/http.go, and the scripting libraries in lib/ will illuminate the mapping from the script APIs to the native Go API.

Table of Contents

• Plugin Loading and Precedence
• Default Configuration
• Calling Convention
  • Environment Variables
  • Reserved Commands
• Plugin Types and Calling Events
  • Command Plugins
  • Authorization Plugins
  • Elevation Plugins
• Using the Terminal Connector
• Plugin Debugging
  • Debug Plugin Command
  • Dump Plugin Command
• Getting Started
  • Starting from a Sample Plugin
  • Using Boilerplate Code
    • Bash Boilerplate
    • Python Boilerplate
    • Ruby Boilerplate
• The Plugin API

Plugin Loading and Precedence

Gopherbot ships with a number of external script plugins in the install directory. These can be overridden by placing a plugin with the same filename in the optional configuration directory.

Default Configuration

Plugin configuration is fully documented in the configuration article; you should be familiar with that document before beginning to write your own plugins.

On start-up and during a reload, the robot will run each external script plugin with an argument of configure. The plugin should respond by writing the plugin default configuration to standard out and exiting with exit code 0. When responding to configure, the plugin shouldn't initialize a robot object or make any API calls, as configure is called without setting robot environment variables.

Calling Convention

The robot calls external plugins by creating a goroutine and exec'ing the external script with a set of environment variables. The external script uses the appropriate library for the scripting language to create a robot object from the environment. The script then examines it's command-line arguments to determine the type of action to take (normally a command followed by arguments to the command), and uses the library to make JSON-over-http calls for executing and returning results from methods. Depending on how the plugin is used, different kinds of events can cause external plugins to be called with a variety of commands and arguments. The most common means of calling an external plugin is for one of it's commands to be matched, or by matching a pattern in an ambient message (one not specifically directed to the robot).

There are two sources of information for an external plugin being called:

• Environment Variables - these should generally only be referenced by the scripting library
• Command Line Arguments - these should be used by the plugin to determine what to do

Environment Variables

Gopherbot sets two primary environment variables of use to the plugin developer:

• GOPHER_CONFIGDIR - the directory where Gopherbot looks for it's configuration
• GOPHER_INSTALLDIR - the directory where the Gopherbot executable resides

In addition, the following two environment variables are set for every script plugin:

• GOPHER_USER - the username of the user who spoke to the robot
• GOPHER_CHANNEL - the channel the user spoke in (empty string indicates a direct message)

Reserved Commands

The first argument to a plugin script is the command. In addition to the configure command, which instructs a plugin to dump it's default configuration to standard out, the following commands are reserved:

• init - After starting the connector and on reload, the robot will call external plugins with a command argument of init. Since all environment variables for the robot are set at that point, it would be possible to e.g. save a robot data structure that could be loaded and used in a cron job.
• authorize - The plugin should check authorization for the user and return Success or Fail
• elevate - The plugin should perform additional authentication for the user and return Success or Fail
• event - This command is reserved for future use with e.g. user presence change & channel join/leave events
• catchall - Plugins with CatchAll: true will be called for commands directed at the robot that don't match a command plugin. Normally these are handled by the compiled-in help plugin, but administrators could override that setting and provide their own plugin with CatchAll: true. Note that having multiple such plugins is probably a bad idea.

Plugin Types and Calling Events

There are (currently) three different kinds of external plugin:

• Command / Message Plugins - these are called by the robot in respond to messages the user sends
• Authorization Plugins - these plugins encapsulate the logic for authorizing specific users to use specific commands, and are called by the robot during authorization processing
• Elevation Plugins - these plugins perform some variety of multi-factor authentication for higher assurance of user identity, and are called by the robot during elevation processing

Command Plugins

A command plugin's configuration specifies CommandMatchers and MessageMatchers that associate regular expressions with plugin commands:

MessageMatchers:
- Command: help
  Regex: '^(?i:help)$'
CommandMatchers:
- Regex: (?i:remember ([-\w .,!?:\/]+))
  Command: remember
  Contexts: [ "item" ]

Whenever a CommandMatcher regex matches a command given to the robot, or a MessageMatcher matches an ambient message, the robot calls the plugin script with the first argument being the matched Command, and subsequent arguments corresponding to the regex capture groups (which may in some cases be an empty string). Command plugins should normally exit with status 0 (bot.Normal), or non-zero for unusual error conditions that may require an administrator to investigate. The robot will notify the user whenever a command plugin exits non-zero, or when it emits output to STDERR.

Authorization Plugins

To separate command logic from user authorization logic, Gopherbot supports the concept of an authorization plugin. The main robot.yaml can define a specific plugin as the DefaultAuthorizer, and individual plugins can be configured to override this value by specifying their own Authorizer plugin. If a plugin lists any commands in it's AuthorizedCommands config item, or specifies AuthorizeAllCommands: true, then the robot will call the authorizer plugin with a command of authorize, followed by the following arguments:

• The name of the plugin for which authorization is being requested
• The optional value of AuthRequire, which may be interpreted as a group or role
• The plugin command being called followed by any arguments passed to the command

Based on these values and the User and Channel values from the robot, the authorization plugin should evaluate whether a user/plugin is authorized for the given command and exit with one of:

• bot.Succeed (1) - authorized
• bot.Fail (2) - not authorized
• bot.MechanismFail (3) - a technical issue prevented the robot from determining authorization

Note that exiting with bot.Normal (0) or other values will result in an error and failed authentication.

Additionally, authorization plugins may provide extra feedback to the user on Fail or MechanismFail so they can have the issue addressed, e.g. "Authorization failed: user not a member of group 'foo'". In some cases, however, authorization plugins may not have a full Gopherbot API library; they could be written in C, and thus not be able to interact with the user.

Elevation Plugins

Elevation plugins provide the means to request additional authentication from the user for commands where higher assurance of identity is desired. The main robot.yaml can specify an elevation plugin as the DefaultElevator, which can be overridden by a given plugin specifying an Elevator. When the plugin lists commands as ElevatedCommands or ElevateImmediateCommands, the robot will call the appropriate elevator plugin with a command of elevate and a first argument of true or false for immediate. The elevator plugin should interpret immediate == true to mean MFA is required every time; when immediate != true, successful elevation may persist for a configured timeout period.

Based on the result of the elevation determination, the plugin should have an exit status one of:

• bot.Succeed (1) - elevation succeeded
• bot.Fail (2) - elevation failed
• bot.MechanismFail (3) - a technical issue prevented the robot from processing the elevation request

Note that exiting with bot.Normal (0) or other value will result in an error being logged and elevation failing.

Additionally, the elevation plugin may provide extra feedback to the user when elevation isn't successful to indicate the nature of the failure.

Using the Terminal Connector

Interacting with your bot in a chat app might not always be convenient or fast; to simplify testing and plugin development, Gopherbot includes a terminal connector that emulates a chat service with multiple users and channels, with a sample configuration in the cfg/term/ directory. You'll probably want to copy the directory and modify it for your own use (mainly configuring the plugins you're developing), but it can be used by using the -c <configpath> option:

[gopherbot]$ ./gopherbot -c cfg/term/
2018/04/13 18:07:52 Initialized logging ...
2018/04/13 18:07:52 Starting up with config dir: cfg/term/, and install dir: /home/user/go/src/github.com/lnxjedi/gopherbot
2018/04/13 18:07:52 Debug: Loaded installed conf/robot.yaml
2018/04/13 18:07:52 Debug: Loaded configured conf/robot.yaml
Terminal connector running; Type '|c?' to list channels, '|u?' to list users
c:general/u:alice -> |ubob
Changed current user to: bob
c:general/u:bob -> ;ping
general: @bob PONG
c:general/u:bob -> |ualice
Changed current user to: alice
c:general/u:alice -> |crandom
Changed current channel to: random
c:random/u:alice -> ;quit
random: @alice Adios
[gopherbot]$

Plugin Debugging

The most common problem for plugin authors is the robot does nothing after sending it a message, or the robot just says Sorry, that didn't match any commands I know, ....

This can be due to a number of issues:

• The plugin didn't load because of various configuration problems
• The robot isn't in the channel, and doesn't hear the message
• The plugin isn't visible because of channel, user, or other restrictions
• The user message doesn't match a regex for the plugin
• The plugin runs, but does nothing

To track down these issues easily, Gopherbot has the builtin administrator commands debug plugin and dump plugin. Make sure your username / handle is listed in the AdminUsers list in robot.yaml for your development environment.

Debug Plugin Command

Gopherbot has a builtin command for plugin debugging that can help quickly pinpoint most problems. Turning on plugin debugging will initiate a reload, then send debugging information about a plugin in direct messages. If verbose is enabled, you will get debugging information for every message you send, or every command sent to the robot by another user. You can see an example of plugin debugging here with the terminal connector:

[gopherbot]$ ./gopherbot
2018/04/18 15:43:01 Initialized logging ...
2018/04/18 15:43:01 Starting up with config dir: /home/user/.gopherbot, and install dir: /home/user/go/src/github.com/lnxjedi/gopherbot
2018/04/18 15:43:01 Debug: Loaded installed conf/robot.yaml
2018/04/18 15:43:01 Debug: Loaded configured conf/robot.yaml
Terminal connector running; Type '|c?' to list channels, '|u?' to list users
c:general/u:alice -> ;ruby me!
general: @alice Sorry, that didn't match any commands I know, or may refer to a command that's not available in this channel; try 'floyd, help <keyword>'
c:general/u:alice -> ;help debug
general: Command(s) matching keyword: debug
floyd, debug plugin <pluginname> (verbose) - turn on debugging for the named plugin, optionally verbose

floyd, stop debugging - turn off debugging
c:general/u:alice -> ;debug plugin rubydemo
general: Debugging enabled for rubydemo (verbose: false)
c:general/u:alice -> ;ruby me!
(dm:alice): 2018/04/18 03:43:15 DEBUG rubydemo: plugin is NOT visible to user alice in channel general; channel 'general' is not on the list of allowed channels: random
general: @alice Sorry, that didn't match any commands I know, or may refer to a command that's not available in this channel; try 'floyd, help <keyword>'
c:general/u:alice -> |crandom
Changed current channel to: random
c:random/u:alice -> ;ruby me to the max!
(dm:alice): 2018/04/18 03:43:44 DEBUG rubydemo: plugin is visible to user alice in channel random
(dm:alice): 2018/04/18 03:43:44 DEBUG rubydemo: Checking 7 command matchers against message: "ruby me to the max!"
(dm:alice): 2018/04/18 03:43:44 DEBUG rubydemo: Not matched: (?i:bashecho ([.;!\d\w-, ]+))
(dm:alice): 2018/04/18 03:43:44 DEBUG rubydemo: Not matched: (?i:ruby( me)?!?)
(dm:alice): 2018/04/18 03:43:44 DEBUG rubydemo: Not matched: (?i:listen( to me)?!?)
(dm:alice): 2018/04/18 03:43:44 DEBUG rubydemo: Not matched: (?i:remember(?: (slowly))? ([-\w .,!?:\/]+))
(dm:alice): 2018/04/18 03:43:44 DEBUG rubydemo: Not matched: (?i:recall ?([\d]+)?)
(dm:alice): 2018/04/18 03:43:44 DEBUG rubydemo: Not matched: (?i:forget ([\d]{1,2}))
(dm:alice): 2018/04/18 03:43:44 DEBUG rubydemo: Not matched: (?i:check me)
random: @alice Sorry, that didn't match any commands I know, or may refer to a command that's not available in this channel; try 'floyd, help <keyword>'
c:random/u:alice -> ;ruby me!
(dm:alice): 2018/04/18 03:43:49 DEBUG rubydemo: plugin is visible to user alice in channel random
(dm:alice): 2018/04/18 03:43:49 DEBUG rubydemo: Checking 7 command matchers against message: "ruby me!"
(dm:alice): 2018/04/18 03:43:49 DEBUG rubydemo: Not matched: (?i:bashecho ([.;!\d\w-, ]+))
(dm:alice): 2018/04/18 03:43:49 DEBUG rubydemo: Matched command regex '(?i:ruby( me)?!?)', command: ruby
(dm:alice): 2018/04/18 03:43:49 DEBUG rubydemo: Running plugin with command 'ruby' and arguments: [ me]
random: Sure, Alice!
random: I'll ruby you, but not right now - I'll wait 'til you're least expecting it...
(dm:alice): 2018/04/18 03:43:51 DEBUG rubydemo: Plugin finished with return value: Normal

NOTE: If your plugin is disabled with a mysterious File not found error, be sure you've got the appropriate scripting language installed. If the first line in your plugin is e.g.:

#!/usr/bin/ruby

... you'll get File not found if /usr/bin/ruby isn't present on the system.

Dump Plugin Command

To view a plugin's default or final configuration, you can use the dump plugin command:

c:general/u:alice -> ;help dump
general: Command(s) matching keyword: dump
floyd, dump plugin (default) <plugname> - dump the current or default configuration for the plugin (direct message only)

floyd, dump robot - dump the current configuration for the robot (direct message only)
c:general/u:alice -> |c
Changed current channel to: direct message
c:(direct)/u:alice -> dump plugin rubydemo
(dm:alice): AdminCommands: null
AllChannels: false
AllowDirect: true
AuthRequire: ""
AuthorizeAllCommands: false
AuthorizedCommands: null
Authorizer: ""
CatchAll: false
... (MUCH more)

Getting Started

Starting from a Sample Plugin

The simplest way for a new plugin author to get started is to:

• Disable the demo plugin for your chosen scripting language (if enabled) in <config dir>/conf/robot.yaml
• Copy the demo plugin to <config dir>/plugins/<newname>(.extension)
• Enable your new plugin in robot.yaml and give it a descriptive Name

Using Boilerplate Code

Each supported scripting language has a certain amount of "boilerplate" code required in every command plugin; generally, the boilerplate code is responsible for:

• Loading the appropriate version of the Gopherbot library from $GOPHER_INSTALLDIR/lib
• Defining and providing the default config
• Instantiating a Robot object with a library call Normally this is followed by some form of case / switch statement that performs different functions based on the contents of the first argument, a.k.a. the "command".

Bash Boilerplate

#!/bin/bash -e

source $GOPHER_INSTALLDIR/lib/gopherbot_v1.sh

COMMAND=$1
shift

configure(){
  cat <<"EOF"
<yaml config document>
EOF
}

case "$COMMAND" in
	"configure")
		configure
		;;
...
esac

NOTE: Bash doesn't have an object-oriented API

Python Boilerplate

#!/usr/bin/python3

import os
import sys
from gopherbot_v2 import Robot # use _v1 for python2

bot = Robot()

default_config = '''
<yaml config document>
'''

executable = sys.argv.pop(0)
command = sys.argv.pop(0)

if command == "configure":
    print default_config
...

Ruby Boilerplate

#!/usr/bin/ruby

# boilerplate
require 'gopherbot_v1'

bot = Robot.new()

defaultConfig = <<'DEFCONFIG'
<yaml config document>
DEFCONFIG

command = ARGV.shift()

case command
when "configure"
	puts defaultConfig
	exit
...
end

The Plugin API

Gopherbot has a rich set of methods (functions) for interacting with the robot / user. Here we break down the API into sets of related functions:

• Attribute Lookup Methods - for retrieving names, email addresses, etc.
• Sending Messages and Replies - for sending messages to the users
• Prompting for Input - for getting replies from the user
• Brain Methods and Memories - for storing and retrieving long- and short-term memories
• Utility Methods - a collection of miscellaneous useful functions, like Pause() and Log()

Getting Information About Users and the Robot

The Get*Attribute(...) family of methods can be used to get basic chat service directory information like first and last name, email address, etc. GetSenderAttribute and GetBotAttribute take a single argument, the name of the attribute to retrieve. The lesser-used GetUserAttribute takes two arguments, the user and the attribute. The return value is an object with Attribute and RetVal members. RetVal will be one of Ok, UserNotFound or AttributeNotFound.

User Attributes

The available attributes for a user / sender:

• name (handle)
• fullName
• email
• firstName
• lastName
• phone
• internalID (protocol internal representation)

Bot Attributes

The available attributes for the bot:

• name
• alias
• fullName / realName
• contact / admin / adminContact
• email
• protocol (e.g. "slack")
• internalID (protocol internal representation)

Note: the values for most of these are configured in conf/robot.yaml

Code Examples

Bash

USEREMAIL=$(GetSenderAttribute email)
if [ $? -ne $GBRET_Ok ]
then
  Say "I was unable to look up your email address"
else
  Say "Your email address is $USEREMAIL"
fi

Python

# In some cases you might forego error checking
bot.Say("You can send email to %s" % bot.GetBotAttribte("email"))
botNameAttr = bot.GetBotAttribute("fullName")
if botNameAttr.ret == Robot.Ok:
  bot.Say("My full name is %s" % botNameAttr)
else:
  bot.Say("I don't even know what my name is!")

Ruby

# In some cases you might forego error checking
bot.Say("You can send email to #{bot.GetBotAttribute("email")}")
botNameAttr = bot.GetBotAttribute("fullName")
if botNameAttr.ret == Robot::Ok
  bot.Say("My full name is #{botNameAttr}")
else
  bot.Say("I don't even know what my name is!")
end

Gopherbot comes with brain methods allowing plugin authors to store information long-term information like a TODO list, or short-term contextual information, such as a particular list item under discussion. An important supplement to this guide can be found in the example scripting plugins in the plugins/ directory, and the links and lists Go plugins in the goplugins directory (from the source, not included in the distributed .zip files).

Table of Contents

• Memory Scoping
• Long-Term Memories
  • Code Examples
  • Sample Transcript
• Short-Term Memories
  • Method Summary
  • Code Examples
  • Sample Transcript

Memory Scoping

Long-term memories are scoped per-plugin by key, and so the data is not shareable between plugins. Short-term memories are stored for each user/channel combination, and can be freely shared between plugins.

Sharing Memories

For multiple jobs and/or plugins to share the same memory namespace, create an entry in the robot's NameSpaces:, and set the NameSpace: for each job or plugin.

Simple Task Memory NameSpace

By default, simple tasks in a job or plugin pipeline inherit the NameSpace from the pipeline, allowing memory-using tasks to be re-used. If a particular task needs to use the same memories for different plugins and jobs, create a unique entry for the task in the robot's NameSpaces:, and set that task's NameSpace: accordingly.

Long-Term Memories

The following methods are available for manipulating long-term memories:

• CheckoutDatum(key, RWflag) - returns a complex data item (memory) with a short-term exclusive lock on the datum if RW is true
• CheckinDatum(memory) - signals the robot to release the lock without updating
• UpdateDatum(memory) - updates the memory and releases the lock

When checking out a memory read-write, the engine grants a short-term lock on the memory lasting 1-2 seconds. For tasks where the time between checkout and update could be longer than one second, you should check the return value on UpdateDatum - DatumLockExpired indicates another waiting task took the lock. It's worth noting that for low-contention memories, a given task can keep the memory checked out indefinitely and still update successfully; the 1-2 seconds is guaranteed, but the lock isn't actually lost unless another task checks out the memory read-write. This also means that a read-only checkout may get an outdated memory.

Long-Term Memory Code Examples

The memory stored can be an arbitrarily complex data item; a hash, array, or combination - anything that can be serialized to/from JSON. The example plugins for Python and Ruby implement a remember function that remembers a list (array) of items. For the code examples, we'll start with a memory whose key is memory, and has two items defined by this snippet of JSON:

["the Alamo", "Ferris Bueller"]

The examples will check if the memory exists, add "the answer is 42", and then update the memory.

Note that long-term memory commands aren't current implemented for bash.

Python

memory = bot.CheckoutDatum("memory", True)
if memory.exists:
    memory.datum.append("the answer is 42")
else:
    memory.datum = [ thing ]
ret = bot.UpdateDatum(memory)
if ret != Robot.Ok:
    bot.Say("Uh-oh, I must be gettin' old - having memory problems!")

Ruby

Note: the Ruby example is a little more complicated; adapted from plugins/rubydemo.rb.

memory = bot.CheckoutDatum("memory", true)
remembered = false
if memory.exists
  if memory.datum.include?(thing)
    bot.Say("That's already one of my fondest memories")
    bot.CheckinDatum(memory)
  else
    remembered = true
    memory.datum.push("the answer is 42")
  end
else
  remembered = true
  memory.datum = [ "the answer is 42" ]
end
if remembered
  ret = bot.UpdateDatum(memory)
  if ret != Robot::Ok
    bot.Say("Dang it, having problems with my memory")
  end
end

Long-Term Memory Sample Transcript

Using the terminal connector, you can see the remember function in action:

c:general/u:alice -> floyd, remember the answer is 42
general: Ok, I'll remember "the answer is 42"
c:general/u:alice -> |ubob
Changed current user to: bob
c:general/u:bob -> floyd, recall
general: Here's everything I can remember:
#1: the Alamo
#2: Ferris Bueller
#3: the answer is 42

From the transcript you can see that alice added the item to the list, which was then visible to bob. The links and lists plugins are more useful, and allow easy sharing of bookmark items or TODO lists, for example.

Short-Term Memories

Short term memories are simple key -> string values stored for each user / channel combination, and expiring after a time. The best example of this uses the built-in links and lists plugins, shown in this example using the terminal plugin:

c:general/u:alice -> link tuna casserole to https://www.allrecipes.com/recipe/17219/best-tuna-casserole/, floyd
general: Link added
c:general/u:alice -> add it to the dinner meals list
c:general/u:alice -> floyd
general: Ok, I added tuna casserole to the dinner meals list
c:general/u:alice -> floyd
general: Yes?
c:general/u:bob -> floyd, pick a random item from the dinner meals list
general: Here you go: tuna casserole
c:general/u:bob -> look it up, floyd
general: Here's what I have for "tuna casserole":
https://www.allrecipes.com/recipe/17219/best-tuna-casserole/: tuna casserole

Here, the robot is using short-term memories several times. When I forgot to address my command to the robot, the command add it to the dinner meals list was stored in short-term memory for user alice in the general channel; then, when I typed the robot's name, it checked short-term memory for the last thing alice said (stored automatically). Then, the links plugin stored tuna casserole in the item short-term contextual memory; when I used it in the lists command, the lists plugin checked the item short-term memory (see contexts in the plugin config for lists) and substituted the value from the short-term memory.

Method Summary

These methods are available for short-term memories:

• Remember(key, value) - associate the string value to key, always returns Ok
• RememberContext(context, value) - store a short-term contextual memory for use with other plugins
• Recall(key) - return the short-term memory associated with key, or the empty string when the memory doesn't exist

Note that the short-term memory API doesn't have complicated return values. They are always stored in the robot's working memory and never persisted, and expire after several minutes - so plugins should always be prepared to get blank return values.

Short-Term Memory Code Examples

Note that the short-term memory API is super trivial, so I didn't go to great lengths to provide detailed examples. The bash example comes from the bashdemo.sh plugin.

Bash

Remember "$1" "$2"
Say "I'll remember \"$1\" is \"$2\" - but eventually I'll forget!"

MEMORY=$(Recall "$1")
if [ -z "$MEMORY" ]
then
	Reply "Gosh, I have no idea - I'm so forgetful!"
else
	Say "$1 is $MEMORY"
fi

Python

bot.Remember(key, value)

mem = bot.Recall(key)

Ruby

bot.Remember(key, value)

mem = bot.Recall(key)

Short-Term Memory Sample Transcript

Here you can see the robot's short term memories of Ferris Bueller in action (using the bashdemo.sh plugin):

c:general/u:bob -> floyd, what is Ferris Bueller
general: @bob Gosh, I have no idea - I'm so forgetful!
c:general/u:bob -> floyd, store Ferris Bueller is a Righteous Dude
general: I'll remember "Ferris Bueller" is "a Righteous Dude" - but eventually I'll forget!
c:general/u:bob -> floyd, what is Ferris Bueller
general: Ferris Bueller is a Righteous Dude
c:general/u:bob -> |ualice
Changed current user to: alice
c:general/u:alice -> floyd, what is Ferris Bueller
general: @alice Gosh, I have no idea - I'm so forgetful!
c:general/u:alice -> |ubob
Changed current user to: bob
c:general/u:bob -> floyd, what is Ferris Bueller
general: Ferris Bueller is a Righteous Dude

Table of Contents

• Message Formatting
• Say and Reply
• SendUserMessage, SendChannelMessage and SendUserChannelMessage
• Code Examples
  • Bash
  • Python
  • Ruby

MessageFormat method and Message Formatting

Gopherbot is designed to provide ChatOps functionality for a variety of team chat platforms, with Slack being the first. Due to the technical nature of ChatOps, characters like _, * and ` may need to be rendered in replies to the user; at times omitting these characters (because they cause formatting changes) could remove important information. To provide the plugin author with the most flexibility, Gopherbot supports the notion of three message formats:

• Raw - text sent by a plugin with Raw format is passed straight to the chat platform as-is; this is the default if no other default is specified
• Variable - for the Variable format, the protocol connector should attempt to process the message so that special characters are escaped or otherwise modified to render for the user in a standard variable-width font; for Slack, special characters are surrounded by nulls
• Fixed - the protocol connector should render Fixed format messages in a fixed-width block format

The MessageFormat(raw|variable|fixed) method returns a robot object with the specified format. A plugin can use GetBotAttribute("protocol") to determine the connector protocol (e.g. "slack") to make intelligent decisions about the format to use, or modify the content of raw messages depending on the connection protocol.

Say/SayThread and Reply/ReplyThread

Say and Reply are the staples of message sending. Both are generally used for replying to the person who spoke to the robot, but Reply will also mention the user. Normally, Say is used when the robot responds immediately to the user, but Reply is used when the robot is performing a task that takes more than a few minutes, and the robot needs to direct the message to the user to update them with progress on the task. Both Say and Reply take a message argument, and an optional second format argument that can be variable (the default) for variable-width text, or fixed for fixed-width text. The fixed format is normally used with embedded newlines to create tabular output where the columns will line up. The return value is not normally checked, but can be one of Ok, UserNotFound, ChannelNotFound, or FailedMessageSend.

Additionally, Say and Reply will send a message to the channel for a message sent directly to the channel, or in a thread if the message came in a thread. By using the Thread variants (SayThread, ReplyThread), the robot will create a new message thread if the original message was not already in a thread.

SendUserMessage, SendChannelMessage and SendUserChannelMessage

Say and Reply are actually convenience wrappers for the Send*Message family of methods. SendChannelMessage takes the obvious arguments of channel and message and just writes a message to a channel. SendUserMessage sends a direct message to a user, and SendUserChannelMessage directs the message to a user in a channel by using a connector-specific mention. Like Say and Reply, each of these functions also takes an optional format argument, and uses the same return values.

Code Examples

Bash

# Note that bash isn't object-oriented
Say "I'm sending a message to Bob in #general"
SendUserChannelMessage "bob" "general" "Hi, Bob!"
RETVAL = $?
if [ $RETVAL -ne $GBRET_Ok ]
then
  Log "Error" "Unable to message Bob in #general - return code $RETVAL"
fi

Python

bot.Say("I'm sending a message to Bob in #general")
retval = bot.SendUserChannelMessage("bob", "general", "Hi, Bob!")
if ( retval != Robot.Ok ):
  bot.Log("Error", "Unable to message Bob in #general - return code %d" % retval)

Ruby

bot.Say("I'm sending a message to Bob in #general")
retval = bot.SendUserChannelMessage("bob", "general", "Hi, Bob!")
if retval != Robot::Ok
  bot.Log("Error", "Unable to message Bob in #general - return code %d" % retval)
end

Gopherbot takes a slightly different approach to creating pipelines; pipelines are created by Add/Fail/Final Job/Command/Task family of methods, rather than by fixed configuration directives. This allows flexible configuration of pipelines if desired for e.g. a CI/CD application, or dynamic generation of pipelines based on logic at runtime.

Until more documentation is written, see:

• The Gopherbot Pipeline Source
• The Configuration repository for Floyd, the robot that builds Gopherbot

Table of Contents

• AddTask
• SetParameter

AddTask

The AddTask method ... TODO: finish me!

Bash

AddTask "echo" "hello, world"

Python

ret = bot.AddTask("echo", ["hello", "world"])
bot.AddTask("robot-quit", [])

Ruby

ret = bot.AddTask("echo", ["hello", "world"])
bot.AddTask("robot-quit", [])

SetParameter

Bash

SetParameter "PING_LATENCY" "45ms"

Python

bot.SetParameter("PING_LATENCY", "45ms")

Ruby

bot.SetParameter("PING_LATENCY", "45ms")

The Prompt*ForReply methods make it simple to write interactive plugins where the bot can request additional input from the user.

Table of Contents

• Technical Background
• Prompting Methods
  • Method Arguments
  • Return Values
• Code Examples
  • Bash
  • Python
  • Ruby

Technical Background

Interactive plugins are complicated by the fact that multiple plugins can be running simultaneously and each can request input from the user. Gopherbot handles requests for replies this way:

1. If there are no other plugins waiting for a reply for the given user/channel, the robot emits the prompt and waits to hear back from the user
2. If other plugins are waiting for a reply, the prompt is not emitted and the request goes in to a list of waiters
3. As other plugins get replies (or timeout while waiting), waiters in the list get a RetVal of RetryPrompt, indicating they should issue the prompt request again (this is handled internally in individual scripting libraries)

Prompting Methods

The following methods are available for prompting for replies:

• PromptForReply(regexID string, prompt string) - issue a prompt to whoever/wherever the original command was issued
• PromptUserForReply(regexID string, user string, prompt string) - for prompting the user in a direct message (DM) (for e.g. a password or other sensitive information)
• PromptUserChannelForReply(regexID string, user string, channel string, prompt string) - prompt a specific user in a specific channel (for e.g. getting approval from another user for an action)

Method arguments

The user and channel arguments are obvious; the prompt is the question the robot is asking the user, and should usually end with a ?.

The regexID should correspond to a ReplyMatcher defined in the plugin configuration, (see Plugin Configuration), or one of the built-in regex's:

• Email
• Domain - an alpha-numeric domain name
• OTP - a 6-digit one-time password code
• IPAddr
• SimpleString - Characters commonly found in most english sentences, doesn't include special characters like @, {, etc.
• YesNo

Return Values

Two distinct values are returned from the prompting methods:

1. A RetVal indicating success or error condition - Reply.ret
2. When RetVal == Ok, the matched string is also returned - Reply.reply

In Go, these are returned as two separate values; in most scripting languages, these are returned as a compound object whose string representation is the returned string in Reply.reply (if the RetVal was Ok, otherwise it's the empty string).

Possible values for the RetVal in Reply.ret are:

• Ok - If the user replied and the reply matched the regex identified by regexID
• UserNotFound, ChannelNotFound - When an invalid user / channel is provided
• MatcherNotFound - When an invalid matcher is supplied
• Interrupted - If the user issues a new command to the robot (see NOTE below), too many RetryPrompt values are returned (>3), or the user replies with a single dash: '-' (cancel)
• TimeoutExpired - If the user says nothing for 45 seconds
• UseDefaultValue - If the user replied with a single equal sign (=)
• ReplyNotMatched - When the reply from the user didn't match the supplied regex (the user was probably talking to somebody else)

Code Examples

Bash

# Note that bash isn't object-oriented
REPLY=$(PromptForReply "YesNo" "Do you like kittens?")
if [ $? -ne 0 ]
then
	Reply "Eh, sorry bub, I'm having trouble hearing you - try typing faster?"
else
  if [[ $REPLY == y* ]] || [[ $REPLY == Y* ]]
  then
    Say "No kidding! Me too!"
  else
    Say "Oh, come on - you're kidding, right?!?"
  fi
fi

Python

rep = bot.PromptForReply("YesNo", "Do you like kittens?")
if rep.ret != Robot.Ok:
  bot.Say("Eh, sorry bub, I'm having trouble hearing you - try typing faster?")
else:
  reply = rep.__str__()
  if re.match("y.*", reply, flags=re.IGNORECASE):
    bot.Say("No kidding! Me too!")
  else:
    bot.Say("Oh, come on - you're kidding, right?!?")

Ruby

rep = bot.PromptForReply("YesNo", "Do you like kittens?")
if rep.ret != Robot::Ok
  bot.Say("Eh, sorry bub, I'm having trouble hearing you - try typing faster?")
else
  reply = rep.to_s()
  if /y.*/i =~ reply
    bot.Say("No kidding! Me too!")
  else
    bot.Say("Oh, come on - you're kidding, right?!?")
  end
end

Log Method

Besides the logging that Gopherbot does on it's own, plugins can also emit log messages with one of the following log levels:

• Trace - for fine-grained logging of all actions
• Debug - for emitting debugging info
• Info - the default log level
• Audit - for auditable events - NOTE: Audit events are always logged regardless of the current log level
• Warn - for potentially harmful events
• Error - for errors
• Fatal - emit fatal error and cause robot to exit(1)

Bash

Log "Error" "The robot broke"

Python

bot.Log("Error", "The robot broke")

Ruby

bot.Log(:error, "The robot broke")
# or
bot.Log("Error", "The robot broke")

Symbols look better and work just fine.

Pause Method

Every language has some means of sleeping / pausing, and this method is provided as a convenience to plugin authors and implemented natively. It takes a single argument, time in seconds.

Bash

Say "Be back soon!"
Pause 2
Say "... aaaand I'm back!"

Python

bot.Say("Be back soon!")
bot.Pause(2)
bot.Say("... aaaand I'm back!")

Ruby

bot.Say("Be back soon!")
bot.Pause(2)
bot.Say("... aaaand I'm back!")

Gopherbot Pipelines

Whenever a chat message matches a command plugin, or your robot runs a job, a new pipeline is created; or, more accurately, three pipelines are created:

• The primary pipeline does all the real work of the job or plugin, and stops on any failed task.
• The final pipeline is responsible for cleanup work, and all tasks in this pipeline always run, in reverse of the order added; see the section on the final pipeline for more information.
• The fail pipeline runs only after a failure in the primary pipeline.

NOTE: Jobs, plugins and simple tasks are all instances of a 'task'; the main differences are that 'simple' tasks are meant to be the "worker bees" for job and plugin tasks, which create new pipeline sets. A simple task never creates a new pipeline (though it may add tasks to the current set), it only serves as an element for pipelines created by jobs or plugins.

Job, Plugin and Task Configuration

Every job, plugin and task needs to be listed in robot.yaml (or an included file). See the chapter on robot configuration for more about this topic.

Pipeline State

The main state items that connect one element in a pipeline to the next are:

• The robot object, generated at the start of the pipeline - this includes a channel where the pipeline is being run, and if started interactively, the user that issued the command.
• The pipeline environment - these are environment variables provided to external scripts, or retrievable with GetParameter(...) in the Go API. Any task in the pipeline can also use the SetParameter(...) API call to set/update the environment for tasks that follow in the pipeline; for instance, the ssh-init task will set $SSH_OPTIONS that should be used with ssh commands later in the pipeline. See the section on environment variables.
• The pipeline working directory and it's contents - normally used for software builds, this is generally set early in the pipeline with SetWorkingDirectory, with a path relative to $GOPHER_WORKSPACE.

The Primary Pipeline

For those with previous experience with CI/CD and pipelines, the primary pipeline is analogous to the pipelines you've worked with before. Each task in the pipeline runs until a task fails or the pipeline completes successfully.

Unlike statically configured pipelines, once a job or plugin starts a pipeline, the AddTask and AddCommand API calls are used to add tasks to this pipeline, and can vary, for instance, based on environment variables such as $GOPHERCI_BRANCH in a software build.

In the case of most plugins, there is only a single task in the primary pipeline, the plugin itself. In the case of most jobs, the job task normally adds tasks to each of the pipelines and then exits successfully, at which point the next task in the pipeline runs.

Initializing the Environment

There are a few differences between job and plugin pipelines, one of which is the initial environment. For jobs, any parameters set for the job are propagated to the environment for the entire pipeline; plugins must use the SetParameter(...) API call to propagate values. Environment variables are more thoroughly explained in the section on task environment variables.

Populating the Pipeline Set

During the execution of the primary pipeline, the Add*, Final* and Fail* family of API calls can be used to populate the pipeline set; these calls will fail if used in the final and fail pipelines:

• AddTask, FinalTask and FailTask add simple tasks to the pipeline set.
• AddJob adds another job to the pipeline, optionally with arguments; note, however, that this creates an entirely new child pipeline, with a new environment not inherited from the parent. If the child job pipeline fails, the parent pipeline also fails.
• AddCommand, FinalCommand and FailCommand are mostly special-purpose and generally little used, they add plugin commands to the respective pipelines in the set. One use is in wrapping existing plugins with shortcut commands in another plugin - for instance, when I say to my robot "dinner?", it calls AddCommand "lists" "pick a random item from the dinner meals list"; see Floyd's util plugin. Adding plugins to the pipeline does not create a new child pipeline.

Order of Task Execution

The normal case is having an initial job or plugin task that fully populates the primary, final and fail pipelines. There are some tasks, however, that add one or more additional tasks to the pipeline set. The final and fail pipelines just grow in length, but the behavior of the primary pipeline is special:

• If tasks are added in the last task of the pipeline, the added tasks are just appended and the pipeline runs them as normal.
• If the tasks are added BEFORE the final task in the pipeline, all of the added tasks run before the next task in the original pipeline run. As an example:

Job A:

AddTask A
AddTask B
AddTask C
AddTask D

Task C:

AddTask E
AddTask F

Assuming none of the other tasks add additional tasks, they will run in the order: A, B, C, E, F, D. This allows for slightly more complicated tasks that may, for instance, have several sub-steps in setting up the working directory, and need to all complete before the next task in the job proceeds.

In the case of GopherCI, the localbuild job adds (among others) the following tasks:

• "startbuild" - a task to provide information about the build that is starting
• "run-pipeline" - a task that runs a pipeline specified in the repository
• "finishbuild" - a task to report on how the build ended

The contents of <repository>/.gopherci/pipeline.sh correspond most closely to the contents of e.g. <repository>/.travis.yml - they define the build, test and deploy tasks for the repository-specific pipeline. The task execution order for the primary pipeline insures that all the tasks in the repository pipeline are run before any other tasks added by the build job.

NOTE: "finishbuild" is actually added to the final pipeline so it always runs, however the example still holds.

The Final Pipeline

The final pipeline always runs after the primary pipeline, regardless of whether it failed, and every task in the pipeline runs, regardless of failures. Unlike the primary and fail pipelines, the tasks in the final pipeline run in FILO order - first in, last out. This creates a kind of "bracketing" behavior; the "ssh-init" task, for instance, adds a final task to kill the ssh-agent at the end of the pipeline. If the "ssh-init" task runs first, the "kill" task will run last; if a following task performs some kind of initialization / setup on a remote host, and adds a final task to clean up, this insures that the remote cleanup occurs before the ssh-agent is killed.

Note that there are several environment variables that are set at the end of the primary pipeline that can be examined and used for reporting in the final pipeline. For an example of this, see the finishbuild task that runs in the final pipeline of a GopherCI build.

The Fail Pipeline

The fail pipeline is a straight-forward pipeline of actions to take in the event of a failure in the primary pipeline. There are a host of environment variables that are set in the event of a primary pipeline failure, and fail tasks can use these for reporting the nature of the failure. Every task in the fail pipeline runs, regardless of individual task failures, in the order they were added.

Note on Parameters and Environment Variables: The Gopherbot documentation uses environment variable and parameter somewhat interchangeably; this is due to configured and set parameters being made available to external scripts as environment variables.

Task Environment Variables

Each time a task is run, a custom environment is generated for that task. For external tasks such as ssh-init, these values are actually set as environment variables for the process. Go tasks access these values with the GetParameter() API call.

The precedence of environment variables seen by a given task is determined by the algorithm in bot/run_pipelines.go:getEnvironment(). The various environment sources are listed here, in order from lowest to highest priority.

NameSpace Parameters

Various tasks, plugins and jobs can be configured to share parameters by defining NameSpaces in conf/robot.yaml, and setting the NameSpace parameter for a given task to the shared namespace. For instance, the ssh-init task and ssh-admin plugin both get access to the BOT_SSH_PHRASE environment variable via the ssh namespace.

Task Parameters

Individual tasks, plugins and jobs can also have Parameters defined. If present, these override any parameters set for a shared namespace.

Pipeline Parameters

Any time a job is run, the parameters for that job (including those inherited from it's namespace, if any) initialize the environment variables for the pipeline as a whole, and are available to all tasks in the pipeline. Parameters set in the pipeline override task and namespace parameters for any tasks run in the pipeline, allowing specific job parameters to override defaults for the task.

Plugins and Pipelines

Plugins can also start a new pipeline, but plugin parameters are not automatically added to the pipeline. Plugins can, however, explicity publish parameters to the pipeline with the SetParameter() API call.

Plugin tasks can also be added to a pipeline with the AddCommand(), FinalCommand() and FailCommand() API calls. Unlike tasks, plugins only inherit parameters from the pipeline when they are configured with Privileged: true in robot.yaml.

Repository Parameters

If a given Job calls the ExtendNamespace() API to start a build, the parameters for that repository set in conf/repositories.yaml overwrite any values in the current pipeline, which then behave as Pipeline Parameters as above.

SetParameter()

Parameters set with the SetParameter() API call overwrite the current value for a pipeline. Parameters set with SetParameter() have the highest priority, and will always apply to later tasks in the pipeline.

Included Tasks

NOTE: This section is incomplete!

Gopherbot ships with a selection of available pipeline tasks, listed here alphabetically. Note that the given examples use bash syntax for simplicity; for ruby and python see the chapter on the Gopherbot API.

email-log - privileged

Usage:

• FinalTask email-log joe@example.com emily@example.com frank bob
• FinalTask email-log

Normally used in the fail and final pipelines, emails a copy of the pipeline log. Can also be used with AddTask in the main pipeline, but content will be incomplete.

Notes on the current implementation:
The email function call takes a byte slice for the email body, rather than an io.Reader, meaning the entire body of the message is read in to memory. To limit memory use, Gopherbot allocates a 10MB line-based circular buffer, with maximum 16KB-long lines (terminated by \n), and reads the log to that buffer for emailing. Logs that are longer than 10MB will only send the last 10MB of the log, and lines longer than 16KB will be truncated.

pause-brain - privileged

Usage: AddTask pause-brain

Pause brain operations for backups and restores. After half a minute, the brain automatically resumes; best practice is to add resume-brain after the backup/restore task.

pause-brain

Usage: AddTask pause <seconds>

Causes the pipeline to pause for a number of seconds before proceeding to the next task. Useful when triggering an external process that there's no easy way to poll for completion.

restart-robot - privileged

Usage: AddTask restart-robot

Allow a privileged pipeline to queue a restart of the robot after the pipeline completes. Heavily used by bootstrap and setup plugins. NOTE: This behavior means that tasks added after restart-robot are actually completed before the restart. In practice, the current pipeline will change the state of the robot, and after restarting a plugin init function will check for the state change (e.g. presence of .restore file) and start a new pipeline.

resume-brain - privileged

Usage: AddTask resume-brain

Resume brain functions after backup/restore.

robot-quit - privileged

Usage: AddTask robot-quit

Special purpose task; once called, the robot will exit as soon as all pipelines have completed, and not allow new pipelines to start.

rotate-log - privileged

Usage: AddTask rotate-log <extension>

Useful for e.g. nightly log rotation jobs, when the robot has been started with e.g.: gopherbot -d -l robot.log. For example:

AddTask rotate-log "log.$(date +%a)"

This would keep daily logs of the form robot.log.Mon, etc.

send-message

Usage: AddTask send-message <str> (...)

Have the robot send a message in the channel, normally to report status of a build. Can be a single long string, or multiple strings; in the case of multiple strings they will be joined with a space separator.

tail-log

Usage: AddTask tail-log or FailTask tail-log

Dump the last 2k chars (in even line increments) to the channel. Normally this is used as a FailTask to show the source of the failure.

Appendix

Reference material for Gopherbot administrators and developers.

A - Gopherbot Installation Archive

Up-to-date with v2.6

This appendix describes the contents of the Gopherbot install archive.

Files

• gopherbot - the main executable, both a daemon and a command-line interface
• cbot.sh - bash script for setting up, running and developing robots in a container
• gb-install-links.sh - trivial utility for creating symlinks to the above

Directories

• conf/ - the default yaml configuration files, merged with / overridden by individual robots
  • conf/robot.yaml - the primary configuration file for a robot
  • conf/plugins/ - default configuration for distributed plugins
  • conf/jobs/ - default configuration for distributed jobs
• lib/ - API libraries for bash, python and ruby
• plugins/ - default external script plugins
• plugins/samples - sample plugins that show API usage but aren't otherwise very useful
• tasks/ - a collection of default pipeline task scripts
• jobs/ - a collection of default jobs for robot management (backup/restore) and CI/CD
• helpers/ - helper scripts not directly called by the robot
• resources/ - miscellaneous useful bits for a running robot, also the Containerfiles used for publishing the stock containers
• robot.skel/ - the initial configuration for new robots, analogous to the contents of /etc/skel
• licenses/ - licenses for other packages used by Gopherbot, as required

Appendix A: Protocols

Gopherbot communicates with users via different protocols, and extensions can modify their behavior and provide protocol-specific functionality based on values provided to the extension. For Go extensions, this is provided in the robot.Message struct provided by the GetMessage() method. For external scripts, this is provided in the GOPHER_PROTOCOL environment variable.

The design of Gopherbot is meant for mainly protocol-agnostic functionality. Running jobs and querying infrastructure should operate in much the same way whether the protocol is Slack or Terminal. However, some teams may wish to create protocol-specific extensions, and accept the risk of a more difficult transition should the team switch their primary chat platform.

A.1 Slack

The first and best-supported protocol is Slack. Developers wishing to support new protocols should consider Slack the "gold standard". Gopherbot uses the slack-go/slack library.

The Message struct for Slack will have an .Protocol value of robot.Slack, and .Incoming pointer to a robot.ConnectorMessage struct:

• Protocol: "slack"
• MessageObject: *slack.MessageEvent
• Client: *slack.Client

A.2 Rocket.Chat

Rocket.Chat is currently the only other network-based team chat protocol for Gopherbot, coded as a fallback in the event that Slack changes the API in a ChatOps un-friendly way. When it was written, the Go library for Rocket.Chat lacked some basic functionality, so it was forked to an internal version. This protocol is not currently well supported, and could use some TLC from a developer using the protocol.

Terminal

The terminal connector is the second-best supported connector, and will remain supported for the life of Gopherbot. It is built-in to the main binary, and heavily used for developing robot extensions prior to deployment in a network-connected chat protocol.

Test

The test protocol is only used for the Gopherbot integration test suite. See the test/ subdirectory for details on it's operation. The test protocol will also remain supported for the life of Gopherbot.

Nullconn

The null connector is not actually able to communicate with users, and was written to support installation and bootstrapping code. In theory, the null connector could be used in production as an alternative to e.g. cron, running jobs developed with the terminal connector; however, no known users are doing this and it's currently unsupported for this use.