1 Introduction
2 Install and Configure Watchdog
2.1 Requirements
2.2 Installation
2.3 E-Mail Server Configuration
2.4 SSH Configuration
2.5 Cluster Configuration
3 Watchdog Overview
3.1 Modules
3.2 Basic XML structure
4 Detailed XML format explanation
4.1 Process blocks
4.2 Dependencies
4.3 Execution environments
4.4 Global constants
4.5 Environment variables
4.6 Mail notification
4.7 Standard streams and working directory
4.8 Task actions
4.9 Simple calculations
4.10 Multiple module search folders
4.11 Custom success and error checker
5 Creating custom modules
5.1 Input parameter definition
5.2 Output parameter definition
5.3 Binary call command and other settings
5.4 Assign a name to the new module
5.5 Putting it all together
5.6 Other matters
6 Extend Watchdog's functionality
6.1 Virtual file systems for task actions
6.2 XML Plugins
7 Docker
7.1 Install the Watchdog Docker image
7.2 Sharing of files
7.3 Port forwarding
7.4 How to use the Docker Watchdog image
7.5 Use Docker in modules
1 Introduction Here, we present Watchdog, a WMS for the automated and distributed analysis of large-scale experimental data. Watchdog is implemented in Java and is thus platform-independent.

Main features include:

• straightforward processing of replicate data
• support for distributed computer systems
• remote storage support
• customizable error detection
• manual intervention into workflow execution
• a GUI for workflow construction using pre-defined modules
• a helper script for creating new module definitions
• no restriction to specific programming languages
• provides a flexible plugin system for extending without modifying the original sources

2 Install and Configure Watchdog 2.1 Requirements Watchdog is platform independent as it is written in Java and requires JRE (Java Runtime Environment) 1.8 or higher (the GUI is build on JavaFX that is currently not part of most OpenJDK builds). Most of the modules that are delivered with Watchdog require a unix based system and were tested on SUSE Linux. They might not run without any changes on other unix systems or Windows but users can define custom modules that are compatible with their installed software and operating system. Each of the modules might require additional software to be installed. These requirements can be checked for the modules that are delivered with Watchdog with the help of

helper_scripts/dependencyTest.sh

as described below. 2.2 Installation The installation of Watchdog is very easy. Simply extract the provided archive into a folder of your choice using

tar xfvz watchdog.tar.gz

The folder must be accessible for remote or cluster executors if you plan to use some.

Alternatively Watchdog can be installed automatically via conda using

conda install -c bioconda watchdog-wms

. In that case be binaries are named

watchdog-cmd

and

watchdog-gui

while the rest of the files is located in

${PREFIX}/share/watchdog-wms-${VERSION}

. If you want to use Watchdog with Docker, read section 7.

In the next few lines the content of each folder is explained:

• core_lib: some core functions that can be used in bash module scripts
• doc: contains Watchdog's documentation in html-format
• examples: contains the examples that are also presented in the documentation
• helper_scripts: scripts for generating new modules, configure the examples or testing of all modules
• modules: must contain all modules that should be used in workflows
• test_data: contains some test data that is used by multiple modules
• tmp: is used for Watchdog's temporary files
• webserver_data: data which is accessed by the internal webserver
• xsd: definition of the module and workflow in xsd format

To test if all software required by installed modules that call bash scripts is available the script

helper_scripts/dependencyTest.sh

can be executed. It checks if all requirements are installed that are also enforced by the module itself. Only dependencies that are defined in the bash script itself stored in the variable named

$USED_TOOLS

are detected by the script. The system must be able to locate the dependencies by using the PATH variable. Moreover, all R and perl packages that are used in scripts are checked.

In order to test if all modules that provide tests work as expected on your system you can run

helper_scripts/moduleTest.sh

. If you want to test the examples which are discussed in this manual, you can configure them by running:

helper_scripts/configureExamples.sh -i /path/to/install/folder/of/watchdog [-m your@mail-adress.com]

(mail attribute (-m) is optional)

Afterwards the configured examples will be located in

/path/to/install/folder/of/watchdog/examples/

and can be executed (from the watchdog installation directory) using the following command:

./watchdog.sh -x examples/filename.xml

or alternatively

java -jar jars/watchdog.jar -x examples/filename.xml

For instance:

./watchdog.sh -x examples/workflow1_basic_information_extraction.xml

If you want to use the workflow designer (GUI), you can start it by using (from the watchdog installation directory):

./workflowDesigner.sh

or alternatively

java -jar jars/WatchdogDesigner.jar 

2.3 E-Mail Server Configuration As Watchdog will send e-mails it needs a working mail configuration. If you don't want Watchdog to send e-mails, simply don't use the

mail

attribute of the

tasks

tag. In that case the content of the mails with be printed to the standard output stream.

By default a server listening on SMTP port 25 is expected that accepts mails without authentication. In order to use another configuration the parameter

-mailConfig

of Watchdog can be used. It expects a tab-separated file that contains information on how to connect to the mail server using the SMTP protocol. If the mail server expects some authentication we strongly suggest to use a mail account that was explicitely created for the use with Watchdog as the password is stored unencrypted. Example 1 shows a configuration for a gmail account. More information about the variables that can be used can be found here. An template mail config file that can be edited can be found in

examples/mail_config

once the examples are configured as described above. 2.4 SSH Configuration Watchdog supports execution of tasks via ssh on remote hosts. In order to use that feature a private ssh key must be provided. It is strongly recommended that the private key is protected by a passphrase. In that case the passphrase must be entered after Watchdog was started and will be hold encrypted in memory until the passphrase is needed.

A key pair that can be used for ssh authentification can be generated using the tool

ssh-keygen

that is part of

openssh

. If you need further information you can find many online tutorials that explain how to use a private key for ssh authentication. E.g. How To Set Up SSH Keys and SSH/OpenSSH/Keys 2.5 Cluster Configuration Watchdog supports cluster solutions which provide a

DRMAA

java binding. By default it is bundled with a DRMAA binding for the

sun grid engine

(SGE 6.1).

The following environment variables must be set correctly in order to communicate with the SGE:

• SGE_ROOT: path to the installation folder of the SGE
• LD_LIBRARY_PATH: path to the library path of the SGE; in most cases it will be

  $SGE_ROOT/lib/lx24-amd64

  or

  $SGE_ROOT/lib/lx24-x86

Basically there are two ways to change the default cluster extension in order to use another DRMAA solution than the SGE:

• dynamically by adding arguments to the jar invocation:
  1. set class name of DRMAA Sessionfactory via -Dorg.ggf.drmaa.SessionFactory=classname
  2. add DRMAA java binding to class path via -cp

     /path/to/drmaaImplementation.jar

• permanently by changing Watchdog's jar file:
  1. jar files can be opened and edited with every tool that supports zip files
  2. replace name of DRMAA Sessionfactory stored in

     /META-INF/services/org.ggf.drmaa.SessionFactory

  3. add class files of the DRMAA java binding to Watchdog's jar file

Probably further settings are needed which depend on the used DRMAA library. 3 Watchdog Overview 3.1 Modules Modules represent re-usable components that perform certain tasks, e.g. compression of files or creating histograms. Watchdog is delivered with a set of predefined modules. Additionally the user has the possibility to define own modules as described in section 5. The modules are stored in the

modules

directory located in the root folder of the Watchdog installation. Each module is stored in its own folder and consists at least of an XSD file with the name of the module. The XSD file contains a definition of the parameters which can be set in the XML format and the tools which are executed in the background when the module is used. Example 2 shows parts of the XSD definition of a module which are important to know for the user. At the beginning parameters and flags which are accepted by the module are defined in an element named

sleepTaskParameterType

(5-9). In this case only one parameter named

wait

is defined that must occur exactly once (7). The type of the parameter is specified at the bottom of the example (17-23). In this case the parameter is a string that must match a regex pattern, which first accepts numbers followed by a letter as optional suffix (19-21). Additionally values that are placeholders for constants or variables are allowed by the first

matches()

function whereby the user must take care that the replaced value is valid with regard to the second part of the specification (20).
The attribute

name

of the element in line 14 defines how the module can be referenced in the XML file. In this example the module can be called using the name

sleepTask

. 3.2 Basic XML structure Tasks which should be executed by Watchdog must be defined in an XML file. In the following the structure of the XML file is presented. The expression

?Task

is used to refer to a task which is not further specified. In general this syntax is used if some attributes are valid for all classes that inherit from that class type. Within the following examples these variables are user-specific and contain therefore no concrete values:

{%INSTALL%} - path to the root installation directory of Watchdog
{%MAIL%} - email adress of the user
{%EXAMPLE_DATA%} - path to the folder in which the example data is located

You already have configured your examples by calling the script

helper_scripts/configureExamples.sh

as described in 2. Example 3 shows a most basic XML file that contains only a single task named

sleep

. Every XML file that should be parsed by Watchdog must contain a

watchdog

element as root element (2). The attribute

watchdogBase

of it must refer to the folder in which Watchdog was installed. The attribute

isTemplate

prevents Watchdog from executing workflows that contain variables that must be set by the user and is removed automatically by the configure script. Afterwards as childs of

tasks

the tasks which should be executed must be defined (5). Each task must contain an

id

and

name

attribute (8). With the

parameter

element, values can be assigned to the parameters of the task, which have to be specified in the XSD file of the module. Flags are activated by using

flagName

true

/flagName

or

flagName

1

/flagName

while parameters can be set with

paramName

value

/paramName

(10). In the following sections the structure of the XML format is described in greater detail. 4 Detailed XML format explanation In the following sections the complete range of functions of Watchdog's XML format is explained. The following elements must be defined as child elements of the

settings

element before the

tasks

element begins and are valid within the complete XML file:

• processBlock

  - process a task with varying parameters (see 4.1)

• executors

  - define different executor environments (see 4.3)

• constants

  - defines constants that substitute placeholders (see 4.4)

• environments

  - define or update environment variables (see 4.5)

• modules

  - define multiple module include directories (see 4.10)

Apart from the

parameter

element, the following elements are allowed in

?Task

elements:

• environment

  - define or update environment variables (see 4.5)

• dependencies

  - define dependencies between tasks (see 4.2)

• streams

  - define location of standard streams and set a working directory (see 4.7)

• checkers

  - usage of custom success or error checkers (see 4.11)

• actions

  - define task actions that are performed before or after tasks execution (see 4.8)

4.1 Process blocks Watchdog is able to process multiple tasks of the same type, which differ only in some parameter values, without the need to define all of these tasks separately. This function is referred to as process blocks while the tasks created by an process block are called subtasks of the task. There are four different possibilities to define process blocks as childs of the

processBlock

element:

• processSequence

  - argument is numeric

• processFolder

  - argument is a path to a file

• processInput

  - multiple arguments obtained from dependencies

• processTable

  - multiple arguments stored in a tab-separated file with names of variables stored in the first line

When the

processBlock

attribute of a task is set the argument of the process folder or sequence is substituted at run time within

parameter

,

streams

,

checkers

,

actions

and

environment

elements in the following manner:

• processSequence

  - []/{}/() -> number

• processFolder

  - {} -> absolute path to the file

• processFolder

  - () -> absolute path to the parent folder of the file

• processFolder

  - [] -> name of the file

• processFolder

  - [

  n

  ]/{

  n

  } ->

  n

  suffixes of the filename are truncated using . as separator

• processFolder

  - (

  n

  ) ->

  n

  suffixes of the parent folder are truncated using / as separator

• processFolder

  - ([{

  n

  ,

  sep

  }]) -> suffixes of the value are truncated using

  sep

  as separator (might also be a regex)

• processTable

  - ([{

  $COL_NAME

  }]) -> value stored in the column named

  $COL_NAME

• processTable

  - ([{

  $COL_NAME

  ,

  n

  ,

  sep

  }]) -> value stored in the column named

  $COL_NAME

  but with suffix truncation as described above

• processInput

  - ([{

  $RET_NAME

  }]) -> return value of a dependency with the name

  $RET_NAME

• processInput

  - ([{

  $RET_NAME

  ,

  n

  ,

  sep

  }]) -> return value of a dependency with the name

  $RET_NAME

  but with suffix truncation as described above

If a task depends on two tasks, which return variables with the same name, the return value of the task with the smaller id will be overwritten. Deviating from this, return values from separate dependencies will overwrite the ones from global dependencies if both use the same name for a variable. Example 4 shows three different ways how process blocks can be specified. First a

processSequence

named

qualities

is defined that creates the numbers 1,3,5,7 and 9 (7). In the next line a

processFolder

is defined that will process all files stored in

{%EXAMPLE_DATA%}/spec

that end with

.spec

(8). The syntax which must be used in the

pattern

attribute is the same as in bash. If a

processFolder

is a child element of a

baseFolder

, the

folder

attribute of the

processFolder

will be prefixed with the

folder

attribute of the

baseFolder

(9-14). The attribute

disableExistenceCheck

that is enabled for the

processFolder

with the name

gzFiles

causes Watchdog not to force the existence of the folder when it is started (12).
The task with

id

1

will compress all

.txt

files in the folders

{%EXAMPLE_DATA%}/txt

and

{%EXAMPLE_DATA%}/other_txt

and store them in

{%EXAMPLE_DATA%}/txt_zipped

(20-25). Whereas, the task with

id

2

will compress a file with different quality values (28-40). The compressed files will be stored with

txtFile1_q

as prefix and the used quality as suffix in

{%EXAMPLE_DATA%}/qualityTest

(34). Additionally, an environment variable with the name

QUALITY

is set which also contains the set quality (38). The sleep task at the end of the example shows how the colums of a

processTable

can be used as input (43-57). If the variable is of numeric type it can also be used within simple calculations which are presented in 4.9 (55). 4.2 Dependencies By default all tasks specified in the XML document are independent from each other. That implies that all tasks are scheduled at the same time if no other constraints exist. It is possible to define dependencies between tasks using the

depends

element that expects as value the id or name of an already defined task. The element must be a child of a

dependencies

element. Without any arguments the task will not be scheduled until all (sub)tasks of the dependencies have finished successfully. By setting the

separate

argument to

true

a subtask can depend only on the corresponding subtask the task depends on. This option is only meaningful if both tasks are process block tasks and work on the same input set or a transformed version of it. In example 5 a

compress

task with id

2

is defined which depends on the before defined

sleep

task (23-32). Additionally, a task, which will decompress the compressed files immediately after the compression is finished, is defined (35-45). In order to achieve this behavior the

separate

attribute is set to

true

(43). Because the

.txt

ending of the original filename was cropped and a

.gz

ending was added, only the first part of the filename is considered as specified in the

prefixName

attribute (26,43). 4.3 Execution environments By default the tasks are executed one after the other locally on the host which runs Watchdog. It is possible to define different execution environments using the

executors

element. Possible environments:

• local

  - task is executed on the local host

• remote

  - task is executed on a remote host using ssh

• cluster

  - task is executed on a computer cluster

In example 6 four different execution environments are defined as childs of the

executors

element (6-11). Tasks scheduled on first executor will run on the host on which Watchdog was started (7). The executor with the name

defaultCluster

is used by default and runs on the

short.q

queue of the computer cluster (8). The next executor is reserved for high performance tasks because it reserves 4 slots on the cluster with each slot consuming three gigabyte of main memory (9). In order not to occupy the complete computing power the attribute

maxRunning

is set to four which means that a maximum of four tasks will run simultaneously on that execution environment. In line 10 an example for a remote executor is given which executes tasks via ssh using a host named

superComputer

.
Afterwards the same sleep task is defined as in the first example and will run on the local executor (16). The other executors can be tested once you adapted them to your local infrastructure (see 2.4 and 2.5). 4.4 Global constants A constant can be defined globally using the

const

elements which must be a child of a

constants

element. The parent element itself must be a child of the

settings

environment. Every

const

element must own a unique name which is set with the

name

attribute. The value of the constant is stored between the opening and closing element tag.

${NAME_OF_CONSTANT}

is substituted with the corresponding constant in every attribute or text content. Only the

watchdogBase

attribute of

watchdog

, the

default

attribute of

?Executor

and the

id

attribute of

?Task

and within

depends

elements can not be substituted.

Currently, there is one pre-defined constant named

${TMP}

which is substituted within

?Task

tags with the working directory of the executor that will execute the task. In example 7 three constants are defined (6-10). The constant named

${WAIT_TIME}

is used as wait time in the sleep task (21). The other two constants are used to construct the standard output file path (18). 4.5 Environment variables Some tools expect specific environment variables to be set correctly. For example the PATH variable is important because executable programs are located only in directories defined by that variable. The environment variables which are set on the host running Watchdog can be simply inherited. With help of the

var

element new variables can be defined or updated. The name of the variable must be defined with the

name

attribute while the value is stored between the opening and closing element tag. The parent element of each

var

element must be a

environment

element which also owns a

name

attribute. This

name

attribute is used to link the environment with a task using the

environment

attribute all tasks possess. It is also possible to define environment variables locally within task definitions. If local and global variables with the same name are set, the local ones override the global variables.

The following environment variables are set by Watchdog by default:

• IS_WATCHDOG_JOB: if module was executed by Watchdog this value is set to

  1

• WATCHDOG_CORES: number of reserved cores if task runs on a cluster environment
• WATCHDOG_MEMORY: number of total reserved memory in megabyte if task runs on a cluster environment

In example 8 an environment named

pathEnv

is defined in which the variable

PATH

is updated (8). The entry

~/software/bin

is added at the beginning of the

PATH

variable and after the default seperator character the previous value is kept. The

environment

attribute of the

env

task is set to the name of the previously defined environment (17). Additionally two local environment variables are defined (23-26). The first one replaces the default shell with

/bin/sh

while the second one updates a variable called

TEST

using an alternative separator. 4.6 Mail notification By default Watchdog informs the user only when an error occurs during the execution of a task or if an error was detected afterwards. But these behaviour can be changed using the

notify

attribute of tasks. In example 9 different notification options are presented. The first defined task is the simple sleep task from the previous examples for which the

notify

attribute is set to

enabled

(14). Once the task is finished a mail will be sent to the address the user specified in the

mail

attribute of the

tasks

element (12). The second defined task is based on a process block named

sleepTime

and causes Watchdog to inform the user as soon as a subtask is finished because the

notify

attribute is set to

subtask

(7, 21). 4.7 Standard streams and working directory By default the stdout and stderr stream of the tool which is executed by watchdog is not saved. This can be changed by setting a path via the

stdout

or

stderr

element. It is also possible to use a file as input via the

stdin

element. Additionally, a working directory can be set by using the

workingDir

element. When this is done also relative path for

stdout

,

stderr

and

stdin

are allowed. Other than usual, the elements must occour in the same order as they are listed in the following:

workingDir

,

stdout

,

stderr

and

stdin

(but each of them is optional) In example 10 the working directory is set to

/tmp

(14). Afterwards, the standard output of the

sleep

task is written to the file

{%EXAMPLE_DATA%}/sleepTest.out

(15). The standard error stream is appended at the end of a file named

/tmp/sleepTest.err

(16). 4.8 Task actions Additional operations can be executed before and after the actual task is executed. Currently a set of IO operations are implemented that can be used to roll out data the task depends on or to clean up once the task is finished. This feature is especically usefull when Watchdog is running in slave mode (see 4.3) in order to reduce load for the shared file system as some files might be requirements for different tasks. Hence, these files would have to be transfered multiple times from the shared file system to the host executing the task.
Moreover, files stored on remote files systems can be up- or downloaded by Watchdog. By default, virtual file systems based on the protocols File, HTTP, HTTPS, FTP, FTPS and SFTP as well as the main memory (RAM) are supported. These virtual file systems are provided by the Commons Virtual File System project of the Apache Software Foundation. Examples for valid URIs of these file systems can be can be found here. However, any file system with an implementation of the

FileProvider

can also be included by the user as described in 6.1.

Task actions are defined in an

actions

tag as child of

?Task

. Slave mode is automatically activated if a task action is used. Currently six different IO operations are implemented:

• createFile

  - creates an empty file

• createFolder

  - create an empty folder

• copyFile

  - copies a file

• copyFolder

  - copies a folder (with content)

• deleteFile

  - deletes a file

• deleteFolder

  - deletes a folder (with content)

The event after which the task actions are executed must be defined using the

time

attribute each

actions

tag owns. The following arguments are available:

• beforeTask

  - before the task is executed

• afterTask

  - after the task is executed

• onSuccess

  - when the task was successfully executed

• onFailure

  - when task execution failed

• beforeTerminate

  - before Watchdog or a slave terminates itself

In example 11 a file name

/tmp/watchdog_file_to_compress.tmp

is compressed using gzip (5-14). Before the compress task is executed, the task action defined within the

actions

tag is executed because the

time

attribute is set to

beforeTask

(11-13). The task action copies the file stored in

{%INSTALL%}/examples/example_task_actions.xml

to

/tmp/watchdog_file_to_compress.tmp

(12). 4.9 Simple calculations Within a

?Task

element simple calculations can be preformed using the

$(expr)

construct whereby

expr

must be a numerical equation. The following operators are supported: +, -, *, /, ^, ² and ³. Additional the brackets

()

are provided. Moreover in case of a

processSequence

i

is replaced by the current value of the process sequence. In the more general case of a

processBlock

x

is substituted by an increasing number starting at

1

. The result of all calculations is rounded to five decimal places or converted to an integer if it is one. In example 12 the usage of the

$(expr)

construct is shown. The wait time for the sleep task is calculated based on the input numbers of the

processSequence

(7, 16). In the second gzip task the number of the subtask is used to name the standard output files (23). 4.10 Multiple module search folders By default Watchdog tries to locate modules in a folder named

modules/

stored in the installation directory of Watchdog. By using the

modules

element as child of

settings

, additional folders can be added. In example 13 the default directory is changed to

myCustomFolder/

with the

defaultFolder

attribute (7). Moreover, in line 8 an additional folder is added to Watchdog's search path. Watchdog will now try to locate modules in the directories

{%INSTALL%}/myCustomFolder/

and

/home/additionalModules/

. In order to test that example you must create a new folder, copy the

sleep

module from Watchdog's module folder and adapt the path in line 7 or 8 to match that folder. 4.11 Custom success and error checker In some cases the exit code of a command is not a reliable indicator whether the command was executed successfully or not. For example some tools return as exit code zero regardless of whether the command succeeded or failed. Furthermore, a command could succeed technically but the desired result is not obtained (e.g. wrong index used for mapping of RNA-seq data results in a very low mapping rate). In order to handle such cases the user has the option to implement custom success and error checkers in Java that are executed by Watchdog once a task has terminated. Two steps must be performed to use custom checkers: implementation in Java and invocation in the XML workflow.

Interfaces for checkers are stored in the package de.lmu.ifi.bio.watchdog.interfaces. Basically a function returning a boolean value that indicates whether the task succeeded or failed must be implemented. The constructor must accept as first argument a object of the type Task that contains information about the task that was finished. Additional arguments of type Boolean, Integer, Double or String can be passed via the XML definition.

Example 14 shows an example of how an success checker can be added to a task by using the

checkers

element as a child of

?Task

(12-18). In addition to the location of the compiled Java class and the full class name arguments can be passed to the constructor of the class. In this example one variable of type

string

is passed to the constructor of the success checker using the

cArg

element (16). Once the task is finished, the checkers are evaluated in the same order as they were added in the XML workflow. In cases in which simultaneously success and error were detected, the task will be treated as failed. In this example the success checker will ensure that the file

{%INSTALL%}/examples/mail_config exists

and is not empty (15-16) 5 Creating custom modules In the following the steps that are needed to create a custom module named

nameOfModule

are explained. Basic XSD skills are needed to understand how things work together. Modules are defined in XSD format and should have the basic structure showed in example 15. To actually create modules the script

helper_scripts/createNewModule.sh

can be used and modified by hand as not all settings can be configured by it. As a first step, a folder with the name

nameOfModule

must be created in the

modules

folder. That folder must contain a file named

nameOfModule.xsd

which will hold the actual module definition. 5.1 Input parameter definition Example 16 shows how input parameters and flags can be defined. In line 4 and 5 two parameters are defined while line 6 creates a flag. Using the

minOccurs

and

maxOccurs

attributes it can be specified how often a parameter can be used. Also parameters can have different types which are enforced during the validation of the XML workflows. Some pre-defined types are

• paramBoolean (for flags)
• paramString
• paramInteger
• paramDouble

These types accept by default booleans, strings, integers or doubles but also allow all values that are substituted by Watchdog. Moreover own parameter types can be defined as showed in example 17. 5.2 Output parameter definition Optionally a module can return output parameters that can be used for the following tasks as input. Return parameters must be written to a file in the format {%VAR_NAME%} TAB {%VALUE%}. The name of the file to write into is automatically sent to the module by Watchdog using the

returnFilePathParameter

parameter. If you want to change this default parameter name see 5.3. Line 6 in example 18 specifies that the module must return a parameter named

outputParam1

of type

x:string

. The module itself must ensure that the parameters are written physically before the module exits or otherwise Watchdog will terminate itself. In case of a bash script which is executed, two functions named

writeParam2File

and

blockUntilFileIsWritten

defined in

core_lib/functions.sh

can be used. 5.3 Binary call command and other settings Now the command that will be executed can be defined. Example 19 specifies that a script named

nameOfModule.sh

that is stored in

modules/nameOfModule

will be called. In addition to that, the following settings can be modified:

• binName

  : name of the command which should be called

• preBinCommand

  : command that is added before the binName; e.g. interpreter

• isWatchdogModule

  : by default the command must be located in modules/

  binName

  ; if this parameter is false the command must point to a absolute binary or be reachable via the PATH environment variable

• returnFilePathParameter

  : name of the parameter that is used to store the return values

• paramFormat

  : defines how names of parameters are prefixed; (do not print parameter name, - or --); default:

  --

• spacingFormat

  : defines how names of parameters and values are spaced; default:

  blank

• quoteFormat

  : defines how values are quoted; default:

  single quoting

• separateFormat

  : defines the separator string between multiple occurrences of the same parameter; default:

  ,

The last three arguments can also be used for each parameter separately. 5.4 Assign a name to the new module Finally, a name must be assigned to the module. This can simply be done by creating a element with the attribute

name

of type

nameOfModuleType

and

substitutionGroup

set to

abstractTask

. Example 20 shows the needed line for the example module. Afterwards the type of the task is defined (5-15). If no output parameters are used line 10 can be omitted. 5.5 Putting it all together Example 21 shows the definition of the complete module. 5.6 Other matters Exit codes: The module developer must ensure that a command does only exit with exit status 0 if the command was executed sucessfully. File

core_lib/exitCodes.sh

contains some exit codes which names are also included in mail notifications if they are used. Custom exit codes can be easily added.

Error messages: Watchdog can detect by default error messages in standard out and standard error streams if they begin with

[ERROR]

. The errors are only stored if standard out and error files are saved to disk using the

streams

tag. If an error was detected but the exit code was 0 the command will also fail.

Module test: A script named

test_nameOfModule.sh

can also be part of the module. It is automatically called, if the user calls

helper_scripts/moduleTest.sh

. Also the module folder might contain some test data in the folder

test_data

. For simple test cases the bash function

testExitCode

can be used to test, if an input leads to the expected output. 6 Extend Watchdog's functionality In the following sections two different ways to extend Watchdog's functionality are described.

• Virtual File Systems that can be used within task actions (see 6.1)
• XML Plugins that add new

  ?Executor

  and

  ?ProcessBlock

  elements (see 6.2)

6.1 Virtual file systems for task actions With the help of task actions, file system operations can be performed before and after tasks (see 4.8). By default, virtual file systems based on the protocols File, HTTP, HTTPS, FTP, FTPS and SFTP as well as the main memory (RAM) are supported. These virtual file systems are provided by the Commons Virtual File System project of the Apache Software Foundation.

In order to add a new virtual file system, a class that implements the

VFSRegister

interface can be addded to the jar-file. The class will be automatically loaded by Watchdog and the new virtual file system will be useable without other modifications. The following four methods must be implemented for the interface:

• getFileProvider

  - must return an instance of the

  FileProvider

  interface as defined in the Commons Virtual File System project

• getURLSchemes

  - returns the url schemes that should be used in combination with that

  FileProvider

  (e.g.

  ftp

  )

• getMimeTypes

  - sets schemes that should be used for specific mimetypes

• getExtensions

  - sets schemes that should be used for specific file extension

The class

SimpleVFSRegister

can be extended if an instance of the

FileProvider

class can be created without arguments. Then only the name of the

FileProvider

class and the URL schemes that should be used must be defined. Example 22 shows how the virtual FTP file system is integrated in Watchdog by using the

FtpsFileProvider

class of the Commons Virtual File System project. 6.2 XML Plugins Watchdog provides a flexible plugin system that allows extending Watchdog by additional types of executors and process blocks without modifying the original Java classes. Essentially, this means creating a new XML element for use in Watchdog workflows as well as implementing additional Java classes that provide the functionality for this element. In brief, you have to do the following to use the plugin system:

• create an XSD file describing the new element and its parent element for use in Watchdog workflows
• Extend a few abstract classes
• Add class files for the new classes to the Watchdog jar-file and copy the new XSD file to a sub-directory of the Watchdog installation directory

In the Watchdog command-line version, all non-abstract classes in the Watchdog jar-file that extend the

XMLParserPlugin

abstract class are loaded dynamically during workflow execution. Currently, this is restricted to XML parsers for the generic type

ProcessBlock

or

ExecutorInfo

. The XML parser for a new XML element provides the functionality to parse this element in a workflow (i.e. a new executor or process block type) and to create a new object representing the corresponding element type. Here, the four most important functions of the

XMLParserPlugin

abstract class that have to be implemented are:

• getNameOfParseableTag

  : returns the name of the element the class can parse

• getNameOfParentTag

  : returns the name of the parent element of this element

• getXSDDefinition

  : returns the path to the XSD file describing this element (relative to the xsd sub-directory of the Watchdog directory)

• parseElement

  : implements the actual parsing process.

The last function creates an object of a class representing the new element. This class has to implement the interfaces

XMLDataStore

and

XMLPlugin

, for instance by extending one of the abstract classes

ProcessBlock

or

ExecutorInfo

or one of their subclasses.
For use in the Workflow designer GUI of Watchdog, two additional requirements have to be met:

• An FXML file has to be provided describing how the attributes of the new element type are represented graphically. FXML is an XML-based markup language for describing the layout of a user interface in a JavaFX application.
• Classes extending

  PluginView

  and

  PluginViewController

  have to be implemented for testing whether the input is valid and for loading and saving data to and from XML.

All executors and process blocks integrated in Watchdog are using this plugin system. Hence, examples how to implement a new XML element can be found in the package

de.lmu.ifi.bio.watchdog.xmlParser.plugins

of the Java source code. 7 Docker In order to run a Docker image, Docker must be installed and configured correctly as descibed here. 7.1 Install the Watchdog Docker image A Watchdog image for Docker can be obtained from hub.docker.com. The image is rebuild automatically by the Bioconda project once a new version is released on Bioconda.

You can download the latest version of the image with

docker pull klugem/watchdog-wms

. Within the Docker image the environment variable

WATCHDOG_HOME

is set automatically to the installation directory of Watchdog (required for the

watchdogBase

attribute). The

-useEnvBase

flag of the command line version can be used to override the

watchdogBase

attribute of the XML workflow with the value stored in

WATCHDOG_HOME

. Moreover, the installation directory of Watchdog is mounted under

/watchdog

within the Docker image. 7.2 Sharing of files In order to exchange files with the host system, the -v or -mount option of Docker can be used. These option can be used multiple times.

docker run -v source_folder_or_file_on_host:destination_folder_or_file[:ro] image command

More information can be found in the documentation of Docker. 7.3 Port forwarding In order to use the build-in webserver of Watchdog, the port used by the webserver must be forwarded to the host running the Docker container.

The command

docker run -p 8090:8080 image command

maps the container port 8080 (TCP) to the port 8090 (TCP) on the Docker host. More information can be found in the documentation of Docker. 7.4 How to use the Docker Watchdog image The examples within the Docker image are automatically configured when {%Nwatchdog-cmd%N} is started the first time and are stored in

/watchdog/examples

. The command

docker run -h localhost -p 8080:8080 klugem/watchdog-wms watchdog-cmd -useEnvBase -x /watchdog/examples/​example_basic_sleep.xml

executes the example described in 3.2 and forwards the webserver port to the host port 8080.

Alternatively, it is possible to run a workflow that is stored on the host system as described in 7.2. Ensure that all files used in the workflow are made accessible within the Docker image. 7.5 Use Docker in modules A Docker image can also be used in a module. The module

bowtie2Docker

implements an example module that uses the Docker image of Bowtie 2 that is provided by Bioconda and hosted on quay.io. The Docker image will be automatically downloaded if it is not found locally.

Make sure that the Docker daemon is installed and running before you test this example. Example 23 shows how the

bowtie2Docker

module can be used with the provided example data. The test data that is shipped with Bowtie 2 is stored in the folder

example_data

of the module (6). Log files are written to

/tmp/bowtie2.docker.test.[out|err]

(13, 14) while the mapped reads are stored in SAM format in

/tmp/bowtie2.docker.test.sam

(20).