R: Default function to tokenize

default_tokenize {lime} | R Documentation

## Default function to tokenize

### Description

This tokenizer uses `stringi::stri_split_boundaries()` to tokenize a `character` vector. To be used with [explain.character()'.

### Usage

    default_tokenize(text)

### Arguments

`text` | text to tokenize as a `character` vector

### Value

a `character` vector.

### Examples

    data('train_sentences')
    default_tokenize(train_sentences$text[1])

[Package lime version 0.5.1 Index]