R: Simple tokenization functions for string splitting

tokenizers {text2vec} | R Documentation

## Simple tokenization functions for string splitting

### Description

Few simple tokenization functions. For more comprehensive list see `tokenizers` package: https://cran.r-project.org/package=tokenizers. Also check `stringi::stri_split_*`.

### Usage

    word_tokenizer(strings, ...)

    char_tokenizer(strings, ...)

    space_tokenizer(strings, sep = " ", xptr = FALSE, ...)

### Arguments

`strings` | `character` vector
`...` | other parameters (usually not used - see source code for details).
`sep` | `character`, `nchar(sep)` = 1 - split strings by this character.
`xptr` | `logical` tokenize at C++ level - could speed-up by 15-50%.

### Value

`list` of `character` vectors. Each element of list contains vector of tokens.

### Examples

    doc = c("first  second", "bla, bla, blaa")
    # split by words
    word_tokenizer(doc)
    #faster, but far less general - perform split by a fixed single whitespace symbol.
    space_tokenizer(doc, " ")

[Package text2vec version 0.5.1 Index]