R: Simple tokenization functions for string splitting

tokenizers {text2vec}

R Documentation

Simple tokenization functions for string splitting

Description

Few simple tokenization functions. For more comprehensive list see tokenizers package: https://cran.r-project.org/package=tokenizers. Also check stringi::stri_split_*.

Usage

word_tokenizer(strings, ...)

char_tokenizer(strings, ...)

space_tokenizer(strings, sep = " ", xptr = FALSE, ...)

Arguments

`strings`	`character` vector
`...`	other parameters (usually not used - see source code for details).
`sep`	`character`, `nchar(sep)` = 1 - split strings by this character.
`xptr`	`logical` tokenize at C++ level - could speed-up by 15-50%.

Value

list of character vectors. Each element of list contains vector of tokens.

Examples

doc = c("first  second", "bla, bla, blaa")
# split by words
word_tokenizer(doc)
#faster, but far less general - perform split by a fixed single whitespace symbol.
space_tokenizer(doc, " ")

[Package text2vec version 0.5.1 Index]