R: quanteda tokenizers

tokenize_internal {quanteda} | R Documentation

## quanteda tokenizers

### Description

Internal methods for tokenization providing default and legacy methods for text segmentation.

### Usage

    tokenize_word(x, split_hyphens = FALSE, verbose = quanteda_options("verbose"))

    tokenize_word1(x, split_hyphens = FALSE, verbose = quanteda_options("verbose"))

    tokenize_character(x, ...)

    tokenize_sentence(x, ..., verbose = FALSE)

    tokenize_fasterword(x, ...)

    tokenize_fastestword(x, ...)

### Arguments

`x` | (named) character; input texts
`split_hyphens` | logical; if `TRUE`, split words that are connected by hyphenation and hyphenation-like characters in between words, e.g. `"self-aware"` becomes `c("self", "-", "aware")`
`verbose` | if `TRUE`, print timing messages to the console
`...` | used to pass arguments among the functions

### Value

a list of characters corresponding to the (most conservative) tokenization, including whitespace where applicable; except for `tokenize_word1()`, which is a special tokenizer for Internet language that includes URLs, #hashtags, @usernames, and email addresses.

### Examples

    ## Not run:
    txt <- c(doc1 = "Tweet https://quanteda.io using @quantedainit and #rstats.",
             doc2 = "The £1,000,000 question.",
             doc4 = "Line 1.\nLine2\n\nLine3.",
             doc5 = "?",
             doc6 = "Self-aware machines! \U0001f600")
    tokenize_word(txt)
    tokenize_word(txt, split_hyphens = TRUE)
    tokenize_word2(txt, split_hyphens = FALSE)
    tokenize_word2(txt, split_hyphens = TRUE)
    tokenize_fasterword(txt)
    tokenize_fastestword(txt)
    tokenize_sentence(txt)
    tokenize_character(txt[2])

    ## End(Not run)

[Package quanteda version 3.0.0 Index]