R: Obtain token table from text

tokenize {styler} | R Documentation

## Obtain token table from text

### Description

`utils::getParseData()` is used to obtain a flat parse table from `text`.

### Usage

    tokenize(text)

### Arguments

`text` | A character vector.

### Details

Apart from the columns provided by `utils::getParseData()`, the following columns are added:

- A column "short" with the first five characters of "text".

  - A column "pos_id" for (positional id) which can be used for sorting (because "id" cannot be used in general). Note that the nth value of this column corresponds to n as long as no tokens are inserted.

  - A column "child" that contains nests.

### Value

A flat parse table

[Package styler version 1.0.2 Index]