R: Remove sentences based on their token lengths or a pattern...

corpus_trim {quanteda} | R Documentation

## Remove sentences based on their token lengths or a pattern match

### Description

Removes sentences from a corpus or a character vector shorter than a specified length.

### Usage

    corpus_trim(
      x,
      what = c("sentences", "paragraphs", "documents"),
      min_ntoken = 1,
      max_ntoken = NULL,
      exclude_pattern = NULL
    )

    char_trim(
      x,
      what = c("sentences", "paragraphs", "documents"),
      min_ntoken = 1,
      max_ntoken = NULL,
      exclude_pattern = NULL
    )

### Arguments

`x` | corpus or character object whose sentences will be selected.
`what` | units of trimming, `"sentences"` or `"paragraphs"`, or `"documents"`
`min_ntoken, max_ntoken` | minimum and maximum lengths in word tokens (excluding punctuation)
`exclude_pattern` | a stringi regular expression whose match (at the sentence level) will be used to exclude sentences

### Value

a corpus or character vector equal in length to the input. If the input was a corpus, then the all docvars and metadata are preserved. For documents whose sentences have been removed entirely, a null string (`""`) will be returned.

### Examples

    txt <- c("PAGE 1. This is a single sentence.  Short sentence. Three word sentence.",
             "PAGE 2. Very short! Shorter.",
             "Very long sentence, with multiple parts, separated by commas.  PAGE 3.")
    corp <- corpus(txt, docvars = data.frame(serial = 1:3))
    corp

    # exclude sentences shorter than 3 tokens
    corpus_trim(corp, min_ntoken = 3)
    # exclude sentences that start with "PAGE <digit(s)>"
    corpus_trim(corp, exclude_pattern = "^PAGE \\d+")

    # trimming character objects
    char_trim(txt, "sentences", min_ntoken = 3)
    char_trim(txt, "sentences", exclude_pattern = "sentence\\.")

[Package quanteda version 3.1.0 Index]