R: String metrics in 'stringdist'

stringdist-encoding {stringdist} | R Documentation

## String metrics in stringdist

### Description

This page gives an overview of encoding handling in stringst.

### Encoding in stringdist

All character strings are stored as a sequence of bytes. An encoding system relates a byte, or a short sequence of bytes to a symbol. Over the years, many encoding systems have been developed, and not all OS's and softwares use the same encoding as default. Similarly, depending on the system R is running on, R may use a different encoding for storing strings internally.

The stringdist package is designed so users in principle need not worry about this. Strings are converted to `UTF-32` (unsigned integer) by default prior to any further computation. This means that results are encoding-independent and that strings are interpreted as a sequence of symbols, not as a sequence of pure bytes. In functions where this is relevant, this may be switched by setting the `useBytes` option to `TRUE`. However, keep in mind that results will then likely depend on the system R is running on, except when your strings are pure ASCII. Also, for multi-byte encodings, results for byte-wise computations will usually differ from results using encoded computations.

Prior to stringdist version 0.9, setting `useBytes=TRUE` could give a significant performance enhancement. Since version 0.9, translation to integer is done by C code internal to stringdist and the difference in performance is now negligible.

### Unicode normalisation

In `utf-8`, the same (accented) character may be represented as several byte sequences. For example, an u-umlaut can be represented with a single byte code or as a byte code representing `'u'` followed by a modifier byte code that adds the umlaut. The stringi package of Gagolevski and Tartanus offers unicode normalisation tools.

### Some tips on character encoding and transliteration

Some algorithms (like soundex) are defined only on the printable ASCII character set. This excludes any character with accents for example. Translating accented characters to the non-accented ones is a form of transliteration. On many systems running R (but not all!) you can achieve this with

`iconv(x,to="ASCII//TRANSLIT")`,

where `x` is your character vector. See the documentation of `iconv` for details.

The `stringi` package (Gagolewski and Tartanus) should work on any system. The command `stringi::stri_trans_general(x,"Latin-ASCII")` transliterates character vector `x` to ASCII.

### References

- The help page of `Encoding` describes how R handles encoding.

- The help page of `iconv` has a good overview of base R's encoding conversion options. The capabilities of `iconv` depend on the system R is running on. The stringi package offers platform-independent encoding and normalization tools.

### See Also

- Functions using re-encoding: `stringdist`, `stringdistmatrix`, `amatch`, `ain`, `qgrams`

- Encoding related: `printable_ascii`

[Package stringdist version 0.9.4.6 Index]