R: Multi-file datasets

Dataset {arrow} | R Documentation

## Multi-file datasets

### Description

Arrow Datasets allow you to query against data that has been split across multiple files. This sharding of data may indicate partitioning, which can accelerate queries that only touch some partitions (files).

A `Dataset` contains one or more `Fragments`, such as files, of potentially differing type and partitioning.

For `Dataset$create()`, see `open_dataset()`, which is an alias for it.

`DatasetFactory` is used to provide finer control over the creation of `Dataset`s.

### Factory

`DatasetFactory` is used to create a `Dataset`, inspect the Schema of the fragments contained in it, and declare a partitioning. `FileSystemDatasetFactory` is a subclass of `DatasetFactory` for discovering files in the local file system, the only currently supported file system.

For the `DatasetFactory$create()` factory method, see `dataset_factory()`, an alias for it. A `DatasetFactory` has:

- `$Inspect(unify_schemas)`: If `unify_schemas` is `TRUE`, all fragments will be scanned and a unified Schema will be created from them; if `FALSE` (default), only the first fragment will be inspected for its schema. Use this fast path when you know and trust that all fragments have an identical schema.

- `$Finish(schema, unify_schemas)`: Returns a `Dataset`. If `schema` is provided, it will be used for the `Dataset`; if omitted, a `Schema` will be created from inspecting the fragments (files) in the dataset, following `unify_schemas` as described above.

`FileSystemDatasetFactory$create()` is a lower-level factory method and takes the following arguments:

- `filesystem`: A FileSystem

- `selector`: A FileSelector

- `format`: A string identifier of the format of the files in `path`. Currently "parquet" and "ipc"/"arrow"/"feather" (aliases for each other) are supported. For Feather, only version 2 files are supported.

### Methods

A `Dataset` has the following methods:

- `$NewScan()`: Returns a ScannerBuilder for building a query

- `$schema`: Active binding that returns the Schema of the Dataset; you may also replace the dataset's schema by using `ds$schema <- new_schema`. This method currently supports only adding, removing, or reordering fields in the schema: you cannot alter or cast the field types.

`FileSystemDataset` has the following methods:

- `$files`: Active binding, returns the files of the `FileSystemDataset`

- `$format`: Active binding, returns the FileFormat of the `FileSystemDataset`

`UnionDataset` has the following methods:

- `$children`: Active binding, returns all child `Dataset`s.

### See Also

`open_dataset()` for a simple interface to creating a `Dataset`

[Package arrow version 0.17.1 Index]