R: Feature Tranformation - Word2Vec (Estimator)

ft_word2vec {sparklyr} | R Documentation

## Feature Tranformation – Word2Vec (Estimator)

### Description

Word2Vec transforms a word into a code for further natural language processing or machine learning process.

### Usage

    ft_word2vec(x, input_col, output_col, vector_size = 100L, min_count = 5L,
      max_sentence_length = 1000L, num_partitions = 1L, step_size = 0.025,
      max_iter = 1L, seed = NULL, dataset = NULL,
      uid = random_string("word2vec_"), ...)

    ml_find_synonyms(model, word, num)

### Arguments

`x` | A `spark_connection`, `ml_pipeline`, or a `tbl_spark`.
`input_col` | The name of the input column.
`output_col` | The name of the output column.
`vector_size` | The dimension of the code that you want to transform from words. Default: 100
`min_count` | The minimum number of times a token must appear to be included in the word2vec model's vocabulary. Default: 5
`max_sentence_length` | (Spark 2.0.0+) Sets the maximum length (in words) of each sentence in the input data. Any sentence longer than this threshold will be divided into chunks of up to `max_sentence_length` size. Default: 1000
`num_partitions` | Number of partitions for sentences of words. Default: 1
`step_size` | Param for Step size to be used for each iteration of optimization (> 0).
`max_iter` | The maximum number of iterations to use.
`seed` | A random seed. Set this value if you need your results to be reproducible across repeated calls.
`dataset` | (Optional) A `tbl_spark`. If provided, eagerly fit the (estimator) feature "transformer" against `dataset`. See details.
`uid` | A character string used to uniquely identify the feature transformer.
`...` | Optional arguments; currently unused.
`model` | A fitted `Word2Vec` model, returned by `ft_word2vec()`.
`word` | A word, as a length-one character vector.
`num` | Number of words closest in similarity to the given word to find.

### Details

When `dataset` is provided for an estimator transformer, the function internally calls `ml_fit()` against `dataset`. Hence, the methods for `spark_connection` and `ml_pipeline` will then return a `ml_transformer` and a `ml_pipeline` with a `ml_transformer` appended, respectively. When `x` is a `tbl_spark`, the estimator will be fit against `dataset` before transforming `x`.

When `dataset` is not specified, the constructor returns a `ml_estimator`, and, in the case where `x` is a `tbl_spark`, the estimator fits against `x` then to obtain a transformer, which is then immediately used to transform `x`.

### Value

The object returned depends on the class of `x`.

- `spark_connection`: When `x` is a `spark_connection`, the function returns a `ml_transformer`, a `ml_estimator`, or one of their subclasses. The object contains a pointer to a Spark `Transformer` or `Estimator` object and can be used to compose `Pipeline` objects.

- `ml_pipeline`: When `x` is a `ml_pipeline`, the function returns a `ml_pipeline` with the transformer or estimator appended to the pipeline.

- `tbl_spark`: When `x` is a `tbl_spark`, a transformer is constructed then immediately applied to the input `tbl_spark`, returning a `tbl_spark`

`ml_find_synonyms()` returns a DataFrame of synonyms and cosine similarities

### See Also

See http://spark.apache.org/docs/latest/ml-features.html for more information on the set of transformations available for DataFrame columns in Spark.

Other feature transformers: `ft_binarizer`, `ft_bucketizer`, `ft_chisq_selector`, `ft_count_vectorizer`, `ft_dct`, `ft_elementwise_product`, `ft_feature_hasher`, `ft_hashing_tf`, `ft_idf`, `ft_imputer`, `ft_index_to_string`, `ft_interaction`, `ft_lsh`, `ft_max_abs_scaler`, `ft_min_max_scaler`, `ft_ngram`, `ft_normalizer`, `ft_one_hot_encoder`, `ft_pca`, `ft_polynomial_expansion`, `ft_quantile_discretizer`, `ft_r_formula`, `ft_regex_tokenizer`, `ft_sql_transformer`, `ft_standard_scaler`, `ft_stop_words_remover`, `ft_string_indexer`, `ft_tokenizer`, `ft_vector_assembler`, `ft_vector_indexer`, `ft_vector_slicer`

[Package sparklyr version 0.8.0 Index]