Creates Global Vectors matrix factorization model.

Description

GloVe matrix factorization model. Model can be trained via fully can asynchronous and parallel AdaGrad with $fit_transform() method.

Usage

GloVe

Format

R6Class object.

Fields

components

represents context embeddings

shuffle

logical = FALSE by default. Defines shuffling before each SGD iteration. Generally shuffling is a good idea for stochastic-gradient descent, but from my experience in this particular case it does not improve convergence.

Usage

For usage details see Methods, Arguments and Examples sections.

glove = GloVe$new(rank, x_max, learning_rate = 0.15,
                          alpha = 0.75, lambda = 0.0, shuffle = FALSE)
glove$fit_transform(x, n_iter = 10L, convergence_tol = -1,
              n_threads = getOption("rsparse_omp_threads", 1L), ...)
glove$components

Methods

$new(rank, x_max, learning_rate = 0.15, alpha = 0.75, lambda = 0, shuffle = FALSE)

Constructor for Global vectors model. For description of arguments see Arguments section.

$fit_transform(x, n_iter = 10L, convergence_tol = -1, n_threads = getOption("rsparse_omp_threads", 1L), ...)

fit Glove model given input matrix x

Arguments

glove

A GloVe object

x

An input term co-occurence matrix. Preferably in dgTMatrix format

n_iter

integer number of SGD iterations

rank

desired dimension for the latent vectors

x_max

integer maximum number of co-occurrences to use in the weighting function. see the GloVe paper for details: http://nlp.stanford.edu/pubs/glove.pdf

learning_rate

numeric learning rate for SGD. I do not recommend that you modify this parameter, since AdaGrad will quickly adjust it to optimal

convergence_tol

numeric = -1 defines early stopping strategy. We stop fitting when one of two following conditions will be satisfied: (a) we have used all iterations, or (b) cost_previous_iter / cost_current_iter - 1 < convergence_tol. By default perform all iterations.

alpha

numeric = 0.75 the alpha in weighting function formula : f(x) = 1 if x > x_max; else (x/x_max)^alpha

lambda

numeric = 0.0 regularization parameter

init

list(w_i = NULL, b_i = NULL, w_j = NULL, b_j = NULL) initialization for embeddings (w_i, w_j) and biases (b_i, b_j). w_i, w_j - numeric matrices, should number of #rows = rank, #columns - expected number of rows/columns in input matrix. b_i, b_j = numeric vectors, should have length of # expected number of rows/columns in input matrix

See Also

http://nlp.stanford.edu/projects/glove/

Examples

temp = tempfile()
download.file('http://mattmahoney.net/dc/text8.zip', temp)
text8 = readLines(unz(temp, "text8"))
it = itoken(text8)
vocabulary = create_vocabulary(it)
vocabulary = prune_vocabulary(vocabulary, term_count_min = 5)
v_vect = vocab_vectorizer(vocabulary)
tcm = create_tcm(it, v_vect, skip_grams_window = 5L)
glove_model = GloVe$new(rank = 50, x_max = 10, learning_rate = .25)
# fit model and get word vectors
word_vectors_main = glove_model$fit_transform(tcm, n_iter = 10)
word_vectors_context = glove_model$components
word_vectors = word_vectors_main + t(word_vectors_context)