PivotalR (version 0.1.18.3.1)

madlib.lda: Wrapper for MADlib's Latent Dirichilet Allocation

Description

This function is a wrapper for MADlib's Latent Dirichlet Allocation. The computation is parallelized by MADlib if the connected database is distributed. Please refer to MADlib documentation for details of the algorithm implementation [1].

Usage

madlib.lda(data, topic_num, alpha, beta, iter_num = 20,
  nstart = 1, best = TRUE,...)

Arguments

data

An object of db.obj class. This is the database table containing the documents on which the algorithm will train. The text of each document should be tokenized into 'words'.

topic_num

Number of topics.

alpha

Dirichlet parameter for the per-doc topic multinomial.

beta

Dirichlet parameter for the per-topic word multinomial.

iter_num

Number of iterations.

nstart

Number of repeated random starts.

best

If TRUE only the model with the minimum perplexity is returned.

…

Other optional parameters. Not implemented.

Value

An lda.madlib object or a list of them, which is a list that contains the following items:

assignments

The per-document topic assignments.

document_sums

The per-document topic counts.

model_table

The db.table object for accessing the model table in the database.

output_table

The db.table object for accessing the output table in the database.

tf_table

The db.table object for accessing the term frequency table in the database.

topic_sums

The per-topic sum of assignments.

topics

The per-word association with topics.

References

[1] Documentation of LDA in the latest MADlib release, http://madlib.incubator.apache.org/docs/latest/group__grp__lda.html

See Also

predict.lda.madlib is used for prediction-labelling test documents using a learned lda.madlib model.

perplexity.lda.madlib is used for computing the perplexity of a learned lda.madlib model.

Examples

# NOT RUN {
<!-- %% @test .port Database port number -->
<!-- %% @test .dbname Database name -->
## set up the database connection
## Assume that .port is port number and .dbname is the database name
cid <- db.connect(port = .port, dbname = .dbname, verbose = FALSE)

dat <- db.data.frame("__madlib_pivotalr_lda_data__", conn.id = cid,
  verbose = FALSE)

output.db <- madlib.lda(dat, 2,0.1,0.1, 50)

perplexity.db <- perplexity.lda.madlib(output.db)
print(perplexity.db)

## Run LDA multiple times and get the best one
output.db <- madlib.lda(dat, 2,0.1,0.1, 50, nstart=2)
perplexity.db <- perplexity.lda.madlib(output.db)
print(perplexity.db)

## Run LDA multiple times and keep all models
output.db <- madlib.lda(dat, 2,0.1,0.1, 50, nstart=2, best=FALSE)

perplexity.db <- perplexity.lda.madlib(output.db[[1]])
print(perplexity.db)

perplexity.db <- perplexity.lda.madlib(output.db[[2]])
print(perplexity.db)

db.disconnect(cid)
# }