textshape (version 1.7.5)

split_token: Split Tokens

Description

Split tokens.

Usage

split_token(x, ...)
# S3 method for default
split_token(x, lower = TRUE, ...)
# S3 method for data.frame
split_token(x, text.var = TRUE, lower = TRUE, ...)

Value

Returns a list of vectors of tokens or an expanded data.table with tokens split apart.

Arguments

x

A data.frame or character vector with tokens.

lower

logical. If TRUE the words are converted to lower case.

text.var

The name of the text variable. If TRUE split_token tries to detect the text column with tokens.

...

Ignored.

Examples

(x <- c(
    "Mr. Brown comes! He says hello. i give him coffee.",
    "I'll go at 5 p. m. eastern time.  Or somewhere in between!",
    "go there"
))
split_token(x)
split_token(x, lower=FALSE)

data(DATA)
split_token(DATA)
split_token(DATA, lower=FALSE)

## Larger data set
split_token(hamlet)