Learn R Programming

syuzhet (version 1.0.4)

get_tokens: Word Tokenization

Description

Parses a string into a vector of word tokens.

Usage

get_tokens(text_of_file, pattern = "\\W")

Arguments

text_of_file

A Text String

pattern

A regular expression for token breaking

Value

A Character Vector of Words