tknz_sent()
splits text into sentences using a list of
single character delimiters, specified by the parameter EOS
.
Specifically, when an EOS token is found, the next sentence begins at the
first position in the input string not containing any of the EOS tokens
or white space (so that entries like "Hi there!!!"
or
"Hello . . ."
are both recognized as a single sentence).
If keep_first
is FALSE
, the delimiters are stripped off from
the returned sequences, which means that all delimiters are treated
symmetrically.
In the absence of any EOS
delimiter, tknz_sent()
returns the input as is, since parts of text corresponding to different
entries of the input vector x
are understood as parts of separate
sentences.