This layer has basic options for managing text in a Keras model. It
transforms a batch of strings (one example = one string) into either a list
of token indices (one example = 1D tensor of integer token indices) or a
dense representation (one example = 1D tensor of float values representing
data about the example's tokens). This layer is meant to handle natural
language inputs. To handle simple string inputs (categorical strings or
pre-tokenized strings) see layer_string_lookup().
The vocabulary for the layer must be either supplied on construction or
learned via adapt(). When this layer is adapted, it will analyze the
dataset, determine the frequency of individual string values, and create a
vocabulary from them. This vocabulary can have unlimited size or be capped,
depending on the configuration options for this layer; if there are more
unique values in the input than the maximum vocabulary size, the most
frequent terms will be used to create the vocabulary.
The processing of each example contains the following steps:
Standardize each example (usually lowercasing + punctuation stripping)
Split each example into substrings (usually words)
Recombine substrings into tokens (usually ngrams)
Index tokens (associate a unique int value with each token)
Transform each example using this index, either into a vector of ints or a dense float vector.
Some notes on passing callables to customize splitting and normalization for this layer:
Any callable can be passed to this Layer, but if you want to serialize
this object you should only pass functions that are registered Keras
serializables (see register_keras_serializable()
for more details).
When using a custom callable for standardize, the data received
by the callable will be exactly as passed to this layer. The callable
should return a tensor of the same shape as the input.
When using a custom callable for split, the data received by the
callable will have the 1st dimension squeezed out - instead of
list("string to split", "another string to split"), the Callable will
see c("string to split", "another string to split").
The callable should return a tf.Tensor of dtype string
with the first dimension containing the split tokens -
in this example, we should see something like list(c("string", "to", "split"), c("another", "string", "to", "split")).
Note: This layer uses TensorFlow internally. It cannot be used as part of the compiled computation graph of a model with any backend other than TensorFlow. It can however be used with any backend when running eagerly. It can also always be used as part of an input preprocessing pipeline with any backend (outside the model itself), which is how we recommend to use this layer.
Note: This layer is safe to use inside a tf.data pipeline
(independently of which backend you're using).
layer_text_vectorization(
object,
max_tokens = NULL,
standardize = "lower_and_strip_punctuation",
split = "whitespace",
ngrams = NULL,
output_mode = "int",
output_sequence_length = NULL,
pad_to_max_tokens = FALSE,
vocabulary = NULL,
idf_weights = NULL,
sparse = FALSE,
ragged = FALSE,
encoding = "utf-8",
name = NULL,
...
)get_vocabulary(object, include_special_tokens = TRUE)
set_vocabulary(object, vocabulary, idf_weights = NULL, ...)
The return value depends on the value provided for the first argument.
If object is:
a keras_model_sequential(), then the layer is added to the sequential model
(which is modified in place). To enable piping, the sequential model is also
returned, invisibly.
a keras_input(), then the output tensor from calling layer(input) is returned.
NULL or missing, then a Layer instance is returned.
Object to compose the layer with. A tensor, array, or sequential model.
Maximum size of the vocabulary for this layer. This should
only be specified when adapting a vocabulary or when setting
pad_to_max_tokens=TRUE. Note that this vocabulary
contains 1 OOV token, so the effective number of tokens is
(max_tokens - 1 - (1 if output_mode == "int" else 0)).
Optional specification for standardization to apply to the input text. Values can be:
NULL: No standardization.
"lower_and_strip_punctuation": Text will be lowercased and all
punctuation removed.
"lower": Text will be lowercased.
"strip_punctuation": All punctuation will be removed.
Callable: Inputs will passed to the callable function, which should be standardized and returned.
Optional specification for splitting the input text. Values can be:
NULL: No splitting.
"whitespace": Split on whitespace.
"character": Split on each unicode character.
Callable: Standardized inputs will passed to the callable function, which should be split and returned.
Optional specification for ngrams to create from the
possibly-split input text. Values can be NULL, an integer
or list of integers; passing an integer will create ngrams
up to that integer, and passing a list of integers will
create ngrams for the specified values in the list.
Passing NULL means that no ngrams will be created.
Optional specification for the output of the layer.
Values can be "int", "multi_hot", "count" or "tf_idf",
configuring the layer as follows:
"int": Outputs integer indices, one integer index per split
string token. When output_mode == "int",
0 is reserved for masked locations;
this reduces the vocab size to max_tokens - 2
instead of max_tokens - 1.
"multi_hot": Outputs a single int array per batch, of either
vocab_size or max_tokens size, containing 1s in all elements
where the token mapped to that index exists at least
once in the batch item.
"count": Like "multi_hot", but the int array contains
a count of the number of times the token at that index
appeared in the batch item.
"tf_idf": Like "multi_hot", but the TF-IDF algorithm
is applied to find the value in each token slot.
For "int" output, any shape of input and output is supported.
For all other output modes, currently only rank 1 inputs
(and rank 2 outputs after splitting) are supported.
Only valid in INT mode. If set, the output will
have its time dimension padded or truncated to exactly
output_sequence_length values, resulting in a tensor of shape
(batch_size, output_sequence_length) regardless of how many tokens
resulted from the splitting step. Defaults to NULL. If ragged
is TRUE then output_sequence_length may still truncate the
output.
Only valid in "multi_hot", "count",
and "tf_idf" modes. If TRUE, the output will have
its feature axis padded to max_tokens even if the number
of unique tokens in the vocabulary is less than max_tokens,
resulting in a tensor of shape (batch_size, max_tokens)
regardless of vocabulary size. Defaults to FALSE.
Optional. Either an array of strings or a string path to a
text file. If passing an array, can pass a list, list,
1D NumPy array, or 1D tensor containing the string vocabulary terms.
If passing a file path, the file should contain one line per term
in the vocabulary. If this argument is set,
there is no need to adapt() the layer.
An R vector, 1D numpy array, or 1D tensor of inverse document frequency weights with equal length to vocabulary. Must be set if output_mode is "tf_idf". Should not be set otherwise.
Boolean. Only applicable to "multi_hot", "count", and
"tf_idf" output modes. Only supported with TensorFlow
backend. If TRUE, returns a SparseTensor
instead of a dense Tensor. Defaults to FALSE.
Boolean. Only applicable to "int" output mode.
Only supported with TensorFlow backend.
If TRUE, returns a RaggedTensor instead of a dense Tensor,
where each sequence may have a different length
after string splitting. Defaults to FALSE.
Optional. The text encoding to use to interpret the input
strings. Defaults to "utf-8".
String, name for the object
For forward/backward compatability.
If TRUE, the returned vocabulary will include the padding and OOV tokens, and a term's index in the vocabulary will equal the term's index when calling the layer. If FALSE, the returned vocabulary will not include any padding or OOV tokens.
This example instantiates a TextVectorization layer that lowercases text,
splits on whitespace, strips punctuation, and outputs integer vocab indices.
max_tokens <- 5000 # Maximum vocab size.
max_len <- 4 # Sequence length to pad the outputs to.
# Create the layer.
vectorize_layer <- layer_text_vectorization(
max_tokens = max_tokens,
output_mode = 'int',
output_sequence_length = max_len)
# Now that the vocab layer has been created, call `adapt` on the
# list of strings to create the vocabulary.
vectorize_layer %>% adapt(c("foo bar", "bar baz", "baz bada boom"))
# Now, the layer can map strings to integers -- you can use an
# embedding layer to map these integers to learned embeddings.
input_data <- rbind("foo qux bar", "qux baz")
vectorize_layer(input_data)
## tf.Tensor(
## [[4 1 3 0]
## [1 2 0 0]], shape=(2, 4), dtype=int64)This example instantiates a TextVectorization layer by passing a list
of vocabulary terms to the layer's initialize() method.
vocab_data <- c("earth", "wind", "and", "fire")
max_len <- 4 # Sequence length to pad the outputs to.
# Create the layer, passing the vocab directly. You can also pass the
# vocabulary arg a path to a file containing one vocabulary word per
# line.
vectorize_layer <- layer_text_vectorization(
max_tokens = max_tokens,
output_mode = 'int',
output_sequence_length = max_len,
vocabulary = vocab_data)
# Because we've passed the vocabulary directly, we don't need to adapt
# the layer - the vocabulary is already set. The vocabulary contains the
# padding token ('') and OOV token ('[UNK]')
# as well as the passed tokens.
vectorize_layer %>% get_vocabulary()
## [1] "" "[UNK]" "earth" "wind" "and" "fire"# ['', '[UNK]', 'earth', 'wind', 'and', 'fire']
Other preprocessing layers:
layer_auto_contrast()
layer_category_encoding()
layer_center_crop()
layer_discretization()
layer_equalization()
layer_feature_space()
layer_hashed_crossing()
layer_hashing()
layer_integer_lookup()
layer_max_num_bounding_boxes()
layer_mel_spectrogram()
layer_mix_up()
layer_normalization()
layer_rand_augment()
layer_random_brightness()
layer_random_color_degeneration()
layer_random_color_jitter()
layer_random_contrast()
layer_random_crop()
layer_random_flip()
layer_random_grayscale()
layer_random_hue()
layer_random_posterization()
layer_random_rotation()
layer_random_saturation()
layer_random_sharpness()
layer_random_shear()
layer_random_translation()
layer_random_zoom()
layer_rescaling()
layer_resizing()
layer_solarization()
layer_stft_spectrogram()
layer_string_lookup()
Other layers:
Layer()
layer_activation()
layer_activation_elu()
layer_activation_leaky_relu()
layer_activation_parametric_relu()
layer_activation_relu()
layer_activation_softmax()
layer_activity_regularization()
layer_add()
layer_additive_attention()
layer_alpha_dropout()
layer_attention()
layer_auto_contrast()
layer_average()
layer_average_pooling_1d()
layer_average_pooling_2d()
layer_average_pooling_3d()
layer_batch_normalization()
layer_bidirectional()
layer_category_encoding()
layer_center_crop()
layer_concatenate()
layer_conv_1d()
layer_conv_1d_transpose()
layer_conv_2d()
layer_conv_2d_transpose()
layer_conv_3d()
layer_conv_3d_transpose()
layer_conv_lstm_1d()
layer_conv_lstm_2d()
layer_conv_lstm_3d()
layer_cropping_1d()
layer_cropping_2d()
layer_cropping_3d()
layer_dense()
layer_depthwise_conv_1d()
layer_depthwise_conv_2d()
layer_discretization()
layer_dot()
layer_dropout()
layer_einsum_dense()
layer_embedding()
layer_equalization()
layer_feature_space()
layer_flatten()
layer_flax_module_wrapper()
layer_gaussian_dropout()
layer_gaussian_noise()
layer_global_average_pooling_1d()
layer_global_average_pooling_2d()
layer_global_average_pooling_3d()
layer_global_max_pooling_1d()
layer_global_max_pooling_2d()
layer_global_max_pooling_3d()
layer_group_normalization()
layer_group_query_attention()
layer_gru()
layer_hashed_crossing()
layer_hashing()
layer_identity()
layer_integer_lookup()
layer_jax_model_wrapper()
layer_lambda()
layer_layer_normalization()
layer_lstm()
layer_masking()
layer_max_num_bounding_boxes()
layer_max_pooling_1d()
layer_max_pooling_2d()
layer_max_pooling_3d()
layer_maximum()
layer_mel_spectrogram()
layer_minimum()
layer_mix_up()
layer_multi_head_attention()
layer_multiply()
layer_normalization()
layer_permute()
layer_rand_augment()
layer_random_brightness()
layer_random_color_degeneration()
layer_random_color_jitter()
layer_random_contrast()
layer_random_crop()
layer_random_flip()
layer_random_grayscale()
layer_random_hue()
layer_random_posterization()
layer_random_rotation()
layer_random_saturation()
layer_random_sharpness()
layer_random_shear()
layer_random_translation()
layer_random_zoom()
layer_repeat_vector()
layer_rescaling()
layer_reshape()
layer_resizing()
layer_rnn()
layer_separable_conv_1d()
layer_separable_conv_2d()
layer_simple_rnn()
layer_solarization()
layer_spatial_dropout_1d()
layer_spatial_dropout_2d()
layer_spatial_dropout_3d()
layer_spectral_normalization()
layer_stft_spectrogram()
layer_string_lookup()
layer_subtract()
layer_tfsm()
layer_time_distributed()
layer_torch_module_wrapper()
layer_unit_normalization()
layer_upsampling_1d()
layer_upsampling_2d()
layer_upsampling_3d()
layer_zero_padding_1d()
layer_zero_padding_2d()
layer_zero_padding_3d()
rnn_cell_gru()
rnn_cell_lstm()
rnn_cell_simple()
rnn_cells_stack()