This layer maps a set of arbitrary integer input tokens into indexed integer
output via a table-based vocabulary lookup. The layer's output indices will
be contiguously arranged up to the maximum vocab size, even if the input
tokens are non-continguous or unbounded. The layer supports multiple options
for encoding the output via output_mode, and has optional support for
out-of-vocabulary (OOV) tokens and masking.
The vocabulary for the layer must be either supplied on construction or
learned via adapt(). During adapt(), the layer will analyze a data set,
determine the frequency of individual integer tokens, and create a
vocabulary from them. If the vocabulary is capped in size, the most frequent
tokens will be used to create the vocabulary and all others will be treated
as OOV.
There are two possible output modes for the layer. When output_mode is
"int", input integers are converted to their index in the vocabulary (an
integer). When output_mode is "multi_hot", "count", or "tf_idf",
input integers are encoded into an array where each dimension corresponds to
an element in the vocabulary.
The vocabulary can optionally contain a mask token as well as an OOV token
(which can optionally occupy multiple indices in the vocabulary, as set
by num_oov_indices).
The position of these tokens in the vocabulary is fixed. When output_mode
is "int", the vocabulary will begin with the mask token at index 0,
followed by OOV indices, followed by the rest of the vocabulary. When
output_mode is "multi_hot", "count", or "tf_idf" the vocabulary will
begin with OOV indices and instances of the mask token will be dropped.
Note: This layer uses TensorFlow internally. It cannot be used as part of the compiled computation graph of a model with any backend other than TensorFlow. It can however be used with any backend when running eagerly. It can also always be used as part of an input preprocessing pipeline with any backend (outside the model itself), which is how we recommend to use this layer.
Note: This layer is safe to use inside a tf.data pipeline
(independently of which backend you're using).
layer_integer_lookup(
object,
max_tokens = NULL,
num_oov_indices = 1L,
mask_token = NULL,
oov_token = -1L,
vocabulary = NULL,
vocabulary_dtype = "int64",
idf_weights = NULL,
invert = FALSE,
output_mode = "int",
sparse = FALSE,
pad_to_max_tokens = FALSE,
name = NULL,
...
)The return value depends on the value provided for the first argument.
If object is:
a keras_model_sequential(), then the layer is added to the sequential model
(which is modified in place). To enable piping, the sequential model is also
returned, invisibly.
a keras_input(), then the output tensor from calling layer(input) is returned.
NULL or missing, then a Layer instance is returned.
Object to compose the layer with. A tensor, array, or sequential model.
Maximum size of the vocabulary for this layer. This should
only be specified when adapting the vocabulary or when setting
pad_to_max_tokens=TRUE. If NULL, there is no cap on the size of
the vocabulary. Note that this size includes the OOV
and mask tokens. Defaults to NULL.
The number of out-of-vocabulary tokens to use.
If this value is more than 1, OOV inputs are modulated to
determine their OOV value.
If this value is 0, OOV inputs will cause an error when calling
the layer. Defaults to 1.
An integer token that represents masked inputs. When
output_mode is "int", the token is included in vocabulary
and mapped to index 0. In other output modes,
the token will not appear in the vocabulary and instances
of the mask token in the input will be dropped.
If set to NULL, no mask term will be added. Defaults to NULL.
Only used when invert is TRUE. The token to return
for OOV indices. Defaults to -1.
Optional. Either an array of integers or a string path to a
text file. If passing an array, can pass a list, list,
1D NumPy array, or 1D tensor containing the integer vocabulary terms.
If passing a file path, the file should contain one line per term
in the vocabulary. If this argument is set,
there is no need to adapt() the layer.
The dtype of the vocabulary terms, for example
"int64" or "int32". Defaults to "int64".
Only valid when output_mode is "tf_idf".
A list, list, 1D NumPy array, or 1D tensor or the same length
as the vocabulary, containing the floating point inverse document
frequency weights, which will be multiplied by per sample term
counts for the final TF-IDF weight.
If the vocabulary argument is set, and output_mode is
"tf_idf", this argument must be supplied.
Only valid when output_mode is "int".
If TRUE, this layer will map indices to vocabulary items
instead of mapping vocabulary items to indices.
Defaults to FALSE.
Specification for the output of the layer. Values can be
"int", "one_hot", "multi_hot", "count", or "tf_idf"
configuring the layer as follows:
"int": Return the vocabulary indices of the input tokens.
"one_hot": Encodes each individual element in the input into an
array the same size as the vocabulary,
containing a 1 at the element index. If the last dimension
is size 1, will encode on that dimension.
If the last dimension is not size 1, will append a new
dimension for the encoded output.
"multi_hot": Encodes each sample in the input into a single
array the same size as the vocabulary,
containing a 1 for each vocabulary term present in the sample.
Treats the last dimension as the sample dimension,
if input shape is (..., sample_length),
output shape will be (..., num_tokens).
"count": As "multi_hot", but the int array contains
a count of the number of times the token at that index
appeared in the sample.
"tf_idf": As "multi_hot", but the TF-IDF algorithm is
applied to find the value in each token slot.
For "int" output, any shape of input and output is supported.
For all other output modes, currently only output up to rank 2
is supported. Defaults to "int".
Boolean. Only applicable to "multi_hot", "count", and
"tf_idf" output modes. Only supported with TensorFlow
backend. If TRUE, returns a SparseTensor
instead of a dense Tensor. Defaults to FALSE.
Only applicable when output_mode is "multi_hot",
"count", or "tf_idf". If TRUE, the output will have
its feature axis padded to max_tokens even if the number
of unique tokens in the vocabulary is less than max_tokens,
resulting in a tensor of shape (batch_size, max_tokens)
regardless of vocabulary size. Defaults to FALSE.
String, name for the object
For forward/backward compatability.
Creating a lookup layer with a known vocabulary
This example creates a lookup layer with a pre-existing vocabulary.
vocab <- c(12, 36, 1138, 42) |> as.integer()
data <- op_array(rbind(c(12, 1138, 42),
c(42, 1000, 36))) # Note OOV tokens
out <- data |> layer_integer_lookup(vocabulary = vocab)
out
## tf.Tensor(
## [[1 3 4]
## [4 0 2]], shape=(2, 3), dtype=int64)Creating a lookup layer with an adapted vocabulary
This example creates a lookup layer and generates the vocabulary by analyzing the dataset.
data <- op_array(rbind(c(12, 1138, 42),
c(42, 1000, 36))) # Note OOV tokens
layer <- layer_integer_lookup()
layer |> adapt(data)
layer |> get_vocabulary() |> str()
## List of 6
## $ : int -1
## $ : num 42
## $ : num 1138
## $ : num 1000
## $ : num 36
## $ : num 12Note that the OOV token -1 have been added to the vocabulary. The remaining tokens are sorted by frequency (42, which has 2 occurrences, is first) then by inverse sort order.
layer(data)
## tf.Tensor(
## [[5 2 1]
## [1 3 4]], shape=(2, 3), dtype=int64)Lookups with multiple OOV indices
This example demonstrates how to use a lookup layer with multiple OOV indices. When a layer is created with more than one OOV index, any OOV tokens are hashed into the number of OOV buckets, distributing OOV tokens in a deterministic fashion across the set.
vocab <- c(12, 36, 1138, 42) |> as.integer()
data <- op_array(rbind(c(12, 1138, 42),
c(37, 1000, 36))) # Note OOV tokens
out <- data |>
layer_integer_lookup(vocabulary = vocab,
num_oov_indices = 2)
out
## tf.Tensor(
## [[2 4 5]
## [1 0 3]], shape=(2, 3), dtype=int64)Note that the output for OOV token 37 is 1, while the output for OOV token 1000 is 0. The in-vocab terms have their output index increased by 1 from earlier examples (12 maps to 2, etc) in order to make space for the extra OOV token.
One-hot output
Configure the layer with output_mode='one_hot'. Note that the first
num_oov_indices dimensions in the ont_hot encoding represent OOV values.
vocab <- c(12, 36, 1138, 42) |> as.integer()
data <- op_array(c(12, 36, 1138, 42, 7), 'int32') # Note OOV tokens
layer <- layer_integer_lookup(vocabulary = vocab,
output_mode = 'one_hot')
layer(data)
## tf.Tensor(
## [[0 1 0 0 0]
## [0 0 1 0 0]
## [0 0 0 1 0]
## [0 0 0 0 1]
## [1 0 0 0 0]], shape=(5, 5), dtype=int64)Multi-hot output
Configure the layer with output_mode = 'multi_hot'. Note that the first
num_oov_indices dimensions in the multi_hot encoding represent OOV tokens
vocab <- c(12, 36, 1138, 42) |> as.integer()
data <- op_array(rbind(c(12, 1138, 42, 42),
c(42, 7, 36, 7)), "int64") # Note OOV tokens
layer <- layer_integer_lookup(vocabulary = vocab,
output_mode = 'multi_hot')
layer(data)
## tf.Tensor(
## [[0 1 0 1 1]
## [1 0 1 0 1]], shape=(2, 5), dtype=int64)Token count output
Configure the layer with output_mode='count'. As with multi_hot output,
the first num_oov_indices dimensions in the output represent OOV tokens.
vocab <- c(12, 36, 1138, 42) |> as.integer()
data <- rbind(c(12, 1138, 42, 42),
c(42, 7, 36, 7)) |> op_array("int64")
layer <- layer_integer_lookup(vocabulary = vocab,
output_mode = 'count')
layer(data)
## tf.Tensor(
## [[0 1 0 1 2]
## [2 0 1 0 1]], shape=(2, 5), dtype=int64)TF-IDF output
Configure the layer with output_mode='tf_idf'. As with multi_hot output,
the first num_oov_indices dimensions in the output represent OOV tokens.
Each token bin will output token_count * idf_weight, where the idf weights
are the inverse document frequency weights per token. These should be
provided along with the vocabulary. Note that the idf_weight for OOV
tokens will default to the average of all idf weights passed in.
vocab <- c(12, 36, 1138, 42) |> as.integer()
idf_weights <- c(0.25, 0.75, 0.6, 0.4)
data <- rbind(c(12, 1138, 42, 42),
c(42, 7, 36, 7)) |> op_array("int64")
layer <- layer_integer_lookup(output_mode = 'tf_idf',
vocabulary = vocab,
idf_weights = idf_weights)
layer(data)
## tf.Tensor(
## [[0. 0.25 0. 0.6 0.8 ]
## [1. 0. 0.75 0. 0.4 ]], shape=(2, 5), dtype=float32)To specify the idf weights for oov tokens, you will need to pass the entire vocabulary including the leading oov token.
vocab <- c(-1, 12, 36, 1138, 42) |> as.integer()
idf_weights <- c(0.9, 0.25, 0.75, 0.6, 0.4)
data <- rbind(c(12, 1138, 42, 42),
c(42, 7, 36, 7)) |> op_array("int64")
layer <- layer_integer_lookup(output_mode = 'tf_idf',
vocabulary = vocab,
idf_weights = idf_weights)
layer(data)
## tf.Tensor(
## [[0. 0.25 0. 0.6 0.8 ]
## [1.8 0. 0.75 0. 0.4 ]], shape=(2, 5), dtype=float32)When adapting the layer in "tf_idf" mode, each input sample will
be considered a document, and IDF weight per token will be
calculated as:
log(1 + num_documents / (1 + token_document_count)).
Inverse lookup
This example demonstrates how to map indices to tokens using this layer.
(You can also use adapt() with inverse = TRUE, but for simplicity we'll
pass the vocab in this example.)
vocab <- c(12, 36, 1138, 42) |> as.integer()
data <- op_array(c(1, 3, 4,
4, 0, 2)) |> op_reshape(c(2,-1)) |> op_cast("int32")
layer <- layer_integer_lookup(vocabulary = vocab, invert = TRUE)
layer(data)
## tf.Tensor(
## [[ 12 1138 42]
## [ 42 -1 36]], shape=(2, 3), dtype=int64)Note that the first index correspond to the oov token by default.
Forward and inverse lookup pairs
This example demonstrates how to use the vocabulary of a standard lookup layer to create an inverse lookup layer.
vocab <- c(12, 36, 1138, 42) |> as.integer()
data <- op_array(rbind(c(12, 1138, 42), c(42, 1000, 36)), "int32")
layer <- layer_integer_lookup(vocabulary = vocab)
i_layer <- layer_integer_lookup(vocabulary = get_vocabulary(layer),
invert = TRUE)
int_data <- layer(data)
i_layer(int_data)
## tf.Tensor(
## [[ 12 1138 42]
## [ 42 -1 36]], shape=(2, 3), dtype=int64)In this example, the input token 1000 resulted in an output of -1, since
1000 was not in the vocabulary - it got represented as an OOV, and all OOV
tokens are returned as -1 in the inverse layer. Also, note that for the
inverse to work, you must have already set the forward layer vocabulary
either directly or via adapt() before calling get_vocabulary().
Other categorical features preprocessing layers:
layer_category_encoding()
layer_hashed_crossing()
layer_hashing()
layer_string_lookup()
Other preprocessing layers:
layer_auto_contrast()
layer_category_encoding()
layer_center_crop()
layer_discretization()
layer_equalization()
layer_feature_space()
layer_hashed_crossing()
layer_hashing()
layer_max_num_bounding_boxes()
layer_mel_spectrogram()
layer_mix_up()
layer_normalization()
layer_rand_augment()
layer_random_brightness()
layer_random_color_degeneration()
layer_random_color_jitter()
layer_random_contrast()
layer_random_crop()
layer_random_flip()
layer_random_grayscale()
layer_random_hue()
layer_random_posterization()
layer_random_rotation()
layer_random_saturation()
layer_random_sharpness()
layer_random_shear()
layer_random_translation()
layer_random_zoom()
layer_rescaling()
layer_resizing()
layer_solarization()
layer_stft_spectrogram()
layer_string_lookup()
layer_text_vectorization()
Other layers:
Layer()
layer_activation()
layer_activation_elu()
layer_activation_leaky_relu()
layer_activation_parametric_relu()
layer_activation_relu()
layer_activation_softmax()
layer_activity_regularization()
layer_add()
layer_additive_attention()
layer_alpha_dropout()
layer_attention()
layer_auto_contrast()
layer_average()
layer_average_pooling_1d()
layer_average_pooling_2d()
layer_average_pooling_3d()
layer_batch_normalization()
layer_bidirectional()
layer_category_encoding()
layer_center_crop()
layer_concatenate()
layer_conv_1d()
layer_conv_1d_transpose()
layer_conv_2d()
layer_conv_2d_transpose()
layer_conv_3d()
layer_conv_3d_transpose()
layer_conv_lstm_1d()
layer_conv_lstm_2d()
layer_conv_lstm_3d()
layer_cropping_1d()
layer_cropping_2d()
layer_cropping_3d()
layer_dense()
layer_depthwise_conv_1d()
layer_depthwise_conv_2d()
layer_discretization()
layer_dot()
layer_dropout()
layer_einsum_dense()
layer_embedding()
layer_equalization()
layer_feature_space()
layer_flatten()
layer_flax_module_wrapper()
layer_gaussian_dropout()
layer_gaussian_noise()
layer_global_average_pooling_1d()
layer_global_average_pooling_2d()
layer_global_average_pooling_3d()
layer_global_max_pooling_1d()
layer_global_max_pooling_2d()
layer_global_max_pooling_3d()
layer_group_normalization()
layer_group_query_attention()
layer_gru()
layer_hashed_crossing()
layer_hashing()
layer_identity()
layer_jax_model_wrapper()
layer_lambda()
layer_layer_normalization()
layer_lstm()
layer_masking()
layer_max_num_bounding_boxes()
layer_max_pooling_1d()
layer_max_pooling_2d()
layer_max_pooling_3d()
layer_maximum()
layer_mel_spectrogram()
layer_minimum()
layer_mix_up()
layer_multi_head_attention()
layer_multiply()
layer_normalization()
layer_permute()
layer_rand_augment()
layer_random_brightness()
layer_random_color_degeneration()
layer_random_color_jitter()
layer_random_contrast()
layer_random_crop()
layer_random_flip()
layer_random_grayscale()
layer_random_hue()
layer_random_posterization()
layer_random_rotation()
layer_random_saturation()
layer_random_sharpness()
layer_random_shear()
layer_random_translation()
layer_random_zoom()
layer_repeat_vector()
layer_rescaling()
layer_reshape()
layer_resizing()
layer_rnn()
layer_separable_conv_1d()
layer_separable_conv_2d()
layer_simple_rnn()
layer_solarization()
layer_spatial_dropout_1d()
layer_spatial_dropout_2d()
layer_spatial_dropout_3d()
layer_spectral_normalization()
layer_stft_spectrogram()
layer_string_lookup()
layer_subtract()
layer_text_vectorization()
layer_tfsm()
layer_time_distributed()
layer_torch_module_wrapper()
layer_unit_normalization()
layer_upsampling_1d()
layer_upsampling_2d()
layer_upsampling_3d()
layer_zero_padding_1d()
layer_zero_padding_2d()
layer_zero_padding_3d()
rnn_cell_gru()
rnn_cell_lstm()
rnn_cell_simple()
rnn_cells_stack()