Inputs are a list with 2 or 3 elements:
A query tensor of shape (batch_size, Tq, dim).
A value tensor of shape (batch_size, Tv, dim).
A optional key tensor of shape (batch_size, Tv, dim). If none
supplied, value will be used as a key.
The calculation follows the steps:
Calculate attention scores using query and key with shape
(batch_size, Tq, Tv).
Use scores to calculate a softmax distribution with shape
(batch_size, Tq, Tv).
Use the softmax distribution to create a linear combination of value
with shape (batch_size, Tq, dim).
layer_attention(
object,
use_scale = FALSE,
score_mode = "dot",
dropout = 0,
seed = NULL,
...
)The return value depends on the value provided for the first argument.
If object is:
a keras_model_sequential(), then the layer is added to the sequential model
(which is modified in place). To enable piping, the sequential model is also
returned, invisibly.
a keras_input(), then the output tensor from calling layer(input) is returned.
NULL or missing, then a Layer instance is returned.
Object to compose the layer with. A tensor, array, or sequential model.
If TRUE, will create a scalar variable to scale the
attention scores.
Function to use to compute attention scores, one of
{"dot", "concat"}. "dot" refers to the dot product between the
query and key vectors. "concat" refers to the hyperbolic tangent
of the concatenation of the query and key vectors.
Float between 0 and 1. Fraction of the units to drop for the
attention scores. Defaults to 0.0.
An integer to use as random seed in case of dropout.
For forward/backward compatability.
inputs: List of the following tensors:
query: Query tensor of shape (batch_size, Tq, dim).
value: Value tensor of shape (batch_size, Tv, dim).
key: Optional key tensor of shape (batch_size, Tv, dim). If
not given, will use value for both key and value, which is
the most common case.
mask: List of the following tensors:
query_mask: A boolean mask tensor of shape (batch_size, Tq).
If given, the output will be zero at the positions where
mask==FALSE.
value_mask: A boolean mask tensor of shape (batch_size, Tv).
If given, will apply the mask such that values at positions
where mask==FALSE do not contribute to the result.
return_attention_scores: bool, it TRUE, returns the attention scores
(after masking and softmax) as an additional output argument.
training: Python boolean indicating whether the layer should behave in
training mode (adding dropout) or in inference mode (no dropout).
use_causal_mask: Boolean. Set to TRUE for decoder self-attention. Adds
a mask such that position i cannot attend to positions j > i.
This prevents the flow of information from the future towards the
past. Defaults to FALSE.
Attention outputs of shape (batch_size, Tq, dim).
(Optional) Attention scores after masking and softmax with shape
(batch_size, Tq, Tv).
Other attention layers:
layer_additive_attention()
layer_group_query_attention()
layer_multi_head_attention()
Other layers:
Layer()
layer_activation()
layer_activation_elu()
layer_activation_leaky_relu()
layer_activation_parametric_relu()
layer_activation_relu()
layer_activation_softmax()
layer_activity_regularization()
layer_add()
layer_additive_attention()
layer_alpha_dropout()
layer_auto_contrast()
layer_average()
layer_average_pooling_1d()
layer_average_pooling_2d()
layer_average_pooling_3d()
layer_batch_normalization()
layer_bidirectional()
layer_category_encoding()
layer_center_crop()
layer_concatenate()
layer_conv_1d()
layer_conv_1d_transpose()
layer_conv_2d()
layer_conv_2d_transpose()
layer_conv_3d()
layer_conv_3d_transpose()
layer_conv_lstm_1d()
layer_conv_lstm_2d()
layer_conv_lstm_3d()
layer_cropping_1d()
layer_cropping_2d()
layer_cropping_3d()
layer_dense()
layer_depthwise_conv_1d()
layer_depthwise_conv_2d()
layer_discretization()
layer_dot()
layer_dropout()
layer_einsum_dense()
layer_embedding()
layer_equalization()
layer_feature_space()
layer_flatten()
layer_flax_module_wrapper()
layer_gaussian_dropout()
layer_gaussian_noise()
layer_global_average_pooling_1d()
layer_global_average_pooling_2d()
layer_global_average_pooling_3d()
layer_global_max_pooling_1d()
layer_global_max_pooling_2d()
layer_global_max_pooling_3d()
layer_group_normalization()
layer_group_query_attention()
layer_gru()
layer_hashed_crossing()
layer_hashing()
layer_identity()
layer_integer_lookup()
layer_jax_model_wrapper()
layer_lambda()
layer_layer_normalization()
layer_lstm()
layer_masking()
layer_max_num_bounding_boxes()
layer_max_pooling_1d()
layer_max_pooling_2d()
layer_max_pooling_3d()
layer_maximum()
layer_mel_spectrogram()
layer_minimum()
layer_mix_up()
layer_multi_head_attention()
layer_multiply()
layer_normalization()
layer_permute()
layer_rand_augment()
layer_random_brightness()
layer_random_color_degeneration()
layer_random_color_jitter()
layer_random_contrast()
layer_random_crop()
layer_random_flip()
layer_random_grayscale()
layer_random_hue()
layer_random_posterization()
layer_random_rotation()
layer_random_saturation()
layer_random_sharpness()
layer_random_shear()
layer_random_translation()
layer_random_zoom()
layer_repeat_vector()
layer_rescaling()
layer_reshape()
layer_resizing()
layer_rnn()
layer_separable_conv_1d()
layer_separable_conv_2d()
layer_simple_rnn()
layer_solarization()
layer_spatial_dropout_1d()
layer_spatial_dropout_2d()
layer_spatial_dropout_3d()
layer_spectral_normalization()
layer_stft_spectrogram()
layer_string_lookup()
layer_subtract()
layer_text_vectorization()
layer_tfsm()
layer_time_distributed()
layer_torch_module_wrapper()
layer_unit_normalization()
layer_upsampling_1d()
layer_upsampling_2d()
layer_upsampling_3d()
layer_zero_padding_1d()
layer_zero_padding_2d()
layer_zero_padding_3d()
rnn_cell_gru()
rnn_cell_lstm()
rnn_cell_simple()
rnn_cells_stack()