vocabulary_size

Used in <code>textrecipes::step_tokenize_sentencepiece()</code> and
<code>textrecipes::step_tokenize_bpe()</code>.

Many models contain tuning parameters (i.e. parameters that
cannot be directly estimated from the data). These tools can be used
to define objects for creating, simulating, or validating values for
such parameters.

Hannah Frick

dials

Tools for Creating Tuning Parameter Values

Max Kuhn

Posit Software, PBC 

vocabulary_size function

<dl><dt>range</dt>
<dd>A two-element vector holding the defaults for the smallest and
largest possible values, respectively. If a transformation is specified,
these values should be in the transformed units.</dd>
<dt>trans</dt>
<dd>A <code>trans</code> object from the <code>scales</code> package, such as
<code>scales::transform_log10()</code> or <code>scales::transform_reciprocal()</code>. If not provided,
the default is used which matches the units used in <code>range</code>. If no
transformation, <code>NULL</code>.</dd></dl>

Arguments

Number of tokens in vocabulary — vocabulary_size

<dl>

<dt>range</dt>
<dd>A two-element vector holding the defaults for the smallest and
largest possible values, respectively. If a transformation is specified,
these values should be in the transformed units.</dd>


<dt>trans</dt>
<dd>A <code>trans</code> object from the <code>scales</code> package, such as
<code>scales::transform_log10()</code> or <code>scales::transform_reciprocal()</code>. If not provided,
the default is used which matches the units used in <code>range</code>. If no
transformation, <code>NULL</code>.</dd>

</dl>

vocabulary_size: Number of tokens in vocabulary

Description

Usage

Arguments

Examples