The col_vals_increasing() validation function, the
expect_col_vals_increasing() expectation function, and the
test_col_vals_increasing() test function all check whether column values in
a table are increasing when moving down a table. There are options for
allowing NA values in the target column, allowing stationary phases (where
consecutive values don't change), and even on for allowing decreasing
movements up to a certain threshold. The validation function can be used
directly on a data table or with an agent object (technically, a
ptblank_agent object) whereas the expectation and test functions can only
be used with a data table. The types of data tables that can be used include
data frames, tibbles, database tables (tbl_dbi), and Spark DataFrames
(tbl_spark). Each validation step or expectation will operate over the
number of test units that is equal to the number of rows in the table (after
any preconditions have been applied).
col_vals_increasing(
x,
columns,
allow_stationary = FALSE,
decreasing_tol = NULL,
na_pass = FALSE,
preconditions = NULL,
actions = NULL,
step_id = NULL,
label = NULL,
brief = NULL,
active = TRUE
)expect_col_vals_increasing(
object,
columns,
allow_stationary = FALSE,
decreasing_tol = NULL,
na_pass = FALSE,
preconditions = NULL,
threshold = 1
)
test_col_vals_increasing(
object,
columns,
allow_stationary = FALSE,
decreasing_tol = NULL,
na_pass = FALSE,
preconditions = NULL,
threshold = 1
)
A data frame, tibble (tbl_df or tbl_dbi), Spark DataFrame
(tbl_spark), or, an agent object of class ptblank_agent that is
created with create_agent().
The column (or a set of columns, provided as a character vector) to which this validation should be applied.
An option to allow pauses in decreasing values. For
example if the values for the test units are [80, 82, 82, 85, 88] then
the third unit (82, appearing a second time) would be marked with fail
when allow_stationary is FALSE (the default). Using allow_stationary = TRUE will result in all the test units in [80, 82, 82, 85, 88] to be
marked with pass.
An optional threshold value that allows for movement of
numerical values in the negative direction. By default this is NULL but
using a numerical value with set the absolute threshold of negative travel
allowed across numerical test units. Note that setting a value here also
has the effect of setting allow_stationary to TRUE.
Should any encountered NA values be considered as passing
test units? This is by default FALSE. Set to TRUE to give NAs a pass.
An optional expression for mutating the input table
before proceeding with the validation. This is ideally as a one-sided R
formula using a leading ~. In the formula representation, the . serves
as the input data table to be transformed (e.g., ~ . %>% dplyr::mutate(col = col + 10).
A list containing threshold levels so that the validation step
can react accordingly when exceeding the set levels. This is to be created
with the action_levels() helper function.
One or more optional identifiers for the single or multiple
validation steps generated from calling a validation function. The use of
step IDs serves to distinguish validation steps from each other and provide
an opportunity for supplying a more meaningful label compared to the step
index. By default this is NULL, and pointblank will automatically
generate the step ID value (based on the step index) in this case. One or
more values can be provided, and the exact number of ID values should (1)
match the number of validation steps that the validation function call will
produce (influenced by the number of columns provided), (2) be an ID
string not used in any previous validation step, and (3) be a vector with
unique values.
An optional label for the validation step. This label appears in the agent report and for the best appearance it should be kept short.
An optional, text-based description for the validation step. If
nothing is provided here then an autobrief is generated by the agent,
using the language provided in create_agent()'s lang argument (which
defaults to "en" or English). The autobrief incorporates details of the
validation step so it's often the preferred option in most cases (where a
label might be better suited to succinctly describe the validation).
A logical value indicating whether the validation step should
be active. If the validation function is working with an agent, FALSE
will make the validation step inactive (still reporting its presence and
keeping indexes for the steps unchanged). If the validation function will
be operating directly on data (no agent involvement), then any step with
active = FALSE will simply pass the data through with no validation
whatsoever. Aside from a logical vector, a one-sided R formula using a
leading ~ can be used with . (serving as the input data table) to
evaluate to a single logical value. With this approach, the pointblank
function has_columns() can be used to determine whether to make a
validation step active on the basis of one or more columns existing in the
table (e.g., ~ . %>% has_columns(vars(d, e))). The default for active
is TRUE.
A data frame, tibble (tbl_df or tbl_dbi), or Spark
DataFrame (tbl_spark) that serves as the target table for the expectation
function or the test function.
A simple failure threshold value for use with the
expectation (expect_) and the test (test_) function variants. By
default, this is set to 1 meaning that any single unit of failure in data
validation results in an overall test failure. Whole numbers beyond 1
indicate that any failing units up to that absolute threshold value will
result in a succeeding testthat test or evaluate to TRUE. Likewise,
fractional values (between 0 and 1) act as a proportional failure
threshold, where 0.15 means that 15 percent of failing test units results
in an overall test failure.
For the validation function, the return value is either a
ptblank_agent object or a table object (depending on whether an agent
object or a table was passed to x). The expectation function invisibly
returns its input but, in the context of testing data, the function is
called primarily for its potential side-effects (e.g., signaling failure).
The test function returns a logical value.
If providing multiple column names to columns, the result will be an
expansion of validation steps to that number of column names (e.g.,
vars(col_a, col_b) will result in the entry of two validation steps). Aside
from column names in quotes and in vars(), tidyselect helper functions
are available for specifying columns. They are: starts_with(),
ends_with(), contains(), matches(), and everything().
This validation function supports special handling of NA values. The
na_pass argument will determine whether an NA value appearing in a test
unit should be counted as a pass or a fail. The default of na_pass = FALSE means that any NAs encountered will accumulate failing test units.
Having table preconditions means pointblank will mutate the table just
before interrogation. Such a table mutation is isolated in scope to the
validation step(s) produced by the validation function call. Using
dplyr code is suggested here since the statements can be translated to
SQL if necessary. The code is most easily supplied as a one-sided R
formula (using a leading ~). In the formula representation, the . serves
as the input data table to be transformed (e.g.,
~ . %>% dplyr::mutate(col_a = col_b + 10)). Alternatively, a function could
instead be supplied (e.g.,
function(x) dplyr::mutate(x, col_a = col_b + 10)).
Often, we will want to specify actions for the validation. This argument,
present in every validation function, takes a specially-crafted list
object that is best produced by the action_levels() function. Read that
function's documentation for the lowdown on how to create reactions to
above-threshold failure levels in validation. The basic gist is that you'll
want at least a single threshold level (specified as either the fraction of
test units failed, or, an absolute value), often using the warn_at
argument. This is especially true when x is a table object because,
otherwise, nothing happens. For the col_vals_*()-type functions, using
action_levels(warn_at = 0.25) or action_levels(stop_at = 0.25) are good
choices depending on the situation (the first produces a warning when a
quarter of the total test units fails, the other stop()s at the same
threshold level).
Want to describe this validation step in some detail? Keep in mind that this
is only useful if x is an agent. If that's the case, brief the agent
with some text that fits. Don't worry if you don't want to do it. The
autobrief protocol is kicked in when brief = NULL and a simple brief will
then be automatically generated.
A pointblank agent can be written to YAML with yaml_write() and the
resulting YAML can be used to regenerate an agent (with yaml_read_agent())
or interrogate the target table (via yaml_agent_interrogate()). When
col_vals_increasing() is represented in YAML (under the top-level steps
key as a list member), the syntax closely follows the signature of the
validation function. Here is an example of how a complex call of
col_vals_increasing() as a validation step is expressed in R code and in
the corresponding YAML representation.
# R statement
agent %>%
col_vals_increasing(
columns = vars(a),
allow_stationary = TRUE,
decreasing_tol = 0.5,
na_pass = TRUE,
preconditions = ~ . %>% dplyr::filter(a < 10),
actions = action_levels(warn_at = 0.1, stop_at = 0.2),
label = "The `col_vals_increasing()` step.",
active = FALSE
)# YAML representation
steps:
- col_vals_increasing:
columns: vars(a)
allow_stationary: true
decreasing_tol: 0.5
na_pass: true
preconditions: ~. %>% dplyr::filter(a < 10)
actions:
warn_fraction: 0.1
stop_fraction: 0.2
label: The `col_vals_increasing()` step.
active: false
In practice, both of these will often be shorter as only the columns
argument requires a value. Arguments with default values won't be written to
YAML when using yaml_write() (though it is acceptable to include them with
their default when generating the YAML by other means). It is also possible
to preview the transformation of an agent to YAML without any writing to disk
by using the yaml_agent_string() function.
2-13
The analogous function that moves in the opposite direction:
col_vals_decreasing().
Other validation functions:
col_exists(),
col_is_character(),
col_is_date(),
col_is_factor(),
col_is_integer(),
col_is_logical(),
col_is_numeric(),
col_is_posix(),
col_schema_match(),
col_vals_between(),
col_vals_decreasing(),
col_vals_equal(),
col_vals_expr(),
col_vals_gte(),
col_vals_gt(),
col_vals_in_set(),
col_vals_lte(),
col_vals_lt(),
col_vals_make_set(),
col_vals_make_subset(),
col_vals_not_between(),
col_vals_not_equal(),
col_vals_not_in_set(),
col_vals_not_null(),
col_vals_null(),
col_vals_regex(),
conjointly(),
rows_distinct()