Learn R Programming
maestro (version 0.5.3)

MaestroPipeline: Class for an individual maestro pipeline A pipeline is defined as a single R script with a schedule or input

Description

Class for an individual maestro pipeline A pipeline is defined as a single R script with a schedule or input

Class for an individual maestro pipeline A pipeline is defined as a single R script with a schedule or input

Arguments

Methods

Method new()

Create a new Pipeline object

Usage

MaestroPipeline$new(
  script_path,
  pipe_name,
  frequency = NA_character_,
  start_time = lubridate::NA_POSIXct_,
  tz = NA_character_,
  hours = NULL,
  days = NULL,
  months = NULL,
  skip = FALSE,
  log_level = "INFO",
  inputs = NULL,
  outputs = NULL
)

Arguments

script_path

path to the script

pipe_name

name of the pipeline

frequency

frequency of the pipeline (e.g., 1 day)

start_time

start time of the pipeline

tz

time zone of the pipeline

hours

specific hours of the day

days

specific days of week or month

months

specific months of year

skip

whether to skip the pipeline regardless of scheduling

log_level

log level of the pipeline

inputs

names of pipelines that this pipeline is dependent on for input

outputs

names of pipelines for which this pipeline is a dependency

Returns

MaestroPipeline object

Method print()

Prints the pipeline

Usage

MaestroPipeline$print()

Returns

print

Method run()

Runs the pipeline

Usage

MaestroPipeline$run(
  resources = list(),
  log_file = tempfile(),
  quiet = FALSE,
  log_file_max_bytes = 1e+06,
  .input = NULL,
  cli_prepend = "",
  log_to_console = FALSE,
  ...
)

Arguments

resources

named list of arguments and values to pass to the pipeline

log_file

path to the log file for logging

quiet

whether to silence console output

log_file_max_bytes

maximum bytes of the log file before trimming

.input

input values from upstream pipelines

cli_prepend

text to prepend to cli output

log_to_console

whether or not to output statements in the console (FALSE is to suppress and append to log)

...

additional arguments (unused)

Returns

invisible

Method get_pipe_name()

Get the pipeline name

Usage

MaestroPipeline$get_pipe_name()

Returns

pipeline_name

Method get_schedule()

Get the schedule as a data.frame

Usage

MaestroPipeline$get_schedule()

Returns

data.frame

Method check_timeliness()

Check whether a pipeline is scheduled to run based on orchestrator frequency and current time

Usage

MaestroPipeline$check_timeliness(
  orch_unit,
  orch_n,
  check_datetime = lubridate::now(),
  ...
)

Arguments

orch_unit

unit of the orchestrator (e.g., day)

orch_n

number of units of the frequency

check_datetime

datetime against which to check the timeliness of the pipeline (should almost always be now)

...

unused

Returns

MaestroPipeline

Method get_status()

Get status of the pipeline as a data.frame

Usage

MaestroPipeline$get_status()

Returns

data.frame

Method get_status_chr()

Get status of the pipeline as a string

Usage

MaestroPipeline$get_status_chr()

Returns

character

Method get_outputs()

Names of pipelines that receive input from this pipeline

Usage

MaestroPipeline$get_outputs()

Returns

character

Method get_inputs()

Names of pipelines that input into this pipeline

Usage

MaestroPipeline$get_inputs()

Returns

character

Method get_artifacts()

Get artifacts (return values) from the pipeline

Usage

MaestroPipeline$get_artifacts()

Returns

list

Method get_errors()

Get list of errors from the pipeline

Usage

MaestroPipeline$get_errors()

Returns

list

Method get_warnings()

Get list of warnings from the pipeline

Usage

MaestroPipeline$get_warnings()

Returns

list

Method get_messages()

Get list of messages from the pipeline

Usage

MaestroPipeline$get_messages()

Returns

list

Method update_inputs()

Update the inputs of a pipeline

Usage

MaestroPipeline$update_inputs(inputs)

Arguments

inputs

character vector of inputting pipeline names

Returns

list

Method update_outputs()

Update the outputs of a pipeline

Usage

MaestroPipeline$update_outputs(outputs)

Arguments

outputs

character vector of outputting pipeline names

Returns

list

Method clone()

The objects of this class are cloneable with this method.

Usage

MaestroPipeline$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.