exams_eval: Auxiliary Tools for Evaluating Exams

Description

Generation of various helper functions for evaluating exams.

Usage

exams_eval(partial = TRUE, negative = FALSE,
  rule = c("false2", "false", "true", "all", "none"))

Value

exams_eval returns a list with the input parameters

partial, negative, and rule along with the following functions:

checkanswer: function with arguments (correct, answer, tolerance = 0, and type = NULL. It checks whether answer (sufficiently) matches correct or not. It returns 1 for correct, -1 for wrong and 0 for not attempted. In case of partial = TRUE, the functions returns a vector for multiple-choice questions.
pointvec: function with arguments correct = NULL and type = NULL. It computes the vector of points for correct and wrong answers, respectively.
pointsum: function with arguments (correct, answer, tolerance = 0, and type = NULL. It computes the overall number of points.

All of the functions require at least the correct answer and optionally the exercise type (num, mchoice/schoice, or string). By default, the

type is inferred from correct which works automatically except in a few edge cases (e.g., to correctly autodetect a string exercise the correct answer must contain at least one character that is not 0 or 1).

Arguments

partial: logical. Should multiple-choice (mchoice) answers be evaluated as a whole pattern (partial = FALSE) or should partial credits be assigned to each of the choices (partial = TRUE)?
negative: logical or numeric. Handling of negative points for an exercise, for details see below.
rule: character specifying which rule to use for negative partial credits (i.e., only relevant for multiple-choice answers when partial = TRUE).

Details

The function exams_eval is a convenience wrapper for specifying various types of evaluation policies. It returns a set of auxiliary functions that may be useful in the evaluation of exams.

Exercises of types num, string, or schoice can essentially be just correct or wrong. In the former case they will give 100 percent of all points, in the latter either 0 percent or some negative percentage can be assigned. Setting negative = TRUE is equivalent to setting either negative = 1 or equivalently negative = -1, which all signal that 100 percent of the points for the exercise should be subtracted. Other percentages are also possible, e.g., negative = 0.25, which would be a natural choice for "schoice" questions with five answer alternatives. Not that when using an evaluation strategy with negative points for wrong answers, the system that collects the participants' answers should distinguish between ‘solved incorrectly’ and ‘not attempted’ (which should always yield 0 percent).

Moreover, for mchoice (multiple-choice) answers the evaluation policy can either pertain to the answer pattern as a whole (which can be correct or wrong, see above) or it can employ a partial credit strategy. In the latter case, each selected correct choice will yield the fraction 1/ncorrect of points. When an incorrect choice is selected, it should lead to negative points. Five strategies are currently implemented: "false" uses 1/nwrong while "false2" uses 1/max(nwrong, 2); "true" uses 1/ncorrect (so that each wrong selection cancels one correct selection); "all" uses 1 (so that a single wrong selection cancels all correct selections); and "none" uses 0 (so that wrong selections have no effect at all). When aggregating the partial percentages, the overall points can become negative. By setting negative a lower bound can be set: negative = TRUE sets no bound while negative = FALSE sets the bound to zero. Any other numeric value could be set as well, e.g., negative = 0.25.

The functions returned by exams_eval try to automatically infer the type of exercise based on the correct answer. However, this cannot always infer the type reliably (e.g., the number 10 vs. the string 10 vs. a multiple-choice question with two elements, true and false). Specifically, multiple-choice vs. single-choice cannot be distinguished automatically. Hence, it is better to explicitly indicate the exercise type with the type argument.

Evaluations for cloze exercises have to be built by appropriately reusing the building blocks for num/string/schoice/mchoice. For example, the components of cloze exercises have to be evaluated individually and then aggregated as desired. Different evaluations for different item types may be set as in: exams2qti12(..., eval = eval1, schoice = list(eval = eval2)). Then eval = eval1 is used as the default for all exercise types except schoice where eval = eval2 is used.

Thus, exams_eval might not give the complete finished evaluation policy for an entire exam but supplies the most important building blocks for setting this up ‘by hand’. Internally, exams_eval is also used by exams2moodle, exams2qti12 and exams2blackboard for writing the evaluation specifications in the respective XML specifications.

Examples

Run this code

## binary evaluation policy with solutions being either correct
## or wrong: partial = FALSE, negative = FALSE
ee <- exams_eval(partial = FALSE, negative = FALSE)

## points that can be achieved are 0/1
ee$pointvec()

## checkanswer() returns 1 for correct, -1 for incorrect and 0 for missing answer
ee$checkanswer(1.23, 1.23)
ee$checkanswer(1.23, "1.23")
ee$checkanswer(1.23, "1,23")
ee$checkanswer(1.23, 1.24)
ee$checkanswer(1.23, 1.24, tolerance = 0.01)
ee$checkanswer(1.23, NA)
ee$checkanswer(1.23, NULL)
ee$checkanswer(1.23, "")

## similarly for logical (mchoice/schoice) answers
## (which allows either string or logical specification)
ee$checkanswer("10000", "10000")
ee$checkanswer(c(TRUE, FALSE, FALSE, FALSE, FALSE), c(TRUE, FALSE, FALSE, FALSE, FALSE))
ee$checkanswer(c(TRUE, FALSE, FALSE, FALSE, FALSE), "10000")
ee$checkanswer("10000", "01000")
ee$checkanswer("10000", "11000")

## and analogously for strings
ee$checkanswer("foo", "foo")
ee$checkanswer("foo", "bar")
ee$checkanswer("foo", "")

## obtain points achieved
ee$pointsum("10000", "10000")
ee$pointsum("10000", "01000")
ee$pointsum("10000", "00000")
ee$pointsum("10000", NA)

## ---------------------------------------------------------
## evaluation policy with -25% penalty for wrong answers
ee <- exams_eval(partial = FALSE, negative = -0.25)

## points that can be achieved are 1/-0.25 (or zero)
ee$pointvec()

## obtain points achieved
ee$pointsum("10000", "10000")
ee$pointsum("10000", "01000")
ee$pointsum("10000", "00000")
ee$pointsum("10000", NA)
ee$pointsum(1.23, 1.23)
ee$pointsum(1.23, 2.34)
ee$pointsum(1.23, NA)
ee$pointsum(1.23, 1.24)
ee$pointsum(1.23, 1.24, tolerance = 0.1)

## ---------------------------------------------------------
## default evaluation policy with partial points
## (but without negative points overall)
ee <- exams_eval()

## points that can be achieved are 1/3 (1/#true)
## or -1/2 (1/#false)
ee$pointvec("10101")

## obtain points achieved
ee$pointsum("10101", "10101")
ee$pointsum("10101", "10100")
ee$pointsum("10101", "11100")
ee$pointsum("10101", "01010")
ee$pointsum("10101", "00000")

## show individual answer check
ee$checkanswer("10101", "10101")
ee$checkanswer("10101", "10100")
ee$checkanswer("10101", "11100")
ee$checkanswer("10101", "01010")
ee$checkanswer("10101", "00000")

## numeric/string answers are not affected by partial=TRUE
ee$checkanswer(1.23, 1.23)
ee$pointsum(1.23, 1.23)
ee$checkanswer(1.23, 2.34)
ee$pointsum(1.23, 2.34)

## ---------------------------------------------------------
## evaluation policy with partial points
## (and with up to -25% negative points overall)
ee <- exams_eval(partial = TRUE, negative = -0.25)

## points that can be achieved are 1/3 (1/#true)
## or -1/2 (1/#false)
ee$pointvec("10101")

## obtain points achieved
ee$pointsum("10101", "10101")
ee$pointsum("10101", "01010")
ee$pointsum("10101", "00000")

## show individual answer check
ee$checkanswer("10101", "10101")
ee$checkanswer("10101", "10100")
ee$checkanswer("10101", "11100")
ee$checkanswer("10101", "01010")
ee$checkanswer("10101", "00000")

## numeric/string answers are not affected by partial=TRUE
ee$pointsum(1.23, 1.23)
ee$pointsum(1.23, 2.34)