Does user code for an exercise return the expected feedback?

expect_feedback() is designed for use in the -tests chunk of interactive learnr exercises. When used in a test chunk, it will evaluate the exercise as if user_code were the user's submitted code, comparing the feedback returned by the exercise's checking function and code with the expected feedback values.

In other words, expect_feedback() will check that the user_code returns the expected feedback. You may check that

the user code is marked as correct (or incorrect)
the feedback message matches the expected message
the feedback is of the expected type and location
additional attributes of the feedback, specified in ..., are as expected.

When used in a -tests chunk, it's best to include expect_feedback() inside testthat::test_that() to provide a description of the behavior being tested:

Usage

expect_feedback(
  user_code,
  correct = NULL,
  message = NULL,
  type = NULL,
  location = NULL,
  error = FALSE,
  ...,
  perl = FALSE,
  fixed = FALSE,
  ignore.case = FALSE,
  .exercise = NULL,
  .envir = NULL,
  .label = NULL,
  .evaluate_global_setup = FALSE
)

Arguments

user_code

The code a user might have submitted to the exercise. This code should be a single unquoted expression. If the user code requires multiple expressions, it should be wrapped in curly braces ({}).

user_code does not support rlang injection operators. To provide user_code as text, provide the code as .exercise$code and provide nothing for user_code.

correct

[logical(1)]
Whether the user code is expected to be correct. If NULL, the correctness of the user code is not checked.

message

[character(1)]
A string or regular expression that should match the feedback returned for user_code. If NULL, the feedback message is not checked. messages is passed to testthat::expect_match() as regexp.

type

[character(1)]
The expected feedback type. Should be one of "success", "info", "warning", "error", or "custom". If NULL, the feedback type is not checked.

location

[character(1)]
The expected feedback location. Should be one of "append", "prepend", or "replace". If NULL, the feedback location is not checked.

error

[logical(1)] or [list()]
Is the user_code expected to result in an internal error? When FALSE, the default, expect_feedback() will check that an error is not attached to the feedback, i.e. that $feedback$error is NULL in the resulting feedback.

When an error is expected, set error = TRUE (like the knitr chunk option) or provide a list that describes the expected error. If a list is provided, expect_feedback() will test only the items of feedback$error that overlap with the list given to error.

...

Additional named properties of the feedback to be compared with items of the same name in the returned feedback object.

perl

logical. Should Perl-compatible regexps be used?

fixed

logical. If TRUE, pattern is a string to be matched as is. Overrides all conflicting arguments.

ignore.case

if FALSE, the pattern matching is case sensitive and if TRUE, case is ignored during matching.

.exercise

The exercise object. Within a -tests chunk, the exercise will be found automatically and the .exercise argument should not be provided. In other testing code contexts, an exercise created with mock_exercise() can be used.

.envir

[environment]
The parent environment in which the exercise should be evaluated. Only required for testing outside of a -tests chunk.

.label

[character(1)]
A label for the expectation; by default the exercise label will be used.

.evaluate_global_setup

[logical(1)]
Whether to evaluate the global setup code for the exercise.

Value

Invisibly returns the evaluated exercise result for user_code. A list with items: feedback, error_message, timeout_exceeded, and html_output. If the exercise includes checking code, the feedback item will contain the feedback result as a list including at least the items correct, message, type, and location.

Examples

# Here's an example exercise where students would be asked
# to subset `letters` to its first four elements. There's
# some very basic checking code (but normally you'd want to
# use {gradethis}).

ex <- mock_exercise(
  user_code = "",
  solution_code = "letters[1:4]",
  check = '
  is_correct <- identical(last_value, solution)
  list(
    correct = is_correct,
    message = if (is_correct) {
      "Great!"
    } else if (length(last_value) != 4) {
      "I expected a vector with four items."
    } else {
      "Try again!"
    }
  )
  ',
  exercise.checker = "function(last_value, check_code, solution_code, ...) {
    solution <- eval(parse(text = solution_code))
    eval(parse(text = check_code))
  }"
)

# A small helper function to better preview the result from
# evaluating an example user submission.
preview_result <- function(res) {
  res$feedback$html <- format(res$feedback$html)
  res$html_output <- format(res$html_output)
  str(res)
}

# A passing test for the correct solution
res <- expect_feedback(
  letters[1:4],
  correct = TRUE,
  message = "Great!",
  .exercise = ex
)
preview_result(res)
#> List of 4
#>  $ feedback        :List of 5
#>   ..$ correct : logi TRUE
#>   ..$ message : chr "Great!"
#>   ..$ type    : chr "success"
#>   ..$ location: chr "append"
#>   ..$ html    : chr "<div role=\"alert\" class=\"alert alert-success\">Great!</div>"
#>  $ error_message   : NULL
#>  $ timeout_exceeded: logi FALSE
#>  $ html_output     : chr "\n\n\n<pre><code>[1] &quot;a&quot; &quot;b&quot; &quot;c&quot; &quot;d&quot;</code></pre>"
#>  - attr(*, "class")= chr "learnr_exercise_result"

# A passing test for an incorrect submission
res <- expect_feedback(
  letters[1:3],
  correct = FALSE,
  message = "four items",
  .exercise = ex
)
preview_result(res)
#> List of 4
#>  $ feedback        :List of 5
#>   ..$ correct : logi FALSE
#>   ..$ message : chr "I expected a vector with four items."
#>   ..$ type    : chr "error"
#>   ..$ location: chr "append"
#>   ..$ html    : chr "<div role=\"alert\" class=\"alert alert-danger\">I expected a vector with four items.</div>"
#>  $ error_message   : NULL
#>  $ timeout_exceeded: logi FALSE
#>  $ html_output     : chr "\n\n\n<pre><code>[1] &quot;a&quot; &quot;b&quot; &quot;c&quot;</code></pre>"
#>  - attr(*, "class")= chr "learnr_exercise_result"

# A failing test for an incorrect submission
# Here we were expecting a different message
tryCatch(
  expect_feedback(
    letters[1:3],
    correct = FALSE,
    message = "Try again!",
    .exercise = ex
  ),
  error = function(err) {
    # Downgrade the error to a regular message
    # so that the error doesn't break our documentation
    message(conditionMessage(err))
  }
)
#> Feedback `message` from 'ex' does not match "Try again!".
#> Actual value: "I expected a vector with four items\."

Does user code for an exercise return the expected feedback?

Usage

Arguments

Value

See also

Examples