Auditing Shiny apps

Audit Shiny apps with {shinytest2}

Have you ever dreamed of profiling, load testing your Shiny app at each commit, without having to manually run any script?

In this vignette, we’ll see how one can design and automate Shiny apps audit pipelines with shinytest2.

Introduction

This case study consists of analyzing a particularly non-optimized app, whose code is defined below. This app simulates a stiff oscillator, also known as Van der Pol model. Under the hood, this system (composed of 2 differential equations) is integrated with the deSolve package. Stiff systems need a much smaller time step than classic systems, thereby requiring more time to solve:

van_der_pol <- function(t, y, mu) {
  d_x <- y[2]
  d_y <- mu * (1 - y[1]^2) * y[2] - y[1]
  list(c(X = d_x, Y = d_y))
}

server <- function(input, output) {
  output$brussels <- renderPlot({
    y0 <- c(X = input$X, Y = input$Y)
    times <- seq(0, 1000, .01)
    out <- ode(y0, times, van_der_pol, input$mu)
    par(mfrow = c(1, 1))
    plot(out[, 2:3], type = "l", xlab = "X", ylab = "Y", main = "state diagram")
  })
}

ui <- fluidPage(
  headerPanel("Van der Pol oscillator"),
  sidebarLayout(
    sidebarPanel(
      h3("Init values"),
      numericInput("X", label = "X", min = 0.0, max = 5,  value = 1, step = 0.2),
      numericInput("Y", label = "Y", min = 0.0, max = 5,  value = 1, step = 0.2),

      h3("Parameters"),
      numericInput("mu", label = "mu", min = 0.0, max = 5,  value = 1, step = 0.1)
    ),
    mainPanel(
      h3("Simulation results"),
      plotOutput("brussels")
    )
  )
)

shinyApp(ui = ui, server = server)

As a first test, you can run the app by calling runApp(system.file("vig-apps/non-optimized-app/", package = "shinytest2")) locally and notice how slow it is to perform one computation. The next question is, how does this app scale? The answer is likely “No it doesn’t!”, but we would like to know exactly how bad it is. This is where load testing comes to play.

Load testing with {shinytest2}

Shiny app load testing aims at running multiple identical user sessions in parallel and measure the resulting app response. It answers many questions, such as:

What is the time necessary to get to the homepage?
How much time does it take to perform the first computation?
Will all session end at the same time?

A load test is composed a three phases:

Similarly to shinytest2, shinyloadtest records a user session with shinyloadtest::record_session().
This session is subsequently replayed by a Java-based tool, shinycannon, able to simulate multiple sessions during a chosen amount of time.
Sessions are analyzed by shinyloadtest::load_runs() and an HTML report generated with shinyloadtest::shinyloadtest_report().

In general, the first step is done manually, that is, you play with the app as if you were a real business user and stop the session when satisfied. However, because of shinytest2 headless capabilities, we could manipulate the app with shinytest2 helpers such as set_inputs or raw JavaScript code to achieve the same goal.

There is, however, a tiny technical obstacle to overcome. By default, shinytest2 starts the app on a given port and the headless browser, namely Chrome, is then connected to the same port. This would be an issue with shinyloadtest since the recorder does not listen to the same port, which is 8600. In practice, we’ll have to:

Start the Shiny app as a background process on a given port.
Fire the load test recorder on port 8600.
Connect Chrome to the recorder on port 8600.

Launch the background app

You may have already noticed that when launching a Shiny app, you can’t run anything else in the R console while the app is live. The explanation is pretty simple: R performs tasks sequentially and can only perform one calculation at a time.

How do we start the app without blocking the main R process? We leverage the callr package, which exposes a convenient API to start R processes in the background, that is, without blocking the main R process. The code below shows how to start a Shiny app located at path on a specific port and run the load test recorder on the same port:

# Main shiny app
shiny_bg <- function(path, port) {
  options(shiny.port = port)
  shiny::runApp(path)
}

# Start recorder
recorder_bg <- function(port) {
  shinyloadtest::record_session(
    target_app_url = sprintf("http://127.0.0.1:%s", port),
    host = "127.0.0.1",
    port = 8600,
    output_file = "recording.log",
    open_browser = FALSE
  )
}

We can pass this to the start_r_bg() function:

start_r_bg <- function(fun, path = NULL, port = 3515) {

  # remove NULL elements
  args <- Filter(Negate(is.null), list(path = path, port = port))

  process <- callr::r_bg(
    func = fun,
    args = args,
    stderr= "",
    stdout = ""
  )

  while (any(is.na(pingr::ping_port("127.0.0.1", 3515)))) {
    message("Waiting for Shiny app to start...")
    Sys.sleep(0.1)
  }

  attempt::stop_if_not(
    process$is_alive(),
    msg = "Unable to launch the subprocess"
  )

  process
}

where r_bg() starts a background R process, passing the corresponding function and parameters. Besides, we provide some log elements and safety guard in case the app can’t start. To launch the app and recorder we can call:

target <- start_r_bg(shiny_bg, path = system.file("vig-apps/non-optimized-app/", package = "shinytest2"))
# Listening on 127.0.0.1:3515
recorder <- start_r_bg(recorder_bg)
# Listening on 127.0.0.1:8600

Connect Chrome

The previous part was the most technical step. Now, we only have to start a Chrome headless browser on port 8600, where the load test recorder runs. You’ll notice that shinytest2 also supports remote urls. We should increase the value of load_timout to 15 seconds to help us avoid producing this annoying shinytest2 warning:

{shinytest2} R  info 15:25:37.36 Error while initializing AppDriver:
Shiny app did not become stable in 10000ms.

# Start AppDriver with recorder url
chrome <- shinytest2::AppDriver$new("http://127.0.0.1:8600", load_timeout = 15 * 1000)

If your running under Linux OS, below shows the list of R processes running so far in the background:

$ netstat -lntp

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8600          0.0.0.0:*               LISTEN      361115/R
tcp        0      0 127.0.0.1:3515          0.0.0.0:*               LISTEN      361101/R
tcp        0      0 127.0.0.1:44179         0.0.0.0:*               LISTEN      359555/google-chrom

If everything is successful, you should see a Client connected message in the R console. We then change the mu parameter input without forgetting the timeout:

app$set_inputs(mu = 4, timeout_ = 15 * 1000)

We can inspect the logs with the help of chrome$get_log(), to check whether everything run smoothly:

app$get_logs()
#> {shinytest2} R  info 14:27:00.80 Start AppDriver initialization
#> {shinytest2} R  info 14:27:00.82 Creating new ChromoteSession
#> {shinytest2} R  info 14:27:02.20 Navigating to Shiny app
#> {shinytest2} R  info 14:27:02.57 Injecting shiny-tracer.js
#> {chromote}   JS info 14:27:02.61 shinytest2; jQuery not found
#> {chromote}   JS info 14:27:02.63 shinytest2; Loaded
#> {shinytest2} R  info 14:27:02.64 Waiting for Shiny to become ready
#> {chromote}   JS info 14:27:02.72 shinytest2; jQuery found
#> {chromote}   JS info 14:27:02.73 shinytest2; Waiting for shiny session to connect
#> {chromote}   JS info 14:27:03.12 shinytest2; Connected
#> {shinytest2} R  info 14:27:03.17 Waiting for Shiny to become idle for 200ms within 10000ms
#> {chromote}   JS info 14:27:03.18 shinytest2; Waiting for Shiny to be stable
#> {chromote}   JS info 14:27:03.27 shinytest2; shiny:busy
#> {chromote}   JS info 14:27:06.28 shinytest2; shiny:idle
#> {chromote}   JS info 14:27:06.30 shinytest2; shiny:value brussels
#> {chromote}   JS info 14:27:06.48 shinytest2; Shiny has been idle for 200ms
#> {shinytest2} R  info 14:27:06.48 Shiny app started
#> {shinytest2} R  info 14:30:17.79 Setting inputs: 'mu'', ''X'', ''Y'
#> {chromote}   JS info 14:30:17.81 shinytest2; inputQueue: adding mu
#> {chromote}   JS info 14:30:17.81 shinytest2; inputQueue: adding X
#> {chromote}   JS info 14:30:17.81 shinytest2; inputQueue: adding Y
#> {chromote}   JS info 14:30:17.82 shinytest2; inputQueue: flushing mu
#> {chromote}   JS info 14:30:17.83 shinytest2; inputQueue: flushing X
#> {chromote}   JS info 14:30:17.83 shinytest2; inputQueue: flushing Y
#> {chromote}   JS info 14:30:17.87 shinytest2; shiny:busy
#> {chromote}   JS info 14:30:20.42 shinytest2; shiny:idle
#> {chromote}   JS info 14:30:20.46 shinytest2; shiny:value brussels
#> {shinytest2} R  info 14:30:20.47 Finished setting inputs. Timedout: FALSE

Once satisfied, we may close the headless connection to stop the recorder:

# clean
app$stop()
# needed to avoid
# java.lang.IllegalStateException: last event in log not a
# WS_CLOSE (did you close the tab after recording?)
Sys.sleep(2)

Replay with shinycannon

If you remember, the final step consists of replaying the main session in parallel. We start shinycannon with 5 workers and wait:

target_url <- "http://127.0.0.1:3515"
workers <- 5
system(
  sprintf(
    "shinycannon recording.log %s --workers %s --loaded-duration-minutes 2 --output-dir run1",
    target_url, workers
  )
)

# shinycannon replay
#2022-05-09 14:37:18.549 INFO [thread00] - Detected target application type: R/Shiny
#2022-05-09 14:37:18.560 INFO [thread00] - Waiting for warmup to complete
#2022-05-09 14:37:18.554 INFO [thread01] - Warming up
#2022-05-09 14:37:18.562 INFO [progress] - Running: 0, Failed: 0, Done: 0
#2022-05-09 14:37:23.563 INFO [progress] - Running: 1, Failed: 0, Done: 0

Some error message in the shinycannon logs can be explained by failures during the shinytest2 driver initialization. If this error persist, best practice is to restart R, cleanup everything and start again.

Report generation

The report generation is quite straightforward:

# Close the running app
target$kill()

# Treat data and generate report
df <- shinyloadtest::load_runs("run1")
shinyloadtest::shinyloadtest_report(
  df,
  "public/index.html",
  self_contained = TRUE,
  open_browser = FALSE
)

We don’t forget to clean the target app so as to kill the underlying process. Note that if you want to deploy the report on GitHub Pages, you have to name it index.html.

Automating with GitHub Actions

Now, it is time to integrate the current pipeline in a CI/CD workflow. For convenience, we wrap all the previous steps in a single function:

## File: R/audit-app.R
record_loadtest <- function(path, timeout = 15, workers = 5) {
  message("\n---- BEGIN LOAD-TEST ---- \n")
  # start app + recorder
  target <- start_r_bg(shiny_bg, path = path)
  recorder <- start_r_bg(recorder_bg)

  # start headless chrome (points to recorder!).
  # AppDriver also support remote urls.
  app <- shinytest2::AppDriver$new(
    "http://127.0.0.1:8600",
    load_timeout = timeout * 1000
  )

  app$set_inputs(mu = 4, timeout_ = timeout * 1000)

  # clean
  app$stop()
  # needed to avoid
  # java.lang.IllegalStateException: last event in log not a
  # WS_CLOSE (did you close the tab after recording?)
  Sys.sleep(2)

  # shinycannon (maybe expose other params later ...)
  target_url <- "http://127.0.0.1:3515"
  system(
    sprintf(
      "shinycannon recording.log %s --workers %s --loaded-duration-minutes 2 --output-dir run1",
      target_url, workers
    )
  )

  target$kill()

  # Treat data and generate report
  df <- shinyloadtest::load_runs("run1")
  shinyloadtest::shinyloadtest_report(
    df,
    "public/index.html",
    self_contained = TRUE,
    open_browser = FALSE
  )
}

Below is the necessary GitHub Actions yaml file. Overall, this will:

Run each time code is pushed to GitHub.
Run in parallel on 3 different Ubuntu flavors.
Install R, some system dependencies, and R packages. Install shinycannon tools for load-testing.
Deploy the load test report to GitHub Pages.

on:
  push:
    branches: [main, master]
  pull_request:
    branches: [main, master]

name: shiny-loadtest-ci

jobs:
  shiny-loadtest-ci:
    runs-on: ${{ matrix.config.os }}
    permissions:
      contents: write

    name: ${{ matrix.config.os }} (${{ matrix.config.r }})

    strategy:
      fail-fast: false
      matrix:
        config:
          - {os: ubuntu-latest,   r: 'devel', http-user-agent: 'release'}
          - {os: ubuntu-latest,   r: 'release'}
          - {os: ubuntu-latest,   r: 'oldrel-1'}

    env:
      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
      R_KEEP_PKG_SOURCE: yes

    steps:
      - uses: actions/checkout@v2
      - uses: r-lib/actions/setup-r@v2
      - uses: r-lib/actions/setup-r-dependencies@v2
        with:
          cache-version: 2
          extra-packages: |
            any::shinyloadtest
            any::lubridate
            any::DT
            any::callr
            any::shinytest2
            any::deSolve
            any::attempt

      - name: Install shinycannon 💥
        run: |
          sudo bash -c 'apt-get update; apt-get install -y default-jre-headless; apt-get clean; rm -rf /var/lib/apt/lists/*'
          wget https://github.com/rstudio/shinycannon/releases/download/v1.1.3/shinycannon_1.1.3-dd43f6b_amd64.deb
          sudo dpkg -i ./*.deb

      - name: Run load test 🏥
        shell: Rscript {0}
        run: |
            source("R/audit-app.R")
            record_loadtest(path = "app.R");

      - name: Deploy to GitHub pages 🚀
        if: github.event_name != 'pull_request'
        uses: JamesIves/github-pages-deploy-action@4.1.4
        with:
          clean: false
          branch: gh-pages
          folder: public

If you want to test it on your end, below are lines of code to setup a RStudio project and link it to GitHub:

# Inside the project
# Use audit GitHub Actions workflow
usethis::use_github_action(url = "https://raw.githubusercontent.com/rstudio/shinytest2/main/inst/gha/app-audit.yaml")
# Copy audit script
dir.create("R", showWarnings = FALSE)
file.copy(system.file("gha/audit-app.R", package = "shinytest2"), "R/audit-app.R")
# Create your app.R file
file.create("app.R")

message("TODO: User - Copy in your app code to `app.R`!")

# To test it locally
source("R/audit-app.R")
record_loadtest("app.R")

# Test the audit with GitHub Actions
usethis::use_description()
usethis::use_git()
usethis::use_github()
usethis::use_github_pages()

If you test it locally within a package, you may want to ignore public, recording.log by adding them in the .Rbuildignore file, which will avoid unnecessary warnings during any future devtools::check():

usethis::use_build_ignore(c("public", "recording.log"))

An example is available here.

As shown in the above report, especially in the session duration tab, the app is clearly not able to handle 5 simultaneous user due to the very large computations being performed by the Shiny server:

The red area is over represented for most sessions, which means many users have to wait significant amount of time before seeing the app. This is a critical issue, most users will leave the app at this moment.
The blue area is also large, showing that most time is spent for computations. Given the single-threaded nature of R, there are high chances other users might be blocked by already ongoing computations. This is also frustrating, as people may think the app has crashed.

Even though out of this vignette scope, one quick and significant optimization would be to set a caching system with shiny::bind_cache(). As shown in the following figure, adding cache is so fast that shinycannon was able to start more than 1000 sessions in 2 minutes.

David Granjon