Large quarto projects and speed

At work I am often asked to share my analysis and keep updating it as more data comes in. In the beginning, I used powerpoints because it’s easy to copy paste plots from positron / vscode directly. But as the volume of data coming grew, it became clear that I needed a better system so that I spent less time updating analysis (necessary, but boring) and more time doing new analysis.

I thought Quarto would be a good choice for this project, and I always wanted to use Quarto for something bigger. And for the most part, it did.

With the release of release of Positron IDE and it’s tight integration for Quarto, I felt like it queitly has become a daily tool. Plus I enjoy the added benefit of sounding like a hardcore programmer now that I can say I use vscode as my daily IDE of choice ¹.

One reason I chose quarto for this project was because it can handle multiple R/python environments, and caching, which I anticipated to help as website, analysis needs, and data grows - and they always do. The nice thing about quarto and .qmd notebooks specifically is that it also enables code-driven development and multi-author collaboration possible. Though, after 5 years of PhD and 3 years of industry work, I think I can count on one hand the number of times I had to “collaboratively” edit the same analysis notebook. I ponder if the all the hours spent figuring out the confusing differences between git terms like branch, forks, detached HEAD, HEAD~1/HEAD^ / HEAD~ (they are all the same), origin/main vs local main was worth it.

I was not unaware of the problems around performance with quarto. I had already read the numerous discussion in the past about limitations of serial/parallel rendering [5]. Perhaps because there were community workarounds, and maybe also because the Quarto team was aware and stated plans to improve performance [6], I felt like Quarto was a good choice for my project.

Figure 1: An optimistic render time by size of quarto project where the average page-level render time is 1-3 seconds. See render_in_isolation.R for code

Fighting with Quarto

I knew Quarto had performance problems before I had started. At least, I knew in this sort of vague way that every software has certain “problems”. I had read the github discussions and community workarounds but the exact details were muddy. Well, I now know very clearly that there are two issues with scaling Quarto to large websites.

The first issue is that quarto generates intermediate files during render, meaning that rendering a .qmd in parallel is not supported by default.

This issue I was already aware of. It appeared to me that easy workarounds exist: copy qmd into a new environmnet and render there, copy outputs back. And this is indeed what proposed community solutions contained - though not completely generalizable to my specific project (website + usage of includes [7]). It was relatively easy to ask Claude to generate me a more robust solution that worked well in my own personal project. I will share that solution here, in case it might benefit others, who also encounter troubles with the workarounds provided on the quarto github issues/discussions.

So the first issue was easy, the solution enables parallel renders, combined with freeze means that computations only happen in a very event-driven and dependency-depenedent manner (hello good friend r-targets).

The second issue is that even with quarto’s freeze, which I would describe as caching ², Quarto still takes 1-2 seconds to render each page on full website renders. Meaning that it will take at minimum 2 minutes to render the full website of 120 pages. In practice, because I have a combination of pages, some that utilize freezing and others that do benefit from re-rendering every time, pre- and post- render scripts, the total time it takes is reaching 10-15 minutes. Combine that with local vs deployment previews, it can take a few hours to make simple changes to a deployed website. On the stakeholder-management side, it is hard to explain to a user that their “small change” will be live in a few hours.

I empathize with them, why does it need to take so long to re-render the whole website, especially when the changes involve no code or underlying data changes?

Unfortunately, I’m not aware of any solutions to this limitation. So today, I handle the full website renders by offloading that to a server with Prefect to run renders on schedule and on events like through GHA - this was not trivial, and is the end result of a long process of trying other approaches. If I look at the amount of time I spent writing analysis versus figuring out how to automate the boring stuff, I wonder if an job title more reflective of my work would be “dev ops” or “data” engineer. For that, I have to thank the quarto team for leading me to this hat-swapping opportunity.

Why I still stand by my choice

Quarto, despite your flaws, I still think you were the best choice for my situation. The problems I listed were not gating, the website runs, I can write my analyses from the comfort of R/python, and make some nice looking webpages that my collaborators appreciate. Quarto has a very active community and developer team which means there are gems and tips³ hidden all over that are delightful to discover.

I do wish that I spent less time waiting for my website to render, but at the same, I am grateful for learning a little bit more about DevOps, orchestration, and automation in general. I wonder for example, if Quarto didn’t have these performance limitations, when would I have learned skills?

Technical Appendix: parallel quarto render solution

Why includes complicates parallel rendering

A .qmd document can be treated like a function: parameterized and reused as a shared component that other .qmd documents call into. For example, I might have a penguins.qmd that analyzes penguin body measurements, and a series of per-species documents that each include penguins.qmd, passing in a different species as a parameter.

This is a great way to follow DRY principles in an “analysis notebook” workflow: write the analysis once, reuse it across many pages. But it comes that it complicates parallelizing the renders. Rendering a .qmd produces intermediate files during the render process. When two documents that both include the same shared .qmd render at the same time, their intermediate state collides: each process overwrites the other’s files mid-render, and both renders error out.

One community workaround) symlinks the .qmd under a unique name, renders that, then cleans up:

ln -s template.qmd page_A.qmd
quarto render page_A.qmd -P counter:1 --output page_A.html
unlink page_A.qmd

Though this works for standalone parameterized reports, but in a Quarto website project the output needs to land in the _site/ or output directory. Thus,

render_in_isolation()

The idea is: for each .qmd you want to render, create a fresh tempdir, symlink in the project resources it needs, render inside that sandbox, and copy the results back.

flowchart TD
    A["Project directory<br/>(_quarto.yml, R/, data/, …)"]
    B["render_in_isolation('page.qmd')"]
    C["tempdir('/tmp/render-XXXX')"]
    D["Symlink project resources<br/>into tempdir"]
    E["Symlink target .qmd<br/>into tempdir"]
    F["quarto::quarto_render('page.qmd')"]
    G["_freeze/page/ produced<br/>in tempdir"]
    H["Results copied back<br/>to project _freeze/ and _site/"]
    I["tempdir deleted"]
    J["Done"]

    A --> B
    B --> C
    C --> D
    C --> E
    D --> F
    E --> F
    F --> G
    G --> H
    H --> I
    I --> J

Pairing with {targets} to run in parallel:

library(targets)
source("render_in_isolation.R")

list(
  tar_target(
    pages,
    c("reports/page1.qmd", "reports/page2.qmd", "reports/page3.qmd")
  ),
  tar_target(
    rendered,
    render_in_isolation(pages), # instead of quarto::render
    pattern = map(pages)
  )
)

The full source is in render_in_isolation.R. Here are the key pieces.

The main logic

render_in_isolation <- function(
  qmd_path,
  project_dir = ".",
  symlinks = NULL,
  keep_failed = FALSE,
  output_dir = "_site",
  ...
) {
  start <- Sys.time()
  project_dir <- fs::path_abs(project_dir)
  tempdir_path <- tempfile("render-", tmpdir = "/tmp")
  fs::dir_create(tempdir_path)

  render_ok <- FALSE
  original_wd <- getwd()

  on.exit(
    {
      setwd(original_wd)
      if (fs::dir_exists(tempdir_path) && !(keep_failed && !render_ok)) {
        fs::dir_delete(tempdir_path)
      }
    },
    add = TRUE
  )

  links <- if (is.null(symlinks)) DEFAULT_SYMLINKS else symlinks
  links <- c(links, list_root_templates(project_dir))

  # Sibling "_*" files next to the target qmd (e.g. a shared template used
  # via includes) also need to be symlinked in. A missing qmd_dir is reported
  # as a normal render failure below rather than crashing here.
  qmd_dir <- fs::path_dir(qmd_path)
  if (qmd_dir != ".") {
    qmd_dir_files <- tryCatch(
      fs::dir_ls(fs::path(project_dir, qmd_dir), type = "file", all = FALSE),
      error = function(e) character(0)
    )
    for (file_path in qmd_dir_files) {
      file_name <- fs::path_file(file_path)
      if (startsWith(file_name, "_")) {
        links <- c(links, fs::path(qmd_dir, file_name))
      }
    }
  }

  # The render target itself may already be one of the underscore-prefixed
  # templates symlinked above; dedupe so it's only linked once.
  links <- unique(links)
  links <- links[links != qmd_path]
  for (link in links) {
    symlink_if_exists(link, project_dir, tempdir_path)
  }

  setwd(tempdir_path)

  result <- tryCatch(
    {
      target_src <- fs::path(project_dir, qmd_path)
      if (!fs::file_exists(target_src)) {
        cli::cli_abort("Target qmd does not exist: {.path {target_src}}")
      }
      target_dest <- fs::path(tempdir_path, qmd_path)
      fs::dir_create(fs::path_dir(target_dest), recurse = TRUE)
      if (fs::link_exists(target_dest)) {
        fs::file_delete(target_dest)
      }
      fs::link_create(fs::path_abs(target_src), target_dest)

      quarto::quarto_render(qmd_path, ...)
      copy_freeze_back(qmd_path, tempdir_path, project_dir)
      copy_output_back(qmd_path, tempdir_path, project_dir, output_dir)
      list(error = NULL, success = TRUE)
    },
    error = function(e) list(error = conditionMessage(e), success = FALSE)
  )
  render_ok <- isTRUE(result$success)

  list(
    qmd = qmd_path,
    success = result$success,
    error = result$error,
    tempdir = if (keep_failed && !result$success) tempdir_path else NULL,
    duration_sec = as.numeric(difftime(Sys.time(), start, units = "secs"))
  )
}

Three principles drive the design:

Symlinking project resources

The function populates a tempdir with symbolic links to everything a Quarto render might need:

DEFAULT_SYMLINKS <- c(
  "_quarto.yml",
  "_brand.yml",
  ".Rprofile",
  "R",
  "DESCRIPTION",
  "NAMESPACE",
  "renv",
  "renv.lock",
  "brand",
  "data",
  "includes",
  "img",
  "figures",
  "figure",
  "styles.css",
  "scripts",
  "references.bib"
)

Note

This is a non-exhaustive list of resources that I needed for my own website - add more as needed

It also auto-detects any underscore-prefixed file at the project root (_metadata.yml, _page_template.qmd, etc.) and any sibling underscore-prefixed files sitting next to the target .qmd. This is the piece that actually solves the includes problem from earlier: when species-A.qmd and species-B.qmd both include the shared penguins.qmd, each gets its own symlinked copy of penguins.qmd in its own sandboxed tempdir, so their intermediate render state can never collide. Missing paths are silently skipped, and if the render target itself happens to be one of the auto-detected templates, it’s only linked once.

Freeze and output management

After quarto::quarto_render() completes, the function copies _freeze/<page>/ and the rendered HTML (from the tempdir’s _site/) back into the real project. This is what makes freeze: true work across parallel renders — each page’s freeze data arrives back to the right place without conflicts.

Error handling

The function returns a list with $success, $error, $tempdir, and $duration_sec. With keep_failed = TRUE, the tempdir is preserved for debugging and its path is returned in $tempdir. Temp directories are created in /tmp (not R’s session temp), so they survive the R process exiting.

References

[1]

quarto-dev, “HPC parallel render bug.” 2024. Available: https://github.com/quarto-dev/quarto-cli/issues/5861

[2]

quarto-dev, 2024. Available: https://github.com/orgs/quarto-dev/discussions/4684

[3]

quarto-devs, “Parameterized parallel report conflicts.” 2024. Available: https://github.com/quarto-dev/quarto-cli/issues/4730

[4]

quarto-dev, “Parallel rendering feature request.” 2024. Available: https://github.com/orgs/quarto-dev/discussions/2749

[5]

D. Mackinlay, “Quarto is slow.” Available: https://danmackinlay.name/notebook/quarto_websites.html

[6]

Posit, “What’s next for quarto 2.” 2026. Available: https://opensource.posit.co/blog/2026-04-06_whats-next-quarto-2/

[7]

Quarto, “Includes.” Available: https://quarto.org/docs/authoring/includes.html

Footnotes

1. well technically I should say it is a vscode-fork, still cool!

2. but there is actually a specific Quarto cache feature that is different from freeze, still I think it is easy to think of them both as caching because in essence they are

3. caddy file-server --listen :4210 --root _site is SO much faster than quarto preview, tysm D. Mackinlay