Comparing Anthropic’s vs Genentech’s Bioinformatics LLM Benchmarks

A deep dive on Anthropic and Genentech’s bioinformatic LLM benchmarks, BioMysteryBench and CompBioBench.

Every week a new large language model (LLM) is released, and with it comes a battery of numbers to prove its incremental improvement over existing models. These numbers come from “benchmarks”: a series of evaluations that domain experts carefully create. Recently, two very different companies released bioinformatics-specific benchmarks: Genentech¹, one of the oldest and most successful pharma companies, and Anthropic², a top AI company. The two companies couldn’t be more different, and I think it shows in their benchmarks.

In this post, I look at some of the questions and evaluations in these benchmarks and share some thoughts about the quality, relevance, and why I think these results show that I’m not going to be replaced by AI anytime soon.

Anthropic and Genentech’s benchmarking philosophy

There are many ways to design LLM benchmarks. Previous bioinformatics benchmarks like BixBench evaluate whether LLM conclusions match up with scientists’. But in science, often there are multiple reasonable approaches to a question that might end up leading to differing but reasonable conclusions. Other benchmarks like scBench are fairly prescriptive in the series of “steps” that a model must take to complete a task, but that hampers creativity, which is a core tenet of innovative scientific research.

It’s both notable and validating that Anthropic and Genentech, independently and concurrently, decided on the same strategy for model evaluation. Both benchmarks are designed such that there is a single data-verifiable ground truth answer, and evaluations are non-prescriptive in how to arrive at that answer. Multiple paths are not necessarily encouraged, but models are free to “wander” along different paths to reach their conclusions.

For example, both benchmarks have many questions classified as “metadata retrieval”. These types of questions involve presenting some data (transcriptomic, epigenetic, etc.) to the model and asking questions about the sample identity: Are these samples male/female? What tissue is this scRNA-seq sample from?. These are good questions because they are data-verifiable: the identity of the sample is known but kept hidden from the model, and they are relevant; they require leveraging biological and technological knowledge, and can require interaction with additional datasets to reach the answer (e.g. mapping transcript identifiers to genes using a public database).

Another type of question that CompBioBench uses relies on synthetic and augmented data. For example, genomic content from contaminant species can be added to an existing dataset, and then the model is tasked with identifying the contaminant species. These are also “data-verifiable”, and have the added benefit that only the authors know the answer to the question. In metadata retrieval, where samples are often pulled from published datasets, models have a small possibility of “cheating”: for example, by finding the answer by matching the paper itself, circumventing any usage of bioinformatics in its solution.

The last notable difference is how the answer is scored. CompBioBench uses an “exact string match” of the agent’s answer and the ground truth, so the tasks are very explicit about the answer format, with some tolerance for numerical answers. BioMysteryBench, on the other hand, uses an LLM-as-a-judge. LLM-as-a-judge is an approach where a model is relied on to interpret an answer, and decide if the answer is correct or not.

I initially thought a deterministic evaluation was clearly the better approach. But after researching the LLM-as-a-judge approach, I learned: it’s very common, with advantages and disadvantages³, and ultimately I’m unsure if one approach is significantly better than the other.

How rigorously these benchmarks are documented

It’s worth first discussing the difference in how these two teams chose to publish their results.

Genentech published their benchmark on bioRxiv, which is a popular preprint server in the bioinformatics field. Their benchmark, problem statement, and results are presented in standard scientific article format. This means everything is deeply documented. And the article itself is hosted by bioRxiv, meaning a non-affiliated organization is ensuring the original content stays accessible⁴.

In contrast, Anthropic announced BioMysteryBench with a blog post. As you can imagine, the difference between a published article and a blog post is vast. There is no standard format for a blog post like there is for a journal article. There is no in-depth section describing how the questions were developed; there are no supplementary figures, and in general it is really hard to dig into details.

The Genentech team, I imagine, is likely preparing the article for submission at a peer-reviewed journal. This means transparency is not optional; it’s required and enforced to some degree by the journal and the peer reviews. They must describe their methods fully; they must defend their decisions; and they must be as comprehensive in scope as reasonably possible. Some journals even enforce reproducibility and open-source code⁵. CompBioBench showed that all top models performed similarly, including Anthropic’s, but also GPT from OpenAI, and Gemini from Google (might I mention: similar performance at a much lower cost!). Anthropic chose to evaluate and publish only their own models.

Coverage over bioinformatics subdomains

Bioinformatics is a big field. There are highly specialized methods depending on species, tissue, and assay type. CompBioBench publishes a distribution of question types (each benchmark has 99-100 questions). BioMysteryBench has no details on distribution but does mention:

In developing this eval, questions were primarily derived from raw or minimally processed DNA or RNA sequencing data since this is where many biological processing pipelines begin (WGS, scRNA-seq, methylation, ChIP-seq, metagenomics, Hi-C), and also included several questions drawn from proteomics and metabolomics.

To compare coverage over different types of bioinformatics tasks, I asked DeepSeek to map Anthropic’s questions onto the domain types that CompBioBench uses⁶:

benchmarkData = FileAttachment("benchmark-comparison.csv").csv({typed: true})

domainOrder = ["Single-cell", "Epigenomics", "Genomics", "Transcriptomics", "Microbiology", "Human Genetics", "Machine Learning", "Proteomics", "Spatial", "Others"]
styleOrder = ["Metadata Recovery", "Synthetic/Augmented Data", "Routine Analysis", "Retrieval", "Tooling"]
benchmarkOrder = ["CompBioBench", "BioMysteryBench"]
benchmarkColor = ({ domain: benchmarkOrder, range: ["#000000", "#aaaaaa"] })

function benchmarkChart(categoryType, order) {
  const chartData = benchmarkData.filter(d => d.category_type === categoryType)
  return Plot.plot({
    width: 680,
    height: order.length * 34 + 40,
    marginLeft: 200,
    marginRight: 50,
    style: { background: "transparent", fontSize: "13px" },
    fx: { domain: benchmarkOrder, label: null},
    y: { domain: order, label: null, tickSize: 0},
    x: { label: "Number of questions", grid: true, tickSize: 0},
    color: benchmarkColor,
    marks: [
      Plot.barX(chartData, { x: "count", y: "category", fx: "benchmark", fill: "benchmark" }),
      Plot.text(chartData, { x: "count", y: "category", fx: "benchmark", text: d => d.count, dx: 16 }),
      Plot.tip(chartData, Plot.pointerY({
        x: "count",
        y: "category",
        fx: "benchmark",
        title: d => `${d.benchmark}\n${d.category}: ${d.count} questions (${d.pct}%)`
      })),
      Plot.ruleX([0])
    ]
  })
}

html`<div style="font-weight: 600; margin: 0.5rem 0 0.25rem;">Domain</div>${benchmarkChart("Domain", domainOrder)}`

html`<div style="font-weight: 600; margin: 1rem 0 0.25rem;">Question Style</div>${benchmarkChart("Question Style", styleOrder)}`

BioMysteryBench has only one single-cell RNA-seq question! Single-cell RNA-seq might be one of the most important novel technologies developed in the last decade [4], and we are at a point where the methods are mature, datasets are plentiful, and so are published, peer-reviewed, well-cited articles. Lacking evaluations on this critical assay type is like evaluating AI at generating accurate physics simulations but saying let’s leave out the third dimension.

What I do like about Anthropic’s work is that they more heavily skewed towards bulk RNA-seq (“Transcriptomics” in the graphs): it’s sort of the bread and butter of next-generation sequencing assays, because of its low cost and mature methods, so it’s highly relevant. BioMysteryBench also seems to cover other data types that are notably lacking in CompBioBench, like proteomics and microbiology.

Spatial transcriptomics is another data type with minimal coverage: CompBioBench includes only two spatial questions, and BioMysteryBench has none. Spatial is becoming an increasingly important readout in life sciences research and is developing at a rapid pace [5]. Many are going to be using AI for spatial analysis, so a more comprehensive benchmark on spatial would be helpful.

The reported numbers

Before diving into individual questions, it’s worth looking at the results from both publications.

	CompBioBench	BioMysteryBench
Questions	100	99 (76 human-solvable, 23 human-difficult)7
What models are evaluated	Claude Code Opus 4.6, Codex GPT 5.4, Gemini 3.1 Pro	Claude Mythos Preview, Opus 4.7, Sonnet 4.6
Best overall	Codex CLI (GPT-5.4): 83% Accuracy8	Opus 4.7: 78.9% on human-solvable
Scoring	Deterministic string matching	LLM-as-judge with cheat detection

Question difficulty

Moving on to the actual content of these benchmarks, I was curious how hard these questions are. I looked through a bunch of the hardest questions, and here I will just talk about one question from each.

First, Genentech’s CompBioBench has difficulty ratings 1-5 (5 is hardest). I could not actually find these difficulty ratings for any of their published questions on Hugging Face. The only questions with known difficulty are therefore the ones chosen as “selected examples” in their article’s main text.

Anthropic’s BioMysteryBench has a much simpler binary categorization: human-solvable and human-difficult (or human-unsolvable).

On the most difficult problems (levels 4-5 on CompBioBench and “human-unsolvable” on BioMysteryBench), top models got 49-69% accuracy on CompBioBench, but only 19-27% on BioMysteryBench’s questions. This suggests that BioMysteryBench has harder questions, which may mean that their benchmark will stay relevant longer, as models get better and better.

CompBioBench Question Difficulty 4/5

Since I couldn’t find a difficult rating for CompBioBench questions, I had to use one of the “selected examples” described in the text. Notably, there is one 5/5 difficult problem in their dataset but it’s not described in the main text.

This is a difficulty 4/5 ATAC-seq question:

Q. atac.doublet.q1.bed.gz contains a fragment file from a scATAC-seq sample (hg38). Barcodes with low read counts have been filtered out. Look for doublets. Return the barcode of any one doublet, if any. Respond only with the label of the barcode in the format: “AAATGGAACGTTAAAG-1”, or “None”.

Doublet detection is a standard step in most single-cell workflows. The authors note that all model attempts involved applying standard doublet filtering, but the correct answer required using a more advanced doublet detection package called AMULET [6] ⁹. AMULET is able to detect the homotypic doublets, which occur when a droplet includes two cells that are the same cell type. Because they are the same cell type, “standard” doublet detection programs which rely on expression patterns cannot detect these.

The impact of failing to detect homotypic doublets is not discussed. In general, leaving a doublet consisting of two cells of the same cell type with the same functional readouts is likely minimally impactful on the analysis. The cases where homotypic doublets might have a large impact are pretty rare, and likely other analyses (like looking at read depth) will inform how much attention is needed to this issue. This question is graded as a simple binary: whether the model provided the correct barcode or not. But I would argue that the “wrong” answers were not that wrong.

I pull this example out because I thought it was one of the few that might be improved or replaced. In general, I found most of CompBioBench’s questions relevant, the instructions clear, and the evaluations also clear. I think reliance on synthetic and augmented data is smart; there is not really a way for the model to cheat like in BioMysteryBench, when all the information for the solution must be derived from the data itself.

BioMysteryBench Question “Human Unsolvable”

First off, on the term “Human Unsolvable”: the questions are designed by a human expert; they are all solvable questions with self-contained information. What they mean here is that another set of independent human experts could not get the right answer for these questions. I think the term “Human Unsolvable” is not technically correct, and is probably the result of some marketing strategy. They want people to think their models can be smarter than humans.

It allows for “superhuman” question generation. By sourcing problems derived from controllable properties of data, BioMysteryBench does not depend on humans being able to solve the problems. In particular, BioMysteryBench contains a handful of problems that—despite having objective, ground-truth solutions—humans found difficult or impossible to solve on their own.

I found the BioMysteryBench data easier to download. Here I worked through one of the unsolvable questions myself.

This microarray dataset contains expression data from various ovarian and pancreatic cancer biopsies ran on GPL 887 and GPL 4133 platforms, respectively. Based on the expression profile of each sample, assign a sex (male, female, or unknown) to each sample. The final output should be in the format of “Sample; sex” (eg Sample 1; male)

I picked this one because at first glance it’s trivially easy and a fairly relevant problem: identifying the sex of samples can often be helpful in determining sample mix-ups, which is an unfortunate but inevitable occurrence.

Identifying sex from any kind of genomic data is a common procedure. It relies on both biological knowledge (how sex differences translate at the genome level) and an understanding of the technology used. There are several parallel paths that one could take to verify sex; stronger defendable answers will involve using multiple lines of evidence.

For RNA-seq, or in this case microarrays, one approach could involve visualizing genes on the sex chromosomes, such as XIST ¹⁰, which will display sex-specific differential expression.

However, the problem is complicated by multiple tissues, multiple platforms, and cancerous signatures.

Even though gene expression sex differences are large, they might not be surfaced unless accounting for platform/batch and tissue effects. To be safe, we can start by identifying tissue.

The question gives a big hint: it implies one set of tissue samples was measured on one array (ovarian = GPL887), and the other tissues on another (pancreatic = GPL4133). We can see this clearly in the fact that samples 1-6 have non-missing values for a large number of probes that are not captured on the other array.

The other big piece of information in this question is that it tells you that one set of samples is ovarian. Ovaries are female reproductive organs. Meaning, if you identify which samples are ovarian, you know the sex of these samples. So, identifying tissue identity gives you half the answer already.

In conclusion, my general plan for solving this question was:

Identify platform → determines tissue → ovary samples = female, remaining pancreatic samples determine sex by gene expression.

Identify platform

So, first we can map which probe sets correspond to which array, and that will tell us which set of samples is ovarian and which are pancreatic (if the question is correct).

Download and compare the probesets for GPL887 and GPL4133 microarrays. Based on the systematic missing values by probeset for some samples of the Cancers.csv, determine the platform for each sample. Return a data frame that has sample id in one column, and platform (GPL887 vs GPL4133) in another column. Also print the number of probes in both platforms, and the intersection in both.

The microarray platform GPL887 has 17011 features. GPL4133 has 25547 features. The intersection between the two platforms is 15396 features.

platformOrder = ["GPL887", "GPL4133"]
sampleDomain = platformMembership.map(d => d.sample).filter((s, i, a) => a.indexOf(s) === i).sort((a, b) => a - b)

platformMembershipLong = platformMembership.flatMap(d => [
  { sample: d.sample, platform: d.platform, metric: "non-missing", value: d.pct_non_missing },
  { sample: d.sample, platform: d.platform, metric: "missing", value: 100 - d.pct_non_missing }
])

baseFont = 9
axisTitleFont = baseFont * 1.3
facetTitleFont = baseFont * 1.5

platformMembershipPlot = Plot.plot({
  width: 700,
  height: 480,
  marginLeft: 30,
  style: { background: "transparent", fontSize: `${baseFont}px` },
  fx: { domain: platformOrder, axis: null },
  y: { domain: sampleDomain, padding: 0, label: null },
  x: { domain: [0, 100], label: null },
  color: {
    domain: ["non-missing", "missing"],
    range: ["#000000", "#888888"],
    legend: true
  },
  marks: [
    Plot.axisFx({ fontSize: facetTitleFont, label: null }),
    Plot.axisY({ ticks: [1, 7, 10, 20, 30, 40, 49], tickSize: 0, fontSize: baseFont, label: null }),
    Plot.axisX({ tickSize: 0, fontSize: baseFont, label: null }),
    Plot.ruleX([0, 100], { stroke: "#999" }),
    Plot.barX(platformMembershipLong, {
      x: "value",
      y: "sample",
      fx: "platform",
      fill: "metric",
      insetTop: 0,
      insetBottom: 0,
      strokeWidth: 0,
      title: d => `${d.sample} · ${d.platform}\n${d.metric}: ${d.value.toFixed(1)}%`
    })
  ]
})

html`<div style="font-size:${axisTitleFont}px; margin-bottom:0.25rem;"><strong>y:</strong> Sample &nbsp; <strong>x:</strong> % of platform's probeset</div>${platformMembershipPlot}`

Based on probe coverage across the two microarray platforms:

• samples 1-6 are GPL4133 and are pancreatic
• remaining samples 7-49 are GPL887 and ovarian, and therefore female

Sexing pancreatic cancer samples

Now we can turn our effort towards determining the sex of the pancreatic cancer samples 1-6.

Here we can determine the sex of these samples based on the expression patterns of good old XIST, a transcript that is highly expressed in females and not expressed in males. To be extra sure, we can also include a Y-chromosome gene RPS4Y1 (expected to be expressed in males, not in females). To be extra extra sure, we can include a larger panel of X-chromosome genes (with PCA + clustering).

Subset to samples 1-6 (pancreatic cancer). Generate three graphs, 1. y-axis samples, x-axis XIST expression, barchart. 2. same as graph 1, but x-axis is RPS4Y1 expression. 3. PCA on samples using the top 100 variable XY probes. Each point is a sample.

subsetASampleDomain = subsetAMarkers.map(d => d.sample).sort((a, b) => a - b)

function subsetABarChart(gene) {
  return Plot.plot({
    width: 180,
    height: 220,
    style: { background: "transparent", fontSize: "11px" },
    x: { label: `${gene} expression`, grid: true,  tickSize: 0},
    y: { label: "Sample", domain: subsetASampleDomain, padding: 0.2,  tickSize: 0},
    marks: [
      Plot.barX(subsetAMarkers, {
        x: gene,
        y: "sample",
        fill: "#616161",
        title: d => `Sample ${d.sample}\n${gene}: ${d[gene].toFixed(2)}`
      }),
      Plot.ruleX([0])
    ]
  })
}

subsetAPcaScatter = Plot.plot({
  width: 180,
  height: 220,
  style: { background: "transparent", fontSize: "11px" },
  x: { label: "PC1", tickSize: 0},
  y: { label: "PC2", tickSize: 0},
  marks: [
    Plot.dot(subsetAXyPca, {
      x: "pc1",
      y: "pc2",
      r: 5,
      fill: "#616161",
      title: d => `Sample ${d.sample}\nPC1: ${d.pc1.toFixed(2)}, PC2: ${d.pc2.toFixed(2)}`
    }),
    Plot.ruleX([0]),
    Plot.ruleY([0])
  ]
})

html`<div style="display:flex; flex-wrap:wrap; justify-content:center; gap:1.5rem; align-items:flex-start;">
  <div><div style="font-weight:600; margin-bottom:0.25rem;">XIST expression</div>${subsetABarChart("XIST")}</div>
  <div><div style="font-weight:600; margin-bottom:0.25rem;">RPS4Y1 expression</div>${subsetABarChart("RPS4Y1")}</div>
  <div><div style="font-weight:600; margin-bottom:0.25rem;">PCA, top 100 variable XY probes</div>${subsetAPcaScatter}</div>
</div>`

Based on XIST and RPS4Y1 expression, samples 2-3 are likely female, and samples 1, 4, 5, 6 are male. However, the PCA on all XY genes does not separate samples 2/3 from the rest. Cancer is an extremely abnormal process that might be associated with gene expression changes obscuring the expression of sex-specific genes ¹¹.

Generate a plot that shows the a density function of Y chromosome transcript expression - 1 density line for each sample, and color samples 2 and 3 a different color from the rest.

highlightColor = ({ domain: ["Samples 2, 3", "1, 4-6"], range: ["#000000", "#bbbbbb"], legend: true })

subsetAYDensityPlot = Plot.plot({
  width: 500,
  height: 280,
  style: { background: "transparent", fontSize: "11px" },
  x: { label: "Y-chromosome transcript expression" },
  y: { label: "Density", grid: true, tickSize: 0},
  color: highlightColor,
  marks: [
    Plot.line(subsetAYDensity, {
      x: "x",
      y: "density",
      stroke: d => [2, 3].includes(d.sample) ? "Samples 2, 3" : "1, 4-6",
      z: "sample",
      title: d => `Sample ${d.sample}`
    }),
    Plot.ruleY([0])
  ]
})

html`<div style="display:flex; justify-content:center;"><div><div style="font-weight:600; margin-bottom:0.25rem;">Y-chromosome transcript density per sample</div>${subsetAYDensityPlot}</div></div>`

Y-chromosome transcripts are specific to males. The density plot shows that samples 2 and 3 have mostly background-level signal at Y-chromosome transcripts, indicating that these samples are likely female. They appear distinct from samples 1, 4-6, which have a “flatter” distribution, with more Y-chromosome transcripts having above-background levels of expression.

This analysis finalizes our answers. In the table below, I show every sample in this dataset, a column corresponding to each analysis, and final classification (“My Call”). I then added Anthropic’s answer, and another column indicating if I got the answer right for each sample (I didn’t, let’s see why).

Generate a heatmap that shows male/female based on my analysis (platform, tissue, marker analysis), annotate with anthropics answer. Calculate a final accuracy (% correct).

grayShades = ({
  platform: { GPL4133: "#dddddd", GPL887: "#f5f5f5" },
  tissue: { pancreatic: "#dddddd", ovarian: "#f5f5f5" }
})
sexColors = ({ male: "#dbe6f2", female: "#f3dde3", unknown: "#eeeeee" })
matchColors = ({ match: "#000000", "no match": "#ffffff" })

xistScale = d3.scaleSequential(d3.interpolateGreys).domain(d3.extent(subsetAFinalCalls, d => d.XIST))
rps4y1Scale = d3.scaleSequential(d3.interpolateGreys).domain(d3.extent(subsetAFinalCalls, d => d.RPS4Y1))

function textColorFor(bg) {
  return d3.lab(bg).l < 55 ? "white" : "black"
}

function finalCallsHeatmap() {
  const columns = [
    { key: "platform", label: "Platform", colors: grayShades.platform, group: "My analysis" },
    { key: "tissue", label: "Tissue", colors: grayShades.tissue, group: "My analysis" },
    { key: "XIST", label: "XIST", scale: xistScale, group: "My analysis" },
    { key: "RPS4Y1", label: "RPS4Y1", scale: rps4y1Scale, group: "My analysis" },
    { key: "my_call", label: "My call", colors: sexColors, group: "My analysis" },
    { key: "anthropic", label: "Anthropic", colors: sexColors, group: null },
    { key: "match", label: "Match", colors: matchColors, group: null }
  ]

  const table = document.createElement("table")
  table.style.cssText = "border-collapse: collapse; font-size: 9px;"

  const groupRow = document.createElement("tr")
  groupRow.appendChild(document.createElement("th"))
  let i = 0
  while (i < columns.length) {
    const group = columns[i].group
    let span = 1
    while (i + span < columns.length && columns[i + span].group === group && group != null) span++
    const th = document.createElement("th")
    th.textContent = group ?? ""
    th.colSpan = group ? span : 1
    // Border only under the actual grouped columns, so the group's real
    // span is visible instead of one line running under the whole table.
    const border = group ? "border-bottom:1px solid #999;" : ""
    th.style.cssText = `padding:1.8px 6px; font-weight:600; text-align:center; ${border}`
    groupRow.appendChild(th)
    i += group ? span : 1
  }
  table.appendChild(groupRow)

  const headerRow = document.createElement("tr")
  headerRow.appendChild(document.createElement("th"))
  for (const col of columns) {
    const th = document.createElement("th")
    th.textContent = col.label
    th.style.cssText = "padding:1.8px 6px; font-weight:600; text-align:center;"
    headerRow.appendChild(th)
  }
  table.appendChild(headerRow)

  for (const d of subsetAFinalCalls) {
    const tr = document.createElement("tr")
    const sampleTd = document.createElement("td")
    sampleTd.textContent = d.sample
    sampleTd.style.cssText = "padding:0.9px 6px; font-weight:600;"
    tr.appendChild(sampleTd)
    for (const col of columns) {
      const value = d[col.key]
      const td = document.createElement("td")
      let bg, text
      if (col.scale) {
        bg = value == null || Number.isNaN(value) ? "#f5f5f5" : col.scale(value)
        text = value == null || Number.isNaN(value) ? "–" : value.toFixed(1)
      } else {
        bg = col.colors[String(value)] ?? "#f5f5f5"
        text = String(value)
      }
      td.textContent = text
      td.title = `Sample ${d.sample} · ${col.label}: ${text}`
      td.style.cssText = `padding:0.9px 6px; text-align:center; background:${bg}; color:${textColorFor(bg)};`
      tr.appendChild(td)
    }
    table.appendChild(tr)
  }
  return table
}

function accuracyCard(d) {
  const box = document.createElement("div")
  box.style.cssText = "border:1px solid black; padding:0.5rem 0.75rem; font-family:monospace; font-size:13.8px; min-width:180px;"
  const label = document.createElement("div")
  label.style.cssText = "font-weight:600; margin-bottom:0.25rem;"
  label.textContent = d.label
  const value = document.createElement("div")
  value.textContent = `Accuracy = ${d.correct}/${d.n} (${d.pct}%)`
  box.append(label, value)
  return box
}

html`<div style="display:flex; flex-direction:column; align-items:center;">
${finalCallsHeatmap()}
<div style="display:flex; flex-wrap:wrap; justify-content:center; gap:0.75rem; margin-top:1rem;">
  ${subsetAAccuracyCards.map(accuracyCard)}
</div>
</div>`

Several wrong answers in BioMysteryBench

The overall accuracy of my classification was 46.9%, worse than random guessing. But in this section, let me convince you that I’m right: the answer itself is broken.

1. The answer contains several samples whose sex is “unknown”

This is problematic for two reasons:

• it suggests that the sex is unverifiable from the provided data. While confounding by tissue and technical artifacts may indeed obscure the sex-specific patterns that could be used to determine this answer, it’s highly arguable that sex can still be determined with a reasonable level of confidence, especially given that whole transcriptome profiles were provided.
• having an “unknown” category as part of the answer suggests that the task involves providing some level of “confidence” in identifying the label, and choosing an appropriate threshold to classify samples as having too low confidence. How do we, or the models, know what a level of “reasonable confidence” is? In practice these kinds of thresholds depend entirely on outside context (e.g. the impact of a false negative or false positive). Here no context is provided to inform the level of thresholding.

2. The answer itself states that ovarian samples were run on one array. Males don’t have ovaries, females do. Instead, the answer suggests that even after figuring out which samples are ovarian, some of these are actually male samples, or have unknown sex.

Obviously, the question-answer is a total dud. It should be retracted. I noticed that since publication, BioMysteryBench now has 90/99 of the original questions. If we look at the reasons cited for the removal of 9 questions, we can see a troubling pattern:

• hb011 Answer key wrong
• hb014 Answer key contradicted by every expert and model rerun
• hb040 Infeasible

It goes on. An additional update modified 24 questions to fix phrasing and format.

The number of questions that were in the unsolvable set that have been removed is 7, meaning only 10/17 remain. If we recompute (via napkin math) the accuracy excluding those flawed questions, the performance is likely closer to 50-80%. And I just found another question that is broken. How many more questions in BioMysteryBench are flawed?

It’s important to note that there is extensive quality control that Anthropic claims went into task creation. This means that a significant percentage of the final questions made it through their quality control process (>10% of questions are flawed, >25% needed modification). That is deeply concerning and calls into question the integrity of their benchmark and the results described.

Conclusions

It’s difficult to create good benchmarks. There’s no standard recipe, and every benchmark has a different approach. The ones described in this post had some commonalities in their goals and approaches: the teams created tasks with verifiable ground truth answers. The execution, though, had some big differences.

Genentech’s CompBioBench is high quality, and how they published it was highly transparent and reproducible. BioMysteryBench, unfortunately, feels clumsily put together. There are too many errors. The work feels sloppy.

Despite these differences, I think they converge on the overall message that LLMs are indeed highly capable at performing advanced bioinformatics tasks. Using LLMs, though, is a skill in itself. Look at the prompts in these questions, and it’s clear that only an experienced domain expert can generate such specific instructions, and also be in a position to review and check a model’s output. In this post itself, I worked through one of the harder problems using AI as a tool to perform individual analysis steps. I think this is the best way to use these tools, as tools, not as junior assistants. LLMs are a multiplier; they require a skilled hand to be used effectively.

I think that LLMs are not necessarily replacing us scientists. There’s too much contextual information that can make or break an analysis. And LLMs are too prone to sycophancy; they are too agreeable. But LLMs are certainly making bioinformatics more accessible. That’s generally a good thing, but there are also new challenges this brings. If LLMs let more people run these analyses without the years of training that used to be a prerequisite, fewer people will be in a position to catch it when the model is wrong. I feel simultaneously empowered by the capabilities LLMs have enabled, but also concerned about the future of science and bioinformatics.

References

[1]

S. Nair et al., “Agentic systems are adept at solving well-scoped, verifiable problems in computational biology.” bioRxiv preprint, 2026. doi: 10.64898/2026.04.06.716850.

[2]

Anthropic, “Evaluating claude for bioinformatics with BioMysteryBench.” 2026. Available: https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench

[3]

J. Gu et al., “A survey on LLM-as-a-judge.” arXiv preprint, 2024. doi: 10.48550/arXiv.2411.15594.

[4]

B. Van de Sande et al., “Applications of single-cell RNA sequencing in drug discovery and development,” Nature Reviews Drug Discovery, vol. 22, no. 6, pp. 496–520, 2023, doi: 10.1038/s41573-023-00688-4.

[5]

T.-Y. Chen, L. You, J. A. U. Hardillo, and M.-P. Chien, “Spatial transcriptomic technologies,” Cells, vol. 12, no. 16, p. 2042, 2023, doi: 10.3390/cells12162042.

[6]

A. Thibodeau et al., “AMULET: A novel read count-based method for effective multiplet detection from single nucleus ATAC-seq data,” Genome Biology, vol. 22, no. 1, p. 252, 2021, doi: 10.1186/s13059-021-02469-x.

[7]

A. Sadagopan, I. T. Nasim, J. Li, M. Achom, C.-Z. Zhang, and S. R. Viswanathan, “Somatic XIST activation and features of X chromosome inactivation in male human cancers,” Cell Systems, vol. 13, no. 11, pp. 932–944.e5, 2022, doi: 10.1016/j.cels.2022.10.002.

Footnotes

1. Genentech, a 50-year-old pharma company with dozens of approved drugs and hundreds of clinical trials, released CompBioBench [1] on April 4th, 2026. The reputation of the authors’ affiliations sets expectations for their dataset. This is a benchmark created by some of the most experienced practitioners in the industry.

2. BioMysteryBench [2] was created by Anthropic, possibly one of the most important companies in the world right now. They are known for producing the world’s leading LLM models. Their investments in life sciences research are highly intriguing. We will see what the payoff (if any) will be; we are still in early days.

3. One advantage of the LLM-as-a-judge approach is that due to model reasoning capabilities, LLMs can catch “cheating”. An example of cheating in bioinformatics might be if a model chooses to solve the task, not by using a bioinformatics approach, like classifying a transcriptomic profile against known gene signatures, but instead by finding the answer through some shortcut, like matching a sample’s identity to a published database or article [3].

4. And unlike a blog, which can be taken down or edited without any trace of original content, a preprint server is managed by a third party organization (bioRxiv is managed by Cold Spring Harbor), which includes keeping the original published article accessible and has a system for managing updates.

5. Genentech published a GitHub repo that hosts the code to fully reproduce their results for CompBioBench: Genentech/compbiobench-runner, which is an amazing exemplar of transparency and reproducibility. This type of practice is a big step in building trust in their results.

6. Some categories I relabeled to more generally capture the questions across both: Population Genetics → Human Genetics, Structure → Proteomics. Microbiology contains only BioMysteryBench questions; CompBioBench has none.

7. Currently, BMB is at 90 questions, with 9 questions being removed in a post-publication review.

8. CompBioBench compared 4 Claude models, OpenAI’s GPT 5.4, and Google’s Gemini 3.1 Pro. The results show that the top models from these three companies had similar performance (within a few percentage points). But GPT 5.4 and Gemini 3.1 Pro cost half as much real dollars per task as Anthropic’s models. Anthropic’s BioMysteryBench only includes their own models in their benchmark.

9. It surprises me that standard scATAC-seq workflows ignore homotypic doublets because in other assay types like scRNA-seq, doublet detection is standard. (Why is this not already part of the standard workflow?)

10. a phenomenon where females downregulate X-chromosome-derived gene products, as a result of having one more X chromosome than males.

11. In fact, XIST is expressed in 4% of male cancer samples. See figure 1G, pancreatic cancer expression of XIST in males vs females [7].