Oxford Nanopore direct RNA sequencing can detect multiple RNA modifications on individual molecules, providing a view of RNA biology that is difficult to obtain with other sequencing technologies. Here we present updated benchmarking and validation results from a synthetic ground-truth dataset, designed to measure modified-base identification at single-read, single-base resolution.

The release includes unmodified controls alongside 6-methyladenosine (m6A), 5-methylcytosine (m5C), pseudouridine (pseU), inosine, and 2’-O-methylation (2’Ome). As the modified state of every test site is known, these synthetic constructs allow us to measure model performance directly rather than infer it from biological samples.

This dataset builds on our earlier RNA validation release, but extends the design to cover every 5-mer sequence context, the five-base window centred on each test site. The 32 synthetic strands are each 100 nucleotides long and contain eight modified positions, separated by eight bases. Together, the resulting 256 target positions provide a more even and demanding test of the sequence diversity that models encounter in practical RNA analyses.

Release details

Raw nanopore data, references, annotations, and basecalls are available from our open data archive. Download the release with:

aws s3 sync --no-sign-request s3://ont-open-data/rna-mod-validation-all5mer-2026.07 rna-mod-validation-all5mer-2026.07

Each control and modified sample is supplied as both a full and a subset POD5 file, the container format for raw nanopore signal data, with matching basecalls and reference annotations. The full set contains the first 20 million reads from each sequencing run; the balanced subset is used for the downstream validation comparisons. The archive also includes commands.sh, a concise recipe for regenerating the results shown here, so that the raw signal can be inspected or the analysis reproduced with the supplied references and basecalls. A full manifest is available in the release archive, although the per-sample naming convention is consistent across controls and modified bases.

For more information and help downloading data from our open dataset archive, see the Datasets Tutorials page.

Methods note

Validation summaries were generated by aligning Dorado basecalls with the Dorado aligner using the short-read (sr) preset, then running modkit validate with its default filtering. Modified-base model version numbers track the canonical basecaller release with which they are supplied, so they should be interpreted as paired versions. In general, we recommend using the latest canonical basecaller with its linked modified-base models.

Tools and models

Dorado v2.0.1 for basecalling and alignment.
- The models used here are the v6.0.0 canonical basecaller model together with their associated v1 modified base models. See the Dorado model list for available downloads via the dorado download command.
modkit v0.6.4 for the validation and modified base filtering steps.

v6 SUP model validation: overview

We begin with the v6 Super accuracy (SUP) all-context modified-base models, our recommended RNA model family when the highest accuracy is required. Results are reported per read and per site, so each call represents one molecule observed at a known reference position. We use the standard modkit validate procedure to remove the lowest-confidence 10% of calls before reporting the results.

The confusion matrices below retain each of these individual calls rather than collapsing results by site or aggregating neighbouring bases within a read. Their diagonals therefore show correct calls, while the off-diagonal cells reveal the classes with which a base is most often confused. Across the canonical-base cohorts, the v6 SUP model maintains high accuracy and a low rate of false modified-base calls, despite the deliberately broad sequence-context panel. These results provide an overview of performance across the complete all-context panel. The detailed benchmarks that follow isolate individual modified bases and compare model generations and model types. Figure 1 shows the confusion matrices for the four canonical-base cohorts.

Figure 1. Confusion matrices for canonical A, C, G, and U calls from the v6 SUP all-context model.

Detailed binary benchmarks

The overview above considers all output classes together. The binary benchmarks below focus separately on m6A, inosine, m5C, and pseudouridine, providing a direct comparison between model generations, even as newer releases add targets such as 2’Ome. Before validation, we used modkit modbam adjust-mods to remove non-target modified-base outputs and keep each comparison specific to its intended target.

The all-5-mer design expands the validation space to include more of the contexts encountered in practice. The following accuracy and false-positive-rate views provide complementary measures of performance across this broader panel.

Accuracy across modified bases

Accuracy is the proportion of calls that match the known truth in the balanced validation set. We report filtered accuracy after removing the lowest-confidence calls with modkit validate. Figure 2 and the accompanying table compare the v6 and v5.1 models for each modified base. Figure 2 shows that the v6 models improve accuracy across the complete comparison set. Inosine and pseU are the most accurately identified targets, while m5C remains the most challenging of the four.

Figure 2. Filtered per-read, per-site accuracy for individual RNA modified-base comparisons.

False positive rate across modified bases

The false positive rate (FPR) is the proportion of unmodified calls that are incorrectly labelled as modified. It therefore complements accuracy by showing how often a model produces a spurious modified-base call. Figure 3 and the accompanying table compare the FPR for the v6 and v5.1 models across the four modified bases. Figure 3 shows that the v6 models reduce the FPR relative to v5.1 for every target in this comparison. Inosine has the lowest FPR, whereas m5C has the highest.

Figure 3. Filtered false positive rate for individual RNA modified-base comparisons.

Mapping performance

Mapping performance determines the number of reads available for downstream site-level validation. We therefore compared the proportion of RNA reads mapped successfully by each model. The full read set was used for this comparison to avoid any influence from model-selected reads in the subset.

Figure 4 compares mapping rates across the four basecallers. It shows that the v6 models map more reads than their v5.1 counterparts, with the largest gain for inosine; across the modified bases, SUP also maps more reads than HAC.

Figure 4. Mapping rates across RNA basecallers.

Higher mapping rates carry more validation sites forward into the downstream comparisons. To ensure that the four basecallers are assessed on the same evidence, we then restrict the analysis to reads and sites shared by all of them. Figure 5 reports filtered accuracy on this shared subset. It shows that the v6 models and the larger SUP models achieve higher accuracy with lower false positive rates.

Figure 5. Filtered accuracy for the reads and sites shared by all four basecallers.

Taken together, these results show that the newer models improve both mapping yield and accuracy on matched reads, while the all-context design ensures that those gains are tested against more difficult examples.

DRACH context validation

DRACH motifs are biologically important and are a recurring focus in RNA modified-base model training datasets, where D represents A, G, or U; R represents A or G; and H represents A, C, or U. Restricting the m6A benchmark to DRACH positions therefore provides a complementary view of performance in a context that matters both biologically and operationally. In this comparison, the v6 SUP and HAC models both perform extremely well, with SUP retaining a small advantage. Figure 6a shows filtered accuracy in the DRACH context, while Figure 6b shows the corresponding FPR.

Figure 6. a) Filtered accuracy for m6A calls in the DRACH context. b) Filtered false positive rate for m6A calls in the DRACH context.

Summary

This release provides an all-5-mer synthetic ground-truth dataset for benchmarking RNA modified-base models across a broad range of sequence contexts. The v6 models improve accuracy and reduce false positive rates relative to v5.1 across the detailed binary comparisons. They also map more reads, increasing the evidence available for downstream site-level validation. Together, these results support the use of the current v6 models for accurate RNA modified-base analysis.