Simbench 2 Dashboard

Given the number of simulation tools designed for single-cell RNA-seq data, it is critical to have a benchmark system that compares their relative performance under same set of data and same set of criteria SimBench together with this website is developed for the exact purpose. We invite all interested researchers to run their methods on the set of data we have curated, upload their result to this website and participate in this public leaderboard of scRNA-seq simulation tools

Overview of SimBench

Collection of 35 datasets across a range of protocols and tissue types
18 published methods
Evaluated 16 different measures corresponding to four key aspects
35 x 18 x 16 = 10080 measures

The datasets are the collection of 35 datasets across a range of protocols and tissue types

There are two major groups of protocols UMI (deep) and read-based (shallow), including:

10x
CEL-seq2
Drop-seq
MARS-seq
SCRB-seq
Smart-seq2

Used two collections of data, including both UMI and read-based protocols

PBMC collection

10x
Celseq
Dropseq
inDrops
SCRB-seq
Seqwell

Cortex collection

10x
DroNc-seq
Sci-RNA-seq
Smart-seq

Protocol
Tissues
Data collection

Simulation datasets are commonly used for methods development and methods benchmark, eg, batch correction, normalization

Our evaluation:

Kernel Density Based Global Two-Sample Comparison Test (KDE test)

where f1 and f2 is the kernel density estimate

Advantage of KDE test:

Result can be summarized quantitatively
Can be used for both univariate and multivariate distribution
Non – parametric test via kernel density estimation

Not all parameters are equally easy to estimate

Scaled all the kde values
applied median over all datasets for each method
The boxplot shows criteria with larger range of kde in methods has lower values

Collection of Data

Protocol
Tissues
Collection of Data

Evaluation - Methods scores

Score Plot
Score Heatmap
Evaluation - Methods scores

Boxplot Exploration
Heatmap Exploration

Dataset Selection

Filter datasets based on

Select number of cells in the data:

Select number of cell types in the data:

Select the datasets to rank the methods

Method Selection

Select Method

Criteria selection

SelectCriteria

Benchmarking Selection

Filter datasets based on

Select number of cells in the data:

Select number of cell types in the data:

Select the datasets to rank the methods

Benchmarking Table and Plot

Instructions

Please submit two input files to upload your results, the files are result file and method files.

The format for result file is in csv with the following columns: method, sum_kde_zstat, parameter, dataset, criteria.

The format for method file is in csv with the following columns: method, est.multi.groups, sim.multi.groups, sim.custom.DE.

Once your files are uploaded successfully, you will be able to see the summary result in benchmarking tab

Input Files

Uploaded File

Citation

Please follow the link to cite this paper

Data and instructions

Download the set of curated data at link . Details about the data can be found in the package and can also be browsed under the shiny tab "More"

Run your own method
Evaluate the simulated data against the real data across the set of criteria we have curated at link.
Upload the result to the panel Submit your data.