Diese Seite ist aus Gründen der Barrierefreiheit optimiert für aktuelle Browser. Sollten Sie einen älteren Browser verwenden, kann es zu Einschränkungen der Darstellung und Benutzbarkeit der Website kommen!
zur Startseite Institut für
					  Statistik

Benchmarking

In statistical learning benchmarking is the methodology of comparing learners or algorithms with respect to a certain performance measure. The benchmarking process abstractly consists of three levels: Setup, Execution and Analysis. In each level different statistical and computational aspects play a role.

This website gives some insights about the computational implementation -- a comprehensive list of all our work on this topic can be found at the project site of the working group.

R package

R-Forge repository: http://r-forge.r-project.org/projects/benchmark/

useR! 2008 source code release

benchmark_0.01.tar.gz

(Psycho-)Analysis of Benchmark Experiments

Supplemental resources: Artificial toy example, Application example (data, precalculated tree).

Use current development version of the benchmark package.

Interactive bench plot

The interactive bench plot is a prototype of a bench plot version with interactive elements. The implementation is based on the R packages iplots and icp; we have patched the CRAN versions of iplots and its related packages: iplots, icp, rJava.

beplot.icp.R is a first version; it is a proof-of-concept and the idea is to release a stable version within the first CRAN version of the benchmark package.

The uci621 benchmark experiment

The uci621 is a benchmark experiment we used in our article Exploratory and Inferential analysis of benchmark experiments to illustrate our methods. Here we show how to reproduce the results for each section using the benchmark package version 0.01. The raw data and the corresponding Rnw files are available on demand.

Setup and Execution:
The uci621 benchmark experiment
Section 3.1:
Exploratory analysis of one data set and one performance measure
Section 3.2:
Inferential analysis of one data set and one performance measure
Section 3.3:
Overall analysis of one data set and different performance measures
Section 4.1:
Exploratory analysis of one performance measure and more than one data set
Section 4.2:
Consensus of one performance measure and more than one data set
Manuel J. A. Eugster, 2007-2009. Website created with TT2