Towards better estimation of statistical significance when comparing evolutionary algorithms

Maxim Buzdalov

doi:10.1145/3319619.3326899

Towards better estimation of statistical significance when comparing evolutionary algorithms

Maxim Buzdalov

ITMO University

Research output: Chapter in Book/Report/Conference proceeding › Conference Proceeding (Non-Journal item)

1 Citation (Scopus)

Abstract

The use of well-established statistical testing procedures to compare the performance of evolutionary algorithms often yields pessimistic results. This requires increasing the number of independent samples, and thus the computation time, in order to get results with the necessary precision.

We aim at improving this situation by developing statistical tests that are good in answering typical questions coming from benchmarking of evolutionary algorithms. Our first step, presented in this paper, is a procedure that determines whether the performance distributions of two given algorithms are identical for each of the benchmarks. Our experimental study shows that this procedure is able to spot very small differences in the performance of algorithms while requiring computational budgets which are by an order of magnitude smaller (e.g. 15x) compared to the existing approaches.

Original language	English
Title of host publication	GECCO '19
Subtitle of host publication	Proceedings of the Genetic and Evolutionary Computation Conference Companion
Editors	Manuel López-Ibáñez
Publisher	Association for Computing Machinery
Pages	1782-1788
Number of pages	7
ISBN (Print)	978-1-4503-6748-6
DOIs	https://doi.org/10.1145/3319619.3326899
Publication status	Published - 13 Jul 2019
Externally published	Yes
Event	GECCO 2019: The Genetic and Evolutionary Computation Conference - Prague, Czech Republic Duration: 13 Jul 2019 → 17 Jul 2019 https://gecco-2019.sigevo.org

Conference

Conference	GECCO 2019: The Genetic and Evolutionary Computation Conference
Country/Territory	Czech Republic
City	Prague
Period	13 Jul 2019 → 17 Jul 2019
Internet address	https://gecco-2019.sigevo.org

Keywords

multiple comparisons
statistical significance

Access to Document

10.1145/3319619.3326899

Permanent link

Persistent link

Cite this

@inproceedings{0dda476f5d9444e0bea2126925316684,

title = "Towards better estimation of statistical significance when comparing evolutionary algorithms",

abstract = "The use of well-established statistical testing procedures to compare the performance of evolutionary algorithms often yields pessimistic results. This requires increasing the number of independent samples, and thus the computation time, in order to get results with the necessary precision.We aim at improving this situation by developing statistical tests that are good in answering typical questions coming from benchmarking of evolutionary algorithms. Our first step, presented in this paper, is a procedure that determines whether the performance distributions of two given algorithms are identical for each of the benchmarks. Our experimental study shows that this procedure is able to spot very small differences in the performance of algorithms while requiring computational budgets which are by an order of magnitude smaller (e.g. 15x) compared to the existing approaches.",

keywords = "multiple comparisons, statistical significance",

author = "Maxim Buzdalov",

year = "2019",

month = jul,

day = "13",

doi = "10.1145/3319619.3326899",

language = "English",

isbn = "978-1-4503-6748-6",

pages = "1782--1788",

editor = "L{\'o}pez-Ib{\'a}{\~n}ez, {Manuel }",

booktitle = "GECCO '19",

publisher = "Association for Computing Machinery",

address = "United States of America",

note = "GECCO 2019: The Genetic and Evolutionary Computation Conference ; Conference date: 13-07-2019 Through 17-07-2019",

url = "https://gecco-2019.sigevo.org",

}

Buzdalov, M 2019, Towards better estimation of statistical significance when comparing evolutionary algorithms. in M López-Ibáñez (ed.), GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference Companion. Association for Computing Machinery, pp. 1782-1788, GECCO 2019: The Genetic and Evolutionary Computation Conference, Prague, Czech Republic, 13 Jul 2019. https://doi.org/10.1145/3319619.3326899

Towards better estimation of statistical significance when comparing evolutionary algorithms. / Buzdalov, Maxim.
GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference Companion. ed. / Manuel López-Ibáñez. Association for Computing Machinery, 2019. p. 1782-1788.

Research output: Chapter in Book/Report/Conference proceeding › Conference Proceeding (Non-Journal item)

TY - GEN

T1 - Towards better estimation of statistical significance when comparing evolutionary algorithms

AU - Buzdalov, Maxim

PY - 2019/7/13

Y1 - 2019/7/13

N2 - The use of well-established statistical testing procedures to compare the performance of evolutionary algorithms often yields pessimistic results. This requires increasing the number of independent samples, and thus the computation time, in order to get results with the necessary precision.We aim at improving this situation by developing statistical tests that are good in answering typical questions coming from benchmarking of evolutionary algorithms. Our first step, presented in this paper, is a procedure that determines whether the performance distributions of two given algorithms are identical for each of the benchmarks. Our experimental study shows that this procedure is able to spot very small differences in the performance of algorithms while requiring computational budgets which are by an order of magnitude smaller (e.g. 15x) compared to the existing approaches.

AB - The use of well-established statistical testing procedures to compare the performance of evolutionary algorithms often yields pessimistic results. This requires increasing the number of independent samples, and thus the computation time, in order to get results with the necessary precision.We aim at improving this situation by developing statistical tests that are good in answering typical questions coming from benchmarking of evolutionary algorithms. Our first step, presented in this paper, is a procedure that determines whether the performance distributions of two given algorithms are identical for each of the benchmarks. Our experimental study shows that this procedure is able to spot very small differences in the performance of algorithms while requiring computational budgets which are by an order of magnitude smaller (e.g. 15x) compared to the existing approaches.

KW - multiple comparisons

KW - statistical significance

UR - https://www.scopus.com/record/display.uri?eid=2-s2.0-85070657847&origin=resultslist

U2 - 10.1145/3319619.3326899

DO - 10.1145/3319619.3326899

M3 - Conference Proceeding (Non-Journal item)

SN - 978-1-4503-6748-6

SP - 1782

EP - 1788

BT - GECCO '19

A2 - López-Ibáñez, Manuel

PB - Association for Computing Machinery

T2 - GECCO 2019: The Genetic and Evolutionary Computation Conference

Y2 - 13 July 2019 through 17 July 2019

ER -

Towards better estimation of statistical significance when comparing evolutionary algorithms

Abstract

Conference

Keywords

Access to Document

Permanent link

Other files and links

Fingerprint

Cite this