Skip to Main content Skip to Navigation
Conference papers

Metrics: Towards a Unified Library for Experimenting Solvers

Abstract : When developing a SAT solver, one of the most important parts is to\nperform experiments so as to evaluate its performance.\nMost of the time, this process remains the same, so that everybody collects\nalmost the same statistics about the solver execution.\nHowever, how many scripts are there to retrieve experimental data and draw\nscatter or cactus plots?\nProbably as many as researchers in the domain.\nBased on this observation, this paper introduces Metrics, a Python library,\naiming to unify and make easier the analysis of solver experiments.\nThe ambition of Metrics is to provide a complete toolchain from the≠xecution of the solver to the analysis of its performance.\nIn particular, this library simplifies the retrieval of experimental data\nfrom many different inputs (including the solver's output), and provides\na nice interface for drawing commonly used plots, computing statistics\nabout the execution of the solver, and effortlessly organizing them (e.g., ∋n Jupyter notebooks).\nIn the end, the main purpose of Metrics is to favor the sharing and \nreproducibility of experimental results and their analysis.
Document type :
Conference papers
Complete list of metadata
Contributor : Fabien Delorme <>
Submitted on : Tuesday, July 27, 2021 - 1:33:46 PM
Last modification on : Thursday, September 9, 2021 - 3:10:20 PM


  • HAL Id : hal-03301303, version 1



Thibault Falque, Romain Wallon, Hugues Wattez. Metrics: Towards a Unified Library for Experimenting Solvers. 11th International Workshop on Pragmatics of SAT (POS'20), 2020, Alghero, Italy. ⟨hal-03301303⟩



Record views