Evolution of Colibri2
The releases of Colibri2 on the benchmarks that guide its development, run
on the same cluster with 60 seconds and 4 GB per goal:
- Why3 goals: 48485 proof obligations of Why3
programs (the Why3 gallery of examples, multi-precision arithmetic,
sequences...), translated by the Why3 driver of Colibri2. Why3 is the main
target of Colibri2. Without the 711 goals that use the conversions between
bit-vectors and integers of the current driver, which 0.4 and 0.5 do not
know.
- SMT-LIB files: 3125 problems in SMT-LIB format:
linear real arithmetic (QF_LRA, 1386 problems), difference logic with
uninterpreted functions (QF_UFIDL, 628), arrays (QF_AX, 551) and 557
verification problems on sequences. Floating-point and bit-vector problems
are not in this set.
| Release | Date |
Why3 goals proved | |
SMT-LIB solved | |
Time on the 40089 Why3 goals all prove |
Wrong answers |
| 0.4 |
2023-02-06 |
41013 |
|
887 |
|
53505 s |
4 |
| 0.5 |
2026-02-25 |
45070 |
|
1881 |
|
13805 s |
0 |
| 0.6 |
2026-06-22 |
45140 |
|
1876 |
|
14712 s |
0 |
| 0.7 |
2026-11-16 |
45912 |
|
1959 |
|
14261 s |
0 |
Wrong answers: sat on a goal known to be valid, not counted as
solved. The four of 0.4 are on bit-vectors.
Floating-point arithmetic
The floating-point problems of SMT-LIB, with the same limits:
- Without quantifiers: 57842 problems of the logics
QF_FP, QF_BVFP, QF_FPLRA, QF_BVFPLRA, QF_ABVFPLRA... Most of them come from
the tests of the semantics of the operations by C. Wintersteiger (39994
problems) and from the symbolic execution of C programs by KLEE (17156).
- With quantifiers: 872 problems of the logics
FP, BVFP, BVFPLRA, UFFPDTNIRA... (SPARK programs, Ultimate Automizer,
Alive2). The 3469 other problems of
these logics have no known status and are not counted.
| Release | Date |
Solved without quantifiers | |
Solved with quantifiers | |
Time on the 57526 problems without quantifiers all solve |
Wrong answers |
| 0.5 |
2026-02-25 |
57565 |
|
723 |
|
2988 s |
0 |
| 0.6 |
2026-06-22 |
57564 |
|
722 |
|
3789 s |
0 |
| 0.7 |
2026-11-16 |
57624 |
|
739 |
|
5862 s |
0 |
Wrong answers: an answer that contradicts the known status of the problem.
0.4 is not in this table: it was far behind on the floating-point problems.
The benchmarks and the scripts:
colibri-regression.