Competition Results

Important note about the results

The people or teams mentioned here deserve recognition for their ideas, skills and hard work, which together allowed them to produce the best results of the competition.

The competiton was designed in order to promote research into automated methods for timetabling. It was not designed as a comparison of methods. We must be cautious about drawing strict scientific conclusions from these results, that for example one algorithm is better than another. This is because one competitior might simply be a better programmer, or have spent more time on the problem than another competitor.

If we want to make comparisons between methods then the best way to do this is for the authors to get together, share their skills and programming tricks, and collaborate in conducting experiments.

We must also remember that in the real world there are many ways to decide on the best method to solve a problem. We had to decide upon a criterion for the competition in order to choose a winner, but it could have been many things. For example, computer time might not be an issue in some circumstances, in others it might be a big issue and only one run be allowed. In another situation it might be the reliability of the algorithm that is most important, or the ease with which it can be implemented or combined with other methods.

We should also bear in mind that not all timetabling problems will have the same characteristics as the instances here, and that problem instances with different charcteristics may require different types of algorithm.

Verification

We contacted the competitors who submitted the best results, and asked them to provide information that would help us to verify their results. We ran their programs on our machines, making a single run, using the random seeds provided where relevant and available. We also ran their programs on three new problem instances, in order to demonstrate that the methods were not over fitted to the competition instances. The results we obtained on a single run are listed here (although few scientific conclusions can be drawn from these).

The Top Four

The results of the top four contestants were replicated using our own machines. The first table shows these top four results:

 
Name(s)
Method
Competition Instance Scores
Unseen Instance Scores
1st
Philipp Kostuch
45
25
65
115
102
13
44
29
17
61
44
107
78
52
24
22
86
31
44
7
100
6
72
2nd

Brigitte Jaumard, Jean-François Cordeau and Rodrigo Morales
61
39
77
160
161
42
52
54
50
72
53
110
109
93
62
34
114
38
128
26
86
8
105
3rd
Yuri Bykov
85
42
84
119
77
6
12
32
184
90
73
79
91
36
27
300
79
39
86
0
329
3
84
4th

Luca Di Gaspero and Andrea Schaerf
63
46
96
166
203
92
118
66
51
81
65
119
160
197
114
38
212
40
185
17
97
9
103

Verified Results

For the next group of competitors we were able to independently verfiy that they all had good results, but these were not always so good as they had acheived themselves. In some cases this was because the original seeds were never recorded or have since been lost. In others the difference is probably due to inaccuracies in the benchmark

Name(s)
Method
Competition Instance Scores
Unseen Instance Scores
Halvard Arntzen and Arne Løkketangen
 Claimed Results 132 69 137 228 225 133 161 131 126 148 109 290 223 204 111 90 130 87 206 53 - - -
Verified Results 132 92 170 265 257 133 177 134 139 148 135 290 251 230 140 114 186 87 256 94 145 23 94

Alexandre Dubourg, Benoît Laurent, Emmanuel Long and Benoît Salotti

 Claimed Results 131 97 145 318 281 199 194 128 109 141 132 172 167 233 122 110 176 112 309 139 - - -
Verified Results 148 101 162 350 412 246 228 125 126 147 144 182 192 316 209 121 327 98 325 185 132 51 159
Gustavo Toro, Victor Parada Claimed Results 137 103 156 346 335 243 195 149 147 153 152 219 224 263 174 127 267 102 299 145 - - -
Verified Results 178 103 156 399 336 246 225 210 154 153 169 219 248 267 235 132 313 107 309 185 161 22 173

Roberto Montemanni

 Claimed Results 162 131 205 348 245 138 185 180 145 203 182 242 251 256 137 131 266 114 281 106 - - -
Verfied Results 211 128 213 408 312 169 281 214 164 222 196 282 315 345 185 185 409 153 334 149  204 7 245 

Tomáš Müller

 Claimed Results 172 135 173 380 230 130 53 156 137 216 191 241 349 169 92 154 191 121 428 188 - - -
Verfied Results 257 112 266 441 299 209 99 194 175 308 273 242 364 156 95 171 148 117 414 113  286  31  293

 

Unverifed Results

This group are those people who have not yet sent us the information necessary to verify their results. Only their claimed results are shown.

Name(s)
Method
Competition Instance Scores
David Bredstrom
 114
89 140 213 251 81 38 75 72 130 116 130 155 130 79 88 198 94 187 72

Parham Mousavi

 144 118 157 388 443 322 187 124 119 162 163 220 231 328 236 108 378 121 240 210

 

Other Results

This group contains the excellent results obtained by two teams from the Metaheuristics project. The teams worked to the same rules as the other competitors, but were not officially allowed to enter the competition.

Name(s)
Method
Competition Instance Scores
Unseen Instance Scores

Marco Chiarandini, Mauro Birattari, Olivia Rossi-Doria and Krzysztof Socha
57
31
61
112
86
3
5
4
16
54
38
100
71
25
14
11
69
24
40
0
53
21
75

Krzysztof Socha
65
36
69
138
143
24
24
28
36
75
50
95
79
73
31
23
108
26
108
5
-
-
-