Optimization software
ADMB, R, BUGS, etc.
- NCEAS benchmark (Bolker et al. 2013): ADMB vs. R. vs. BUGS
Obsolete benchmark (1998/2006)
ADMB vs. Gauss vs. Matlab vs. R/S-Plus Summary written by Arni Magnusson, 16 June 2010
Schnute et al. (1998) published a benchmark comparing the performance of different optimization software. They tested ADMB, Gauss, Matlab, and S-Plus with a catch-at-age model by Schnute and Richards (1995), using datasets simulated with S-Plus.
The simulated datasets included T years of data and A age classes, resulting in m=T+A+5 estimated parameters. The computer had a 133 MHz processor and 48 MB RAM, running Windows NT 4.0.
Benchmark trials for two cases of a catch-at-age model, in which T years of data are available for A age classes and the total number of parameters is m=T+A+5.
Case T A m Product ms/call Calls Time Scale 1 20 12 37 ADMB 29 161 4.6 s 1.0
GAUSS 42 4,041 2.8 min 1.0
MATLAB 178 1,936 5.8 min 1.0
S-PLUS 1429 n/a n/a
2 80 15 100 ADMB 131 291 38 s 8.3
GAUSS 167 23,365 1.08 hr 23.1
MATLAB 639 18,360 3.25 hr 33.6
S-PLUS n/a n/a n/a
S-Plus exhausted the computer memory before finishing the calculations, and in smaller examples where evaluations were obtained, S-Plus was much slower than the other products.
ADMB performed much faster and required fewer function calls than its competitors in this benchmark. The reduced number of function calls stems directly from automatic differentiation. One call in ADMB evaluates both the function and its gradient. The other products approximate gradients by making a small change to each estimated parameter, requiring m+1 function calls to achieve the same result. Furthermore, ADMB obtains gradients accurate to machine precision, instead of approximations.
Case 2 scales the problem by a factor of 100/37=2.7, but the computation time increases by a greater factor. ADMB appears more scalable than the other products, probably from the efficiencies of reverse automatic differentiation.
Schnute, J.T. and L.J. Richards. 1995. The influence of error on population estimates from catch-age models. Canadian Journal of Fisheries and Aquatic Sciences 52:2063-2077.
Schnute, J.T., L.J. Richards, and N. Olsen. 1998. In: F. Funk et al. (eds.) Fishery stock assessment models. Fairbanks: Sea Grant Program, pp. 171-184.
Update
In Nov 2006, Dave Fournier ported the S-Plus model to R and benchmarked it against ADMB on newer hardware. With T=80 years and A=15 ages, the updated benchmark corresponds to Case 2 in the original benchmark:
Product ms/call Calls Time Function value ADMB 10.79 278 3 s 3718.693119 R 33.17 56,074 31 min 3717.469 R 33.81 165,000 93 min 3718.693119
In the first run, R was manually interrupted after 31 minutes, to check the function value and parameter estimates. The function value was still far from the optimal value, and one parameter, gamma, was at 0.61, far from the optimal value of 0.42.
In the other run, R was allowed to converge. This took 93 minutes, more than 1000 times longer than ADMB.