NNbenchmark

NNbenchmark was created during the Google Summer of Code program of 2019 and 2020 as a part of open-source organization - The R Project for Statistical Computing. To goal was to verify the convergence of the training algorithms provided in all Neural Network R packages available on CRAN to date. Neural networks must be trained with second order algorithms and not with first order algorithms as many packages seem to do.

The purpose of this project is to verify the quality of the training algorithms in R packages that provide neural network of perceptron type (one input layer, one normalized layer, one hidden layer with non-linear activation function usually tanh(), one normalized layer, one output output layer) for regression purpose i.e. \(NN(X1, ..., Xn) = E[Y]\).

Packages Tested

---

This project has conducted a comprehensive survey of all packages that have the “neural network” keyword in the package title or in the package description.

AMORE	ANN2	appnn	automl	brnn	CaDENCE
DALEX2	DamiaNN	deepnet	elmNNrcpp	ELMR	EnsembleBase
h2o	keras	MachineShop	minpack.lm	monmlp	neuralnet
nlsr	nnet	qrnn	radiant.model	rminer	RSNNS
snnR	TraineR	validann

Evaluation Criteria

We test the neural-network packages on 13 datasets based on the following criteria.

1. Accuracy: the ability to find the global minima, measured by the Root Mean Square Error (RMSE) in a fixed number of iterations.
2. Speed of the training algorithms.
3. Availability of helpful utilities.s
4. Quality of the documentation.

Following are the datasets

Dataset	Rows	Input	Neurons	Parameters
Multivariate
mDette	500	3	5	26
mFriedman	500	5	5	36
mIshigami	500	3	10	51
mRef153	153	5	3	22
Univariate
uDmod1	51	1	6	19
uDmod2	51	1	5	16
uDreyfus1	51	1	3	10
uDreyfus2	51	1	3	10
uGauss1	250	1	5	16
uGauss2	250	1	4	13
uGauss3	250	1	4	13
uNeuroOne	51	1	2	7

Results

	Individual Rating				Global Score
Package	Util	Doc	Call	Algorithm	Time	RMSE
nlsr	*	****	**	NashLM	18	1
rminer	**	***	**	nnet_optim(BFGS)	12	2
nnet	*	***	**	optim (BFGS)	3	3
validann	*	****	**	optim(BFGS)	35	4
	*	****	**	optim(CG)	60	8
	*	****	**	optim(L-BFGS-B)	36	15
	*	****	**	optim(Nelder-Mead)	55	45
	*	****	**	optim(SANN)	20	55
MachineShop	*	***	*	nnet_optim(BFGS)	6	5
traineR	*	**	**	nnet_optim(BFGS)	4	6
radiant.model	**	**	**	nnet_optim(BFGS)	10	7
monmlp	**	***	**	optimx(BFGS)	26	9
	**	***	**	optimx(Nelder-Mead)	32	47
CaDENCE	**	***	**	optim(BFGS)	46	10
	**	***	**	Rprop	56	51
	**	***	**	pso_psoptim	54	54
h2o	**	**		first-order	51	11
EnsembleBase	*	*	**	nnet_optim(BFGS)	5	12
caret	**	***	**	avNNet_nnet_optim(BFGS)	17	13
brnn	**	****	**	Gauss-Newton	8	14
qrnn	**	***	**	nlm()	28	16
RSNNS	**	***	**	Rprop	24	17
	**	***	**	SCG	30	18
	**	***	**	Std_Backpropagation	22	27
	**	***	**	BackpropChunk	26	29
	**	***	**	BackpropMomentum	25	30
	**	***	**	BackpropWeightDecay	29	31
	**	***	**	BackpropBatch	43	49
	**	***	**	Quickprop	45	57
automl	*	***	**	trainwgrad_adam	50	18
	*	***	**	trainwgrad_RMSprop	47	26
	*	***	**	trainwpso	57	43
deepnet	*	***	**	BP	23	18
neuralnet	*	***	**	rprop+	19	21
	*	***	**	rprop-	21	22
	*	***	**	slr	31	31
	*	***	**	sag	41	38
	*	***	**	backprop	37	50
keras	**	*		adamax	48	23
	**	*		adam	42	34
	**	*		nadam	44	36
	**	*		adagrad	58	37
	**	*		adadelta	59	40
	**	*		sgd	48	44
	**	*		rmsprop	37	52
AMORE	*	***	*	ADAPTgdwm	16	24
	*	***	*	ADAPTgd	9	35
	*	***	*	BATCHgdwm	40	39
	*	***	*	BATCHgd	39	41
minpack.lm	*	***	**	Levenberg-Marquardt	15	24
ANN2	**	***	*	rmsprop	14	28
	**	***	*	adam	13	33
	**	***	*	sgd	11	42
deepdive	**	***	**	adam	32	46
	**	***	**	rmsProp	34	53
	**	***	**	momentum	53	56
	**	***	**	gradientDescent	52	58
snnR	**	**	**	SemiSmoothNewton	7	48
elmNNRcpp	**	***	**	ELM	1	59
ELMR	**	***	**	ELM	2	60