file: ~/courses/spring2002/434/feedforward_setup.txt    RWN   02/17/02

The steps to follow, with an example, using MatLab NNET programs for 
handling feedforward networks :

1. setup the network:
	net2 = newff([-3 3; -2 1; 3 9],[4 5 2], {'tansig', ...
'tansig', 'satlin'});
		This uses the new_feedforward call to set up a
	three layer network with three inputs, for which 
	weights will be generated; the three entries in the first [ ]
	give the ranges of the three components of the input
	vectors; the three entries in the next [ ] are the number of 
	neurons in each layer [4 in the first = input layer, 5 in the
	 second = hidden layer, and 2 in the final = output layer]; 
	the next three entries in the { } give the transfer
	functions. There are two more possible entries, BTF and BLF, 
	for which the defaults are to be used: BTF=backpropagation 
	training function = trainlm, BLS=backpropagation learning 
	function = learngdm. To fins out more enter help newff in 
	the MatLab command window.

2. setup the training set of inputs P and corresponding outputs T: 
	P1=[0 0.1 0.2 0.8; 1 -0.6 -2 0.5 ; 3 4 5 6],
	T1=[0.2 -0.1 0 1; 0 1 -1 0.8]

3. check the initial weight and bias values:
	net2.IW{1,1}, net2.b{1}
		returns (something like this; the values result from
     the Nguyen-Widrow initialization net2.layers{1}.initFcn =initnw)
ans =
    0.2407   -1.3503   -0.1871
    0.4333   -0.2799    0.5843
    0.4164    0.8598   -0.4365
    0.3514    1.2300   -0.2170
ans =
   -3.9030
    1.9000
   -2.6005
   -5.1593

	net2.LW{2,1}, net2.b{2}
		returns
ans =

    1.3012    1.2860    0.9660    0.3202
    0.8347   -0.3144   -1.7175   -0.7983
   -0.9773    1.1884   -1.0902   -0.9093
   -0.2718   -1.2742   -0.8566   -1.3970
    1.5132   -0.5113   -1.0469    0.8576


ans =

   -2.0935
   -1.0467
         0
   -1.0467
    2.0935

	net2.LW{3,2}, net2.b{3}
		returns 
ans =

   -0.0386   -0.0239    0.2435   -0.2091    0.2378
    0.2576   -0.0485    0.0150    0.1027   -0.2866


ans =

    0.9020
    0.9020
		note that net2.LW{3,1} returns []

3. next check, by simulating the net, that P1 works with net2:
	Y=sim(net2,P1)
		returns 
Y =

    1.0000    1.0000    0.8125    1.0000
    0.6061    0.7939    0.8205    0.8602

4. now train the network - this uses backpropagation
	net2.trainParam.epochs = 50;
 	net2 = train(net2,P1,T1);
 	Y = sim(net2,P1);
		returns
TRAINLM, Epoch 0/50, MSE 0.779732/0, Gradient 4.5283/1e-10
TRAINLM, Epoch 7/50, MSE 0.25125/0, Gradient 2.7617e-09/1e-10
TRAINLM, Maximum MU reached, performance goal was not met.
	as well as a graph of the performance after 7 epochs. 

To understand more about the results we can look at
		Y, T1, Y-T1

Y  =

    0.2000         0    0.0000    1.0000
    0.0000         0         0    0.8000


T1 =

    0.2000   -0.1000         0    1.0000
         0    1.0000   -1.0000    0.8000


ans =

    0.0000    0.1000    0.0000         0
    0.0000   -1.0000    1.0000   -0.0000

and evaluate the mean squate error by 
		mse(Y-T1)
ans =

   0.2512

Note that this is the sum of the squares of the differences 
of all entries divided by 8 = # of vectors(=4)x size(=2). 

	to find out more we can check the training function
	being used. 
	net2.trainFnc

ans =

trainlm

	then we can access information on trainlm by 
	help trainlm
		especially we find for the default values

	    net.trainParam.epochs     100  Maximum number of epochs to train
 	    net.trainParam.goal         0  Performance goal
 	    net.trainParam.max_fail     5  Maximum validation failures
 	    net.trainParam.mem_reduc    1  Factor to use for
						memory/speed trade off.
 	    net.trainParam.min_grad 1e-10  Minimum performance gradient
 	    net.trainParam.mu       0.001  Initial Mu
 	    net.trainParam.mu_dec     0.1  Mu decrease factor
 	    net.trainParam.mu_inc      10  Mu increase factor
 	    net.trainParam.mu_max    1e10  Maximum Mu
 	    net.trainParam.show        25  Epochs between displays 
						(NaN for no displays)
 	    net.trainParam.time       inf  Maximum time to train in
					seconds

	setting net2.trainParam.min_grad=1e-30  returns on running
	again   net2 = train(net2,P1,T1);
TRAINLM, Epoch 0/50, MSE 0.25125/0, Gradient 2.7617e-09/1e-30
TRAINLM, Epoch 1/50, MSE 0.25125/0, Gradient 2.7617e-09/1e-30
TRAINLM, Maximum MU reached, performance goal was not met.
	along with a flat curve of performance indicating that not
	much more can be done to make this system have a lower 
	mean square error. 

5. A simulink structure for this network can be obtained by 
	entering in the MatLab command window
		gensim(net2)