Threads in Neural Network in Train/Validation/Test

  • Permalink
  • submit to reddit
  • Email
  • Follow


Dear All,
I am using MATLAB R2009a. I have a different answer for train/validation/test from two different method: 

1) I Use: 
[net tr] = train(net,trainV.P,trainV.T,[],[],valV,testV);

 to train the network,   and simulate the different train/validation/test result as: 

normTrainOutput=sim(net,trainV.P,[],[],trainV.T);
normValidateOutput=sim(net,valV.P,[],[],valV.T);
normTestOutput=sim(net,testV.P,[],[],testV.T);

and then i obtained MSE for training validation and test as:

MSETrain=tr.perf(end);
MSEValidate=tr.vperf(end);
MSETest=tr.tperf(end);

And from them finally R2 square value as: 

R2Train=1-NMSETrain
R2Validate=1-NMSEValidate
R2Test=1-NMSETest

And the result i obtained directly from MSEtrain1=mse(normTrainOutput - tn(:XX)), that starts from training interval period i defined. And so on validation and test. Why these two values MSETrain1 and MSETrain differ?

2) I have R2 Train = 0.7738, R2 Validate = 0.7934 and R2 Test = 0.7926. And from the linear regression plot i obtain R train = 0.89584, R validate = 0.81805 and R Test = 0.92432. Does it mean the R2 value of neural network is worst than linear regression model? OR the result i obtained during training = 0.89584 from regression is quite good.

3) Every times i simulate my network, my R2 values sometimes good and sometimes even worst  -ve. How to make it constant, if i assume i get 27 epochs,  hidden neurons  =18 the best R2 value?

Thank You.
0
Reply subodhpaudel (11) 2/21/2013 8:07:08 PM

See related articles to this posting


"Subodh Paudel" <subodhpaudel@gmail.com> wrote in message <kg5upc$pgj$1@newscl01ah.mathworks.com>...
> Dear All,
> I am using MATLAB R2009a. I have a different answer for train/validation/test from two different method: 
> 
> 1) I Use: 
> [net tr] = train(net,trainV.P,trainV.T,[],[],valV,testV);
> 
>  to train the network,   and simulate the different train/validation/test result as: 
> 
> normTrainOutput=sim(net,trainV.P,[],[],trainV.T);
> normValidateOutput=sim(net,valV.P,[],[],valV.T);
> normTestOutput=sim(net,testV.P,[],[],testV.T);

Using norm in the output names is confusing because norm has a special meaning
(help/doc norm) 
> 
> and then i obtained MSE for training validation and test as:
> 
> MSETrain=tr.perf(end);
> MSEValidate=tr.vperf(end);
> MSETest=tr.tperf(end);

I think if tr.stop indicates validation minimum stopping you should 
replace end with end- tr.max_fail or tr.best_epoch.

> And from them finally R2 square value as: 
> 
> R2Train=1-NMSETrain
> R2Validate=1-NMSEValidate
> R2Test=1-NMSETest
> 
> And the result i obtained directly from MSEtrain1=mse(normTrainOutput - tn(:XX)), that starts from training interval period i defined. And so on validation and test. Why these two values MSETrain1 and MSETrain differ?

If tr.stop indicates validation minimum stopping, then the last max_fail epochs should not be included. Find tr.best_epoch ,tr.best_perf,  etc
 
> 2) I have R2 Train = 0.7738, R2 Validate = 0.7934 and R2 Test = 0.7926. And from the linear regression plot i obtain R train = 0.89584, R validate = 0.81805 and R Test = 0.92432. Does it mean the R2 value of neural network is worst than linear regression model? OR the result i obtained during training = 0.89584 from regression is quite good.

If you had chosen the val minimum epoch, I would have expected

R = sqrt( R^2 )
> 
> 3) Every times i simulate my network, my R2 values sometimes good and sometimes even worst  -ve. How to make it constant, if i assume i get 27 epochs,  hidden neurons  =18 the best R2 value?

You get different values because of the random data division and random weight initialization. If you intialize the random number generator to the same state (e.g., 
rng(4151941) ) before data division and weight initialization, you will reproduce runs.

I usually use a double loop over numH candidate values for H and Ntrials weight initialization runs to get five Ntrials X numH sized matrices for numepochs, R2trn, 
R2trna, R2val and R2tst.

Search NEWSGROUP and ANSWERS for greg Ntrials (or other of my characteristic 
variable names MSE00, Neq, Ntrneq, Nw, Hub, R2, R2a,...)

Hope this helps.

Greg 
0
Reply heath (3980) 2/22/2013 1:14:14 AM
comp.soft-sys.matlab 203507 articles. 529 followers. Post

1 Replies
102 Views

Similar Articles

[PageSpeed] 9


  • Permalink
  • submit to reddit
  • Email
  • Follow


Reply:

Similar Artilces:

How do I find out which dataset was used for train, validation and test in a trained neural network?
I'm trying to do a code to train as many networks as I want automaticaly. Nevertheless I'm finding a hard work trying to get the datasets which were used for train, validation and test. What I do, but I don't know if it's right is to use diverand function befor train function. Is it the righway? "Fernando Cunha" <cunha.fr@terra.com.br> wrote in message <hksfg2$rgg$1@fred.mathworks.com>... > I'm trying to do a code to train as many networks as I want automaticaly. Nevertheless I'm finding a hard work trying to get the datasets which were used f...

NEURAL NETWORK TRAINING, VALIDATION AND TESTING
Data = Design + Non-design Design = Train(estimate weights) + Validation(Stop training when MSEval goes thru a minimum). Non-design = Test(Obtain UNBIASED generalization estimate of performance on unseen non-design data). I hope this eliminates a lot of confusion. Greg "Greg Heath" <heath@alumni.brown.edu> wrote in message <l33i8e$ra7$1@newscl01ah.mathworks.com>... > Data = Design + Non-design > > Design = Train(estimate weights) + Validation(Stop training when MSEval goes thru a minimum). > > Non-design = Test(Obtain UNBIASED generaliz...

Training, testing and validating data set in Neural Network
Hi everybody, I'm working with Neural Network toolbox and I had a doubt about how to define the training, testing and validating data set. I mean, how to distribute the 100% of the data into these three categories. Which percentage for each one?? And also, how many tests should i consider for a neural network of 1 hidden layer, with 21 neurons in this layer, 10 in the input and just 1 in the output? is there any way to compute the total number of tests?? I tried everywhere, but i couldnt find any answer, so any advice or help is welcome. thanks in advance, Munoz. On Nov 7...

Neural network: Retrieve training, validation, testing dataset
Hi there, Two questions: 1. I'm using newff and train to create a neural network (LM algorithm). I need to retrieve the training, validation and testing dataset I'm using after the training (I'm using the default 60,20,20 division of the input dataset). I'could obtain those using dividerand in the input dataset (and get a train, val, test input dataset) and then using the indices with divideind to get the corresponding target datasets. But, then, how to use those datasets when calling "train"? Or when doing: [trainP2,valP2,testP2,trainInd2,valInd2,testInd2] ...

Neural Networks Train/Test
Hi, what?s the command line for training, so that i have just train samples and test samples (not using validation) ? I tried changing this without success.. [net,tr] = train(net,trainSamples.P,trainSamples.T,[],[],validateSamples,testSamples); Regards, Andre On Nov 18, 7:40=A0pm, "Andre Santos" <and...@brturbo.com.br> wrote: > Hi, > > what?s the command line for training, so that i have just train samples a= nd test samples (not using validation) ? > > I tried changing this without success.. > > [net,tr] =3D train(net,trainSamples.P,trainSamples.T,[],...

Neural network train problem with validation
i'm Working on modeling very high dimensional data with very few observation, and of course there's no way to obtain a good model in prediction, but i'd like to implement a new approach for assessing neural netowork but i really need a way not to divide the set of dat in training - validation -test, but only training and use the test set for detect the best network in prediction. how can i set the parameter in newff for exclude the validation set? i tried with: net.divideParam.trainRatio = 0.8; net.divideParam.testRatio = 0.2; but it doesn't work. I also split ...

MSE compararison between training and testing for neural network
Helo,i have done neural network for training part. After i get the Mean Square Error(MSE) i want to compare it with MSE in testing part. As I know, 80% for training and 20% for testing, but my problem is I dont know how to start the coding for testing part to get the MSE so i can compare it with training part. Have an idea? Thanks. On Nov 10, 10:03=A0pm, "Reff " <rafi...@yahoo.com> wrote: > Helo,i have done neural network for training part. After i get the Mean S= quare Error(MSE) i want to compare it with MSE in testing part. As I know, = 80% for training and 20% for tes...

needed urgently a neural network train and test code
Dear all, I will be appreciated if any one can help me by a matlab code that recognizes the mnist digits database using the neural network classifier , i need the train and test codes. regards Heba M wrote: > I will be appreciated if any one can help me by a matlab code that > recognizes the mnist digits database using the neural network classifier > , i need the train and test codes. What a coincidence -- I was saying just this morning that I need robust and highly sensitive data classification algorithms for large data sets (16k or more features) with very few...

How to test Neural Network with real world data after training it?
Dear All, I am currently using nprtool for pattern recognition in Matlab version 7.2 (R2011b).I followed the steps in GUI and trained network and saved the network in Matlab work space as net.What is the procedure if I want to test real world data ? If anybody have answer please post it, it will really help me a lot. Thank you all. "Suresh " <dnbdsuresh@gmail.com> wrote in message news:kekt5s$qrb$1@newscl01ah.mathworks.com... > Dear All, > I am currently using nprtool for pattern recognition in Matlab version 7.2 > (R2011b).I followed the steps in GUI a...

Neural Network Validation: Train vs Sim output puzzle
Here are the two functions of the puzzle. [net1,tr,Ytrain,E,Pf,Af] = train(net0,P,T,Pi,Ai,VV,TV) Ysim=sim(net1,P) In the context of regression/prediction. What are the differences between the Ytrain network output from TRAIN and the Output from SIM? In estimation the Ytrain network output is equal to the output from sim. From my understanding both represent the neural net regression fit. In the validation step the Ytrain network output is NOT equal to the output from sim. (even if they are somewhat close) This is my issue. Which one would represent the neural net regression fit (if one ...

data splitting: train&test or train/test/validate
Hi, I have some questions. Anyone willing to answer - big thanks in advance. a)Which one is the best way to split data?Advantages & disadvantages? 1.train & test 2.train/test/validate b)What is the different between test & validate? Aren't they supposed to be the data that different from train set but still fall within the same range? c)How we define the data set is small? eg: I have a set of data contains 6column (represent 5 input & 1 output) which every column has 31 data point. Is this consider small for MLP & RBF network? d)Data splitting come first o...

Can training set be divided further into subsets of training set for training neural networks?
I have read on some forum that there are training methods. one is to train = NN on whole training data at once, 2nd is to train NN in parts like first f= rom 1 to 1000 samples, then 1 to 2000 samples and so on. 3rd is to train NN= in parts like first from 1 to 1000 samples, then 1000 to 2000 samples and = so on. I want to ask that whether all these methods correct and which to us= e when?? On Fri, 4 Jul 2014 01:47:17 -0700 (PDT), Aamir Nawaz <engr.aamir09@gmail.com> wrote: >I have read on some forum that there are training methods. one is to train NN on whole training data...

Neural Network: Predicition equation of the trained network
I am trying to determine the final equation for the trained network. My network is a plain vanilla feedforward network with a single hidden layer. The hidden layer has a logsig squashing function and the output layer has a pure linear transfer function. I would appreciate help in correcting the code in Block 2 where I am attempting to re-create the calculations that the toolbox performs to generate the network predictions. %% BLOCK 1: Network training x= rand( 5, 10) ; y= rand( 1, 10) ; % random training set (for % explanation p...

Testing neural networks
hi all i'm using code below to test network but the Result of subtraction always 0 [n,m]=size(im); % im is matrix of +1,-1 values a2={im(1:n,1:m)}; [Y,Pf,Af]=sim(net,{n,500},{},a2); Y{end}; %comparing q=a2{end}-Y{end}; suma2=sum(sum(cell2mat(a2))); sumq=sum(sum(q)); if sumq/suma2 <0.002 msg('Two results are equal'); else msg('Two results aren't equal'); end On May 1, 4:27=A0pm, "zainab " <zeemo...@yahoo.com> wrote: > hi all > i'm ...

Neural network training
Hi, I like to train a neural network beginning with a predefined weight matrix. However, every time I adjust the matrix and then start the training process the weights seem to be automatically set set to zero first and then get trained. Obviously I am missing something. Otherwise, init functions like "rands" wouldn't make any sense, if the weights are set to zero before training anyway. What am I doing wrong? Thanks in advance for any clarifications! Till Bockemuehl wrote: > Hi, > > I like to train a neural network beginning with a predefined weight > matrix. Ho...

Test the trained network
I want to make a stand alone application that contains two main features, they are Training and Recognizing. When the first input data have trained, I save the network. Then, I trained the network again with the second input data, and I save the network too. When I test the network with test data which is similiar to the first input data, the result is not same with the target of first data, it's like the target of the second data. help me please :'( "Livia Tengor" wrote in message <jq20rn$h02$1@newscl01ah.mathworks.com>... > I want to make a stand alone applicat...

Train Neural network
Hey everybody, I just trained a neural network with the nftool. I noticed that the nftool takes a training and a validation set. I was wondering how you could get the output of both. Course if I say: [net TR out] = train(net,input,target); out is only my trainigset and not the testing and validation set. Thus anyone knows how to solve this problem. thx On Oct 25, 4:54=A0pm, "Stefan Heinen" <s.g.h.hei...@live.com> wrote: > Hey everybody, > > I just trained a neural network with the nftool. I noticed that the nftoo= l takes a training and a validation ...

Training neural network
I am working on Classification of Image Object in type1,Type2 ,type3,type4 using feedforward back propagation NN technique.I have extract 8 feature of 432 object to Matlab so I got input matrices 8*432 matrix. I need to know how to send the data from matrix to the NN, what will be the function used for that. ( if I will take a column from a matrix and send it then come back and get the next column and so on, what is the function used to get this column and send it to the network). I need to khow how to set targate matrix.what are matrix used for targate matrix.what are value set in targa...

Neural Network : train
In supervised learning of Neural Network, after running the train command how to see the new weights and bias. for example: net=newp([-2 2;-2 +2],1); p=[2;2]; t=[0]; net.trainParam.epochs=1; net=train(net,p,t); After running the above, I need to know the new weights and bias. Thanks a lot for your assistance. Mannan. The network object ('net') that is returned by the train function contains all the information you need. Go through the documentation for more information about the "network object". Regards. Mannan wrote: > In supervised learning of Neural Network, aft...

VALIDATION OF A NEURAL NETWORK
Hi, Iam trying to validate my neural network by using the following code: net = newff(minmax(I),{'tansig' 'purelin'}); where I is the validation input. y =sim(net,I) Now,on plotting the validation input versus the validation output,I get a plot which is very unusual. Do I have to rescale my matrix,i.e normalise according to the scale similar to that of my training network ? Any suggestions would be highly appreciated, Many thanks, SK S K wrote: > Hi, > Iam trying to validate Inappropriate use of the term "validate". See the comp.ai.neural-nets FAQ. > ...

Neural Network Validation!!!
i've train my network for recognizing basic shapes such as circle, rectangle, square, diamond and etc. the next part is to do the validation of the neural network, if i am not mistaken. so how do i check for the circularity and rectangularity of the objects for circle and rectangle respectively? p/s: can someone guide me on this please!! i am very new to matlab and neural networks. May Lee wrote: > i've train my network for recognizing basic shapes such as circle, > rectangle, square, diamond and etc. the next part is to do the > validation of the neural network, if i am n...

training a neural network
Hi, Iam training a neural network by using the following code: net = newff(minmax(P),[10 1],{'tansig' 'purelin'},'trainlm'). but my network is not meeting the goal. could anyone please tell me how to meet the goal and what has to be done in order to stop the training. Many thanks, SK S K wrote: > Hi, > > Iam training a neural network by using the following code: > net = newff(minmax(P),[10 1],{'tansig' 'purelin'},'trainlm'). That initializes a network. It doesn't train it. net = train(net,P,T); will train it. > but my ...

how to do validation for neural network?
I'm using this code "[net,tr]=train(net,ptr,ttr,[],[],val,test);" to do validation .Could someone teach me others method couse when i use this code it wil stop early and the mse is more than 0.5. Please advice. Thanx. DanielTB wrote: > > > I'm using this code > "[net,tr]=train(net,ptr,ttr,[],[],val,test);" to do validation > .Could > someone teach me others method couse when i use this code it wil > stop > early and the mse is more than 0.5. Please advice. Thanx. hi, the validation set is supposed to stop training before overgeneralization...

Neural Network Training
Does anybody know of a Neural Network solution in Smalltalk? I'm especially interested in the classification of objects by symbolic attribute vectors (learning by example). Andre Andre Schnoor wrote: > Does anybody know of a Neural Network solution in Smalltalk? I'm > especially interested in the classification of objects by symbolic > attribute vectors (learning by example). > > Andre > Andre - Here are some ideas for you: 1. Smalltalk/X comes with some NN code (I think just backprop though). 2. I had developed a backprop in smalltalk...