Hi, I've a problem concerning the training of a custom neural network. 
I don't know why, but I can't use the transfert function 'hardlim'. In fact, when I use it, there is no learning (no error but 0 iteration of learning). The weights are not changed, they remain to 0.

Can someone help me?

Here are few precisions :
- my neural network has 2 hidden layers. The first has 10 neurons, the second only one.
- The output expected are either 0 or 1 (classification problem) that's why I want to put a 'hardlim' function in the output layer.

Here is my code : I have changed the input in order to have a simple example.
I have tried to change the learning function, nothing is changed. I've tried to use newff function, and the results were the same.


%%%%%%%%%%%%%%%%%%%%%
% Entrees du reseau %
%%%%%%%%%%%%%%%%%%%%%

in=[1 2 3 1 5 1 5 6 7; 1 5 8 2 6 3 8 5 7; 2 8 7 2 10 1 11 12 10];
T=[0 1 1 0 1 0 1 1 1 ];

L=3;

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Construction du reseau %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

net=network;
%net=newff(minmax(in),[10 1]);

    %Caracteristiques de base du reseau;
    net.numInputs=1; %nombre de couche d'entree (1 par defaut);
    net.numLayers=2; %nombre de couches;
    net.inputs{1}.size=L; %nombre de neurones dans la couche d'entrée
    net.layers{1}.size=10;% nombre de neurones dans la couche cachée
    net.layers{2}.size=1;% nombre de neurones dans la couche de sortie

    %Connections des couches
    net.inputConnect(1)=1; %On connect les entrees a la premiere couche
    net.layerConnect(2,1)=1; % On connecte la deuxieme couche a la premiere
    net.outputConnect(2)=1; %c'est a la deuxieme couche que la sortie est connectee
    %net.targetConnect(2)=1; % connexion de la derniere couche a la cible (les fichiers corriges sur lequels on apprend)
 
    %Fonctions de transfert
    net.layers{1}.transferFcn='logsig';
    net.layers{2}.transferFcn='hardlim';
    %hardlim
    %Poids et Seuils
    net.biasConnect=[1;1]; % les deux couches auront des seuils
    net=init(net); %initialisation des poids
    net.initFcn='initlay';   
    
    %Fonction de cout
    net.performFcn='sse'; %ou 'msa'
    
    %Apprentissage
    
    %Fonction Train
    net.trainFcn='traingdm';
    %net.trainFcn='traingdx'; % apprentissage par descente du gradient
    net.trainParam.lr=0.1; %taux d'apprentissage (ou learning rate)
    net.trainParam.mc=0.9; %moment
    net.trainParam.epochs=1000;
    net.trainParam.show=1000;
    
    %Fonction Adapt
    
   % net.adaptFcn='adaptwb';
   % net.inputWeights{1,1}.learnFcn='learngdm';
   % net.layerWeights{2}.learnFcn='learngdm';
   % net.biases{1}.learnFcn='learngdm';
  %  net.adaptParam.passes=10;
    
    
% Etape 3 : simulation et ajustement des poids
net=train(net,in,T);
sim(net,in)

 

"Nicolas" wrote in message <jscjtb$e2t$1@newscl01ah.mathworks.com>...
> Hi, I've a problem concerning the training of a custom neural network. 
> I don't know why, but I can't use the transfert function 'hardlim'. 

It is not continuously differentiable. Therefore, gradient-descent bombs.

> Here are few precisions :
> - my neural network has 2 hidden layers. The first has 10 neurons, the second only one.

No. You have one hidden layer with 10 nodes and one output layer with one node.

> - The output expected are either 0 or 1 (classification problem) that's why I want 
> to put a 'hardlim' function in the output layer.

Hidden 'tansig' 
Output 'logsig' or 'purelin' for training
            round for classification 

> Here is my code : I have changed the input in order to have a simple example.
> I have tried to change the learning function, nothing is changed. I've tried to use 
>newff function, and the results were the same.
>  
> %%%%%%%%%%%%%%%%%%%%%
> % Entrees du reseau %
> %%%%%%%%%%%%%%%%%%%%%

WHEN POSTING PLEASE USE ENGLISH!
 
close all, clear all, clc
 x=[1 2 3 1 5 1 5 6 7; 1 5 8 2 6 3 8 5 7; 2 8 7 2 10 1 11 12 10];
 t=[0 1 1 0 1 0 1 1 1 ];
[ I N] = size(x) 
[O N] =size(t)

% Naive Constant Output Model (output = mean(t))
y00 = mean(t)
MSE00 = mse(t-y00)                                 % 0.222
Nerr00 = numel(find(round(y00)~=t))       % 3
 PctErr00 = 100*Nerr00/N                        % 33.3
 
% Linear Model
W = t/[ones(1,N);x]
y0 = W*[ones(1,N);x]
MSE0=mse(t-y0)                                 % 0.0098
R20 = 1-MSE0/MSE00                      %  0.96 Want R^2 >= 0.99
Nerr0 = numel(find(round(y00)~=t))   % 3
PctErr0 = 100*Nerr0/N                      % 33.3

'NOTE: MSE0 << MSE00  BUT  PCTERR0 = PCTERR00 !'

% Neural Net Model
Neq = N*O                               % No. of equations
%Nw = (I+1)*H+(H+1)*O         No of unknown weights
Hub = floor((N-1)*O/(I+O+1))  %  =1 Maximun H for Neq >= Nw
Ntrials = 10
rng(0)                            % In case you want to duplicate the run
for j = 1:10                   % Validation set stopping allows H > Hub
    H = j
    Nw = (I+1)*H+(H+1)*O
    for i = 1:Ntrials 
        %net = newff(minmax(x),[H O]); OBSOLETE
         net = newff(x,t,H);
        net.trainParam.goal = 0.01*MSE00;  %(R^2 >= 0.99)
        [net tr Y E] = train(net,x,t);
        Nepochs(i,j) = tr.epoch(end);
         MSE = tr.perf(end);
        R2(i,j) = 1-MSE/MSE00;
        Nerr = numel(find(round(Y)~=t));
        PctErr(i,j) = 100*Nerr/N;
    end
end

% Tabulations
format short
Nepochs = Nepochs
R2 = R2
format bank
PctErr = PctErr

Hope this helps.

Greg

"Greg Heath" <heath@alumni.brown.edu> wrote in message <jsd0ic$dtf$1@newscl01ah.mathworks.com>...
> "Nicolas" wrote in message <jscjtb$e2t$1@newscl01ah.mathworks.com>...
> > Hi, I've a problem concerning the training of a custom neural network. 
> > I don't know why, but I can't use the transfert function 'hardlim'. 
> 
> It is not continuously differentiable. Therefore, gradient-descent bombs.
> 
> > Here are few precisions :
> > - my neural network has 2 hidden layers. The first has 10 neurons, the second only one.
> 
> No. You have one hidden layer with 10 nodes and one output layer with one node.
> 
> > - The output expected are either 0 or 1 (classification problem) that's why I want 
> > to put a 'hardlim' function in the output layer.
> 
> Hidden 'tansig' 
> Output 'logsig' or 'purelin' for training
>             round for classification 
> 
> > Here is my code : I have changed the input in order to have a simple example.
> > I have tried to change the learning function, nothing is changed. I've tried to use 
> >newff function, and the results were the same.
> >  
> > %%%%%%%%%%%%%%%%%%%%%
> > % Entrees du reseau %
> > %%%%%%%%%%%%%%%%%%%%%
> 
> WHEN POSTING PLEASE USE ENGLISH!
>  
> close all, clear all, clc
>  x=[1 2 3 1 5 1 5 6 7; 1 5 8 2 6 3 8 5 7; 2 8 7 2 10 1 11 12 10];
>  t=[0 1 1 0 1 0 1 1 1 ];
> [ I N] = size(x) 
> [O N] =size(t)
> 
> % Naive Constant Output Model (output = mean(t))
> y00 = mean(t)
> MSE00 = mse(t-y00)                                 % 0.222
> Nerr00 = numel(find(round(y00)~=t))       % 3
>  PctErr00 = 100*Nerr00/N                        % 33.3
>  
> % Linear Model
> W = t/[ones(1,N);x]
> y0 = W*[ones(1,N);x]
> MSE0=mse(t-y0)                                 % 0.0098
> R20 = 1-MSE0/MSE00                      %  0.96 Want R^2 >= 0.99
> Nerr0 = numel(find(round(y00)~=t))   % 3
> PctErr0 = 100*Nerr0/N                      % 33.3
> 
> 'NOTE: MSE0 << MSE00  BUT  PCTERR0 = PCTERR00 !'
> 
> % Neural Net Model
> Neq = N*O                               % No. of equations
> %Nw = (I+1)*H+(H+1)*O         No of unknown weights
> Hub = floor((N-1)*O/(I+O+1))  %  =1 Maximun H for Neq >= Nw
> Ntrials = 10
> rng(0)                            % In case you want to duplicate the run
> for j = 1:10                   % Validation set stopping allows H > Hub
>     H = j
>     Nw = (I+1)*H+(H+1)*O
>     for i = 1:Ntrials 
>         %net = newff(minmax(x),[H O]); OBSOLETE
>          net = newff(x,t,H);
>         net.trainParam.goal = 0.01*MSE00;  %(R^2 >= 0.99)
>         [net tr Y E] = train(net,x,t);
>         Nepochs(i,j) = tr.epoch(end);
>          MSE = tr.perf(end);
>         R2(i,j) = 1-MSE/MSE00;
>         Nerr = numel(find(round(Y)~=t));
>         PctErr(i,j) = 100*Nerr/N;
>     end
> end
> 
> % Tabulations
> format short
> Nepochs = Nepochs
> R2 = R2
> format bank
> PctErr = PctErr
> 
> Hope this helps.
> 
> Greg

Thanks a lot, I'm going to look attentively your code (for now I don't understand the signification of all lines). 
But I have still an other question : is it possible to use the hardlim function by changing the learning algorithm? 

"Nicolas" wrote in message <jseek3$ih4$1@newscl01ah.mathworks.com>...
> "Greg Heath" <heath@alumni.brown.edu> wrote in message <jsd0ic$dtf$1@newscl01ah.mathworks.com>...
> > "Nicolas" wrote in message <jscjtb$e2t$1@newscl01ah.mathworks.com>...
> Thanks a lot, I'm going to look attentively your code (for now I don't understand the signification of all lines). 
> But I have still an other question : is it possible to use the hardlim function by changing >the learning algorithm? 

Sure. 

Use or write a learning algorithm that doesn't use derivatives to minimize the error.
At each epoch just make backpropagation weight changes proportional to the error. There are many versions in textbooks and on the net. I have posted a few versions along the years in this Newsgroup and/or comp.ai.neural-nets. You can try to find one. However, don't spend to much time on it. It is painfully slow compared to any of the gradient descent algorithms.

Hope this helps.

Greg

"Greg Heath" <heath@alumni.brown.edu> wrote in message <jsd0ic$dtf$1@newscl01ah.mathworks.com>...

---SNIP  

> % Linear Model
> W = t/[ones(1,N);x]
> y0 = W*[ones(1,N);x]
> MSE0=mse(t-y0)                             % 0.0098
> R20 = 1-MSE0/MSE00                      %  0.96 Want R^2 >= 0.99
> Nerr0 = numel(find(round(y00)~=t))   % 3
> PctErr0 = 100*Nerr0/N                     % 33.3
> 
> 'NOTE: MSE0 << MSE00  BUT  PCTERR0 = PCTERR00 !'

CORRECTION:
Nerr0 = numel(find(round(y0)~=t))   % 0
PctErr0 = 100*Nerr0/N                    % 0

Hope this helps.

Greg




 
> % Neural Net Model
> Neq = N*O                               % No. of equations
> %Nw = (I+1)*H+(H+1)*O         No of unknown weights
> Hub = floor((N-1)*O/(I+O+1))  %  =1 Maximun H for Neq >= Nw
> Ntrials = 10
> rng(0)                            % In case you want to duplicate the run
> for j = 1:10                   % Validation set stopping allows H > Hub
>     H = j
>     Nw = (I+1)*H+(H+1)*O
>     for i = 1:Ntrials 
>         %net = newff(minmax(x),[H O]); OBSOLETE
>          net = newff(x,t,H);
>         net.trainParam.goal = 0.01*MSE00;  %(R^2 >= 0.99)
>         [net tr Y E] = train(net,x,t);
>         Nepochs(i,j) = tr.epoch(end);
>          MSE = tr.perf(end);
>         R2(i,j) = 1-MSE/MSE00;
>         Nerr = numel(find(round(Y)~=t));
>         PctErr(i,j) = 100*Nerr/N;
>     end
> end
> 
> % Tabulations
> format short
> Nepochs = Nepochs
> R2 = R2
> format bank
> PctErr = PctErr
> 
> Hope this helps.
> 
> Greg