Learning winner-take-all competition between 
groups of neurons in lateral inhibitory networks 
Xiaohui Xie, Richard Hahnloser and H. Sebastian Seung 
E25-210, MIT, Cambridge, MA 02139 
{xhxie I rhl seung}@mit. edu 
Abstract 
It has long been known that lateral inhibition in neural networks can lead 
to a winner-take-all competition, so that only a single neuron is active at 
a steady state. Here we show how to organize lateral inhibition so that 
groups of neurons compete to be active. Given a collection of poten- 
tially overlapping groups, the inhibitory connectivity is set by a formula 
that can be interpreted as arising from a simple learning rule. Our analy- 
sis demonstrates that such inhibition generally results in winner-take-all 
competition between the given groups, with the exception of some de- 
generate cases. In a broader context, the network serves as a particular 
illustration of the general distinction between permitted and forbidden 
sets, which was introduced recently. From this viewpoint, the computa- 
tional function of our network is to store and retrieve memories as per- 
mitted sets of coactive neurons. 
In traditional winner-take-all networks, lateral inhibition is used to enforce a localized, 
or "grandmother cell" representation in which only a single neuron is active [1, 2, 3, 4]. 
When used for unsupervised learning, winner-take-all networks discover representations 
similar to those learned by vector quantization [5]. Recently many research efforts have 
focused on unsupervised learning algorithms for sparsely distributed representations [6, 7]. 
These algorithms lead to networks in which groups of multiple neurons are coactivated to 
represent an object. Therefore, it is of great interest to find ways of using lateral inhibition 
to mediate winner-take-all competition between groups of neurons, as this could be useful 
for learning sparsely distributed representations. 
In this paper, we show how winner-take-all competition between groups of neurons can be 
learned. Given a collection of potentially overlapping groups, the inhibitory connectivity 
is set by a simple formula that can be interpreted as arising from an online learning rule. 
To show that the resulting network functions as advertised, we perform a stability analysis. 
If the strength of inhibition is sufficiently great, and the group organization satisfies certain 
conditions, we show that the only sets of neurons that can be coactivated at a stable steady 
state are the given groups and their subsets. Because of the competition between groups, 
only one group can be activated at a time. In general, the identity of the winning group 
depends on the initial conditions of the network dynamics. If the groups are ordered by the 
aggregate input that each receives, the possible winners are those above a cutoff that is set 
by inequalities to be specified. 
I Basic definitions 
Let rn groups of neurons be given, where group membership is specified by the matrix 
={10 if the ith neuron is in the ath group 
otherwise 
(1) 
We will assume that every neuron belongs to at least one group 1, and every group contains 
at least one neuron. A neuron is allowed to belong to more than one group, so that the 
groups are potentially overlapping. The inhibitory synaptic connectivity of the network is 
defined in terms of the group membership, 
m 
Jij = (1- ij) = 
if i and j both belong to a group 
otherwise 
(2) 
One can imagine this pattern of connectivity arising by a simple learning mechanism. Sup- 
pose that all elements of J are initialized to be unity, and the groups are presented sequen- 
tially as binary vectors ix,... , m. The ath group is learned through the update 
(3) 
In other words, if neurons i and j both belong to group a, then the connection between 
them is removed. After presentation of all m groups, this leads to Eq. (2). At the start 
of the learning process, the initial state of J corresponds to uniform inhibition, which is 
known to implement winner-take-all competition between individual neurons. It will be 
seen that, as inhibitory connections are removed during learning, the competition evolves 
to mediate competition between groups of neurons rather than individual neurons. 
The dynamics of the network is given by 
d--- + xi = bi + oxi -/? y. Jijxj (4) 
J 
where [z] + = max{z, 0} denotes rectification, o > 0 the strength of self-excitation, and 
/ > 0 the strength of lateral inhibition. 
Equivalently, the dynamics can be written in matrix-vector form as k + x = [b + Wx] +, 
where W = oI -/J includes both self-excitation and lateral inhibition. The state of the 
network is specified by the vector x, and the external input by the vector b. A vector v is 
said to be nonnegative, v _> 0, if all of its components are nonnegative. The nonnegative 
orthant is the set of all nonnegative vectors. It can be shown that any trajectory of Eq. (4) 
starting in the nonnegative orthant remains there. Therefore, for simplicity we will consider 
trajectories that are confined to the nonnegative orthant x _> 0. However, we will consider 
input vectors b whose components are of arbitrary sign. 
2 Global stability 
The goal of this paper is to characterize the steady state response of the dynamics Eq. (4) 
to an input b that is constant in time. For this to be a sensible goal, we need some guarantee 
that the dynamics converges to a steady state, and does not diverge to infinity. This is 
provided by the following theorem. 
Theorem 1 Consider the network Eq. (4). The following statements are equivalent: 
1This condition can be relaxed, but is kept for simplicity. 
1. For any input b, there is a nonempty set of steady states that is globally asymptot- 
ically stable, except for initial conditions in a set of measure zero. 
2. The strength oz of self-excitation is less than one. 
Proof sketch: 
(2) = (1): Ifct < 1, the function --ct)a:Ta:+ a: TJa:--bTa:is bounded below 
 
and radially unbounded in the nonnegative orthant. Furthermore it is nonincreas- 
ing under the dynamics Eq. (4), and constant only at steady states. Therefore it is 
a Lyapunov function, and its local minima are globally asymptotically stable. 
 (1) = (2): Suppose that (2) is false. If oz _> 1, it is possible to choose b and an 
initial condition for a: so that only one neuron is active, and the activity of this 
neuron diverges, so that (1) is contradicted.  
3 Relationship between groups and permitted sets 
In this section we characterize the conditions under which the lateral inhibition of Eq. (4) 
enforces winner-take-all competition between the groups of neurons. That is, the only sets 
of neurons that can be coactivated at a stable steady state are the groups and their subsets. 
This is done by performing a linear stability analysis, which allows us to classify active 
sets using the following definition. 
Definition 1 If a set of neurons can be coactivated by some input at an asymptotically 
stable steady state, it is called permitted. Otherwise, it is forbidden 
Elsewhere we have shown that whether a set is permitted or forbidden depends on the 
submatrix of synaptic connections between neurons in that set[1 ]. If the largest eigenvalue 
of the sub-matrix is less than unity, then the set is permitted. Otherwise, it is forbidden. 
We have also proved that any superset of a forbidden set is forbidden, while any subset of 
a permitted set is also permitted. 
Our goal in constructing the network (4) is to make the groups and their subsets the only 
permitted sets of the network. To determine whether this is the case, we must answer two 
questions. First, are all groups and their subsets permitted? Second, are all permitted sets 
contained in groups? The first question is answered by the following Lemma. 
Lemma 1 All groups and their subsets are permitted. 
Proof: If a set is contained in a group, then there is no lateral inhibition between the 
neurons in the set. Provided that oz < 1, all eigenvalues of the sub-matrix are less than 
unity, and the set is permitted.  
The answer to the second question, whether all permitted sets are contained in groups, is 
not necessarily affirmative. For example, consider the network defined by the group mem- 
bership matrix  = { (1, 1, 0), (0, 1, 1), (1, 0, 1) }. Since every pair of neurons belongs to 
some group, there is no lateral inhibition (J = 0), which means that there are no forbidden 
sets. As a result, (1, 1, 1) is a permitted set, but obviously it is not contained in any group. 
Let's define a ,spurious permitted set to be one that is not contained in any group. For 
example, {1, 1, 1} is a spurious permitted set in the above example. To eliminate all the 
spurious permitted sets in the network, certain conditions on the group membership matrix 
 have to be satisfied. 
Definition 2 The membership  is degenerate if there exists a set of n _> 3 neurons that is 
not contained in any group, but all of its subsets with n - 1 neurons belong to some group. 
Otherwise,  is called nondegenerate. For example,  = { (1, 1, 0), (0, 1, 1), (1, 0, 1) } is 
degenerate. 
Using this definition, we can formulate the following theorem. 
Theorem 2 The neural dynamics Eq. (4) with oz < 1 and/9 > 1 - oz has a spurious 
permitted set if and only if  is degenerate. 
Before we prove this theorem, we will need the following lemma. 
Lemma 2 If / > 1 - oz, any set containing two neurons not in the same group is forbidden 
under the neural dynamics Eq. (4). 
Proof sketch: We will start by analyzing a very simple case, where there are two neu- 
rons belonging to two different groups. Let the group membership be {(1, 0), (0, 1)}. In 
this case, W = {(oz,-/), (-/, oz)}. This matrix has eigenvectors (1, 1)and (1,-1) and 
eigenvalues oz -/9 and oz +/9. Since oz < 1 for global stability and/9 > 0 by definition, the 
(1, 1) mode is always stable. But if/9 > 1 - oz, the (1,- 1) mode is unstable. This means 
that it is impossible for the two neurons to be coactivated at a stable steady state. Since any 
superset of a forbidden set is also forbidden, the general result of the lemma follows. I. 
Proof of Theorem 2 (sketch): 
<=: If  is degenerate, there must exist a set r _> 3 neurons that is not contained in 
any group, but all of its subsets with r - 1 neurons belong to some group. There is 
no lateral inhibition between these r neurons, since every pair of neurons belongs 
to some group. Thus the set containing all r neurons is permitted and spurious. 
=: If there exists a spurious permitted set P, we need to prove that  must be 
degenerate. We will prove this by contradiction and induction. Let's assume  is 
nondegenerate. 
P must contain at least 2 neurons since any one neuron subset is permitted and 
not spurious. By Lemma 2, these 2 neurons must be contained in some group, or 
else it is forbidden. Thus ? must contain at least 3 neurons to be spurious, and 
any pair of neurons in ? belongs to some group by Lemma 2. 
If ? contains at least r neurons and all of its subsets with r - 1 neurons belong 
to some group, then the set with these r neurons must belong to some group, 
otherwise  is degenerate. Thus r must contain at least r q- 1 neurons to be 
spurious, and all its r subsets belong to some group. 
By induction, this implies that ? must contain all neurons in the network, in which 
case, ? is either forbidden or nonspurious. This contradicts with the assumption 
P is a spurious permitted set.  
From Theorem 2, we can easily have the following result. 
Corollary 1 If every group contains some neuron that does not belong to any other group, 
then there is no any ,s)2urious permitted set. 
4 The potential winners 
We have seen that if  is nondegenerate, the active set must be contained in a group, pro- 
vided that lateral inhibition is strong (/9 > 1 - oz). The group that contains the active set 
will be called the "winner" of the competition between groups. The identity of the winner 
depends on the input b, and also on the initial conditions of the dynamics. For a given input, 
we need to characterize which pattern could potentially be the winner. 
Suppose that the group inputs B a = Y4 [bi] i are distinct. Without loss of generality, 
we order the group inputs as B x > ... > Bm. Let's denote the largest input as brnaoe = 
rnaxi{bi} and assume b,ax > O. 
Theorem 3 For nonoverlapping groups, the top c groups with the largest group input could 
end up the winner depending on the initial conditions of the dynamics, where c is deter- 
minedby the equation B c _> (1 - a)/3-bmax > B c+ 
Proof sketch: Suppose the ath group is the winner. For all neurons not in this group to be 
inactive, the self-consistent condition should read 
E [bi] + > 1 - a max{[bj]+} (5) 
If a group containing the neuron with the largest input, this condition can always be sat- 
isfied. Moreover, this group is always in the top c groups. For groups not containing the 
neuron with the largest input, this condition can be satisfied if and only if they are in the 
top c groups. I 
The winner-take-all competition described above holds only for the case of strong inhibi- 
tion/3 > 1 - a. On the other hand, if/3 is small, the competition will be weak and may 
not result in group-winner-take-all. In particular, if/3 < (1 - a)/,, where , is 
the largest eigenvalue of -J, then the set of all neurons is permitted. Since every subset 
of a permitted set is permitted, that means there are no forbidden sets and the network is 
monostable. Hence, group-winner-take-all does not hold. If (1 - a)/, </3 < 1 - a, 
the network has forbidden sets, but the possibility of spurious permitted sets cannot be 
excluded. 
5 Examples 
Traditional winner-take-all network This is a special case of our network with N 
groups, each containing one of the N neurons. Therefore, the group membership matrix  
is the identity matrix, and J = ll T - I, where 1 denotes the vector of all ones. According 
to Corollary 1, only one neuron is permitted to be active at a stable steady state, provided 
that/3 > 1 - a. We refer to the active neuron as the "winner" of the competition mediated 
by the lateral inhibition. 
If we assume that the inputs bi have distinct values, they can be ordered as b > b2 >    > 
bv, without loss of generality. According to Theorem 3, any of the neurons 1 to k can be 
the winner, where k is defined by bk _> (1 - a)/3-b > bk+. The winner depends on 
the initial condition of the network dynamics. In other words, any neuron whose input is 
greater than (1 - a)//3 times the largest input can end up the winner. 
Topographic organization Let the N neurons be organized into a ring, and let every 
set of d contiguous neurons be a group. d will be called the width. For example, in a 
network with N = 4 neurons and group width d = 2, then the membership matrix is 
 = {(1,1,0,0),(0,1,1,0),(0,0,1,1),(1,0,0,1)}. This ring network is similar to the 
one proposed by Ben-Yishai et al in the modeling of orientation tuning of visual cortex[9]. 
Unlike the WTA network where all groups are non-overlapping which implies that  is 
always nondegenerate, in the ring network neurons are shared among different groups,  
will become degenerate when the width of the group is large. To guarantee all permitted 
sets are the subsets of some group, we have the following corollary, which can be derived 
from Theorem 2. 
A D 
I I 
15 15 
5 10 15 5 10 15 
B E 
:'"'"'"'"" ' t 
50 100 150 200 100 200 300 400 
c F 
I 1 
15 15 
5 10 15 5 10 15 20 
Figure 1: Permitted sets of the ring network. The ring network is comprised of 15 neurons with 
oz = 0.4 and fi = 1. In panels A and D, the 15 groups are represented by columns. Black refers to 
active neurons and white refers to inactive neurons. (A) 15 groups of width d = 5. (B) All permitted 
sets corresponding to the groups in A. (C) The 15 permitted sets in B that have no permitted supersets. 
They are the same as the groups in A. (D) 15 groups with width d = 6. (E) All permitted set 
corresponding to groups in D. (F) There are 20 permitted sets in E that have no permitted supersets. 
Note that there are 5 spurious permitted sets. 
Corollary 2 In the ring network with N neurons, if the width d < N/3 + 1, then there is 
no spurious permitted set. 
Fig. (1) shows the permitted sets of a ring network with 15 neurons. From Corollary 2, we 
know that if the group width is no larger than 5 neurons, there will not exist any spurious 
permitted set. In the left three panels of Fig. (1), the group width is 5 and all permitted sets 
are subsets of these groups. However, when the group width is 6 (right three panels), there 
exists 5 spurious permitted sets as shown in panel F. 
As we have mentioned earlier, the lateral inhibition strength/3 plays a critical role in de- 
termining the dynamics of the network. Fig. (2) shows four types of steady states of a ring 
network corresponding to different values of/3. 
6 Discussion 
We have shown that it is possible to organize lateral inhibition to mediate a winner-take-all 
competition between potentially overlapping groups of neurons. Our construction utilizes 
the distinction between permitted and forbidden sets of neurons. 
If there is strong lateral inhibition between two neurons, then any set that contains 
them is forbidden (Lemma 2). Neurons that belong to the same group do not have 
any mutual inhibition, and so they form a permitted set. Because the synaptic con- 
nections between neurons in the same group are only composed of self-excitation, their 
outputs equal their rectified inputs, amplified by the gain factor of 1/(1 - a). Hence 
the neurons in the winning group operate in a purely analog regime. The coexis- 
tence of analog filtering with logical constraints on neural activation represents a form 
of hybrid analog-digital computation that may be especially appropriate for percep- 
tual tasks. It might be possible to apply a similar method to the problem of data re- 
construction using a constrained set of basis vectors. 
combination of basis vec- 
tors could for example im- 
plement sparsity or non- 
negativity constraints. 
As we have shown in The- 
orem 2, there are some de- 
generate cases of overlap- 
ping groups, to which our 
method does not apply. It is 
an interesting open question 
whether there exists a gen- 
eral way of how to translate 
arbitrary groups of coactive 
neurons into permitted sets 
without involving spurious 
permitted sets. 
In the past, a great deal of 
research has been inspired 
by the idea of storing mem- 
ories as dynamical attrac- 
tors in neural networks [ 10]. 
Our theory suggests an al- 
ternative viewpoint, which 
is to regard permitted sets 
as memories latent in the 
synaptic connections. From 
this viewpoint, the contribu- 
The constraints on the linear 
1.2[ =0.087 1.21 =0.0874 I 
1.1 1.illllllll.,l 
0.9 O. 
0.8 0.8 
1 5 10 15 1 5 10 
C D 
31 J=0.088 J=l 
0 
1 5 10 15 
15 
/111 
5 10 15 
Figure 2: Lateral inhibition strength/ determines the behavior 
of the network. The network is a ring network of 15 neurons with 
width d = 5 and where ct = 0.4 and input bi = 1, Vi. These 
panels show the steady state activities of the 15 neurons. (A) 
There are no forbidden sets. (B) The marginal state/ = (1 - 
ct)/A,, = 0.874, in which the network forms a continuous 
attractor. (C) Forbidden sets exist, and so do spurious permitted 
sets. (D) Group-winner-take-all case, no spurious permitted sets. 
tion of the present paper is a method of storing and retrieving memories as permitted sets 
in neural networks. 
References 
[1] R. Hahnloser, R. Sarpeshkar, M. Mahowald, Douglas R., and H.S. Seung. Digital selection and 
analog amplification coexist in an electronic circuit inspired by neocortex. Nature, 3:609-616, 
2000. 
[2] Shun-Ichi Amari and Michael A. Arbib. Competition and Cooperation in Neural Nets', pages 
119-165. Systems Neuroscience. Academic Press, 1977. J. Metzler (ed). 
[3] J. Feng and K.P. Hadeler. Qualitative behaviour of some simple networks. J. Phys. A:, 29:5019- 
5033, 1996. 
[4] Richard H.R. Hahnloser. About the piecewise analysis of networks of linear threshold neurons. 
Neural Networks', 11:691-697, 1998. 
[5] T. Kohonen. Self-Organization and Associative Memory. Springer-Verlag, Berlin, 3 edition, 
1989. 
[6] D. D. Lee and H. S. Seung. Learning the parts of objects by nonnegative matrix factorization. 
Nature, 401:788-91, 1999. 
[7] B.A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning 
a sparse code for natural images. Nature, 381:607-609, 1996. 
[8] R. Ben-Yishai, R. Lev Bar-Or, and H. Sompolinsky. Theory of orientation tuning in visual cortex. 
Proc. Natl. Acad. Sci. USA, 92:3844-3848, 1995. 
[9] J. J. Hopfield. Neurons with graded response have collective properties like those of two-state 
neurons. Proc. Natl. Acad. Sci. USA, 81:3088-3092, 1984. 
