Foundations for a Circuit Complexity Theory of 
Sensory Processing* 
Robert A. Legenstein & Wolfgang Maass 
Institute for Theoretical Computer Science 
Technische Universitit Graz, Austria 
{legi, maass} @igi.tu-graz.ac.at 
Abstract 
We introduce total wire length as salient complexity measure for an anal- 
ysis of the circuit complexity of sensory processing in biological neural 
systems and neuromorphic engineering. This new complexity measure is 
applied to a set of basic computational problems that apparently need to 
be solved by circuits for translation- and scale-invariant sensory process- 
ing. We exhibit new circuit design strategies for these new benchmark 
functions that can be implemented within realistic complexity bounds, in 
particular with linear or almost linear total wire length. 
1 Introduction 
Circuit complexity theory is a classical area of theoretical computer science, that provides 
estimates for the complexity of circuits for computing specific benchmark functions, such 
as binary addition, multiplication and sorting (see, e.g. (Savage, 1998)). In recent years 
interest has grown in understanding the complexity of circuits for early sensory processing, 
both from the biological point of view and from the point of view of neuromorphic engi- 
neering (see (Mead, 1989)). However classical circuit complexity theory has provided little 
insight into these questions, both because its focus lies on a different set of computational 
problems, and because its traditional complexity measures are not tailored to those re- 
sources that are of primary interest in the analysis of neural circuits in biological organisms 
and neuromorphic engineering. This deficit is quite unfortunate since there is growing de- 
mand for energy-efficient hardware for sensory processing, and complexity issues become 
very important since the number n of parallel inputs which such circuits have to handle is 
typically quite large (for example n _> 106 in the case of many visual processing tasks). 
We will follow traditional circuit complexity theory in assuming that the underlying graph 
of each circuit is a directed graph without cycles. The most frequently considered com- 
plexity measures in traditional circuit complexity theory are the number (and types) of 
*Research for this article was partially supported by the the Fonds zur Frrderung der wis- 
senschaftlichen Forschung (FWF), Austria, project P12153, and the NeuroCOLT project of the EC. 
Neural circuits in "wetware" as well as most circuits in analog VLSI contain in addition to 
feedforward connections also lateral and recurrent connections. This fact presents a serious obstacle 
for a direct mathematical analysis of such circuits. The standard mathematical approach is to model 
such circuits by larger feedforward circuits, where new "virtual gates" are introduced to represent the 
state of existing gates at later points in time. 
gates, as well as the depth of a circuit. The latter is defined as the length of the longest 
directed path in the underlying graph, and is also interpreted as the computation time of the 
circuit. The focus lies in general on the classification of functions that can be computed by 
circuits whose number of gates can be bounded by a polynomial in the number r of input 
variables. This implicitly also provides a polynomial - typically quite large - bound on the 
number of "wires" (defined as the edges in the underlying graph of the circuit). 
We proceed on the assumption that the area (or volume in the case of neural circuits) oc- 
cupied by wires is a severe bottleneck for physical implementations of circuits for sensory 
processing. Therefore we will not just count wires, but consider a complexity measure that 
provides an estimate for the total area or volume occupied by wires. In the cortex, neurons 
occupy an about 2 mm thick 3-dimensional sheet of "grey 
matter". There exists a strikingly general upper bound on the 
order of 105 for the number of neurons under any mm 2 of 
cortical surface, and the total length of wires (axons and den- 
drites, including those running in the sheet of "white matter" 
that lies below the grey matter) under any mm 2 of cortical 
surface is estimated to be _< 8kin = 8.106ram (Koch, 1999). 
Together this yields an upper bound of 8'6 -- 
-%--'t = 80  n mm 
for the wire length of the "average" cortical circuit involving 
r neurons. 
In order to arrive at a concise mathematical model we project 
each 3D cortical circuit into 2D, and assume for simplicity 
that its r gates (neurons) occupy the nodes of a grid. Then 
for a circuit with r gates, the total length of the horizontal 
components of all wires is on average _< 80. r mm = 80. r 
 10 5/2 ----- 25300. r grid units. Here, one grid unit is the distance between adjacent nodes on 
the grid, which amounts to 10-5/2mm for an assumed density of 105 neurons per mm 2 of 
cortical surface. Thus we arrive at a simple test for checking whether the total wire length 
of a proposed circuit design has a chance to be biologically realistic: Check whether you 
can arrange its r gates on the nodes of a grid in such a way that the total length of the 
horizontal components of all wires is _< 25300. r grid units. 
More abstractly, we define the following model: 
Gates, input- and output-ports of a circuit are placed on different nodes of a 2-dimensional 
grid (with unit distance 1 between adjacent nodes). These nodes can be connected by 
(unidirectional) wires that run through the plane in any way that the designer wants, in 
particular wires may cross and need not run rectilinearly (wires are thought of as running 
in the 3 dimensional ,sace above the plane, without charge for vertical wire segments) 2. 
We refer to the minimal value of the sum of all wire lengths that can be achieved by any 
such arrangement as the total wire length of the circuit. 
The attractiveness of this model lies in its mathematical simplicity, and in its generality. 
It provides a rough estimate for the cost of connectivity both in artificial (basically 2- 
dimensional) circuits and in neural circuits, where 2-dimensional wire crossing problems 
are apparently avoided (at least on a small scale) since dendritic and axonal branches are 
routed through 3-dimensional cortical tissue. 
There exist quite reliable estimates for the order of magnitudes for the number r of inputs, 
the number of neurons and the total wire length of biological neural circuits for sensory pro- 
cessing, see (Abeles, 1998; Koch, 1999; Shepherd, 1998; Braitenberg and Schiiz, 1998). 3 
2We will allow that a wire from a gate may branch and provide input to several other gates. For 
reasonable bounds on the maximal fan-out (104 in the case of neural circuits) this is realistic both for 
neural circuits and for VLSI. 
3The number of neurons that transmit information from the retina (via the thalamus) to the cortex 
Collectively they suggest that only those circuit architectures for sensory processing are 
biologically realistic that employ a number of gates that is almost linear in the number n 
of inputs, and a total wire length that is quadratic or subquadratic - with the additional re- 
quirement that the constant factor in front of the asymptotic complexity bound has a value 
close to 1. Since most asymptotic bounds in circuit complexity theory have constant fac- 
tors in front that are much larger than 1, one really has to focus on circuit architectures 
with clearly subquadratic bounds for the total wire length. The complexity bounds for cir- 
cuits that can realistically be implemented in VLSI are typically even more severe than for 
"wetware", and linear or almost linear bounds for the total wire length are desirable for that 
purpose. 
In this article we begin the investigation of algorithms for basic pattern recognition tasks 
that can be implemented within this low-level complexity regime. The architecture of such 
circuits has to differ strongly from most previously proposed circuits for sensory process- 
ing, which usually involve at least 2 completely connected layers, since already complete 
connectivity between just two linear size 2-dimensional layers of a feedforward neural net 
requires a total wire length on the order of n s/2 Furthermore a circuit which first se- 
lects a salient input segment consisting of a block of up to m adjacent inputs in some 
2-dimensional map, and then sends this block of _< m inputs in parallel to some central 
"pattern template matcher", typically requires a total wire length of Q(rt 3/2  m) - even 
without taking the circuitry for the "selection" or the template matching into account. 
2 Global Pattern Detection in 2-Dimensional Maps 
For many important sensory processing tasks - such as for vi- 
sual or somatosensory input - the input variables are arranged in 
a 2-dimensional map whose structure reflects spatial relationship 
in the outside world. We assume that local feature detectors are 
able to detect the presence of salient local features in their spe- 
cific "receptive field", such as for example a center which emits 
is estimated to be around 106 (all estimates given are for primates, and they only reflect the order of 
magnitude). The total number of neurons that transmit sensory (mostly somatosensory) information 
to the cortex is estimated to be around 108. In the subsequent sections we assume that these inputs 
represent the outputs of various local feature detectors for r, locations in some 2-dimensional map. 
Thus, if one assumes for example that on average there are 10 different feature detectors for each 
location on this map, one arrives at biologically realistic estimates for r, that lie between 105 and 
10 7 . 
The total number of neurons in the primary visual cortex of primates is estimated to be around 109, 
occupying an area of roughly 104 mm 2 of cortical surface. There are up to 105 neurons under one 
mm 2 of cortical surface, which yields a value of 10 -5/2 mm for the distance between adjacent grid 
points in our model. The total length of axonal and dendritic branches below one mm 2 of cortical 
surface is estimated to be between 1 and 10 km, yielding up to 10 TM mm total wire length for primary 
visual cortex. Thus if one assumes that 100 separate circuits are implemented in primary visual 
cortex, each of them can use 107 neurons and a total wire length of 109 mm. Hence realistic bounds 
for the complexity of a single one of these circuits for visual pattern recognition are 10 7 = r, 7/5 
neurons (for r, = 105), and a total wire length of at most 10 '5 = n 2'a grid units in the framework 
of our model. 
The whole cortex receives sensory input from about 108 neurons. It processes this input with 
about 10 m neurons and less than 10 TM mm total wire length. If one assumes that 10 a separate 
circuits process this sensory information in parallel, each of them processing about 1/10th of the 
input (where again 10 different local feature detectors report about every location in a map), one 
arrives at r, = 106 neurons for each circuit, and each circuit can use on average r* 7/6 neurons and a 
total wire length of 10 '5 < r, 2 grid units in the sense of our model. The actual resources available 
for sensory processing are likely to be substantially smaller, since most cortical neurons and circuits 
are believed to have many other functions besides online sensory processing. 
higher (or lower) intensity than its immediate surrounding, or a high-intensity line segment 
in a certain direction, the end of a line, a junction of line segments, or even more complex 
local visual patterns like an eye or a nose. The ultimate computational goal is to detect 
specific global spatial arrangements of such local patterns, such as the letter "T", or in the 
end also a human face, in a translation- and scale-invariant manner. 
We formalize 2-dimensional global pattern detection problems by assuming that the input 
consists of arrays a = (a,... ,a),b = (b,... ,b), etc. of binary variables that are 
arranged on a 2-dimensional square grid 4. Each index i can be thought of as representing a 
location within some  x v/--square in the outside world. We assume that ai = 1 if and 
only if feature a is detected at location i and that bi = 1 if and only if feature b is detected 
at location i. In our formal model we can reserve a subsquare within the 2-dimensional 
grid for each index i, where the input variables ai, hi, etc. are given on adjacent nodes of 
this grid s . Since we assume that this spatial arrangement of input variables reflects spatial 
relations in the outside word, many salient examples for global pattern detection problems 
require the computation of functions such as 
1, if there exist i and j so that ci = bj = 1 and input location j 
P (a, b) = is above and to the right of input location i 
0, else 
Theorem 2.1 The function P) can be computed- and wimesses i and j with ai = bj = 1 
can be exhibited if they exist - by a circuit with total wire length O(n), consisting of O(n) 
Boolean gates offan-in 2 (andfan-out 2) in depth O(log n. log log n). 
The depth of the circuit can be reduced to O(logn) if one employs threshold gates 6 with 
fan-in log n. This can also be done with total wire length O(n). 
Proof (sketch) At first sight it seems that P needs complete connectivity on the plane 
because of its global character. However, we show that there exists a divide and conquer 
approach with rather small communication cost. 
Divide the input plane into four sub-squares U,... ,U4 (see Figure la). We write 
a,..., a  and b,..., b 4 for the restrictions of the input to these four sub-areas and as- 
sume that the following values have already been computed for each sub-square Ui' 
 The 
 The 
 The 
 The 
 The 
x-coordinate of the leftmost occurrence of feature a in C'i 
x-coordinate of the rightmost occurrence of feature b in C'i 
y-coordinate of the lowest occurrence of feature a in C'i 
y-coordinate of the highest occurrence of feature b in C'i 
value of P)/4(cti,_bi) 
We employ a merging algorithm that uses this information to compute corresponding val- 
ues for the whole input plane. The first four values can be computed by comparison-like 
4Whenever needed we assume for simplicity that r, is such that x/-, log r, etc. are natural num- 
bers. The arrangement of the input variables an the grid will in general leave many nodes empty, 
which can be occupied by gates of the circuit. 
5To make this more formal one can assume that indices i and j represent pairs (i, i2), (j, j2) of 
coordinates. Then "input location j is above and to the right of input location i" means: i < j and 
i2 < j2. The circuit complexity of variations of the function P where one or both of the "<" are 
replaced by "_<" is the same. 
6A threshold gate computes a Boolean function T : {0, 1}  --> {0, 1} of the form 
k 
1 _> wo. 
a) b) 
Figure 1' The 2-dimensional input plane. Occurrences of features in a are indicated by light 
squares, and occurrences of features in b are indicated by dark squares. Divide the input 
area into four sub-squares (a). Merging horizontally adjacent sub-squares (b). Merging 
vertically adjacent sub-squares (c). 
a) b) c) 
Figure 2: The H-tree construction. Black squares represent sub-circuits for the merging 
algorithm. The shaded areas contain the leaves of the tree. The lightly striped areas rep- 
resent busses of wires that run along the edges of the H-Tree. The H-tree H divides the 
input-area into four sub-squares (a). To construct H2, replace the leaves of H by H-trees 
H (b). To construct Hk, replace the leaves of H by H-trees Hk_ (c). 
operations. The computation of P)(a, b) can be sketched as follows: First, check whether 
P)/4(ai, bi) = 1 for some i 6 {1,..., 4}. Then, check the spatial relationships between 
feature occurrences in adjacent sub-squares. When checking spatial relationships between 
features from two horizontally adjacent sub-squares, only the lowest and the highest fea- 
ture occurrence is crucial for the value of PB (see Figure lb). This is true, since the 
x-coordinates are already separated. When checking spatial relationships of features from 
two vertically adjacent sub-squares, only the leftmost and the rightmost feature occurrence 
is crucial for the value of PB (see Figure lc). This is true, since the y-coordinates are 
already separated. When checking spatial relationships of features from the lower left and 
the upper right sub-squares, it suffices to check whether there is an a-feature occurrence 
in the lower left and a b-feature occurrence in the upper right sub-square. Hence, one can 
reduce the amount of information needed from each sub-square to O(log n/4) bits. 
In the remaining part of the proof sketch, we present an efficient layout for a circuit that 
implements this recursive algorithm. We need a layout strategy that is compatible with the 
recursive two-dimensional division of the input plane. We adopt for this purpose a well 
known design strategy: the H-tree (see (Mead and Rem, 1979)). An H-tree is a recursive 
tree-layout on the 2-dimensional plane. Let H denote such a tree with 4  leaves. The 
layout of H is illustrated in Figure 2a. To construct an H-Tree H, build an H-tree H and 
replace its four leaves by H-trees H_ (see Figure 2b,c). 
We need to modify the H-tree construction of Mead and Rum to make it applicable to 
our problem. The inner nodes of the tree are replaced by sub-circuits that implement the 
merging algorithm. Furthermore, each edge of the H-tree is replaced by a "bus" consisting 
of O(lo m) wires if it originates in an area with m inputs. It is not difficult to show that 
this layout uses only linear total wire length. I 
The linear total wire length of this circuit is up to a constant factor optimal for any circuit 
whose output depends on all of its n inputs. Note that most connections in this circuit are 
local, just like in a biological neural circuit. Thus, we see that minimizing total wire length 
tends to generate biology-like circuit structures. 
The next theorem shows that one can compute P) faster (i.e. by a circuit with smaller 
depth) if one can afford a somewhat larger total wire length. This circuit construction, that 
is based on AND/OR gates of limited fan-in A, has the additional advantage that it can not 
just exhibit some pair (i, j) as witness for P)(a,b) = 1 (provided such witness exists), 
but it can exhibit in addition all j that can be used as witness together with some i. This 
property allows us to "chain" the global pattern detection problem formalized through the 
function P), and to decide within the same complexity bound whether for any fixed number 
k of input vectors aO),..., a (k) from {0, 1} ' there exist locations i0),... , i(k) so that 
a  - 1 for m = 1, , k and location i (m+-) lies to the right and above location i  
i(,) -- . . . 
for m = 1,... , k - 1. In fact, one can also compute a k-tuple of witnesses i0),... , i() 
within the same complexity bounds, provided it exists. This circuit design is based on an 
efficient layout for prefix computations. 
Theorem 2.2 For any given n and A E {2,... , v/-} one can compute the function P in 
f)( lg n ' _ 
depth ,  log/x J by a feed-foard circuit consisting of O(n) AND/OR gates offan-in < , 
with total wire length O (n  A  og n 
log A )' I 
Another essential ingredient of translation- and scale-invariant global pattern recognition 
is the capability to detect whether a local feature c occurs in the middle between locations 
i and j where the local features a and b occur. This global pattern detection problem is 
formalized through the following function P]: {0, 1 }an ._> {0, 1}: 
If y a = y b = 1 then P] (a, b, c) = 1, if and only if there exist i, j, k so that input 
location k lies on the middle of the line between locations i and j, and ai = bj = c = 1. 
This function P] can be computed very fast by circuits with the least possible total wire 
length (up to a constant factor), using threshold gates of fan-in up to v/-: 
Theorem 2.3 The function P] can be computed - and witnesses can be exhibited - by a 
circuit with total wire length and area O(n), consisting of O(n) Boolean gates offan-in 2 
and O(v/-) threshold gates offan-in  in depth 7. 
The design of the circuit exploits that the computation of P] can be reduced to the solution 
of two closely related 1-dimensional problems. I 
3 Discussion 
There exists a very large literature on neural circuits for translation-invariant pattern 
recognition see http://www. cnl.salk.eduFwiskott/Bibliographies/Invariances.html. Unfor- 
tunately there exists substantial disagreement regarding the interpretation of existing ap- 
proaches see http://www. ph.tn.tudelft.nl/PRInfo/shift/maillist.html. Virtually all positive 
results are based on computer simulations of small circuits, or on learning algorithms for 
concrete neural networks with a fixed input size n on the order of 20 or 30, without an 
analysis how the required number of gates and the area or volume occupied by wires scale 
up with the input size. The computational performance of these networks is often reported 
in an anecdotical manner. 
The goal of this article is to show that circuit complexity theory may become a useful 
ingredient for understanding the computational strategies of biological neural circuits, and 
for extracting from them portable principles that can be applied to novel artificial circuits 7. 
For that purpose we have introduced the total wire length as an abstract complexity measure 
that appears to be among the most salient ones in this context, and which can in principle 
be applied both to neural circuits in the cortex and to artificial circuitry. We would like to 
argue that only those computational strategies that can be implemented with subquadratic 
total wire length have a chance to reflect aspects of cortical information processing, and 
only those with almost linear total wire length are implementable in special purpose VLSI- 
chips for real-world sensory processing tasks. 8 The relevance of the total wire length of 
cortical circuits has been emphasized by numerous neuroscientists, from Cajal (see for 
example p. 14 in (Cajal, 1995)) to (Chklovskii and Stevens, 2000). On the other hand the 
total wire length of a circuit layout is also closely related to the area required by a VLSI 
implementation of such a circuit (see (Savage, 1998)). 
We have formalized some basic computational problems, that appear to underly various 
translation- and scale-invariant sensory processing tasks, as a first set of benchmark func- 
tions for a circuit complexity theory of sensory processing. We have presented designs for 
circuits that compute these benchmark functions with small - in most cases linear or al- 
most linear - total wire length (and constant factors of moderate size). The computational 
strategies of these circuits differ strongly from those that have been considered in previ- 
ous approaches, which failed to take the limitations imposed by the realistically available 
amount of total wire length into account. 
References 
Abeles, M. (1998). Corticonics: Neural Circuits' of the Cerebral Cortex, Cambridge Univ. Press. 
Braitenberg, V., Schtiz, A. (1998). Cortex: Statistics and Geometry of Neuronal Connectivity, 2nd 
ed., Springer Verlag. 
Cajal, S.R. (1995). Histology of the Nervous System, volumes 1 and 2, Oxford University Press (New 
York). 
Chklovskii, D.B. and Stevens, C.F. (2000). Wiring optimization in the brain. Advances in Neural 
Information Processing Systems vol. 12, MIT Press, 103-107. 
Koch, C. (1999). Biophysics of Computation, Oxford Univ. Press. 
Lazzaro, J., Ryckebusch, S., Mahowald, M. A., Mead, C. A. (1989). Winner-take-all networks of 
O(n) complexity. Advances in Neural Information Processing Systems, vol. 1, Morgan Kauf- 
mann (San Mateo), 703-711. 
Mead, C. and Rem M. (1979). Cost and performance of VLSI computing structures. IEEE J. Solid 
State Circuits' SC- 14(1979), 455-462. 
Mead, C. (1989). Analog VLSI and Neural Systems. Addison-Wesley (Reading, MA, USA). 
Savage, J. E. (1998). Models of Computation: Exploring the Power of Computing. Addison-Wesley 
(Reading, MA, USA). 
Shepherd, G. M. (1998). The Synaptic Organization of the Brain, 2nd ed., Oxford Univ. Press. 
7We do not want to argue that learning plays no role in the design and optimization of circuits 
for specific sensory processing tasks; on the contrary. But one of the few points where the discus- 
sion from http://www.ph.tn.tudelft.nl/PRinfo/shift/maillist.html agreed is that translation- and scale- 
invariant pattern recognition is a task which is so demanding, that learning algorithms have to be 
supported by pre-existing circuit structures. 
aOf course there are other important complexity measures for circuits - such as energy consump- 
tion - besides those that have been addressed in this article. 
