+ +

Audio System for

Technical Readings

T. V. Raman

Cornell University DEC Cambridge Research Lab

Thesis Adviser: D. Gries

+ +

What is formatting?

Information has structure.

• Mathematical expressions. e^{e^{ex}}e^{e^{x}}e^{x}

• Tables.

1 1 1 1 1 1

Formatted output conveys this structure.

+ +

ASTER Audio System for Technical Readings

The next-generation reading machine. Information is display independent.

(L^{A})TEX → Print

Electronic Documents  {↗ \atop ↘}

ASTER → Audio

Different views of the same document.

+ +

Overview

Overcome the passive nature of listening.

• Internalize document structure.

• Render structure in audio.

• Browse information structure.

+ +

Architecture

Electronic documents

⇓ Recognizer ⇓ High-level model

⇓ Audio Formatting ⇓

Audio Rendering

+ Rendering Rules

+ Browsing

=

Audio Documents

+ +

Recognition Is Hard

• A document is not a computer program.

• Written math is ambiguous.

sin 2x = 2 sin x cos x

• Standard parsing techniques can fail. A = {x| x^{2} ∈ (0, 1]}

D = {x| 1 ≤ x ∧ x|y}

+ +

Recognition of structure

High-level models:

• Tree structure for text.

• Quasi-prefix form for mathematics. A node in the quasi-prefix form:

left-sup  accent  superscript
↖  ↑  ↗

math object

↙  ↓  ↘
left-sub  underbar  subscript

Captures ambiguous written mathematics.

+ +

Example: Article

(article
(title <text>)
(author <text>)
(abstract <text>)
(section (title <section title>)
         <section body>
         (subsection ...)
         ...)
<sections>
(references <references>)
)

+ +

Example: Quasi-prefix form

D_{x}^{1}

is represented as :

[MATH
CONTENTS: "D"
CHILDREN: NIL
ATTR: ([SUBSCRIPT
VALUE: [MATH
CONTENTS: "x"
CHILDREN: NIL
ATTR: NIL ]]
[SUPERSCRIPT
VALUE: [NUMBER
CONTENTS: "1"
ATTR: NIL]])
]

+ +

Extending the recognizer

(A ⊗ B)^{T} = A^{T} ⊗ B^{T}

\def\kronecker{\raisebox{1pt}{$ \:\otimes \:$}}

Recognizer extended by :

(define-text-object     :macro-name "kronecker"
:number-args 0
:processing-function kronecker-expand
:precedence  multiplication
:object-name kronecker-product
:super-classes  (binary-operator))

            A \kronecker B

⇓

Recognizer

⇓

(kronecker-product :children( A′ B′))

where A ⟶^{recognizer} A′.

+ +

Defining the audio space

Audio renderings: speech ⊕ sound . Formatter is a Conceptual audio device.

• Formatter parameters define audio space.

• Formatter state is a point in this space.

• Formatter has multiple components.

• Components correspond to subspaces.

• Space = direct sum of subspaces. Changing formatter state ↔ Moving in audio space.

+ +

Audio Formatting Language

AFL is a block structured language. AFL variables encapsulate formatter state. Operators generate points in audio space.

• NEW-BLOCK Introduce a local scope.

• LOCAL-SET-STATE Assignment.

• GLOBAL-SET-STATE Assignment.

+ +

AFL Blocks

Create local copies of all component states. Effect of local assignments seen only within the closest enclosing block. Allows rendering rules to call one another. AFL block is an implicit cobegin statement.

+ +

Component spaces

Dimensions in the speech space: Average-pitch, smoothness, directionality, … Operators: move-by, move-to, step-by, … Dimensions in the sound space: volume, sound, … Operators: select-sound, switch-on, …

+ +

Producing audio renderings

Recognizer \underrightarrow{AFL Rules}

Audio Renderings

A typical rendering rule:

BEGIN-AFL-BLOCK
        set local state;
        render object
END-AFL-BLOCK

Nested objects rendered relative to parent.

+ +

Mathematical notation

Written notation:

• Linearizes implicit tree structure.

• Annotates this with attributes. A parallel audio notation: Independent audio dimensions cue features. Pick a monotonic change of state function f to vary audio state P along these dimensions, where the ordering P < f(P) < f^{2}(P) < ⋯ is perceptible. This is sufficient to cue nested structures.

+ +

Superscripts and subscripts

x^{k_{1}} + x^{k_{2}} + x^{k_{3}} + ⋯ + x^{k_{n}} = 0 x_{k^{1}} + x_{k^{2}} + x_{k^{3}} + ⋯ + x_{k^{n}} = 0

e^{e^{ex}}e^{e^{x}}e^{x}

+ +

Tree structure

Move along a line in speech space spanned by six dimensions.

\frac{a + b}{c + d}

\frac{a}{b + c + d}

a + \frac{b + c}{d + e} + x

Can cue deeply nested structures.

1 + \frac{x}{1+\frac{x}{1+\frac{x}{1+\frac{x}{1+\frac{x}{1+...}}}}}

+ +

Complex notation

Features can be combined: Tree-structure within superscript.

x + y^{\frac{2}{k+1}}

Superscript within tree-structure.

√e^{e^{ex}} + √e^{e^{x}} + √e^{x}

This system of audio notation scales very well.

+ +

Stereo Effects

Directionality is an audio dimension. Moving along it produces stereo.

Let L be the line spanned by the dimensions left-volume and right-volume.

compute-step-size(length(row),L);
BEGIN-AFL-BLOCK
        set speech state  to left speaker;
        LOOP for  element in row do
                render  element ;
                step by 1 along L
        ENDLOOP
END-AFL-BLOCK

A = ( \begin{matrix}a_{11} & a_{12} & … & a_{1n} \\ a_{21} & a_{22} & … & a_{2n} \\ ⋮ & ⋮ & ... & ⋮ \\ a_{m1} & a_{m2} & … & a_{mn} \end{matrix} )

+ +

Classification of audio cues

Speech and non-speech audio cues. Also classified as:

• Fleeting cue Does not last.

• Persistent cue Persists across time. Fleeting cues identify objects. Persistent cues convey environment.

+ +

Rendering Styles

Rendering rules with same name (for different objects) form a style. List of active styles: (special-patterns descriptive simple default)

Alpha-equality commutes with

substitution. If t =_{α} t′ and s =_{α} s′, then

t[s/x] =_{α} t′[s′/x]

. Active rule overrides active style. Activate rules to customize styles.

+ +

Performance

Audio formats both complete and partial text books, technical reports and AMS Bulletins in real time. Text books include:

• Nonlinear Optimization: S. Vavasis.

• Matrix Computation: Golub, Van Loan.

• Analysis of Algorithms: Kozen.

• Discrete Mathematics: Gries, Schneider.

• Commutative Ring Theory: Kaplansky.

+ +

Contributions

• Enable immediate availability of technical documents in audio.

• Internalize written mathematics.

• Render such structure orally.

• Enable active listening.

+ +

Next Step

• Start mass-producing audio documents on cassette.

• Port ASTER to other platforms. Research:

• An editing tool for technical documents.