Lexical analysis and recognition



[Next] [Up] [Previous]
Next: Constructing the quasi-prefix Up: Constructing high-level representations Previous: Constructing high-level representations

Lexical analysis and recognition

  Lisp-CLOS was chosen to implement AsTeR because of its powerful development environment and object-oriented features. However, Lisp-CLOS lacks tools such as lexical analyzers and parser generators, e.g., LEX and YACC. As a convenient way of getting the best of both worlds, we designed a lexical analyzer called lispify in LEX that outputs the input La)TeX source in a canonical list representation. This list is then read in by a recursive descent parser written in Lisp. The general form of this list is [tex2html_wrap5372], where token identifies the type of content encapsulated by the list and body represents the content. The recognizer returns a document object that encapsulates the document instance being recognized. For example, given the La)TeX input

[LVerbatim556]

LISPIFY produces

[LVerbatim561]

LISPIFY handles all of La)TeX concrete syntax.

The recursive descent parser examines the token at the front of the input list and calls a token-specific processing function on the rest of the list. Thus, given the input (token body), the recognizer executes

[displaymath5370]

The technique described so far is sufficient for handling sections, enumerated lists and other textual content.


TV Raman
Thu Mar 9 20:10:41 EST 1995