Lexical analysis and recognition

[Next] [Up] [Previous]
Next: Constructing the quasi-prefix Up: Constructing high-level representations Previous: Constructing high-level representations

Lexical analysis and recognition

  Lisp-CLOS was chosen to implement AsTeR because of its powerful development environment and object-oriented features. However, Lisp-CLOS lacks tools such as lexical analyzers and parser generators, e.g., LEX and YACC. As a convenient way of getting the best of both worlds, we designed a lexical analyzer called lispify in LEX that outputs the input La)TeX source in a canonical list representation. This list is then read in by a recursive descent parser written in Lisp. The general form of this list is [tex2html_wrap5372], where token identifies the type of content encapsulated by the list and body represents the content. The recognizer returns a document object that encapsulates the document instance being recognized. For example, given the La)TeX input


LISPIFY produces


LISPIFY handles all of La)TeX concrete syntax.

The recursive descent parser examines the token at the front of the input list and calls a token-specific processing function on the rest of the list. Thus, given the input (token body), the recognizer executes


The technique described so far is sufficient for handling sections, enumerated lists and other textual content.

TV Raman
Thu Mar 9 20:10:41 EST 1995