Javadoc preservation for Polyglot extensions

We have implemented Javadoc preservation in JL, JL5 and JL7. This means that Javadoc comments (for classes, methods, fields, etc) in the source file will be preserved in the compiler-generated .java file. Since this implementation will not work out-of-the-box for Polyglot extensions, we are outlining the steps required to be performed to support Javadoc preservation in Polyglot extensions.

  1. Your extension's lexer will need to capture Javadoc comments and return them to the parser as tokens. This document ships with the patch that was applied to the Jif lexer to achieve this, which should be representative of the changes required. The patch makes the following changes:

    1. Add a new lexical state called JAVADOC_COMMENT to the existing set of states:
        @@ -32,7 +32,7 @@ import java.util.Set;
         %column
         %char
         
        -%state STRING, CHARACTER, TRADITIONAL_COMMENT, END_OF_LINE_COMMENT
        +%state STRING, CHARACTER, TRADITIONAL_COMMENT, END_OF_LINE_COMMENT, JAVADOC_COMMENT
         
         %{
             StringBuffer sb = new StringBuffer();
                    
    2. Add a commentBegin field to the lexer class:
        @@ -40,6 +40,7 @@ import java.util.Set;
             String path;
             ErrorQueue eq;
             HashMap keywords;
        +    Position commentBegin;
         
             public Lexer_c(java.io.InputStream in, Source file, ErrorQueue eq) {
                 this(new java.io.BufferedReader(new java.io.InputStreamReader(in)),
                    
    3. Add a convenience method for creating JavadocTokens:
        @@ -264,6 +265,10 @@ import java.util.Set;
                 return new StringLiteral(pos(sb.length()), sb.toString(),
                                          sym.STRING_LITERAL);
             }
        +	
        +	private Token javadoc_token() {
        +		return new JavadocToken(pos(sb.length()), sb.toString(), sym.JAVADOC);
        +    }
         
             private String chop(int i, int j) {
                 return yytext().substring(i,yylength()-j);
                    
    4. Add a new transition from <YYINITIAL> to start capturing Javadoc comments:
        @@ -329,6 +334,10 @@ OctalEscape = \\ [0-7]
             /* 3.7 Comments */
             "/*"    { yybegin(TRADITIONAL_COMMENT); }
             "//"    { yybegin(END_OF_LINE_COMMENT); }
        +    "/**"   { yybegin(JAVADOC_COMMENT);
        +              sb.setLength(0);
        +              sb.append(yytext());
        +              commentBegin = pos(); }
         
             /* 3.10.4 Character Literals */
             \'      { yybegin(CHARACTER); sb.setLength(0); }
                    
    5. When done capturing a Javadoc comment, return it to the parser as a JavadocToken:
        @@ -438,6 +447,18 @@ OctalEscape = \\ [0-7]
             .                            { /* ignore */ }
         }
         
        +<JAVADOC_COMMENT> {
        +    "*/"                         { yybegin(YYINITIAL);
        +    							   sb.append(yytext()); 
        +    							   return javadoc_token(); }
        +
        +    <<EOF>>                      { yybegin(YYINITIAL);
        +                                   eq.enqueue(ErrorInfo.LEXICAL_ERROR,
        +                                                  "Unclosed Javadoc comment",
        +                                                  commentBegin); }
        +    [^]                          { sb.append(yytext()); }
        +}
        +
         <CHARACTER> {
             /* End of the character literal */
             \'                           { yybegin(YYINITIAL);
                    
  2. The new interface Documentable was created to mark those AST Nodes which can have a Javadoc comment associated with them. The parser attaches Javadoc comments to the Documentable nodes with which they are associated.

    The following top-level interfaces (and their subclasses) now implement Documentable: TopLevelDecl, ProcedureDecl, FieldDecl, and EnumConstantDecl. This change may mean that some of the AST classes in your extension now implement Documentable. For each such class C in your extension, you will need to modify C's constructors, pretty printer, and factory methods, and adjust the parser to compensate.

    1. Modify C's constructor to take in an additional Javadoc argument, and hand this object to the superclass constructor in the super() call.
    2. If C overrides prettyPrint(), then add a call to javadoc.prettyPrint() before printing rest of the declaration:
        if (javadoc != null) javadoc.prettyPrint(...);
    3. In your extension's NodeFactory, have the factory method for C take an additional Javadoc argument, and hand it to the constructor we modified in Step a.
    4. Update your parser generator (CUP file) to use the factory method we just modified in Step c. Use the BaseParser.javadoc(Position) method to construct Javadoc objects by specifying the start position of the Documentable node under consideration.

    Once Javadoc objects are attached to their corresponding Documentable nodes, and the Documentable classes' prettyPrint() methods are updated, the Javadoc comment associated with every Documentable node will be added to the compiler-generated .java file during compilation.