Javadoc preservation for Polyglot extensions
We have implemented Javadoc preservation in JL, JL5 and JL7. This means
that Javadoc comments (for classes, methods, fields, etc) in the source
file will be preserved in the compiler-generated .java
file. Since this implementation will not work out-of-the-box for
Polyglot extensions, we are outlining the steps required to be
performed to support Javadoc preservation in Polyglot extensions.
-
Your extension's lexer will need to capture Javadoc comments and return them to the parser as tokens. This document ships with the patch that was applied to the Jif lexer to achieve this, which should be representative of the changes required. The patch makes the following changes:
-
Add a new lexical state called
JAVADOC_COMMENT
to the existing set of states:@@ -32,7 +32,7 @@ import java.util.Set; %column %char -%state STRING, CHARACTER, TRADITIONAL_COMMENT, END_OF_LINE_COMMENT +%state STRING, CHARACTER, TRADITIONAL_COMMENT, END_OF_LINE_COMMENT, JAVADOC_COMMENT %{ StringBuffer sb = new StringBuffer();
-
Add a
commentBegin
field to the lexer class:@@ -40,6 +40,7 @@ import java.util.Set; String path; ErrorQueue eq; HashMap keywords; + Position commentBegin; public Lexer_c(java.io.InputStream in, Source file, ErrorQueue eq) { this(new java.io.BufferedReader(new java.io.InputStreamReader(in)),
-
Add a convenience method for creating
JavadocToken
s:@@ -264,6 +265,10 @@ import java.util.Set; return new StringLiteral(pos(sb.length()), sb.toString(), sym.STRING_LITERAL); } + + private Token javadoc_token() { + return new JavadocToken(pos(sb.length()), sb.toString(), sym.JAVADOC); + } private String chop(int i, int j) { return yytext().substring(i,yylength()-j);
-
Add a new transition from
<YYINITIAL>
to start capturing Javadoc comments:@@ -329,6 +334,10 @@ OctalEscape = \\ [0-7] /* 3.7 Comments */ "/*" { yybegin(TRADITIONAL_COMMENT); } "//" { yybegin(END_OF_LINE_COMMENT); } + "/**" { yybegin(JAVADOC_COMMENT); + sb.setLength(0); + sb.append(yytext()); + commentBegin = pos(); } /* 3.10.4 Character Literals */ \' { yybegin(CHARACTER); sb.setLength(0); }
-
When done capturing a Javadoc comment, return it to the parser as
a
JavadocToken
:@@ -438,6 +447,18 @@ OctalEscape = \\ [0-7] . { /* ignore */ } } +<JAVADOC_COMMENT> { + "*/" { yybegin(YYINITIAL); + sb.append(yytext()); + return javadoc_token(); } + + <<EOF>> { yybegin(YYINITIAL); + eq.enqueue(ErrorInfo.LEXICAL_ERROR, + "Unclosed Javadoc comment", + commentBegin); } + [^] { sb.append(yytext()); } +} + <CHARACTER> { /* End of the character literal */ \' { yybegin(YYINITIAL);
-
Add a new lexical state called
-
The new interface
Documentable
was created to mark those AST Nodes which can have a Javadoc comment associated with them. The parser attachesJavadoc
comments to theDocumentable
nodes with which they are associated.The following top-level interfaces (and their subclasses) now implement
Documentable
:TopLevelDecl
,ProcedureDecl
,FieldDecl
, andEnumConstantDecl
. This change may mean that some of the AST classes in your extension now implementDocumentable
. For each such classC
in your extension, you will need to modifyC
's constructors, pretty printer, and factory methods, and adjust the parser to compensate.-
Modify
C's
constructor to take in an additionalJavadoc
argument, and hand this object to the superclass constructor in thesuper()
call. -
If
C
overridesprettyPrint()
, then add a call tojavadoc.prettyPrint()
before printing rest of the declaration:if (javadoc != null) javadoc.prettyPrint(...);
-
In your extension's
NodeFactory
, have the factory method forC
take an additionalJavadoc
argument, and hand it to the constructor we modified in Step a. -
Update your parser generator (CUP file) to use the factory method
we just modified in Step c. Use the
BaseParser.javadoc(Position)
method to constructJavadoc
objects by specifying the start position of theDocumentable
node under consideration.
Once
Javadoc
objects are attached to their correspondingDocumentable
nodes, and theDocumentable
classes'prettyPrint()
methods are updated, the Javadoc comment associated with everyDocumentable
node will be added to the compiler-generated .java file during compilation. -
Modify