SaM I/O

(CORE/TOKENIZER.JAVA, CORE/SAMTOKENIZER.JAVA)

The SaM tokenizer was designed to break text into tokens, and make it easier to parse both high level and assembly programs. The tokenizer's functionality is better described at the SaM Javadoc API documentation. However, the following are the various types of tokens produced by these classes:

• Integers: defined as a sequence of digits starting with either a dash or a digit and containing only digits after the first digit.
• Floating Point Numbers: defined as a sequence of digits starting with either a dash, a period, or a digit and containing only digits and one period after the first digit.
• Words: defined as a letter followed by a sequence of alphanumeric characters or underscores (_) without any whitespace. Also, any numbers that have two or more periods.
• Strings: anything between two quotation signs. Valid escapes (\n, \r. \\, \", \', \xxx, etc...) are evaluated in this case. This is only enabled if the correct option is passed to SamTokenizer.
• Characters: a single character betweeen two apostrophes. Valid escapes (\n, \r. \\, \", \', \xxx, etc...) are evaluated in this case. This is only enabled if the correct option is passed to SamTokenizer.
• Comments: any text on the same line after //. This is only enabled if the correct options is passed to SamTokenizer.
• Operators: defined as any non alphanumeric character and represented by the java type char.
• EOF: Such a token represents that the end of the stream has been reached and that there are no more tokens.