Tokenizer (parsii 4.0 API)

java.lang.Object
- parsii.tokenizer.Lookahead<Token>
- - parsii.tokenizer.Tokenizer

```
public class Tokenizer
extends Lookahead<Token>
```
Turns a stream of characters (Reader into a stream of Token, supporting lookahead.
Reads from the given input and parses it into a stream of tokens. By default all token types defined by Token are supported. Most of the features can be further tweaked by changing the default settings.
By default the tokenizer operates as follows:
- Consume and ignore any whitespace characters (see Char.isWhitepace()
- If the current character starts a line comment, read until the end of the line and ignore all characters consumed.
- If the current character starts a block comment, read until and end of block comment is detected.
- If the current character is a digit, parse a INTEGER, if a decimal separator is found, switch over to a DECIMAL (see Char.isDigit(). Also if the current character is a '-' and the next is a digit, we try to read a number.
- If the current character is a letter, parse an ID (see Char.isLetter(). Once this is complete, check if the ID matches one of the supplied keywords, and convert if necessary.
- If the current character is an opening or closing bracket, a SYMBOL for that single character is returned
- If the current character is one of the special id starters, all valid ID characters (isIdentifierChar(Char) are consumed and returned as SPECIAL_ID
- All other characters, especially all operators, will be read and returned as one SYMBOL. Therefore #++* will be returned as a single symbol.

Field Summary

Fields
Modifier and Type Field and Description

protected LookaheadReader input
- Fields inherited from class parsii.tokenizer.Lookahead
  endOfInputIndicator, endReached, itemBuffer, problemCollector

Fields
Modifier and Type	Field and Description
`protected LookaheadReader`	`input`

Constructor Summary

Constructors
Constructor and Description

Tokenizer(Reader input)
Creates a new tokenizer for the given input

Constructors
Constructor and Description
`Tokenizer(Reader input)` Creates a new tokenizer for the given input

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addError(Position pos, String message, Object... parameters)` Adds a parse error to the internal problem collector.
`void`	`addKeyword(String keyword)` Adds a keyword which is now being recognized by the tokenizer Detection will be case insensitive.
`void`	`addSpecialIdStarter(char character)` Adds character as a special id starter.
`void`	`addSpecialIdTerminator(char character)` Adds character as a special id terminator.
`void`	`addStringDelimiter(char stringDelimiter, char escapeCharacter)` Adds a new string delimiter character along with the character used to escape string within it.
`void`	`addUnescapedStringDelimiter(char stringDelimiter)` Boilerplate method for adding a string delimiter which does not support escape sequences.
`void`	`addWarning(Position pos, String message, Object... parameters)` Adds a warning to the internal problem collector.
`boolean`	`atEnd()` Boilerplate method for `current().isEnd()`
`protected boolean`	`canConsumeThisString(String string, boolean consume)` Checks if the next characters, starting from the current, match the given string.
`void`	`clearStringDelimiters()` Removes all previously registered string delimiters.
`void`	`consumeExpectedKeyword(String keyword)` Consumes the current token, expecting it to be as `KEYWORD` with the given content
`void`	`consumeExpectedSymbol(String symbol)` Consumes the current token, expecting it to be as `SYMBOL` with the given content
`protected Token`	`endOfInput()` Creates the end of input indicator item.
`protected Token`	`fetch()` Fetches the next item from the stream.
`protected Token`	`fetchId()` Reads and returns an identifier
`protected Token`	`fetchNumber()` Reads and returns a number.
`protected Token`	`fetchSpecialId()` Reads and returns a special id.
`protected Token`	`fetchString()` Reads and returns a string constant.
`protected Token`	`fetchSymbol()` Reads and returns a symbol.
`String`	`getBlockCommentEnd()` Returns the string which ends a block comment.
`String`	`getBlockCommentStart()` Returns the string which starts a block comment.
`char`	`getDecimalSeparator()` Returns the decimal separator used in decimal numbers The default separator used is '.'
`char`	`getEffectiveDecimalSeparator()` Returns the decimal separator used in the content of DECIMAL tokens.
`char`	`getGroupingSeparator()` Returns the grouping separator which can be used in numbers for group digits (e.g. in thousands).
`String`	`getLineComment()` Returns the string which starts a line comment.
`protected Token`	`handleKeywords(Token idToken)` Checks if the given identifier is a keyword and returns an appropriate Token
`protected boolean`	`handleStringEscape(char separator, char escapeChar, Token stringToken)` Evaluates an string escape like \n The escape character is already consumed.
`protected boolean`	`isAtBracket(boolean inSymbol)` Determines if the underlying input is looking at a bracket.
`protected boolean`	`isAtEndOfBlockComment()` Checks if the underlying input is looking at a end of block comment If an end of block comment is detected, any characters indicating this are consumed by this method
`protected boolean`	`isAtStartOfBlockComment(boolean consume)` Checks if the underlying input is looking at a start of block comment If a block comment is detected, any characters indicating this are consumed by this method if `consume` is `true` .
`protected boolean`	`isAtStartOfIdentifier()` Determines if the underlying input is looking at a valid character to start an identifier By default, only letters can start identifiers
`protected boolean`	`isAtStartOfLineComment(boolean consume)` Checks if the underlying input is looking at a start of line comment.
`protected boolean`	`isAtStartOfNumber()` Determines if the underlying input is looking at the start of a number.
`protected boolean`	`isAtStartOfSpecialId()` Determines if the underlying input is looking at the start of a special id.
`protected boolean`	`isIdentifierChar(Char current)` Determines if the given Char is a valid identifier part.
`boolean`	`isKeywordsCaseSensitive()` Determines if keywords are case sensitive.
`protected boolean`	`isSymbolCharacter(Char ch)` Determines if the given Char is a symbol character.
`boolean`	`more()` Boilerplate method for `current().isNotEnd()`
`void`	`setBlockCommentEnd(String blockCommentEnd)` Sets the string which ends a block comment.
`void`	`setBlockCommentStart(String blockCommentStart)` Sets the string which stats a block comment.
`void`	`setDecimalSeparator(char decimalSeparator)` Sets the character which is recognized as decimal separator.
`void`	`setEffectiveDecimalSeparator(char effectiveDecimalSeparator)` Sets the decimal separator used in the content of DECIMAL tokens.
`void`	`setGroupingSeparator(char groupingSeparator)` Sets the grouping separator accepting in numbers.
`void`	`setKeywordsCaseSensitive(boolean keywordsCaseSensitive)` Sets the case sensitiveness of keywords.
`void`	`setLineComment(String lineComment)` Sets the string which stats a line comment.
`void`	`setProblemCollector(List<ParseError> problemCollector)` Installs the given problem collector.
`protected void`	`skipBlockComment()` Checks if we're looking at an end of block comment
`protected void`	`skipToEndOfLine()` Read everything upon (and including) the next line break
`void`	`throwOnError()` Throws a `ParseException` if an error occurred while parsing the input.
`void`	`throwOnErrorOrWarning()` Throws a `ParseException` if an error or warning occurred while parsing the input
`String`	`toString()`

Methods inherited from class parsii.tokenizer.Lookahead
consume, consume, current, getProblemCollector, next, next

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - input
```
protected LookaheadReader input
```
- Constructor Detail
  - Tokenizer
```
public Tokenizer(Reader input)
```
    Creates a new tokenizer for the given input
    
    Parameters:
    
    input - the input to parse. The reader will be buffered by the implementation so that it can be effectively read character b character.
- Method Detail
  - setProblemCollector
```
public void setProblemCollector(List<ParseError> problemCollector)
```
    Description copied from class: Lookahead
    
    Installs the given problem collector.
    
    Overrides:
    
    setProblemCollector in class Lookahead<Token>
    
    Parameters:
    
    problemCollector - the problem collector to be from now on
  - endOfInput
```
protected Token endOfInput()
```
    Description copied from class: Lookahead
    
    Creates the end of input indicator item.
    This method will be only called once, as the indicator is cached.
    
    Specified by:
    
    endOfInput in class Lookahead<Token>
    
    Returns:
    
    a special item which marks the end of the input
  - fetch
```
protected Token fetch()
```
    Description copied from class: Lookahead
    
    Fetches the next item from the stream.
    
    Specified by:
    
    fetch in class Lookahead<Token>
    
    Returns:
    
    the next item in the stream or null to indicate that the end was reached
  - isAtStartOfSpecialId
```
protected boolean isAtStartOfSpecialId()
```
    Determines if the underlying input is looking at the start of a special id.
    By default this is one of the given specialIdStarters.
    
    Returns:
    
    true if the current input is the start of a special id, false otherwise
  - isAtStartOfNumber
```
protected boolean isAtStartOfNumber()
```
    Determines if the underlying input is looking at the start of a number.
    By default this is either indicated by a digit or by '-' followed by a digit or a '.' followed by a digit.
    
    Returns:
    
    true if the current input is the start of a numeric constant, false otherwise
  - isAtBracket
```
protected boolean isAtBracket(boolean inSymbol)
```
    Determines if the underlying input is looking at a bracket.
    By default all supplied brackets are checked. If treatSinglePipeAsBracket is true, a single '|' is also treated as bracket.
    
    Parameters:
    
    inSymbol - determines if we're already parsing a symbol or just trying to decide what the next token is
    
    Returns:
    
    true if the current input is an opening or closing bracket
  - canConsumeThisString
```
protected boolean canConsumeThisString(String string,
                                       boolean consume)
```
    Checks if the next characters, starting from the current, match the given string.
    
    Parameters:
    
    string - the string to check
    
    consume - determines if the matched string should be consumed immediately
    
    Returns:
    
    true if the next characters of the input match the given string, false otherwise
  - isAtStartOfLineComment
```
protected boolean isAtStartOfLineComment(boolean consume)
```
    Checks if the underlying input is looking at a start of line comment.
    If a line comment is detected, any characters indicating this are consumed by this method if consume is true.
    
    Parameters:
    
    consume - determines if the matched comment start should be consumed immediately
    
    Returns:
    
    true if the next character(s) of the input start a line comment, false otherwise
  - skipToEndOfLine
```
protected void skipToEndOfLine()
```
    Read everything upon (and including) the next line break
  - isAtStartOfBlockComment
```
protected boolean isAtStartOfBlockComment(boolean consume)
```
    Checks if the underlying input is looking at a start of block comment
    If a block comment is detected, any characters indicating this are consumed by this method if consume is true .
    
    Parameters:
    
    consume - determines if the block comment starter is to be consumed if found or not
    
    Returns:
    
    true if the next character(s) of the input start a block comment, false otherwise
  - isAtEndOfBlockComment
```
protected boolean isAtEndOfBlockComment()
```
    Checks if the underlying input is looking at a end of block comment
    If an end of block comment is detected, any characters indicating this are consumed by this method
    
    Returns:
    
    true if the next character(s) of the input end a block comment, false otherwise
  - skipBlockComment
```
protected void skipBlockComment()
```
    Checks if we're looking at an end of block comment
  - fetchString
```
protected Token fetchString()
```
    Reads and returns a string constant.
    
    Returns:
    
    the parsed string constant a Token
  - handleStringEscape
```
protected boolean handleStringEscape(char separator,
                                     char escapeChar,
                                     Token stringToken)
```
    Evaluates an string escape like \n
    The escape character is already consumed. Therefore the input points at the character to escape. This method must consume all escaped characters.
    
    Parameters:
    
    separator - the delimiter of this string constant
    
    escapeChar - the escape character used
    
    stringToken - the resulting string constant
    
    Returns:
    
    true if an escape was possible, false otherwise
  - isAtStartOfIdentifier
```
protected boolean isAtStartOfIdentifier()
```
    Determines if the underlying input is looking at a valid character to start an identifier
    By default, only letters can start identifiers
    
    Returns:
    
    true if the underlying input is looking at a valid identifier starter, false otherwise
  - fetchId
```
protected Token fetchId()
```
    Reads and returns an identifier
    
    Returns:
    
    the parsed identifier as Token
  - handleKeywords
```
protected Token handleKeywords(Token idToken)
```
    Checks if the given identifier is a keyword and returns an appropriate Token
    
    Parameters:
    
    idToken - the identifier to check
    
    Returns:
    
    a keyword Token if the given identifier was a keyword, the original Token otherwise
  - isIdentifierChar
```
protected boolean isIdentifierChar(Char current)
```
    Determines if the given Char is a valid identifier part.
    By default, letters, digits and '_' are valid identifier parts.
    
    Parameters:
    
    current - the character to check
    
    Returns:
    
    true if the given Char is a valid identifier part, false otherwise
  - fetchSpecialId
```
protected Token fetchSpecialId()
```
    Reads and returns a special id.
    
    Returns:
    
    the parsed special id as Token
  - fetchSymbol
```
protected Token fetchSymbol()
```
    Reads and returns a symbol.
    A symbol are one or two characters, which don't match any other token type. In most cases, this will be operators like + or *.
    
    Returns:
    
    the parsed symbol as Token
  - isSymbolCharacter
```
protected boolean isSymbolCharacter(Char ch)
```
    Determines if the given Char is a symbol character.
    By default these are all non-control characters, which don't match any other class (letter, digit, whitepsace)
    
    Parameters:
    
    ch - the character to check
    
    Returns:
    
    true if the given character is a valid symbol character, false otherwise
  - fetchNumber
```
protected Token fetchNumber()
```
    Reads and returns a number.
    
    Returns:
    
    the parsed number as Token
  - isKeywordsCaseSensitive
```
public boolean isKeywordsCaseSensitive()
```
    Determines if keywords are case sensitive.
    By default, keywords aren't case sensitive. Therefore True and true are the same keyword.
    
    Returns:
    
    true if keywords are case sensitive, false otherwise
  - setKeywordsCaseSensitive
```
public void setKeywordsCaseSensitive(boolean keywordsCaseSensitive)
```
    Sets the case sensitiveness of keywords.
    This must be setup before any call to addKeyword(String) as this will determine internal data structures
    
    Parameters:
    
    keywordsCaseSensitive - true if keywords should be treated as case sensitive, false otherwise (default)
  - addKeyword
```
public void addKeyword(String keyword)
```
    Adds a keyword which is now being recognized by the tokenizer
    Detection will be case insensitive. Only ID tokens (identifiers) are checked against the given keywords, therefore a keyword must be a valid identifier.
    
    Parameters:
    
    keyword - the keyword to be added to the list of known keywords.
  - addSpecialIdStarter
```
public void addSpecialIdStarter(char character)
```
    Adds character as a special id starter. The given character must be followed by a valid id to be recognized as SPECIAL_ID.
    
    Parameters:
    
    character - the character to be added as special id starter
  - addSpecialIdTerminator
```
public void addSpecialIdTerminator(char character)
```
    Adds character as a special id terminator. The given character can be placed at the end of an id to be recognized as SPECIAL_ID.
    
    Parameters:
    
    character - the character to be added as special id terminator
  - clearStringDelimiters
```
public void clearStringDelimiters()
```
    Removes all previously registered string delimiters.
    By default " and ' are registered as string delimiters, where string enclosed by " can have characters escaped by \
  - addStringDelimiter
```
public void addStringDelimiter(char stringDelimiter,
                               char escapeCharacter)
```
    Adds a new string delimiter character along with the character used to escape string within it.
    
    Parameters:
    
    stringDelimiter - the delimiter used to start and end string constants
    
    escapeCharacter - the character used to start an escape sequence or \0 to indicate that escaping is not supported
  - addUnescapedStringDelimiter
```
public void addUnescapedStringDelimiter(char stringDelimiter)
```
    Boilerplate method for adding a string delimiter which does not support escape sequences.
    
    Parameters:
    
    stringDelimiter - the delimiter used to start and end string constants
  - getDecimalSeparator
```
public char getDecimalSeparator()
```
    Returns the decimal separator used in decimal numbers
    The default separator used is '.'
    
    Returns:
    
    the character which is detected as separator between the integer and the decimal part of a number
  - setDecimalSeparator
```
public void setDecimalSeparator(char decimalSeparator)
```
    Sets the character which is recognized as decimal separator.
    
    Parameters:
    
    decimalSeparator - the character to be recognized as decimal separator
  - getEffectiveDecimalSeparator
```
public char getEffectiveDecimalSeparator()
```
    Returns the decimal separator used in the content of DECIMAL tokens.
    The default separator used is '.'. When adapting this for language dependent inputs (e.g. using ',' as decimal separator) this value should probably not be changed, as it is used in the output (content) of the Tokens and has no effect what kind of numbers are being accepted.
    
    Returns:
    
    the decimal separator used in tokens
  - setEffectiveDecimalSeparator
```
public void setEffectiveDecimalSeparator(char effectiveDecimalSeparator)
```
    Sets the decimal separator used in the content of DECIMAL tokens. This can differ from the character set via setDecimalSeparator(char) which is used to recognize decimal numbers. Therefore language dependent input can be parsed with a constant output being language independent.
    
    Parameters:
    
    effectiveDecimalSeparator - the character used as decimal separator in the content of decimal tokens
  - getGroupingSeparator
```
public char getGroupingSeparator()
```
    Returns the grouping separator which can be used in numbers for group digits (e.g. in thousands).
    This character will be accepted in numbers, but ignored (not added to the content). The default value is '_'.
    
    Returns:
    
    the grouping separator accepting in numbers
  - setGroupingSeparator
```
public void setGroupingSeparator(char groupingSeparator)
```
    Sets the grouping separator accepting in numbers.
    
    Parameters:
    
    groupingSeparator - the character which can be used to group digits in numbers
  - getLineComment
```
public String getLineComment()
```
    Returns the string which starts a line comment.
    The default value is '/''/'
    
    Returns:
    
    the string sequence starting a line comment
  - setLineComment
```
public void setLineComment(String lineComment)
```
    Sets the string which stats a line comment.
    
    Parameters:
    
    lineComment - the string used to detect a line comment
  - getBlockCommentStart
```
public String getBlockCommentStart()
```
    Returns the string which starts a block comment.
    The default value is '/''*'
    
    Returns:
    
    the string sequence starting a block comment
  - setBlockCommentStart
```
public void setBlockCommentStart(String blockCommentStart)
```
    Sets the string which stats a block comment.
    
    Parameters:
    
    blockCommentStart - the string used to detect a block comment
  - getBlockCommentEnd
```
public String getBlockCommentEnd()
```
    Returns the string which ends a block comment.
    The default value is '*''/'
    
    Returns:
    
    the string sequence ending a block comment
  - setBlockCommentEnd
```
public void setBlockCommentEnd(String blockCommentEnd)
```
    Sets the string which ends a block comment.
    
    Parameters:
    
    blockCommentEnd - the string used to detect the end of a block comment
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - more
```
public boolean more()
```
    Boilerplate method for current().isNotEnd()
    
    Returns:
    
    true if the current token is not an "end of input" token, false otherwise.
  - atEnd
```
public boolean atEnd()
```
    Boilerplate method for current().isEnd()
    
    Returns:
    
    true if the current token is an "end of input" token, false otherwise.
  - addError
```
public void addError(Position pos,
                     String message,
                     Object... parameters)
```
    Adds a parse error to the internal problem collector.
    It is preferred to collect as much errors as possible and then fail with an exception instead of failing at the first problem. Often syntax errors can be worked out by the parser and we can report a set of errors at once.
    
    Parameters:
    
    pos - the position of the error. Note that Token implements Position. Therefore the current token is often a good choice for this parameter.
    
    message - the message to describe the error. Can contain formatting parameters like %s or %d as defined by String.format(String, Object...)
    
    parameters - Contains the parameters used to format the given message
  - addWarning
```
public void addWarning(Position pos,
                       String message,
                       Object... parameters)
```
    Adds a warning to the internal problem collector.
    A warning indicates an anomaly which might lead to an error but still, the parser can continue to complete its work.
    
    Parameters:
    
    pos - the position of the warning. Note that Token implements Position. Therefore the current token is often a good choice for this parameter.
    
    message - the message to describe the earning. Can contain formatting parameters like %s or %d as defined by String.format(String, Object...)
    
    parameters - Contains the parameters used to format the given message
  - consumeExpectedSymbol
```
public void consumeExpectedSymbol(String symbol)
```
    Consumes the current token, expecting it to be as SYMBOL with the given content
    
    Parameters:
    
    symbol - the expected trigger of the current token
  - consumeExpectedKeyword
```
public void consumeExpectedKeyword(String keyword)
```
    Consumes the current token, expecting it to be as KEYWORD with the given content
    
    Parameters:
    
    keyword - the expected content of the current token
  - throwOnErrorOrWarning
```
public void throwOnErrorOrWarning()
                           throws ParseException
```
    Throws a ParseException if an error or warning occurred while parsing the input
    
    Throws:
    
    ParseException - if an error or warning occurred while parsing.
  - throwOnError
```
public void throwOnError()
                  throws ParseException
```
    Throws a ParseException if an error occurred while parsing the input.
    All warnings which occurred will be ignored.
    
    Throws:
    
    ParseException - if an error occurred while parsing.

Class Tokenizer

Field Summary

Fields inherited from class parsii.tokenizer.Lookahead

Constructor Summary

Method Summary

Methods inherited from class parsii.tokenizer.Lookahead

Methods inherited from class java.lang.Object

Field Detail

input

Constructor Detail

Tokenizer

Method Detail

setProblemCollector

endOfInput

fetch

isAtStartOfSpecialId

isAtStartOfNumber

isAtBracket

canConsumeThisString

isAtStartOfLineComment

skipToEndOfLine

isAtStartOfBlockComment

isAtEndOfBlockComment

skipBlockComment

fetchString

handleStringEscape

isAtStartOfIdentifier

fetchId

handleKeywords

isIdentifierChar

fetchSpecialId

fetchSymbol

isSymbolCharacter

fetchNumber

isKeywordsCaseSensitive

setKeywordsCaseSensitive

addKeyword

addSpecialIdStarter

addSpecialIdTerminator

clearStringDelimiters

addStringDelimiter

addUnescapedStringDelimiter

getDecimalSeparator

setDecimalSeparator

getEffectiveDecimalSeparator

setEffectiveDecimalSeparator

getGroupingSeparator

setGroupingSeparator

getLineComment

setLineComment

getBlockCommentStart

setBlockCommentStart

getBlockCommentEnd

setBlockCommentEnd

toString

more

atEnd

addError

addWarning

consumeExpectedSymbol

consumeExpectedKeyword

throwOnErrorOrWarning

throwOnError