std/lexbase

  Source   Edit

This module implements a base object of a lexer with efficient buffer handling. Only at line endings checks are necessary if the buffer needs refilling.

Types

BaseLexer = object of RootObj
  bufpos*: int               ## the current position within the buffer
  buf*: string               ## the buffer itself
  input: Stream              ## the input stream
  lineNumber*: int           ## the current line number
  sentinel: int
  lineStart: int
  offsetBase*: int
  refillChars: set[char]
the base lexer. Inherit your lexer from this object.   Source   Edit

Consts

EndOfFile = '\x00'
end of file marker   Source   Edit
NewLines = {'\r', '\n'}
  Source   Edit

Procs

proc close(L: var BaseLexer) {....raises: [Exception, IOError, OSError],
                               tags: [WriteIOEffect].}
closes the base lexer. This closes L's associated stream too.   Source   Edit
proc getColNumber(L: BaseLexer; pos: int): int {....raises: [], tags: [].}
retrieves the current column.   Source   Edit
proc getCurrentLine(L: BaseLexer; marker: bool = true): string {....raises: [],
    tags: [].}
retrieves the current line.   Source   Edit
proc handleCR(L: var BaseLexer; pos: int): int {....raises: [IOError, OSError],
    tags: [ReadIOEffect].}
Call this if you scanned over 'c' in the buffer; it returns the position to continue the scanning from. pos must be the position of the 'c'.   Source   Edit
proc handleLF(L: var BaseLexer; pos: int): int {....raises: [IOError, OSError],
    tags: [ReadIOEffect].}
Call this if you scanned over 'L' in the buffer; it returns the position to continue the scanning from. pos must be the position of the 'L'.   Source   Edit
proc handleRefillChar(L: var BaseLexer; pos: int): int {.
    ...raises: [IOError, OSError], tags: [ReadIOEffect].}
Call this if a terminator character other than a new line is scanned at pos; it returns the position to continue the scanning from.   Source   Edit
proc open(L: var BaseLexer; input: Stream; bufLen: int = 8192;
          refillChars: set[char] = NewLines) {....raises: [IOError, OSError],
    tags: [ReadIOEffect].}
inits the BaseLexer with a stream to read from.   Source   Edit