Information about a token read from the underlying stream. More...

Public Attributes
string	tok
	The token itself. More...

TokenType	type
	The token’s type. More...

size_t	start
	The starting byte of the token in the underlying stream. More...

size_t	line_number
	The line number of the first byte of the token in the underlying stream. More...

size_t	line_start_pos
	The stream position of the start of the line of this token. More...

size_t	curr_pos
	The current position in the underlying stream just after reading this token. More...

Detailed Description

Information about a token read from the underlying stream.

Definition at line 113 of file stream-tokenizer.h.

Member Data Documentation

size_t infact::StreamTokenizer::Token::curr_pos

The current position in the underlying stream just after reading this token.

Definition at line 130 of file stream-tokenizer.h.

size_t infact::StreamTokenizer::Token::line_number

The line number of the first byte of the token in the underlying stream.

Definition at line 125 of file stream-tokenizer.h.

size_t infact::StreamTokenizer::Token::line_start_pos

The stream position of the start of the line of this token.

Definition at line 127 of file stream-tokenizer.h.

size_t infact::StreamTokenizer::Token::start

The starting byte of the token in the underlying stream.

Definition at line 123 of file stream-tokenizer.h.

string infact::StreamTokenizer::Token::tok

The token itself.

Definition at line 115 of file stream-tokenizer.h.

TokenType infact::StreamTokenizer::Token::type

The token’s type.

Definition at line 117 of file stream-tokenizer.h.

The documentation for this struct was generated from the following file: