|
Reranker Framework (ReFr)
Reranking framework for structure prediction and discriminative language modeling
|
All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Pages
|
Reranker Framework (ReFr)
Reranking framework for structure prediction and discriminative language modeling
|
Information about a token read from the underlying stream. More...
#include <stream-tokenizer.H>
Public Attributes | |
| string | tok |
| The token itself. More... | |
| TokenType | type |
| The token’s type. More... | |
| size_t | start |
| The starting byte of the token in the underlying stream. More... | |
| size_t | line_number |
| The line number of the first byte of the token in the underlying stream. More... | |
| size_t | curr_pos |
| The current position in the underlying stream just after reading this token. More... | |
Information about a token read from the underlying stream.
Definition at line 110 of file stream-tokenizer.H.
| size_t reranker::StreamTokenizer::Token::curr_pos |
The current position in the underlying stream just after reading this token.
Definition at line 125 of file stream-tokenizer.H.
| size_t reranker::StreamTokenizer::Token::line_number |
The line number of the first byte of the token in the underlying stream.
Definition at line 122 of file stream-tokenizer.H.
| size_t reranker::StreamTokenizer::Token::start |
The starting byte of the token in the underlying stream.
Definition at line 120 of file stream-tokenizer.H.
| string reranker::StreamTokenizer::Token::tok |
The token itself.
Definition at line 112 of file stream-tokenizer.H.
| TokenType reranker::StreamTokenizer::Token::type |
The token’s type.
Definition at line 114 of file stream-tokenizer.H.
1.8.6