A simple class for tokenizing a stream of tokens for the formally specified language used to construct objects for the InFact framework. More...

#include <stream-tokenizer.h>

Classes
struct	Token
	Information about a token read from the underlying stream. More...

Public Types
enum	TokenType { EOF_TYPE, RESERVED_CHAR, RESERVED_WORD, STRING, NUMBER, IDENTIFIER }
	The set of types of tokens read by this stream tokenizer. More...

Public Member Functions
	StreamTokenizer (istream &is, const char *reserved_chars="(){},=;/")
	Constructs a new instance around the specified byte stream. More...

	StreamTokenizer (const string &s, const char *reserved_chars="(){},=;/")
	Constructs a new instance around the specified string. More...

void	set_reserved_words (set< string > &reserved_words)
	Sets the set of “reserved words” used by this stream tokenizer. More...

virtual	~StreamTokenizer ()
	Destroys this instance. More...

string	str ()
	Returns the entire sequence of characters read so far by this stream tokenizer as a newly constructed string object. More...

size_t	tellg () const
	Returns the number of bytes read from the underlying byte stream just after scanning the most recent token, or 0 if this stream is just about to return the first token. More...

size_t	line_number () const
	Returns the number of lines read from the underlying byte stream, where a line is any number of bytes followed by a newline character (i.e., this is ASCII-centric). More...

string	line ()
	Returns a string consisting of the characters read so far of the current line containing the most recently returned token, or the empty string if no tokens have been read yet. More...

size_t	line_start ()
	Returns the stream position of the current line in the underlying byte stream. More...

bool	HasNext () const
	Returns whether there is another token in the token stream. More...

bool	HasPrev () const

string	PeekPrev () const

size_t	PeekPrevTokenLineStart () const
	Returns the stream position of the most recent line start of the previous token, or 0 if this stream is just about to return the first token. More...

size_t	PeekPrevTokenStart () const

TokenType	PeekPrevTokenType () const

string	Next ()
	Returns the next token in the token stream. More...

void	Rewind ()
	Rewinds this token stream to the beginning. More...

void	Rewind (size_t num_tokens)
	Rewinds this token stream by the specified number of tokens. More...

void	Putback ()
	A synonym for Rewind(1). More...

size_t	PeekTokenStart () const
	Returns the next token’s start position, or the byte position of the underlying byte stream if there is no next token. More...

TokenType	PeekTokenType () const
	Returns the type of the next token, or EOF_TYPE if there is no next token. More...

size_t	PeekTokenLineNumber () const
	Returns the line number of the first byte of the next token, or the current line number of the underlying stream if there is no next token. More...

string	Peek () const
	Returns the next token that would be returned by the Next method. More...

Static Public Member Functions
static const char *	TypeName (TokenType token_type)
	Returns a string type name for the specified TokenType constant. More...

Detailed Description

A simple class for tokenizing a stream of tokens for the formally specified language used to construct objects for the InFact framework.

Definition at line 90 of file stream-tokenizer.h.

Member Enumeration Documentation

enum infact::StreamTokenizer::TokenType

The set of types of tokens read by this stream tokenizer.

See Also: PeekTokenType

Enumerator
EOF_TYPE
RESERVED_CHAR
RESERVED_WORD
STRING
NUMBER
IDENTIFIER

Definition at line 95 of file stream-tokenizer.h.

Constructor & Destructor Documentation

infact::StreamTokenizer::StreamTokenizer	(	istream &	is,
		const char *	reserved_chars = `"(){},=;/"`
	)

inline

Constructs a new instance around the specified byte stream.

Parameters

is	the input byte stream for this stream tokenizer to use
reserved_chars	the set of single characters serving as “reserved characters”

Definition at line 139 of file stream-tokenizer.h.

infact::StreamTokenizer::StreamTokenizer	(	const string &	s,
		const char *	reserved_chars = `"(){},=;/"`
	)

inline

Constructs a new instance around the specified string.

Parameters

s	the string providing the stream of characters for this stream tokenizer to use
reserved_chars	the set of single characters serving as “reserved characters”

Definition at line 151 of file stream-tokenizer.h.

virtual infact::StreamTokenizer::~StreamTokenizer ( )

inlinevirtual

Destroys this instance.

Definition at line 164 of file stream-tokenizer.h.

Member Function Documentation

bool infact::StreamTokenizer::HasNext ( ) const

inline

Returns whether there is another token in the token stream.

Definition at line 200 of file stream-tokenizer.h.

bool infact::StreamTokenizer::HasPrev ( ) const

inline

Definition at line 202 of file stream-tokenizer.h.

string infact::StreamTokenizer::line ( )

Returns a string consisting of the characters read so far of the current line containing the most recently returned token, or the empty string if no tokens have been read yet.

Definition at line 193 of file stream-tokenizer.cc.

size_t infact::StreamTokenizer::line_number ( ) const

inline

Returns the number of lines read from the underlying byte stream, where a line is any number of bytes followed by a newline character (i.e., this is ASCII-centric).

Definition at line 182 of file stream-tokenizer.h.

size_t infact::StreamTokenizer::line_start ( )

inline

Returns the stream position of the current line in the underlying byte stream.

This value may be equal to the number of bytes read if the most recently consumed character is a newline and there are no more tokens in the stream.

Definition at line 195 of file stream-tokenizer.h.

string infact::StreamTokenizer::Next ( )

inline

Returns the next token in the token stream.

Definition at line 224 of file stream-tokenizer.h.

string infact::StreamTokenizer::Peek ( ) const

inline

Returns the next token that would be returned by the Next method.

The return value of this method is only valid when HasNext returns true.

Definition at line 292 of file stream-tokenizer.h.

string infact::StreamTokenizer::PeekPrev ( ) const

inline

Definition at line 204 of file stream-tokenizer.h.

size_t infact::StreamTokenizer::PeekPrevTokenLineStart ( ) const

inline

Returns the stream position of the most recent line start of the previous token, or 0 if this stream is just about to return the first token.

Definition at line 211 of file stream-tokenizer.h.

size_t infact::StreamTokenizer::PeekPrevTokenStart ( ) const

inline

Definition at line 215 of file stream-tokenizer.h.

TokenType infact::StreamTokenizer::PeekPrevTokenType ( ) const

inline

Definition at line 219 of file stream-tokenizer.h.

size_t infact::StreamTokenizer::PeekTokenLineNumber ( ) const

inline

Returns the line number of the first byte of the next token, or the current line number of the underlying stream if there is no next token.

Definition at line 285 of file stream-tokenizer.h.

size_t infact::StreamTokenizer::PeekTokenStart ( ) const

inline

Returns the next token’s start position, or the byte position of the underlying byte stream if there is no next token.

Definition at line 272 of file stream-tokenizer.h.

TokenType infact::StreamTokenizer::PeekTokenType ( ) const

inline

Returns the type of the next token, or EOF_TYPE if there is no next token.

Definition at line 278 of file stream-tokenizer.h.

void infact::StreamTokenizer::Putback ( )

inline

A synonym for Rewind(1).

Definition at line 266 of file stream-tokenizer.h.

void infact::StreamTokenizer::Rewind ( )

inline

Rewinds this token stream to the beginning.

If the underlying stream has no tokens, this is a no-op.

Definition at line 249 of file stream-tokenizer.h.

void infact::StreamTokenizer::Rewind ( size_t num_tokens )

inline

Rewinds this token stream by the specified number of tokens.

If the specified number of tokens is greater than the number of tokens read so far, invoking this method will be functionally equivalent to invoking the no-argument Rewind() method.

Definition at line 257 of file stream-tokenizer.h.

void infact::StreamTokenizer::set_reserved_words ( set< string > & reserved_words )

inline

Sets the set of “reserved words” used by this stream tokenizer.

Should be invoked just after construction time.

Definition at line 159 of file stream-tokenizer.h.

string infact::StreamTokenizer::str ( )

inline

Returns the entire sequence of characters read so far by this stream tokenizer as a newly constructed string object.

Definition at line 170 of file stream-tokenizer.h.

size_t infact::StreamTokenizer::tellg ( ) const

inline

Returns the number of bytes read from the underlying byte stream just after scanning the most recent token, or 0 if this stream is just about to return the first token.

Definition at line 175 of file stream-tokenizer.h.

static const char* infact::StreamTokenizer::TypeName ( TokenType token_type )

inlinestatic

Returns a string type name for the specified TokenType constant.

Definition at line 105 of file stream-tokenizer.h.

The documentation for this class was generated from the following files:

src/infact/stream-tokenizer.h
src/infact/stream-tokenizer.cc

Classes

Public Types

Public Member Functions

Static Public Member Functions

Detailed Description

Member Enumeration Documentation

Constructor & Destructor Documentation

Member Function Documentation