reranker::Tokenizer Class Reference

A very simple tokenizer class. More...

#include <tokenizer.H>

Public Member Functions
void	Tokenize (const string &s, vector< string > &toks, const char *delimiters=" \t") const
	Tokenizes the specified string, depositing the results into the specified vector. More...
bool	ParseSpecString (const string &spec, string &class_name, string &init_string, bool &error)
	Parses a specification string of the form "ClassName(init_string)", depositing the results into the specified string reference parameters. More...

Detailed Description

A very simple tokenizer class.

Definition at line 49 of file tokenizer.H.

Member Function Documentation

bool reranker::Tokenizer::ParseSpecString ( const string &  spec, string &  class_name, string &  init_string, bool &  error  )

inline

Parses a specification string of the form "ClassName(init_string)", depositing the results into the specified string reference parameters.

This method will return whether spec was successfully parsed. Note that this method can return false but set error to false if spec is, e.g., all whitespace (which cannot be parsed, but is not an error).

Parameters

[in]	spec	the specifcation to be parsed
[out]	class_name	the class name in the specification string
[out]	init_string	the initialization string obtained from the specification string spec
[out]	error	whether there was an error parsing the specification string spec

Returns

if the specification string spec was successfully parsed by this method with the results placed into class_name and init_string

See Also

Factory::Create(const string&,string&,string&,bool&,bool&)

Definition at line 97 of file tokenizer.H.

void reranker::Tokenizer::Tokenize ( const string &  s, vector< string > &  toks, const char *  delimiters = " \t"  ) const

inline

Tokenizes the specified string, depositing the results into the specified vector.

The default delimiters are space and tab characters (so that this method performs whitespace tokenization).

Parameters

[in]	s	the string to tokenize
[out]	toks	the vector to which to append the results of tokenizing the specified string; this vector is not cleared by this method
[in]	delimiters	the set of delimiter characters to use when tokenizing (defaults to space and tab characters)

Definition at line 62 of file tokenizer.H.

---

The documentation for this class was generated from the following file:

• tokenizer.H