tokenizer.H File Reference

Provides the Tokenizer class. More...

#include <string>
#include <vector>

Go to the source code of this file.

Classes
class	reranker::Tokenizer
	A very simple tokenizer class. More...

Namespaces
	reranker
	Provides reranking models for discriminative modeling, with some special handling for discriminative language models.

Macros
#define	SPACE_CHARS   " \t"

Detailed Description

Provides the Tokenizer class.

Author

dbike.nosp@m.l@go.nosp@m.ogle..nosp@m.com (Dan Bikel)

Definition in file tokenizer.H.

Macro Definition Documentation

#define SPACE_CHARS   " \t"

Definition at line 43 of file tokenizer.H.