Package org.languagetool.tokenizers
Interface Tokenizer
- All Known Subinterfaces:
CompoundWordTokenizer
,SentenceTokenizer
- All Known Implementing Classes:
SimpleSentenceTokenizer
,SRXSentenceTokenizer
,WordTokenizer
public interface Tokenizer
Interface for classes that tokenize text into smaller units.
-
Method Summary
-
Method Details
-
tokenize
-