de.aitools.ie.decomposition.tokenizer
Class Tokenizer

java.lang.Object
  extended by de.aitools.ie.decomposition.tokenizer.Tokenizer
All Implemented Interfaces:
java.util.Iterator<java.lang.String>
Direct Known Subclasses:
CharClassTokenizer, CharNGramTokenizer, EntityTokenizer, LocalizedTokenizer, WhitespaceTokenizer

public abstract class Tokenizer
extends java.lang.Object
implements java.util.Iterator<java.lang.String>


Constructor Summary
Tokenizer()
           
 
Method Summary
 java.lang.String nextAsDeepCopy()
           
 void remove()
           
abstract  void setText(java.lang.String text)
           
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 
Methods inherited from interface java.util.Iterator
hasNext, next
 

Constructor Detail

Tokenizer

public Tokenizer()
Method Detail

setText

public abstract void setText(java.lang.String text)

remove

public void remove()
Specified by:
remove in interface java.util.Iterator<java.lang.String>

nextAsDeepCopy

public java.lang.String nextAsDeepCopy()