public class NameTokenization extends WordTokenization
SymbolTokenization.TokenType
Annotatable.AnnotationForwarder
CHARACTER, FIXEDWIDTH, SEPARATED, UNKNOWN
ANNOTATION
Constructor and Description |
---|
NameTokenization(FiniteAlphabet fab)
Construct a new NameTokenization, defaulting to case-insensitive.
|
NameTokenization(FiniteAlphabet fab,
boolean caseSensitive) |
Modifier and Type | Method and Description |
---|---|
protected void |
finalize() |
protected java.util.Map |
getNameToSymbol() |
Symbol |
parseToken(java.lang.String token)
Returns the symbol for a single token.
|
java.lang.String |
tokenizeSymbol(Symbol s)
Return a token representing a single symbol.
|
getAlphabet, getAnnotation, getTokenType, parseStream, parseString, splitString, tokenizeSymbolList
addChangeListener, addChangeListener, addForwarder, getForwarders, getListeners, isUnchanging, removeChangeListener, removeChangeListener, removeForwarder
clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
addChangeListener, addChangeListener, isUnchanging, removeChangeListener, removeChangeListener
public NameTokenization(FiniteAlphabet fab, boolean caseSensitive)
public NameTokenization(FiniteAlphabet fab)
protected void finalize() throws java.lang.Throwable
finalize
in class java.lang.Object
java.lang.Throwable
protected java.util.Map getNameToSymbol()
public Symbol parseToken(java.lang.String token) throws IllegalSymbolException
SymbolTokenization
The Symbol will be a member of the alphabet. If the token is not recognized as mapping to a symbol, an exception will be thrown.
token
- the token to retrieve a Symbol forIllegalSymbolException
- if there is no Symbol for the tokenpublic java.lang.String tokenizeSymbol(Symbol s) throws IllegalSymbolException
SymbolTokenization
s
- The symbolIllegalSymbolException
- if the symbol isn't recognized.