de.aitools.ie.decomposition.word
Class WordTokenization

java.lang.Object
  de.aitools.ie.decomposition.word.WordTokenization

All Implemented Interfaces:

Decomposition

public class WordTokenization
extends java.lang.Object
implements Decomposition

This class decomposes a given String into words without removing anything that is not a word. So even whitespace characters are returned as tokens. Concatenation of the list of words produces the original string again. For word extraction have a look at the WordDecomposition* classes.

Version:

aitools 2.0 Created on 19.09.2008 $Id: WordTokenization.java,v 1.3 2009-06-19 16:01:46 bege5932 Exp $

Author:

Fabian Loose

See Also:

WordDecompositionICU4J, WordDecompositionDelim, WordDecompositionOpenNLP

Constructor Summary

WordTokenization(java.util.Locale language)
This class decomposes a given String into words without removing anything that is not a word.

Method Summary

java.util.List<Span>	getSpans(java.lang.String text) Analyses a string and split it in parts.
java.util.List<java.lang.String>	getStrings(java.lang.String text, boolean asSubstring) Analyses a string and split it in parts.
static void	main(java.lang.String[] args) Test.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WordTokenization

public WordTokenization(java.util.Locale language)

This class decomposes a given String into words without removing anything that is not a word. So even whitespace characters are returned as tokens. Concatenation of the list of words produces the original string again. For word extraction have a look at the WordDecomposition* classes.

See Also:

WordDecompositionICU4J, WordDecompositionDelim, WordDecompositionOpenNLP

Method Detail

getSpans

public java.util.List<Span> getSpans(java.lang.String text)

Description copied from interface: Decomposition

Analyses a string and split it in parts. The return value is a list of Spans with start/end index in original string.

Specified by:

getSpans in interface Decomposition

Parameters:

text - The original text to decompose.

Returns:

List of Span with start/end index in the original string.

See Also:

Decomposition#getStrings(String, boolean)}

getStrings

public java.util.List<java.lang.String> getStrings(java.lang.String text,
                                                   boolean asSubstring)

Description copied from interface: Decomposition

Analyses a string and split it in parts. The return value is a list of this parts as Strings, either as substrings or string copies dependent on asSubstring parameter.

Specified by:

getStrings in interface Decomposition

Parameters:

text - The original text to decompose.

asSubstring - If true, returned strings in list are substrings of input text else explicit copies are returned. A substring is a pointer to the original string and start/end position. A string copy is an exact copy of the part.
If you are interested just in some parts of the text and don't want to hold the hole text in main memory, you might choose string copies.

Returns:

List of string, as substrings or string copies dependent on asSubstring parameter.

See Also:

Decomposition#getSpans(String)}

main

public static void main(java.lang.String[] args)

Test.

Parameters:

args -