Java StringTokenizer Class – 6 Code Examples

Java StringTokenizer class is used to break a string into tokens based on the specified delimiter. This class implements Enumeration interface.

Java StringTokenizer Constructors

The StringTokenizer class has three constructors.

1. StringTokenizer(String str): This creates a string tokenizer instance with the given string and default delimiter characters. The default delimiter characters are the space character ( ), the tab character (\t), the newline character (\n), the carriage-return character (\r), and the form-feed character (\f). The delimiter characters are not part of the token strings.
2. StringTokenizer(String str, String delimiter): Creates a string tokenizer instance with the specified string and the delimiter. The delimiter string is not part of the generated tokens.
3. StringTokenizer(String str, String delimiter, boolean returnDelims): Similar to the above constructor except that the delimiter string will also be part of the token strings.

StringTokenizer Methods

Some of the useful methods of the StringTokenizer class are:

• hasMoreTokens(): returns true if the string tokenizer has more tokens.
• nextToken(): returns the next token string from the string tokenizer. It throws NoSuchElementException if there are no more tokens in the string tokenizer.
• nextToken(String delimiter): returns the next token string using the given delimiter. After the call, the delimiter string is changed to the default value. If there are no tokens, NoSuchElementException is thrown.
• countTokens(): returns the remaining tokens count in the tokenizer. If it’s called before any call to nextToken() is made, it returns the total numbers of tokens.
• hasMoreElements(), nextElement(): these methods are present to implement Enumeration interface. They call hasMoreTokens() and nextToken() methods internally.

Java StringTokenizer Examples

Let’s look into some real-life usage of the StringTokenizer class.

1. Parsing CSV String to String Tokens

package net.javastring.strings;

import java.util.Iterator;
import java.util.StringTokenizer;

public class JavaStringTokenizer {

	public static void main(String[] args) {
		// parsing CSV string to tokens
		String csv = "Java,Python,Android";

		StringTokenizer st = new StringTokenizer(csv, ",");

		printTokens(st);
	}

	private static void printTokens(StringTokenizer st) {
		while (st.hasMoreTokens()) {
			System.out.println(st.nextToken());
		}
	}
}

Output:

Java
Python
Android

2. StringTokenizer Default delimiter example

String line = "Welcome to JavaString.net\nLearn\tJava\tProgramming";
StringTokenizer st1 = new StringTokenizer(line);
printTokens(st1);

Output:

Welcome
to
JavaString.net
Learn
Java
Programming

3. Available Tokens Count

StringTokenizer st2 = new StringTokenizer("Hi Hello Yes");
System.out.println(st2.countTokens()); // 3
st2.nextToken();
System.out.println(st2.countTokens()); // 2

4. Returning Delimiters in Tokens

StringTokenizer st3 = new StringTokenizer("Hi|Hello|Yes", "|", true);
System.out.println(st3.countTokens());
printTokens(st3);

Output:

5
Hi
|
Hello
|
Yes

Note that the tokens count is 5 because the delimiter string is also returned as a token.

5. Getting StringTokenizer Tokens as Iterator

Since StringTokenizer implements the Enumeration interface, we can call its asIterator() method to create an iterator from its tokens.

StringTokenizer st4 = new StringTokenizer("Hello Java World");
Iterator<Object> iterator = st4.asIterator();

while (iterator.hasNext()) {
	System.out.println(iterator.next());
}

Output:

6. Multiple Delimiters Example

Let’s say we have a string with multiple delimiters. We can use the nextToken(String delimiter) method to handle this scenario.

String data = "1,2#3,4#5,6";
StringTokenizer st5 = new StringTokenizer(data);
int count = 0;
while (st5.hasMoreTokens()) {
	if (count % 2 == 0) {
		System.out.println(st5.nextToken(",").replace("#", ""));
	} else {
		System.out.println(st5.nextToken("#").replace(",", ""));
	}
	count++;
}

Output:

1
2
3
4
5
6

Java StringTokenizer vs split() Method

1. StringTokenizer is a legacy class and it’s retained only for backward compatibility. It’s recommended to use split() method to break a string into multiple parts.
2. String split() method is more powerful because we can utilize regular expressions for the delimiter. For example, we can use split(",|#") for the above example which is much simpler and easy to understand.
3. The code to get all the string tokens from the string tokenizer is a lot when compared to the split() method.

Conclusion

It’s better to know about the StringTokenizer class. But, in most of the real-life scenarios you can get the better code and the same results using the split() method.

References:

• Java StringTokenizer Class API Doc