How to parse text strings in C

Some programs can just process an entire file at once, and other programs need to examine the file line-by-line. In the latter case, you likely need to parse data in each line. Fortunately, the C programming language has a standard C library function to do just that.

The strtok function breaks up a line of data according to “delimiters” that divide each field. It provides a streamlined way to parse data from an input string. Let’s dig further into how to use strtok to parse a string in C:

Read the first token

Suppose your program needs to read a data file, where each line is separated into different fields with a semicolon. For example, one line from the data file might look like this:

102*103;K1.2;K0.5

In this example, store that in a string variable. You might have read this string into memory using any number of methods. For example, you might have a program that read input from a file using fgets like this:

fgets(str, MAX_STR, *file);

Once you have the line in a string, you can use strtok to pull out “tokens.” Each token is part of the string, up to the next delimiter. The basic call to strtok looks like this:

char *strtok(char *string, const char *delim);

The first call to strtok reads the string, adds a zero (also called a “null” or literal \0 value) character at the first delimiter, then returns a pointer to the first token. If the string is already empty, strtok returns NULL.

#include <stdio.h>
#include <string.h> /* strtok */

int main()
{
  char string[] = "102*103;K1.2;K0.5";
  char *token;

  token = strtok(string, ";");

  if (token == NULL) {
    puts("empty string!");
    return 1;
  }

  puts(token);

  return 0;
}

This sample program pulls off the first token in the string, prints it, and exits. If you compile this program and run it, you should see this output:

102*103

102*103 is the first part of the input string, up to the first semicolon. That’s the first token in the string.

Note that calling strtok modifies the string you are examining. If you want the original string preserved, make a copy before using strtok.

The next tokens

Separating the rest of the string into tokens requires calling strtok multiple times until all tokens are read. After parsing the first token with strtok, any further calls to strtok must use NULL in place of the string variable. The NULL allows strtok to use an internal pointer to the next position in the string.

Modify the sample program to read the rest of the string as tokens. Use a while loop to call strtok multiple times until you get NULL.

#include <stdio.h>
#include <string.h>

int main()
{
  char string[] = "102*103;K1.2;K0.5";
  char *token;

  token = strtok(string, ";");

  if (token == NULL) {
    puts("empty string!");
    return 1;
  }

  while (token) {
    /* print the token */
    puts(token);

    /* parse the same string again */
    token = strtok(NULL, ";");
  }

  return 0;
}

By adding the while loop, you can parse the rest of the string, one token at a time. If you compile and run this sample program, you should see each token printed on a separate line, like this:

102*103
K1.2
K0.5

Be careful about delimiters

Using strtok provides a quick and easy way to break up a string into just the parts you’re looking for. You can use strtok to parse all kinds of data, from plain text files to complex data. However, be careful that multiple delimiters next to each other are the same as one delimiter.

For example, if you were reading CSV data (comma-separated values, such as data from a spreadsheet), you might expect a list of four numbers to look like this:

1,2,3,4

But if the third “column” in the data was empty, the CSV might instead look like this:

1,2,,4

This is where you need to be careful with strtok. With strtok, multiple delimiters next to each other are the same as a single delimiter. You can see this by modifying the sample program to call strtok with a comma delimiter:

#include <stdio.h>
#include <string.h>

int main()
{
  char string[] = "1,2,,4";
  char *token;

  token = strtok(string, ",");

  if (token == NULL) {
    puts("empty string!");
    return 1;
  }

  while (token) {
    puts(token);
    token = strtok(NULL, ",");
  }

  return 0;
}

If you compile and run this new program, you’ll see strtok interprets the ,, the same as a single comma and parses the data as three numbers:

1
2
4

Knowing this limitation in strtok can save you hours of debugging. If you need to parse data that might have multiple delimiters in the string, you might consider writing your own parser to break it up in some other way. It’s up to you. More complex programs might need this extra feature, but many do not.

For example, programs that only need to parse “words” from a string, where each word is separated by one or more spaces, can use this feature in strtok to parse words one at a time. But programs that need to be careful about the delimiters, such as reading a CSV file where commas are significant, might need to do it another way.

Let’s keep things simple and stay focused on using strtok in the default way.

Using multiple delimiters

You might wonder why the strtok function uses a string for the delimiter instead of a single character. That’s because strtok can look for different delimiters in the string. For example, a string of text might have spaces and tabs between each word. In this case, you would use each of those “whitespace” characters as delimiters:

#include <stdio.h>
#include <string.h>

int main()
{
  char string[] = "  hello \t world";
  char *token;

  token = strtok(string, " \t");

  if (token == NULL) {
    puts("empty string");
    return 1;
  }

  while (token) {
    puts(token);
    token = strtok(NULL, " \t");
  }

  return 0;
}

Each call to strtok uses both a space and tab character as the delimiter string, allowing strtok to parse the line correctly into two tokens:

hello
world

This feature is very helpful when parsing “words” from a string. Instead of just using spaces as the delimiter between words, a program can use any white space, including spaces, tabs, vertical tabs, and new lines.

Using strtok

The strtok function is a handy way to read and interpret data from strings. Use it in your next project to simplify how you read data into your program.

This article is adapted from Parsing data with strtok in C by Jim Hall, and is republished with the author’s permission.