Can strtok() call strtok(NULL,...) more after returning NULL?

Asked 1 years ago, Updated 1 years ago, 34 views

I saw the following programs:

char*token1=strtok(str, "");
char*token2 = strtok(NULL, '');
char*token3 = strtok(NULL, '');

if(token1==NULL){
    // What to Do Without Tokens
}
else if (token2==NULL) {
    // Handling Single Token
    ...
}
else if(token3==NULL) {
    // Handling Two Tokens
    ...
}
else{
    // What to do if you have more than three tokens?
    ...
}

This program assumes that strtok() can return NULL once and then call strtok(NULL,...).
I couldn't think of such a way of using it, and I thought it was unconsciously undefined, but is this way of using it guaranteed by standards?

For verification purposes, we looked into the specifications.
From ISO/IEC 9899:2011, 7.24.5.8 The strtok function (emphasis is on citation).
In this quote, s1 and s2 represent the first and second arguments of strtok, respectively.

3
The first call in the sequence searches the string pointed to by s1 for the first character
That is not contained in the current separator string pointed to by s2.If no such character
is found, then there are no tokens in the string pointed to by s1 and the strtok function
returns a null pointer
.If such a character is found, it is the start of the first token.

4
The strtok function then searches from there for a character that is contained in the
current separator string.If no such character is found, the current token extensions to the
end of the string pointed to by s1, and subsequential searches for a token will return a null
pointer
.If such a character is found, it is overwritten by a null character, which
terms the current token.Thestrtokfunction saves a pointer to the following
character
, from which the next search for a token will start.

5
Each subsequence call, with an null pointer as the value of the first argument, starts
searching from the saved pointer and behavior as described above.

I considered this statement in the following cases.

Now, the way paragraph 5 was written was ambiguous, but we interpreted the strtok(NULL,...) call as meaning to replace the s1 in paragraphs 3 and 4 with the pointer saved in paragraph 4.

The call after "def" returns matches the pattern in paragraph 4, ssubsequent searches for a token will return a null pointer の and returns NULL."Because ""search"" is plural, no matter how many times you call strtok(NULL,...), it returns NULL.

The call after "def" returns matches the <strtok function returns a null pointer の pattern in paragraph 3, and returns NULL.There is some sort of saved pointer (the end of the string in this case) and there is no update for that pointer, so no matter how many times you call it, it matches the pattern in paragraph 3 and returns NULL.

Similar to the previous section, NULL matches the pattern in paragraph 3.In this case, there is no saved pointer, so the subsequent strtok(NULL,...) is likely undefined.

In some cases, the strtok(NULL,...) after the NULL was returned was returned as NULL, and in other cases it seemed to be undefined.
However, then I feel uncomfortable because it's half done.
What do you think of this? So I thought of two possibilities.

  • There is a mistake in the above interpretation, and the NULL is either returned or undefined.
  • Basically, it should be considered undefined.There is no particular meaning in paragraph 4 that "search" is plural.

However, there was no confirmation of what was right.

For your information, I experimented with how it would be in your environment.

  • OS:Gentoo Linux
  • Compilers: gcc 4.8.4, clang 3.5.0
  • library:glibc2.20
#include<stdio.h>
# include <string.h>

// Output token contents, but "(null)" for NULL.
#define PR_TOKEN(token)do{printf(#token"=%s\n",(token)? token: "(null)");}while(0)

int main (void)
{
    // Case 1: String Ends with Token Characters
    char str1 [ ] = "abc def";
    char*token1_1 = strtok(str1, "");
    char*token1_2 = strtok(NULL, "");
    char*token1_3 = strtok(NULL, "");
    char*token1_4 = strtok(NULL, "");
    PR_TOKEN(token1_1);
    PR_TOKEN(token1_2);
    PR_TOKEN(token1_3);
    PR_TOKEN(token1_4);

    // Case 2: String ends with delimiter
    char str2[ ] = "abc def";
    char*token2_1 = strtok(str2, "");
    char*token2_2 = strtok(NULL, "");
    char*token2_3 = strtok(NULL, "");
    char*token2_4 = strtok(NULL, "");
    PR_TOKEN(token2_1);
    PR_TOKEN(token2_2);
    PR_TOKEN(token2_3);
    PR_TOKEN(token2_4);

    // Case 3: No Token Column
    char str3 [ ] = "";
    char*token3_1 = strtok(str3, "");
    char*token3_2 = strtok(NULL, "");
    char*token3_3 = strtok(NULL, "");
    char*token3_4 = strtok(NULL, "");
    PR_TOKEN(token3_1);
    PR_TOKEN(token3_2);
    PR_TOKEN(token3_3);
    PR_TOKEN(token3_4);

    return 0;
}
token1_1=abc
token1_2 = def
token1_3 = (null)
token1_4 = (null)
token2_1 = abc
token2_2 = def
token2_3 = (null)
token2_4 = (null)
token3_1 = (null)
token3_2=(null)
token3_3=(null)
token3_4 = (null)

For all cases, NULL is returned in the strtok() call after the NULL is returned.
This time, I suspected that case 3 was working well because I still had information stored in case 1 and case 2, so I tried to run case 3 only or enter case 3 immediately after the first strtok() call in case 2.

c

2022-09-30 20:51

1 Answers

7
The strtok function returns a pointer to the first character of a token, or a null pointer if there is no token.

If you do not have the corresponding token, you can consider returning NULL.

For your information,
According to the explanation in the fifth edition of the C Reference Manual,

If all characters belong to the set, strtok returns the null pointer and sets the internal state pointer to the null pointer.

Therefore, in case 3, the internal pointer is set to null, which is different from the questioner's expectation.
Also,

If str and internal state pointers are both null pointers, strtok returns the null pointers and keeps the internal state pointers intact (preparation for when strtok is still called even if all the words are returned).

(*str is the first argument set is the second argument)
As it says,
"After strtok() returns NULL, you can continue to call strtok(NULL,...)"
often returns NULL.


2022-09-30 20:51

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.