I need to use regular expressions to perform line-by-line matching on some text.

I use getline to split multi-line text, dividing it into a string each time a newline character is encountered (without retaining the newline character), and adding it to a container. Even blank lines are counted as a line and added to the container. Whether the final string ends with a newline character or not, it is still counted as one line.

With these few simple lines of code, it achieves this functionality.

std::vector<std::string> split_lines(const std::string& text) {
    std::vector<std::string> lines;
    
    if (text.empty()) {
        return lines;
    }
    
    std::istringstream stream(text);
    std::string line;
    
    while (std::getline(stream, line)) {
        lines.push_back(line);
    }
    return lines;
}

"aaa\nbbb\nccc" // 3 lines: aaa, bbb, ccc
"aaa\n\nbbb\nccc" // 4 lines: aaa, "", bbb, ccc
"aaa\n\nbbb\nccc\n" // 4 lines: aaa, "", bbb, ccc
"\n\n" // 2 lines: "", ""
"aaa\n\n" // 2 lines: "aaa", ""

My regular expression worked correctly when I was doing single-line text matching. It output the right content.

filename: ppp.txt
sha256: bce52ce33d776d18c4830103a736067ddca970ea3636382464025a96df97f048

When I edited multi-line text in Windows Notepad and then performed matching, I found that it couldn't match even a single line. I tried printing the string being matched for each line, and it should output content like this:

loop 0: line: zzzzzzz pattern: %sha256% %filename%

loop 1: line: xxxxxxxx pattern: %sha256% %filename%

loop 2: line: yyyyyyyyyy pattern: %sha256% %filename%

But I found that the output was full of chaos. I wasn’t using multithreading to compete for output, so logically it shouldn’t be messy, yet in reality, it’s a complete mess like this:

loop 0: line: zzzzzzz %sha256% %filename%

line: %sha256% %filename%

line: yyyyyyyyyy

When I was looping through and printing the contents of a vector of multi-line strings in the program, I found that they were output correctly, which surprised me.

Line 0: xxxxxxxx
Line 1: yyyyy
Line 2: zzzzzz

I was using gdb to debug, and when I printed the contents of the multi-line string vector at runtime, I indeed found the problem.

Thread 1 hit Breakpoint 1, parse_filename_and_hash_in_string (
    arg="bce52ce33d776d18c4830103a736067ddca970ea3636382464025a96df97f048 *ppp.txt\r\n\r\npxx.txt\r\n11152ce33d776d18c4830103a736067ddca970ea3636382464025a96df97faaa",
    rules="%sha256% *%filename%") at modules/batch-verification/batverify.cpp:465
465                 allMatched = is_single_line_match(textLines[i+j],ruleLines[j]);
(gdb) print textLines
$1 = std::vector of length 4, capacity 4 = {
  "bce52ce33d776d18c4830103a736067ddca970ea3636382464025a96df97f048 *ppp.txt\r", "\r",
  "pxx.txt\r", "11152ce33d776d18c4830103a736067ddca970ea3636382464025a96df97faaa"}

The gdb output shows that each line string in the vector ends with a \r character. So the problem is quite obvious. The file I edited with Windows Notepad uses line endings of \r\n instead of just \n. The default behavior of std::getline is to read characters until it encounters the newline character \n, which it extracts and discards, but it does not handle \r. As a result, the \r remains at the end of each line, and since my regular expression does not match \r, it fails to match any line successfully.

The reason why printing the container contents earlier appeared correct is probably because the \r character is not displayed in the output. The display garbling in the other place was caused by the \r interfering with the formatting, resulting in messy output.

So I added a check at the place where the text is split into multiple lines, to discard the trailing \r.

static std::vector<std::string> split_lines(const std::string& text) {
    std::vector<std::string> lines;
    
    if (text.empty()) {
        return lines;
    }
    
    std::istringstream stream(text);
    std::string line;
    
    while (std::getline(stream, line)) {
        if(!line.empty()&&line.back()=='\r'){
            line.pop_back();
        }
        lines.push_back(line);
    }
    return lines;
}

In the end, it successfully matched multiple lines of content

filename: ppp.txt
sha256: bce52ce33d776d18c4830103a736067ddca970ea3636382464025a96df97f048
filename: pxx.txt
sha256: 11152ce33d776d18c4830103a736067ddca970ea3636382464025a96df97faaa