My program scans directories. I found that when scanning the directory for the second time, the program results are incorrect. Or it crashes directly. Sometimes some error messages from the sqlite3 library also pop up:

Cannot prepare statement: out of memory

I added some print statements to try to find the location of the error based on the order of the output. However, strangely, after I added these print statements, the program ran normally every time. When I removed these print statements, the program would start having errors and crashing again.

So I started using gdb to debug the program: gdb doesn’t require changing the program’s code. I set some breakpoints, and during the second scan, I found that there was a problem in a function called compare_stage_1:

rc= sqlite3_exec(m_newDB, selectFileSQL.c_str(), on_get_new_file_memory, NULL, &errMsg);
if (rc != SQLITE_OK) {
    std::cout <<__LINE__<< " unable query data: " << errMsg << std::endl;
    sqlite3_free(errMsg);
    return;
}

An error occurred, and m_newDB is a null pointer. I traced it back to its parent function and found that the problematic code is roughly this part:

create_file_memory(g_newMemPath,true,true);
while(!is_memory_scan_finished()){
    //200ms
    g_usleep(200000);
    //other codes
    ...
}

get_memory_database_copy(&m_newDB);

This line of code did not execute correctly

get_memory_database_copy(&m_newDB);

Before this line of code is executed, the value of m_newDB should be a null pointer 0x0, and after execution it should point to the new database. In fact, during the second scan, after this line of code is executed, m_newDB is still a null pointer 0x0.

Thread 35 hit Breakpoint 1, thread_start_compare (data=0x0) at memcomparator.cpp:386
386                     get_memory_database_copy(&m_newDB);
(gdb) print m_newDB
$1 = (sqlite3 *) 0x0
(gdb) n
[New Thread 1140.0x15a8]
388         break;
(gdb) print m_newDB
$2 = (sqlite3 *) 0x0

After that, I restarted the program and tried again. When I hit this breakpoint and entered the function, I found that it judged the m_isDBExist variable as NULL and returned directly. In other words, this database was not created.

DataBase::get_memory_database_copy (this=0x6395e0 <g_memoryDB>, memoryDb=0x638428 <m_newDB>) at database.h:239
239                     if(!m_isDBExist){
(gdb) n
240                             return false;
(gdb) n
267             }

The function that creates the database is create_file_memory. It starts a new thread to create the database.

create_file_memory(std::string directoryPath,bool hasMD5,bool hasSHA256){

    ...

    GError *error = NULL;
    GThread *thread = g_thread_create(
        thread_create_file_memory,
        g_strdup(dirPath.c_str()),
        FALSE,
        &error
    );
    
    if (error) {
        g_printerr("create thread fail: %s\n", error->message);
        g_error_free(error);
    }

}

I set a breakpoint on this thread function thread_create_file_memory. Then I restarted the program and ran it again. When it scanned the second time, I found the problem. In theory, the breakpoint in the thread function should execute first, but it directly skipped it. It went to the later breakpoint, which is the problematic line of code.

get_memory_database_copy(&m_newDB);

Because a waiting loop was added before this line of code, it is supposed to continue executing only after the thread_create_file_memory thread has finished. But now it skips directly. In other words, there is a problem with the task completion check.

create_file_memory(g_newMemPath,true,true);

while(!is_memory_scan_finished()){
    //200ms
    g_usleep(200000);
        
    ...
}
    
get_memory_database_copy(&m_newDB);

The function is_memory_scan_finished determines whether the task is finished through the g_isScanFinished flag.

bool is_memory_scan_finished(){
    g_mutex_lock(g_progressMutex);
    bool ret = g_isScanFinished;
    g_mutex_unlock(g_progressMutex);
    return ret;
}

g_isScanFinished needs to be set to false immediately after the task starts, but in my code, it is set in the thread function thread_create_file_memory. After thread_create_file_memory starts, it sets it to false, and sets it to true when it ends.

gpointer thread_create_file_memory(gpointer data) {
    
    g_mutex_lock(g_progressMutex);
    g_isScanFinished = false;
    g_mutex_unlock(g_progressMutex);

    ...


    g_mutex_lock(g_progressMutex);
    g_isScanFinished = true;
    g_mutex_unlock(g_progressMutex);
    return NULL;
}

However, starting a thread takes some time, and create_file_memory returns immediately after starting the thread. Shortly after, the completion status is checked. After the first scan is finished, g_isScanFinished is directly set to true, but at this time the thread_create_file_memory thread may not have started yet. Therefore, there is no way to set the flag to false. This causes the subsequent code to directly assume that the task is completed. Continuing to execute leads to accessing a null pointer database, causing the program to crash.

The print statements I added earlier might have changed the execution timing, giving the thread enough time to start and set that flag.

The solution is very simple. Just put the code that sets the g_isScanFinished flag inside create_file_memory. create_file_memory is an asynchronous function, but the code inside it is executed on the current thread, so there won't be any issues with skipping code.

void create_file_memory(std::string directoryPath,bool hasMD5,bool hasSHA256){

    g_mutex_lock(g_progressMutex);
    g_isScanFinished = false;
    g_mutex_unlock(g_progressMutex);
    ...

    GError *error = NULL;
    GThread *thread = g_thread_create(
        thread_create_file_memory,
        g_strdup(dirPath.c_str()),
        FALSE,
        &error
    );
    
    if (error) {
        g_printerr("create thread fail: %s\n", error->message);
        g_error_free(error);
    }

}