Under the Hood: Solving SQLite Deadlocks and WhatsApp Reconnect Cycles

As local-first applications grow, managing concurrent state and background workers becomes a major challenge. In Memo, the backend coordinates multiple independent modules: local vector embeddings (RAG), calendar events, active chat logs, and a WhatsApp client connection. All of these subsystems rely on SQLite for local storage.

Recently, we encountered two tricky concurrency bugs during stability reviews: a database deadlock during shutdown, and a state bug that permanently disabled WhatsApp auto-reconnections. Here is a deep dive into how we reproduced and resolved them.

1. The Database Write/Close Deadlock

Memo uses a single write-loop architecture for SQLite. Because SQLite only supports one writer at a time, we serialize all writes by routing database transactions through a Go channel (writeCh).

When the user exits the application, Close() is called to signal the background write loop to terminate, drain the remaining tasks, and release resources. However, under load, the shutdown process would occasionally freeze indefinitely.

Using a goroutine dump, we tracked the hang down to this select block in Write():

// PRE-FIX CODE (Vulnerable to race)
func (db *DB) Write(ctx context.Context, fn func(tx *sql.Tx) error) error {
    done := make(chan error, 1)
    task := writeTask{ctx: ctx, fn: fn, done: done}

    select {
    case db.writeCh <- task:
        // Task sent to queue
    case <-db.ctx.Done():
        return db.ctx.Err()
    }

    return <-done // <-- Goroutine hung here forever
}

Why it happened:

Go's select statement picks pseudo-randomly among ready cases. During application shutdown, db.ctx.Done() is already closed, meaning the second case is ready. But if writeCh has buffer space, the first case is also ready.

If Go randomly chose to send the task to writeCh, the task would enter the channel buffer. However, the background writeLoop had already processed db.ctx.Done(), drained the queue at that exact millisecond, and exited. With the write loop gone, nobody was left to read the task from the buffer or signal the done channel. The calling goroutine remained blocked on <-done forever.

The Fix:

We introduced a read-write mutex (closeMu) and a closed flag to coordinate Write and Close calls.

func (db *DB) Write(ctx context.Context, fn func(tx *sql.Tx) error) error {
    db.closeMu.RLock()
    defer db.closeMu.RUnlock()
    if db.closed {
        return fmt.Errorf("database: closed")
    }

    done := make(chan error, 1)
    task := writeTask{ctx: ctx, fn: fn, done: done}
    
    // Send to writeCh and await done...
    db.writeCh <- task
    return <-done
}

func (db *DB) Close() error {
    db.closeMu.Lock()
    db.closed = true
    db.closeMu.Unlock()

    db.cancel()
    db.wg.Wait()
    return db.sql.Close()
}

By holding a read lock (RLock) for the entire duration of Write(), we guarantee that Close() (which requests a write lock) cannot proceed until all active writes have finished submitting and awaiting their results. Conversely, any Write starting after shutdown is initiated immediately returns an error.

2. WhatsApp Auto-Reconnect Failure

Memo allows users to sync their chats with WhatsApp Web using a local connection. A long-lived background connection will occasionally drop due to network state. We have an autoReconnect loop that attempts to reconnect with exponential backoff.

If the user manually stopped the WhatsApp connection (e.g. toggled the switch in settings) and started it again, the connection would work initially, but if the network dropped later, auto-reconnect would never fire.

Why it happened:

The abort signal for the reconnection loop is stopCh. stopCh and its closing sync.Once wrapper (stopOnce) were initialized in NewClient and never recreated.

When the user called Stop(), we executed stopOnce.Do(func() { close(stopCh) }). If the user subsequently started the client again, Start() did not recreate stopCh or reset the Once blocker. The new auto-reconnect loop immediately saw the closed stopCh from the previous stop event and exited.

The Fix:

We moved stopCh and stopOnce instantiation inside Start() under the startup lock startMu. We also synchronized reads in the auto-reconnection loop:

func (c *Client) Start(ctx context.Context) error {
    c.startMu.Lock()
    // ...
    c.stopCh = make(chan struct{})
    c.stopOnce = sync.Once{}
    c.startMu.Unlock()
    // ...
}

func (c *Client) Stop() {
    c.startMu.Lock()
    defer c.startMu.Unlock()
    c.stopOnce.Do(func() {
        close(c.stopCh)
    })
    // disconnect waClient...
}

func (c *Client) autoReconnect() {
    c.startMu.Lock()
    c.reconnecting = true
    stopCh := c.stopCh // Snapshot the channel under lock
    c.startMu.Unlock()
    
    for _, delay := range backoff {
        select {
        case <-stopCh:
            // Cancel loop
            return
        case <-time.After(delay):
            // Try reconnecting...
        }
    }
}

By snapshotting stopCh before the loop, we prevent data races when Start() reassigns it during a quick reconnect cycle, ensuring the abort signal remains reliable.