BREAKING CHANGE: update: migrate to better-sqlite3

[UnschooledGamer]

Why

sqlite3 has received no updates in the past 2-3 years and has blockers on package updating, like tar/node-tar, which has vulnerabilities.

Better-sqlite3 is benchmarked to be faster, better, and more updated in comparison to sqlite3.

---

I've tested as much as I could; breaking changes could occur because of how the better-sqlite3 API is without callbacks.

To Do

The update schema script needs changes.

[greptile-apps]

Greptile Overview

Greptile Summary

This PR migrates the database layer from sqlite3 (unmaintained for 2-3 years) to better-sqlite3, which is faster and actively maintained. The migration eliminates callback-based async patterns in favor of synchronous API calls.

Key changes:

• Replaced db.all(), db.get(), db.exec() callbacks with db.prepare().all(), .get(), and synchronous db.exec()
• Added directory creation in db.js to prevent startup failures (addresses previous review feedback)
• Fixed comment.js to fetch inserted row separately since insert() no longer returns row data
• Converted numeric values to strings in otp.js for parameter binding compatibility
• Added missing await to Order.insert() in plugin.js
• Updated updateSchema.js with proper synchronous error handling

Issues:

• Entity.execSql now returns empty arrays for write operations (INSERT/UPDATE/DELETE) instead of exposing changes or lastInsertRowid from better-sqlite3's .run() result. This could break any code that needs these values in the future.

Confidence Score: 4/5

• This PR is generally safe to merge with one notable concern about write operation return values
• The migration is well-executed with proper error handling, directory creation, and fixes to API endpoints. However, Entity.execSql now returns empty arrays for write operations instead of exposing the underlying changes/lastInsertRowid from better-sqlite3's .run() result. While current code doesn't appear to use these values, this could cause issues if future code needs insert IDs or affected row counts. All previously reported issues (directory creation, syntax errors) have been addressed.
• server/entities/entity.js - verify that no existing code relies on insert/update return values

Important Files Changed

Filename	Overview
server/lib/db.js	Migrated from sqlite3 to better-sqlite3, added directory creation logic to prevent startup failures
server/entities/entity.js	Converted callback-based sqlite3 API to synchronous better-sqlite3 API with statement preparation and reader detection
server/apis/comment.js	Fixed insert return value handling by fetching inserted comment separately after insert completes
server/apis/plugin.js	Added await to Order.insert and converted numeric increment value to string
updateSchema.js	Migrated from callback-based db.all to synchronous db.exec, added proper error handling for duplicate columns

Sequence Diagram

sequenceDiagram
    participant Client
    participant API as API Endpoint
    participant Entity as Entity Class
    participant DB as better-sqlite3

    Note over DB: Migration from sqlite3<br/>to better-sqlite3

    Client->>API: HTTP Request (e.g., POST /comment)
    
    rect rgb(240, 240, 255)
        Note right of API: Before: async callbacks
        Note right of API: After: synchronous calls
    end
    
    API->>Entity: insert(...columns)
    Entity->>Entity: generateInsertSql()
    Entity->>Entity: execSql(sql, values)
    
    rect rgb(255, 240, 240)
        Note over Entity,DB: Key API Change
        Entity->>DB: db.prepare(sql)
        DB-->>Entity: PreparedStatement
        Entity->>DB: stmt.reader ? stmt.all() : stmt.run()
        
        alt SELECT query (reader=true)
            DB-->>Entity: rows[]
        else INSERT/UPDATE/DELETE (reader=false)
            DB-->>Entity: {changes, lastInsertRowid}
            Note right of Entity: ⚠️ Returns [] instead<br/>of run() result
            Entity-->>API: []
        end
    end
    
    alt comment.js pattern
        Note over API: Insert returns [], so fetch separately
        API->>Entity: get([columns], [where])
        Entity->>DB: db.prepare(sql).all()
        DB-->>Entity: rows[]
        Entity-->>API: rows[]
        API-->>Client: {comment: row}
    else plugin.js pattern
        API->>Entity: increment(column, value, where)
        Entity->>DB: db.prepare(UPDATE...).run()
        DB-->>Entity: {changes}
        Entity-->>API: []
        API-->>Client: response
    end

Loading

[greptile-apps]

_{4 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[UnschooledGamer]

@greptileai Review it again.

[greptile-apps]

_{10 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

Additional Comments (1)

server/apis/plugin.js
Missing await on insert

In the paid-download flow, Order.insert(...) is called without await (and without handling the returned Promise). With the new DB layer being synchronous under the hood but still wrapped in Promises, this means the request can proceed (and later logic can observe no order) before the insert completes, and errors from the insert will be unhandled. Make this await Order.insert(...) (or otherwise handle/rethrow failures).

[UnschooledGamer]

@greptileai Review it again.

[greptile-apps]

_{5 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

increment already accepts a number per entity.js:231, no need to stringify

Suggested change

	await Plugin.increment(Plugin.DOWNLOADS, String(1), [Plugin.ID, id]);
	await Plugin.increment(Plugin.DOWNLOADS, 1, [Plugin.ID, id]);

[UnschooledGamer]

@greptileai Stringifying of them is important here, else downloads in the Database end up as 1.0 or 2.0 instead of 1 or 2.

[greptile-apps]

Write operations (INSERT/UPDATE/DELETE) now return empty array instead of the actual result info (changes, lastInsertRowid). If callers ever need the insert ID or affected row count, they won't have access to it. This could break code that relies on these values.

[UnschooledGamer]

This can be handled in future if needed, Currently it returns empty array for compatibility