testing reducers

[drdozer]

What's the workflow for testing a module's public API? I want to test it during the module build. My wishlist includes:

• run a test or collection of tests in an isolated database environment - different test suites may have isolated environments
• custom database pre-population or starting state for a test
• retain database for failed test
• call into the reducer as if a client
• peek into the db as if on the server
• manage multiple identities so that we can test interactions between different identities interacting with the same db
• module test source lives in the module source tree
• not touch any production environments in any way

So for example, I may want to check that identity A can call a reducer to rename themselves, and that identity B gets notification of A's name change, and that B can not change the name of A. This would require a database with identity A and B pre-initialized and nothing else, and the post-test database state would reflect the updated name for A. I think this means the test would need to create a new DB instance, populate it, then acquire two independent clients for A and B, and potentially allow some state-inspection sql to be run as if inside the database, e.g. to verify the identity name table state.

I am trying to avoid testing the module by either going through random client ux clicking hell, or writing endless "fake" clients that run a particular test but aren't integrated into the build process, or being forced to make tables public to test them.

[cloutiertyler]

This is a fantastic question, and something we'd like to be able to support more fully. We haven't landed on a final design yet, but I appreciate your listing out of the specific usecases you'd like to see, we can definitely try to cover them in a future spacetime test command.

Personally, I would love to be able to spin up a database from a snapshot and run a reducer. It would also be awesome to do fuzz testing/deterministic simulation testing and store database snapshots for the failed tests as regression tests.

Thanks for posting! We will keep you informed.

[drdozer]

Thanks. Automatic collection of regression tests would be really great to have. I would also suggest it may be useful to have support for:

• pre-post reducer snapshot, and validate that running the reducer(s) does generate the post snapshot given the pre snapshot, as an alternative to property-based or assert-based testing.
• support for randomised input tests, each in their own database env as in property testing with generated inputs
• with and without network e.g. run in-process through direct rust/wasm/db interactions vs run inside a mini network where the API is poked through the normal network transport machinery

I realise this is becoming a bit of a bucket for testing ideas rather than a specific request. So there's that.

Is the work on spacetime test all in-house?

[cloutiertyler]

Yes, agreed that would be awesome to have.

It is currently, although in principle it doesn't have to be. We'd just have to coordinate on design.

[aurelien-clu]

BDD feature files with spacetime db preconfigured steps would be great imho, though I suppose this could be implemented outside of spacetime db. 😇

Feature: Player damage and stats tracking

  Background:
    Given a local SpacetimeDB instance is running on "localhost:3000"
    And the module "my-db" is published with "--delete-data"
    And a client is connected to module "my-db"

  Scenario: Player takes damage and stats are updated
    Given the reducer "create_player" is called with:
      | identity    | "player-alice" |
    And the table "player_stats" has no row where "player_id" = "player-alice"
    When the reducer "attack" is called with:
      | target_id | "player-alice" |
      | damage    | 50             |
      | source    | "melee_attack" |
    Then the reducer "attack" last status is "Committed"
    And the table "player_stats" has a row where "player_id" = "player-alice" with:
      | total_damage | 50 |
      | death_count  | 0  |
    And the event table "damage_event" received a row with:
      | entity_id | "player-alice" |
      | damage    | 50             |
      | source    | "melee_attack" |

  Scenario: Player dies when total damage exceeds threshold
    Given the reducer "create_player" is called with:
      | identity    | "player-alice" |
    And the reducer "attack" is called with:
      | target_id | "player-alice" |
      | damage    | 80             |
      | source    | "melee_attack" |
    When the reducer "attack" is called with:
      | target_id | "player-alice" |
      | damage    | 30             |
      | source    | "spell"        |
    Then the reducer "attack" last status is "Committed"
    And the table "player_stats" has a row where "player_id" = "player-alice" with:
      | total_damage | 110 |
      | death_count  | 1   |