Implement V1 recurring work strategy

[alexeygrigorev]

Implement V1 recurring work strategy

Status: pending
Tags: enhancement, work-engine, backend, frontend, portal, data, testing, P1
Depends on: None
Blocks: None

Scope

Implement the V1 recurring work strategy for the DataOps operations workspace. The goal is not only a technical cron scheduler; the operations manager must be able to rely on DataOps to create or surface repeated duties, see them on Operations Home, pause/resume schedules, act on due and overdue work, follow up with people, and export runtime state safely.

This issue should build on the current work-engine primitives:

• RecurringConfig exists with description, cronExpression, assigneeId, and enabled.
• generateRecurringTasks creates source: recurring tasks and skips duplicate {recurringConfigId, date} pairs.
• Template automatic triggers exist through triggerType, triggerSchedule, and triggerLeadDays, with runCron creating bundles and template tasks.
• Task status includes waiting, and waiting tasks can use waitingFor and followUpAt.
• Notifications include follow-up-due, recurring-due, task-due, task-overdue, missing-evidence, stage-change, and automation-failure as V1 types.
• Operations Home currently shows active bundles plus today's tasks and due waiting follow-ups.
• The portal-facing recurring slice already lets operators view/create/pause/resume configs and generate today's recurring tasks manually.

Deliver the remaining V1 behavior across work-engine/ and any portal Operations Home integration needed so recurring work is usable from the daily operating loop.

Operator JTBD Actions

• When the operator starts the day, they can see recurring tasks and generated workflow runs in the same Operations Home queue as today, overdue, follow-up, and at-risk work.
• When a periodic duty is due, the system creates the task or workflow run automatically and records enough context to explain where it came from.
• When a repeated operation should stop temporarily, the operator can pause the schedule without deleting its history; resuming it does not backfill unexpected duplicates unless explicitly requested.
• When Slack invites, Mailchimp backups, tax reports, newsletter preparation, sponsor follow-ups, or routine admin checks are generated, the operator can complete them with proof, comments, or follow-up dates.
• When a generated task depends on someone else, the operator can mark it waiting, set waitingFor, set followUpAt, record follow-up sent, and move it back to todo when a response arrives.
• When generation fails or is skipped because of idempotency, operators/admins can see what happened and rerun safely.
• When runtime data is exported, recurring configs, generated tasks, notifications, and generated workflow bundles preserve stable IDs and relationships without exporting secrets or binary payloads.

Functional Requirements

Schedule Strategy

• Support two V1 recurring mechanisms and make their semantics explicit in code and UI:
  • Simple recurring task configs for standalone duties such as Slack invite handling, Mailchimp backup, monthly Slack dump, bookkeeping checks, sponsor performance follow-up, and routine admin review.
  • Automatic template triggers for workflow bundles such as weekly newsletter preparation, weekly social media schedule, and monthly tax report packages.
• Use five-field cron expressions for both mechanisms. The UI may offer friendly presets, but the stored source of truth must remain cronExpression or triggerSchedule.
• Store enough metadata on generated tasks to identify source: recurring, recurringConfigId, due date, assignee, instructions/proof requirement when configured, and creation/update timestamps.
• Store enough metadata on generated workflow tasks to identify source: template, templateId, bundleId, templateTaskRef, due date, assignee, instructions, required links/files, proof requirements, and stage behavior.
• Date matching must be deterministic in UTC or in a documented application timezone. The chosen timezone must be tested around daily, weekly, and monthly schedules.

Generated Tasks And Workflows

• Simple recurring configs must create task instances on matching schedule windows.
• Automatic templates must create one active bundle per {templateId, anchorDate} and instantiate its tasks once.
• Generated work must appear immediately in Operations Home and the task list after generation.
• Generated recurring tasks must show a visible recurring source label or equivalent context, plus the schedule/config title.
• Generated automatic bundles must create or update a notification that explains the workflow run created, for example newsletter issue, monthly tax report, or social media week.
• Seed or document the initial V1 recurring duties so an operator can create them without guessing:
  • Daily or near-daily Slack invite handling from Airtable.
  • Daily review of new task/card intake.
  • Weekly Mailchimp mailing list backup.
  • Monthly Slack dump.
  • Monthly tax report workflow.
  • Weekly newsletter preparation workflow.
  • Weekly social media schedule workflow.
  • Periodic sponsor performance follow-up.
  • Periodic invoice/receipt/Dropbox/bookkeeping TODO checks.

Pause, Resume, And Deletion Safety

• Recurring configs and automatic template triggers must be pausable/resumable through enabled or equivalent state.
• Paused configs/triggers must not generate new work.
• Resuming a config/trigger must generate only future matching windows by default.
• Historical generated tasks, bundles, notifications, comments, and proof must remain intact after pause/resume.
• Deleting a config must either be disallowed while generated tasks exist or must preserve generated task history without dangling UI errors. Prefer archive/disable behavior over destructive deletion for operator-facing flows.

Due, Overdue, Reminders, And Follow-Ups

• Operations Home must distinguish at minimum:
  • Today's due generated recurring tasks.
  • Overdue generated recurring tasks.
  • Follow-ups due from waiting recurring/template tasks.
  • Active generated workflows at risk because of overdue tasks, waiting follow-ups, or missing required proof.
• Due recurring tasks remain visible until they are done, archived, or intentionally waiting.
• Overdue tasks cannot disappear just because a notification is dismissed.
• Waiting tasks require waitingFor, followUpAt, and a note/comment when moved into waiting state.
• A due waiting task must appear as follow-up due even if the original task date is in the future or past.
• Follow-up actions must record timestamped comment history for follow-up sent, response received, and next follow-up date changes.
• Missing required links/files/proof on generated workflow tasks must block completion and should create or surface missing-evidence context.
• Automation failures must create automation-failure notifications with enough context to retry or file a follow-up issue.

Idempotency And Retry

• Simple recurring tasks must be idempotent per recurring config and schedule window. Re-running generation for the same date range must skip existing generated tasks.
• Automatic workflow generation must be idempotent per template and anchor date. Re-running cron for the same day must not create duplicate bundles or duplicate template tasks.
• Skipped duplicate work must be counted and observable in API responses, logs, or admin/operator-visible notifications where useful.
• Failed generation should not leave partial duplicate state on retry. If a bundle is created but task instantiation fails, the next run must recover deterministically or emit an actionable automation failure.
• Manual generation/backfill, if exposed, must require an explicit date range and show generated/skipped counts before or after execution.

Operator Dashboard And UI

• Operations Home should remain the primary entry point, not the recurring settings page.
• The dashboard must let the operator open a generated recurring task, open a generated workflow bundle, add required proof, mark waiting, record follow-up sent, mark response received, and complete work from the daily queue or workflow context.
• Active workflow cards should show risk indicators derived from generated tasks: overdue count, waiting count, follow-up due count, missing evidence count, and next due task.
• The recurring settings/admin view must show description, schedule, enabled/paused state, assignee, last run or generated count if available, and safe actions for pause/resume/manual generate.
• Empty states must tell the operator what is clear, such as no recurring tasks due, no follow-ups due, or no automation failures.
• UI changes must remain responsive on mobile and desktop and must not introduce horizontal overflow.

Export And Data Safety

• Portable export must include recurring configs and preserve relationships from generated tasks to recurring_config_id, generated workflow tasks to template_id/bundle_id, and notifications to task/bundle/template IDs.
• Exported records must use stable application IDs and parseable dates/timestamps, not DynamoDB key internals.
• Exports must not include secrets, OAuth tokens, cookies, signed URLs, or binary payloads.
• File/artifact proof must export metadata references only, consistent with the V1 data model and data-safety docs.
• Add or update export validation so recurring entities and generated relationships are checked.

Acceptance Criteria

• A simple recurring config with a daily cron creates exactly one source: recurring task for each matching schedule date, with recurringConfigId, due date, assignee when configured, and timestamps.
• Re-running simple recurring generation for the same date range skips duplicates and returns or records generated/skipped counts.
• A paused recurring config does not generate tasks; resuming it preserves existing generated tasks and does not backfill duplicates by default.
• An automatic template with triggerType: automatic, triggerSchedule, and triggerLeadDays creates exactly one active bundle for the computed anchor date and instantiates its task definitions exactly once.
• Re-running automatic template cron for the same schedule window does not create duplicate bundles or duplicate tasks.
• Generated recurring tasks and generated workflow tasks appear on Operations Home in the correct due/overdue/follow-up/at-risk group, not only on the recurring admin page.
• Operations Home shows overdue generated recurring work separately from normal today work.
• A waiting generated task requires waitingFor, followUpAt, and a note/comment; when followUpAt arrives, it appears as follow-up due.
• Recording follow-up sent on a generated waiting task stores a timestamped comment and the next follow-up date without marking the task done.
• Recording response received moves the generated waiting task back to todo and keeps the comment history.
• Completion remains blocked for generated tasks with missing required link, missing required file, or missing required proof, and the UI explains what is missing.
• Generated workflow cards show risk context from their tasks: overdue, waiting/follow-up due, missing evidence, and next due action.
• Automation failures or unrecoverable partial-generation cases produce automation-failure notifications with config/template/date context.
• Dismissing notifications does not hide the underlying overdue, waiting, missing-evidence, or automation-failure state from dashboard/workflow views.
• Recurring config and automatic template state are included in portable export with stable IDs and relationship fields; no secrets or binary payloads are exported.
• The recurring admin UI and Operations Home remain usable on a 390px-wide viewport without horizontal overflow.
• Existing recurring, template, notification, task, bundle, export, and dashboard behavior remains backward compatible with current tests.

Test Scenarios

Scenario: Daily Slack invite task is generated once

Given: an enabled recurring config Handle new Slack invite requests with cron 0 9 * * * and an assignee.
When: recurring generation runs for 2028-02-02 twice.
Then: exactly one task exists for that config/date, it has source: recurring, it appears on Operations Home for that date, and the second run reports a skipped duplicate.

Scenario: Paused Mailchimp backup stays quiet

Given: a weekly Mailchimp backup recurring config exists and has already generated a task for the prior week.
When: the operator pauses it and generation runs on the next matching Wednesday.
Then: no new task is created, the old task remains available, and the UI shows the config as paused.

Scenario: Resume does not surprise-backfill

Given: a monthly Slack dump recurring config was paused for two months.
When: the operator resumes it before the next matching run.
Then: no historical tasks are created automatically, and only the next matching schedule window creates new work unless an explicit backfill range is requested.

Scenario: Newsletter workflow is generated from template schedule

Given: an automatic newsletter template with a weekly cron and lead days.
When: cron runs on a matching schedule date.
Then: one newsletter bundle is created for the computed anchor date, all template tasks are instantiated with due dates and proof requirements, and Operations Home shows the workflow run as active.

Scenario: Monthly tax report workflow is idempotent

Given: an automatic monthly tax report template.
When: cron runs twice for the same schedule date.
Then: one bundle exists for the month, its tasks are not duplicated, and skipped duplicate information is observable.

Scenario: Sponsor follow-up remains visible

Given: a generated sponsor performance follow-up task is marked waiting with waitingFor: Sponsor, a follow-up date, and a note.
When: the follow-up date arrives.
Then: the task appears in Follow-ups due on Operations Home; recording follow-up sent keeps it waiting and stores the next follow-up date; recording response received moves it back to todo.

Scenario: Generated task cannot close without proof

Given: a generated newsletter/social task requires a LinkedIn URL or a report file.
When: the operator tries to complete it before adding the required proof.
Then: completion is blocked, the missing proof is named in the UI, and the workflow remains at risk.

Scenario: Automation failure is visible

Given: recurring generation encounters an invalid config, invalid template trigger, or partial bundle/task creation failure.
When: generation finishes or aborts.
Then: an automation-failure notification is created with config/template/date context and Operations Home/admin UI surfaces it until resolved or acknowledged.

Scenario: Export preserves recurring runtime relationships

Given: recurring configs, generated tasks, automatic bundles, notifications, waiting follow-ups, and proof metadata exist.
When: the portable export command runs.
Then: exported JSONL includes stable IDs and relationship fields for configs/tasks/bundles/notifications/proof metadata, validates successfully, and excludes secrets and binary payloads.

Out of Scope

• Building external integrations with Slack, Airtable, Mailchimp, Dropbox, Google Drive, Google Calendar, Luma, Meetup, Finom, Wise, Revolut, or email APIs.
• Sending real external messages, creating real external backups, or modifying production external accounts.
• Editing source repos ../dtc-operations, ../datatasks, or ../podcast-assistant.
• Creating a full natural-language scheduler or non-cron recurrence engine.
• Replacing the whole workflow/task data model beyond fields needed for this V1 recurring strategy.
• Implementing full audit-event infrastructure if not already present; timestamped comments or existing notification/log records are acceptable for this issue unless audit events already exist locally.
• Importing all historical Trello/spreadsheet recurring data. This issue may seed/document V1 defaults, but bulk migration should be a separate issue if needed.
• Production deployment, commits, merges, and CI monitoring; those happen in later lifecycle steps.

Dependencies

• Product references: .goal-v1.md, docs/operations-manager-platform-jtbd.md, docs/v1-workflow-data-model.md, work-engine/docs/specs.md, work-engine/docs/templates.md.
• Implementation surfaces: work-engine/src/db/recurring.ts, work-engine/src/routes/recurring.ts, work-engine/src/cron/runner.ts, work-engine/src/db/notifications.ts, work-engine/src/public/app.js, recurring/notification/template/export tests, and any portal Operations Home bridge that surfaces work-engine state.
• Existing V1 data-safety/export behavior must be preserved.
• Labels after grooming should include: enhancement, work-engine, backend, frontend, portal, data, testing, P1.

Verification Commands

Run the focused and full local checks that match the touched surface:

cd work-engine
npm run typecheck
npm test
npm run build
npm run test:e2e -- recurring-ui.spec.js
npm run test:e2e -- home-dashboard.spec.js
npm run export:data -- ../.tmp/issue-40-export
npm run validate:export -- ../.tmp/issue-40-export

If dashboard/portal integration outside work-engine/ is changed, also run the relevant portal test/build commands discovered in that package and capture desktop/mobile screenshots for Operations Home.

[alexeygrigorev]

Implemented the first portal-facing recurring operations slice in 450162c.

The portal now lets the operations manager see recurring configs, create daily/weekly/monthly configs with human-friendly controls, pause/resume configs, and generate today's recurring tasks without switching to the standalone work-engine UI.

This does not close the broader strategy issue yet. Remaining strategy work includes automatic scheduled generation, failure notifications, richer recurring templates for each operational source, and migration/seed decisions for existing recurring duties.

[alexeygrigorev]

Grooming complete. I rewrote this from raw intake into an agent-ready V1 implementation spec covering recurring schedules, generated standalone tasks, automatic template workflow runs, pause/resume behavior, due/overdue/follow-up/missing-evidence reminders, idempotency and retry behavior, Operations Home/dashboard expectations, export/data-safety requirements, operator JTBD actions, acceptance criteria, test scenarios, out-of-scope boundaries, dependencies, labels, and verification commands.

I removed needs grooming because the issue is now ready for Software Engineer implementation.