Upstream/Downstream change notifications

[tdari]

Hey, it would be great if users could subscribe to assets and get notified about downstream or upstream changes. This would be especially useful for data teams. For example, if you manage a table in BigQuery and new services start using that table, or if new services begin writing raw files to your upstream storage bucket, you would be notified.

[charlie-haley]

Sounds great! Schema change notifications would be really valuable too.

I think we should let users subscribe to assets via the UI or CLI, with notifications sent through Slack or Email. Users could subscribe to specific events like:

• Lineage changes (new upstream/downstream dependencies)
• Schema changes
• Metadata updates

For notifications about actual data changes (like new writes to a storage bucket), we'd need more infrastructure first - probably data quality rules or checks to evaluate against. That's a probably bigger feature we can explore separately, shouldn't be too difficult to add once a notification system is in place

[tdari]

Schema changes

You got me already, that’s a perfect use case.

I think, at the beginning, notifications could be shown only in the UI (in an inbox similar to GitHub notifications). Later, Slack or other services can be integrated.

[charlie-haley]

I've created an issue here, let me know if there's anything missing you think would be good to add! The external notification system can come later in a separate ticket #60

Also, if there's any of the tasks you're particularly interested in let me know, I might make a start on the initial DB migration and Go package later

[tdari]

@charlie-haley it looks perfect. I'm not yet quite familiar with the repo (also I don't have much experience with Go, still learning) and I don't want to be a blocker. You can work on this if you want. Once the core implementation is done I can help with building new parts on top of it. Thank you.

[charlie-haley]

I've started an initial implementation in main, it has different types of notifications with the option for users to subscribe to individual assets. I'll work on an external notification system next so they can be sent to Slack and Teams