Intent to revive SIP-122 (Integrated monitoring and user activity dashboard)

[hjadm]

Hi @rusackas @TechAuditBI,

SIP-122 (#27620) proposed an integrated monitoring and user activity dashboard, but it was closed due to inactivity and is now locked for comments so I'm opening this to continue the conversation.

I've searched the repository, closed PRs, and the SIP archive, and I couldn't find any implemented work or follow-up related to this proposal as far as I can tell, this gap still exists and community discussions (e.g. #33538) keep pointing back to that issue.

This is a problem I already solve today at my organization by manually extracting data from the metadata database, so I have practical experience with it and I'd like to move this proposal forward.

My plan is to open a new SIP referencing SIP-122, with a complete specification that addresses the points raised in the original discussion:

Metadata-database-agnostic approach (no PostgreSQL-specific materialized views supporting both PostgreSQL and MySQL metastores)
Scope limited to user activity and content usage (explicitly excluding infra metrics like CPU/memory)
Admin-only access by default
Optional, opt-in deployment (e.g. superset load_monitoring), separate from example dashboards
A clear migration plan

Before I write it up: would the maintainers prefer a new SIP, or reopening #27620?

[rusackas]

You're more than welcome to open a new SIP (I'll convert this to a Discussion rather than an Issue in the meantime). I think that'll be cleaner than reopening the old one, especially if you want to own putting it through the SIP process.

That said, I think you'll likely run into similar problems mentione on the other SIP... namely:

1. The scope of what people do and don't want in the dashboard will be largely subjective... people might not want to surface things that include PII
2. Some people will want more/fewer/different things monitored. I'm not sure if there's a "one size fits all" solution here

Database agnosticism and opt-in deployment do sound like important facets though.

There are any other number of things that come through the logging service that people may or may not want in their monitoring/activity dashboards. I guess this all points to the question of if/how it can be configurable/customizable. I have nothing against any of this, just don't know how to make it perfect/universal. I look forward to the SIP if you want to go down that road :)

[hjadm]

Thanks for the thoughtful feedback @rusackas those are fair points.

I think we might be approaching this from different angles though. You're thinking about making a "universal dashboard that makes everyone happy" and yeah, that's impossible. I wouldn't want to try that either.

What I'm thinking is simpler: this isn't really about the dashboard. It's about the data layer.

Every Superset admin eventually needs to answer the same basic questions:

Who logged in and when
How long they spent on the platform
What dashboards/charts are actually being used (and which ones are dead)

That's not subjective it's just facts from the metadata DB. No PII beyond usernames, no opinions, no infrastructure metrics.

If the SIP delivers:

1. A couple of well-structured views/tables in the metadata DB with activity data
2. An example dashboard as a starting point (that admins can clone and adapt however they want)
3. The superset load_monitoring command to opt in

...then the "one size fits all" problem goes away, because Superset itself is the customization layer. The dataset is the size, the fit is whatever the admin builds on top of it.

Happy to draft the SIP with that framing if you think it has legs. Appreciate you keeping the conversation open either way.