-
Notifications
You must be signed in to change notification settings - Fork 9
Milestones
List view
Optimise the SQL profiling weekly aggregation (usp_CompileWeekly and friends) for large (~200k-user) tenants without changing the profiling.* output schema consumed by reports/Usage Analytics/Analytics_DataModel.pbit.
No due date•0/1 issues closedOptimise the O365 audit-log importer (WebJob.Office365ActivityImporter) so a large-tenant (~200k users, tens of millions of audit events/cycle, ~99% of SharePoint events out-of-scope of the org_urls whitelist) full import cycle fits within 24h. The audit phase alone currently runs for days. Verified bottlenecks: - Save path fully serialized (ListBatchProcessor 1-permit semaphore -> static 1-permit _sqlSaveSemaphore); downloads parallelise but SQL save/merge/metadata does not. - Per-batch full scan of audit_events (no index on time_stamp) in ActivityImportCache. - No blob-level checkpoint: consecutive cycles overlap ~99%, so nearly all content blobs are re-downloaded+re-parsed every cycle. - Log noise (per-Graph-page INFO; per-unresolvable-Copilot-context warnings). - Usage-report phase gated by Outlook daily-loader EF6 row-by-row save. Workstream: A. Index audit_events.time_stamp (+ ignored_audit_events.processed_timestamp). B. Blob-level checkpoint table (steady-state + restart resilience). C. Break save serialization (per-worker staging shards / concurrent merges). D. Reduce logging noise. E. Chunk/overlap the Outlook daily-loader save. Plus a DB-backed load test (COLD + WARM) to baseline dev and quantify each change.
No due date•3/3 issues closedA lightweight, in-app version of the Power BI reports (no extra deploy). New Reports tab in the admin SPA with per-area sub-tabs (shown only when the matching import is enabled) charting usage over a configurable window (default 3 months).
No due date•1/2 issues closed