# IntelliView — Content Operations Reports: Guide

The reference for the reporting page at **https://app.rehabessentials.com/reports/**.

> Companion to [`ADMIN-USER-GUIDE.md`](ADMIN-USER-GUIDE.md) and [`AUTHORING-USER-GUIDE.md`](AUTHORING-USER-GUIDE.md). This guide explains **what each number on the Reports page measures, how it's calculated, and what a healthy target looks like** — so you can read the page without reverse-engineering the code.

---

## 1. What this page is for

The Reports page answers operational questions about **how content gets made**, not how students perform:

- How much is the team shipping, and is the rate going up or down?
- Who's productive, who's stuck, and where is quality slipping?
- How fast does a video go from first draft to published?
- Which videos need a human to look at them right now?
- How is the Vimeo migration backlog burning down — and when will it finish?

It is **distinct from the LRS dashboard** (`/lrs/`), which holds student-side xAPI (quiz answers, watch time, mastery). Reports holds **editor/author-side** signals: publishes, drafts, AI generation runs, QA gate results, version churn.

**Access:** admin role only. Editors are redirected to the launcher (the page surfaces other people's productivity, so it's an admin tool today). Global admins see a **Tenant** dropdown to scope or compare; tenant admins are auto-scoped to their own tenant.

---

## 2. The two data sources (why some numbers behave differently)

Everything on the page comes from one of two places. Knowing which is which explains most "why doesn't this match?" questions.

| Source | What it is | Powers |
|---|---|---|
| **`videos/index.json`** | A point-in-time **snapshot** of the catalog — one row per video, fetched fresh on page load. No history. | Catalog overview tiles, Recent publishes table, Content type mix, Source & multilingual coverage, Catalog growth, and the *current backlog* number in burn-down. |
| **Authoring Events worker** (`authoring-events.k-colbo.workers.dev`) | An **append-only event log** — one row each time an editor publishes, drafts, runs AI generation, runs the QA gate, marks a question verified, replaces or restores a video. Has full history. | Team trends, Quality × volume, Editor scorecard (+ drill-in), Cohort onboarding, Recently flagged, AI quality signals, QA hit rate, Version churn, and the *burn rate* in burn-down. |

**The single most important thing to understand:** the snapshot has **no history**, so it only ever shows "how things look right now." The event log only captures events **from the date the events worker shipped forward** (PR #208). Anything published before that cutover exists in the catalog snapshot but has **no events** — so it appears in *Recent publishes* and the *overview tiles*, but the editor who made it won't show velocity/quality numbers for it. That's expected, not a bug. Empty event cards say so inline.

### Date-window filter — what it does and doesn't touch

Every card carries a small badge in its header so you never have to guess: **"Follows window"** (teal) means it reflects the Window chip, **"Fixed · last 12 wks/mo"** (grey) means it always shows a fixed span, and **"Current snapshot"** (blue) means it's the live catalog composition, unaffected by the window.

The **Window** chips (7 / 30 / 90 days, 12 months, All time) at the top filter most of the page. But several sections are deliberately **window-independent** because they need a fixed span to show a trend shape:

| Honors the Window chip | Always a fixed span (ignores the chip) |
|---|---|
| Catalog overview "Active in window" + "Distinct editors" tiles | **Team trends** — always last 12 weeks |
| Recent publishes table | **Editor scorecard 12-week trend column** — always last 12 weeks |
| Editor scorecard counts/velocity/quality | **Cohort onboarding** — defined by its own `?weeks=` (12) |
| Quality × volume scatter | **Vimeo backlog burn-down** — always last 12 weeks |
| Recently flagged videos | Catalog growth chart — always last 12 months |
| AI quality / QA / Version churn cards | |

The **Tenant** filter applies to *everything*.

---

## 3. Catalog overview tiles

Four tiles at the top, computed entirely from the `index.json` snapshot.

| Tile | Measures | How it's calculated | Target |
|---|---|---|---|
| **Total in catalog** | Every video that exists, published or draft. | Count of all index rows in tenant scope. Sub-line splits `published` vs `archived` (Save Draft → archived). | Grows over time; no fixed target. |
| **Needs authoring** | Vimeo imports that have been pulled in but have **no quiz interactions yet** — raw migrated video with no IntelliView value-add. | Count of rows where `_vimeo_id` is present **and** `interactions === 0`. | Trending toward **0**. This is the migration backlog; see the burn-down for the finish-line forecast. |
| **Active in window** | How many videos saw any activity in the selected window. | Count of rows whose most-recent timestamp (`last_modified_at`, falling back to `created_at` / `updated` / `published`) falls in the window. | Higher = busier team; read alongside Team trends. |
| **Distinct editors** | How many people touched content in the window. | Distinct `created_by`/`last_modified_by` identities (keyed by lowercased email) whose stamp falls in the window. | Tracks active headcount; sudden drops are worth a look. |

---

## 4. Team trends (last 12 weeks)

Four mini area charts giving the whole team's direction. **Always 12 weeks**, regardless of the Window chip — the point is to see whether a change (e.g., a new AI prompt) moved the line. Each panel shows the 12-week **total**, a sparkline, and a **▲/▼ % vs prior 4 weeks** arrow (last 4 weeks compared to the 4 weeks before).

| Panel | Measures | How it's calculated | Read it as |
|---|---|---|---|
| **Publishes per week** | Overall shipping velocity. | Weekly count of `video_published` events. | The headline output number. Want it flat-to-up. |
| **QA blockers per week** | Volume of hard problems the pre-publish gate is catching. | Weekly **sum of `details_json.blockers`** across `qa_run` events (the worker's `/events/timeseries` now returns a dedicated `blockers` array). | Spikes can mean *either* more QA discipline *or* a quality dip — cross-check with scorecard. |
| **Manual verify overrides per week** | How often editors override the AI's "Needs Review/Issue" verdict by hand. | Weekly count of `question_marked_verified` events. | A proxy for AI false-positives. Sustained rise = the verifier is too strict (or editors are rubber-stamping). |
| **AI interaction regens per week** | How often the AI quiz generator gets re-run. | Weekly count of `interactions_generated` events. | High = the prompt isn't nailing it first time. Want it falling after prompt improvements. |

---

## 5. Quality × volume scatter

Every editor with at least one publish in the window is a dot, positioned by **publishes (x)** and **quality (y)**. The y-axis is inverted so **fewer QA blockers plots higher** — i.e. "up and to the right = best," matching every other scatter you've seen. Dashed median crosshairs split it into quadrants:

- **Top-right (green): high volume, fewer blockers** — your strongest editors ("unicorns").
- **Bottom-right (red): high volume, lots of blockers** — worth a spot-check; could be rushing.
- **Left half:** lower-volume editors; quality read is less statistically meaningful.

| Axis | Measures | How it's calculated |
|---|---|---|
| **X — Publishes** | Output volume in window. | `video_published` count per editor. |
| **Y — Quality (fewer QA blockers ↑)** | Quality cost per shipped video, inverted so higher = better. | Sum of `details_json.blockers` across that editor's `qa_run` events ÷ their publish count, plotted with 0 blockers at the top. |

The y-axis top is chosen by excluding lone outliers (a Tukey upper fence, Q3 + 1.5·IQR) and capping at the largest "normal" rate plus headroom (floored at 2.0, the "risk" threshold). The headroom keeps the median line and dots off the edges, so the plot doesn't collapse to one color when several editors share the same rate. Editors beyond the fence are **pinned to the bottom with a dashed red ring** and labelled "off-scale" on hover; the axis tick shows the cap with a `+`. When there are ≤14 editors, **each dot is labelled with the editor's name right on the plot**; beyond that it falls back to hover-only. Overlapping dots are fanned out slightly so each stays individually visible.

**Target:** editors clustered in the **top-right / low-y** band. A dot drifting down-right (high volume + rising blockers) is the signal to check in. The axes auto-scale but the y-axis floors at 2.0 so a low-blocker tenant doesn't get visually exaggerated.

---

## 6. Editor scorecard

The core table. One row per editor active in the window; **click a row** to expand a per-video worklist + publish-velocity timeline. Quality columns render a colored chip; rate columns show **"—" when there were no publishes** (so you don't read a fake 0%).

| Column | Measures | How it's calculated | Target / chip thresholds |
|---|---|---|---|
| **Drafts** | Save-Draft actions. | `video_drafted` count. | — (volume context) |
| **Publishes** | Videos shipped live. | `video_published` count. | Higher = more output. |
| **Replaces** | Re-uploads of media on an already-live video. | `video_replaced` count. | Low. A high number means lots of re-cuts. |
| **Time-to-publish** | Median draft→publish turnaround. | For each publish, `published_at − (earliest event ever recorded on that video)`; the **median** per editor. One-off publishes with no prior events (delta = 0) are excluded. `n=` shows the sample size. | Faster is better. The timeline tiers it: **< 1 day = fast, 1–3 days = mid, > 3 days = slow.** |
| **Blockers / publish** | Avg hard QA problems per shipped video. | Sum of QA `blockers` ÷ publishes. | **< 1 = good (green), 1–2 = warn (amber), ≥ 2 = risk (red).** Aim for green. |
| **Manual overrides** | Times the editor hand-marked a question "Verified" over the AI's objection. | `question_marked_verified` count. | Chip uses overrides ÷ publishes: **< 1 = green, 1–2 = amber, ≥ 2 = red.** Some overriding is healthy; a high ratio suggests rubber-stamping or an over-strict verifier. |
| **AI regen videos** | Videos where the AI quiz generator was run more than once. | Distinct videos with `interactions_generated` count > 1. | Lower = prompt working first time. |
| **Caption regens** | Videos where captions were generated more than once. | Distinct videos with `captions_generated` count > 1. | Lower; high = caption source (AssemblyAI/YouTube) struggling. |
| **12-week trend** | This editor's publish cadence shape. | Zero-filled weekly `video_published` counts, **last 12 weeks, window-independent.** | A steady or rising bar pattern. |
| **Last active** | Recency. | Most recent event timestamp, shown relative. | — |

**Export CSV** dumps every editor with the full numeric set (including restores, QA warnings/publish, re-verify breakdown) for the current window + tenant.

### 6.1 Drill-in: per-video worklist

Clicking a row loads that editor's videos touched in the window, **sorted by quality risk (most flags first)**. Each video shows draft/publish/replace counts, AI/caption generation counts, QA blockers, overrides, and **flag chips**. The flag rules (identical here and in *Recently flagged*):

| Flag | Fires when | What it means |
|---|---|---|
| **high churn** | `replaces + restores > 0` | The video's media/config has been swapped or rolled back — re-verify timing. |
| **ai regen** | `interactions_generated > 1` | AI quiz generation was re-run; the first pass didn't satisfy. |
| **caption regen** | `captions_generated > 1` | Captions were regenerated; transcript source was unreliable. |
| **many overrides** | `question_marked_verified ≥ 3` | Three+ questions hand-approved over AI objection on one video. |
| **had qa blockers** | QA `blockers > 0` **and** it was published | A video shipped after the gate caught hard problems — confirm they were truly resolved. |

**Target:** an editor's worklist should be mostly flag-free. Flags aren't automatic failures — they're "look here" markers.

### 6.2 Drill-in: publish velocity timeline

Below the worklist, the editor's last (up to 20) publishes, most recent first, each a bar whose length = its draft→publish duration, colored by the same fast/mid/slow tiers. Publishes with no recorded prior event (common for older imports) show no bar. **Target:** mostly green/short bars; a lengthening trend is an early slowdown signal.

---

## 7. Cohort onboarding

Tracks **new editors** — anyone whose **first-ever publish** was within the last 12 weeks (defined by the worker's `?weeks=` param, **not** the Window chip). Each row lines members up on the same **relative-week axis** (week 1 = the week of their first publish) so a 2-week-old and a 10-week-old editor are comparable. The faded overlay bar on each week is the **cohort average** for that relative week.

| Column | Measures | How it's calculated |
|---|---|---|
| **First publish** | When they started shipping. | `MIN(occurred_at)` over their `video_published` events. |
| **Weeks active** | Tenure since first publish (capped at the cohort window). | `ceil((now − first_publish) / 1 week)`. |
| **Total publishes** | Lifetime output since onboarding. | Sum of their weekly publish buckets. |
| **Weekly cadence** | Ramp shape vs. the cohort average. | Per-week-since-first publish counts; the average curve only includes members who'd actually reached that relative week (so a week-2 editor doesn't drag down the week-12 average). |

**Target:** a new editor's bars should climb toward — and ideally meet — the cohort-average line within the first several weeks. Someone flat at the bottom past week 3–4 may need support.

---

## 8. Recently flagged videos (cross-editor triage)

A single queue of **every video that trips at least one quality flag** in the window, across all editors, newest-touch first. Same flag rules as §6.1. Each entry shows the flag chips, when it was last touched, and the **most-recent editor** who touched it (the "responsible editor" for triage — picked as the editor on the latest event for that video).

**Target:** short list / "clean catalog" empty state. This is the daily "what needs a human" worklist. It honors the Window chip, so narrowing to 7 days gives you a fresh-issues-only view.

---

## 9. Vimeo backlog burn-down

Forecasts when the Vimeo-migration backlog ("Needs authoring" videos) will clear.

| Number | Measures | How it's calculated |
|---|---|---|
| **Backlog now** | Videos still needing interactions. | From the live snapshot: Vimeo-imported rows with `interactions === 0`. |
| **Burn rate** | Videos cleared per week, recent. | Distinct Vimeo videos that received their **first** `interactions_generated` event each week over the last 12 weeks; the rate is the **average of the last 4 weeks**. |
| **Cleared (12wk)** | Total burned down in the trailing quarter. | Sum of the 12 weekly burn counts. |
| **ETA badge** | Finish-line forecast. | `ceil(backlog ÷ burn_rate)` weeks. Shows "stalled" if recent burn is 0, "clears this week" if ≤ 1 week, or "months" beyond ~26 weeks. |

**Target:** burn rate ≥ the rate needed to hit your migration deadline; backlog and ETA both trending down. A "stalled" badge with a non-zero backlog means nobody's authored a Vimeo import in 4 weeks.

---

## 10. AI quality signals

From `/events/summary` (honors the Window chip). Each row has an inline 12-week sparkline.

| Metric | Measures | How it's calculated | Target |
|---|---|---|---|
| **Interaction sets generated** | AI quiz-generation runs. | `interactions_generated` count. | Context for the regen numbers. |
| **Captions generated** | AssemblyAI + YouTube caption fetches. | `captions_generated` count. | Context. |
| **Caption regenerations** | Re-runs beyond the first per video. | For videos with > 1 `captions_generated`, sum of `(count − 1)`; also reports the distinct video count. | Low. High = unreliable caption source. |
| **Manual "Mark Verified" overrides** | AI verdict overridden by hand. | `question_marked_verified` count. | A false-positive proxy — some is fine, a rising trend warrants a verifier-prompt review. |
| **Re-verify attempts** | Times a question was re-run through verification. | `question_re_verified` count; also reports how many **flipped verdict**. | — |
| **Needs Review → Verified on re-verify** | The specific "AI changed its mind in our favor" pattern. | Re-verify events where `prior_verdict ∈ {needs_review, issue}` **and** `new_verdict = verified`. | Watch alongside manual overrides — both are ways an unfavorable AI verdict gets cleared. |

---

## 11. QA hit rate

The pre-publish QA gate's catch rate, from `qa_run` events (honors the Window chip). The gate runs 9 checks; results are logged as blockers / warnings / advisories.

| Metric | Measures | How it's calculated | Target |
|---|---|---|---|
| **QA runs** | How often the gate opened. | `qa_run` count. | Should roughly track publish volume — every publish ideally passes the gate. |
| **Total blockers caught** | Hard problems the gate stopped. | Sum of `details_json.blockers` across runs. | — (the gate doing its job) |
| **Avg blockers per run** | Quality of work *arriving* at the gate. | Total blockers ÷ runs. | **Trending down** = work is cleaner before QA. A high steady number = recurring upstream mistakes. |
| **Avg warnings per run** | Soft issues per run. | Total warnings ÷ runs. | Down over time. |
| **Total advisories** | Non-blocking advisories (e.g., expert-review suggestion). | Sum of `details_json.advisories`. | Informational. |

---

## 12. Version churn

Two counters from `video_replaced` + `video_restored` events (honors the Window chip), each with a sparkline.

| Metric | Measures | How it's calculated | Target |
|---|---|---|---|
| **Replace & Publish** | Media re-uploaded onto an existing live video. | `video_replaced` count. | Low. Spikes follow re-cut batches from review feedback. |
| **Restored from version** | Rollbacks from Version History. | `video_restored` count. | Near **0**. Frequent restores suggest publishes are going out before they're ready. |

---

## 13. Snapshot sections (catalog composition)

These read the `index.json` snapshot directly — current-state composition, not activity.

| Section | Measures | How it's calculated |
|---|---|---|
| **Recent publishes** | Sortable/filterable table of videos active in the window. | One row per index entry in window; sortable by title/author/date/type/interactions/status; searchable by title/ID/author; "Needs authoring" pill when Vimeo + 0 interactions. CSV export included. |
| **Content type mix** | Catalog balance across lecture/lab/simulation/demo/review. | Count + % of each `type` over all rows in tenant scope. |
| **Source & multilingual coverage** | Vimeo-imported vs hand-authored, and multilingual vs English-only. | Vimeo = rows with `_vimeo_id`; multilingual = rows with `languages.length > 1`. Both as count + %. |
| **Catalog growth** | Publishing pace over the last 12 months. | Videos bucketed by `published` (fallback `created_at`) month; bar height relative to the busiest month. |

**Targets** here are strategic, not operational — e.g., a content-mix or multilingual-coverage goal is a business decision, and the growth chart is a pacing check against your publishing plan.

---

## 14. Quick reference — "what's a good number?"

The thresholds already encoded in the page (the chip colors) are the de-facto targets:

| Signal | Good | Watch | Problem |
|---|---|---|---|
| QA blockers per publish | < 1 | 1–2 | ≥ 2 |
| Manual overrides ÷ publishes | < 1 | 1–2 | ≥ 2 |
| Draft→publish time | < 1 day | 1–3 days | > 3 days |
| AI regen videos | 0 | a few | many / rising |
| Caption regens | 0 | a few | many / rising |
| Restores from version | 0 | rare | recurring |
| Needs-authoring backlog | falling to 0 | flat | rising / stalled burn |
| New-editor cadence | meets cohort avg by ~wk 3–4 | slightly below | flat at zero past wk 4 |

> These are **soft** thresholds for prompting a conversation, not SLAs. A flag means "look here," and context (a re-cut batch, a new hire's first week, a known-bad caption source) often explains an outlier benignly.

---

## 15. Troubleshooting

- **A card says "No tool-usage events recorded in this window."** That's the *loaded-but-empty* state — the service answered, there's just nothing in this window (common for windows before event logging shipped). Widen the window. A separate **"Couldn't reach the events service"** message means the fetch actually failed (connection / auth) — sign out and back in. Cards now distinguish the two.
- **An editor shows publishes but "—" for time-to-publish.** Their publishes had no earlier authoring event recorded on that video (common for older imports — the publish was the first event). Velocity needs a draft/QA/generation event *before* the publish to measure against. The "—" tooltip and the column header explain this.
- **Team trends look flat but I know we shipped.** Trends carry a grey **"Fixed · last 12 wks"** badge and ignore the Window chip; if your activity is older than 12 weeks it won't appear here (use *Recent publishes* + *Catalog growth*). The badge on every card tells you whether it follows the window.
- **Numbers differ from the Library.** Reports reads the same `index.json`, but caches nothing — a hard refresh re-fetches. Event-driven numbers come from a separate worker and won't match snapshot counts by design.