Statistical Methods · VERN OS Production Study · June 2026
Complete Statistical Analysis:
Methods, Results & Evidence
All statistical tests applied to the VERN OS conversation dataset (N=7,299 production sessions; N=668 annotated conversations; 19 personas across 3 cohort groups). Tests are reported with full statistics, effect sizes, confidence intervals, and interpretation. Observational design — all findings are associational.
Significant result
Not significant
Descriptive / reliability
Methodological finding
Cautionary / limitation
Significant results
6
of 14 tests at p < 0.05
Not significant
5
reported honestly
Reliability / descriptive
3
Cronbach α, split-half, LR
Primary finding z-score
31.2
graceful exit p < 10⁻¹⁰⁰
Logistic model AUC
0.712
predicting full engagement
Cronbach's α (rubric)
0.089
items measure distinct constructs
All Statistical Tests
Methods, Statistics, and Findings — Complete Reference
| Test & Family | Method & Application | Statistic | p-value | Effect Size & CI | What It Revealed | Verdict | ||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Group 1 — Primary Production Tests (N=7,299 sessions) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Two-Proportion Z-Test
Graceful Exit: VERN vs Control Inferential · Proportion comparison
|
Compared the proportion of sessions ending cleanly (graceful exit flag from production logs) between all VERN personas (n_prod≥20) and all 7 control personas. Normal approximation to binomial. One-tailed (VERN > CTRL). | z = 31.17 SE_diff = 0.0142 |
p < 10⁻¹⁰⁰ effectively 0 |
Δ = +49.4pp 95% CI [46.6, 52.2] VERN: 72.6% (n=6,246) CTRL: 23.2% (n=1,053) |
VERN personas end conversations cleanly at 3.1× the rate of un-orchestrated controls. The gap is not marginal — it is 49 percentage points with an extremely tight CI. BCM architecture produces a structurally different session outcome distribution. | ✓ Significant | ||||||||||||||||||||||||||||||||||||||||||||||||
|
Fisher Exact Test
Graceful Exit: Excl. Luke Inferential · Independence test
|
Luke (n=4,595) represents 74% of VERN sessions. Fisher exact was applied to the Luke-excluded VERN cohort (n=1,651) vs all controls to confirm the result is not driven by a single large persona. | OR = 8.26 z = 24.47 |
p = 2.02×10⁻¹³⁷ | Δ = +48.2pp 95% CI [44.8, 51.5] VERN-excl-Luke: 71.4% CTRL: 23.2% |
Removing Luke does not diminish the finding. The odds ratio of 8.26 means VERN personas (excluding the largest) are 8× more likely to exit gracefully than controls. The result generalises across the entire VERN portfolio. | ✓ Confirmed | ||||||||||||||||||||||||||||||||||||||||||||||||
| Group 2 — Track B Mandate Tests (Entertainment & Simulation) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
One-Sample t-Test
Amber: Sentiment Δ < 0 Inferential · Mean comparison
|
Tests whether Amber's mean annotator-perceived mood change (close_sent − open_sent) is significantly below zero. H₀: μ = 0. One-tailed (less). Mandate: intentionally provoke negative emotional response. | t(16) = −3.347 df = 16 |
p = 0.0020 | mean Δ = −0.412 SD = 0.507, SE = 0.123 95% CI [−0.673, −0.151] |
Amber significantly drives annotator-perceived mood downward. This is the expected and designed outcome for a roast/insult entertainment persona. Combined with 94% in-character rate (binom p=0.00014), confirms VERN executed the negative mandate successfully. | ✓ Mandate confirmed | ||||||||||||||||||||||||||||||||||||||||||||||||
|
One-Sample t-Test
Christine: Sentiment Δ < 0 Inferential · Mean comparison
|
Same design as Amber test. Christine is a horror/paranormal companion with a mandate to produce controlled fear. Tests whether mean mood shift is significantly negative. One-tailed (less). n=50 (two Christine variants merged). | t(49) = −3.256 df = 49 |
p = 0.0010 | mean Δ = −0.260 SD = 0.565, SE = 0.080 95% CI [−0.421, −0.100] |
Christine significantly depresses annotator-perceived mood as designed. Paired with 94% in-character rate across 50 sessions including jailbreak attempts, hostile users, and sexual advances, confirms VERN maintained bidirectional emotional control under adversarial conditions. | ✓ Mandate confirmed | ||||||||||||||||||||||||||||||||||||||||||||||||
| Group 3 — Mood Ceiling/Floor Effect (N=668 annotated conversations) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Spearman Rank Correlation
Open Sentiment → Mood Lift Descriptive · Non-parametric association
|
Tested whether a user's opening mood score (1–5) predicts how much their mood shifts during the conversation (delta_sent). Applied to all 668 annotated conversations. Non-parametric due to ordinal scale. | ρ = −0.2603 n = 668 |
p = 8.30×10⁻¹² | open=1: Δ=+0.75 (n=4) open=2: Δ=+0.571 (n=63) open=3: Δ=+0.201 (n=399) open=4: Δ=−0.071 (n=184) open=5: Δ=−0.056 (n=18) |
Strong structural confound: users arriving distressed (open≤2) have +0.58 mean mood lift; users arriving positive (open≥4) have −0.07. The 5-point scale creates a ceiling/floor that makes between-group mood comparisons unreliable unless baseline is controlled. This is why VERN vs Control mood delta comparison is not a primary test. | ✓ Key confound | ||||||||||||||||||||||||||||||||||||||||||||||||
| Group 4 — Between-Group Comparisons (Annotated Sample) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Mann-Whitney U Test
Engagement: VERN vs Control Inferential · Non-parametric rank test
|
Compared engagement scores (0/0.5/1.0 ordinal) between VERN Track A companion personas (n=297) and all control companion personas (n=100). Two-tailed. Rank-biserial r as effect size. | U = 15,877 n₁=297, n₂=100 |
p = 0.234 | VERN mean: 0.774 CTRL mean: 0.740 rank-biserial r = −0.069 (negligible effect) |
Not significant. Adding Carlos (mean eng=0.80) and Nick (0.80) to the control group raised the control baseline, closing the gap. Direction favours VERN but the effect is negligible. This is an honest null result — well-configured controls can match VERN engagement in the annotated sample. | ⚠ Not significant | ||||||||||||||||||||||||||||||||||||||||||||||||
|
Mann-Whitney U Test
On-Goal Fidelity: VERN vs Control Inferential · Non-parametric rank test
|
Compared on_goal scores (0/0.5/1.0) between all VERN Track A personas and all control personas. One-tailed (VERN > CTRL). Tests whether BCMs improve persona adherence measurably in the annotated sample. | U = 38,833 | p = 0.199 | VERN mean: 0.962 CTRL mean: 0.952 rank-biserial r = −0.021 (negligible) |
Not significant. Both groups cluster at the ceiling of the 3-point scale (on_goal=1.0), leaving little room to detect a difference. The metric is effective as a failure-detector but lacks sensitivity for between-group comparisons when both groups are near ceiling. | ⚠ Ceiling effect | ||||||||||||||||||||||||||||||||||||||||||||||||
|
Two-Proportion Z-Test
Task Goal Achievement Inferential · Proportion comparison
|
Compared nominal goal_achieved rates between VERN task personas (excl. Craig — see note) and control task personas (Ronnie, Becky, Denise) in the annotated sample. Two-tailed. | z = 0.496 | p = 0.310 | VERN: 91/118 = 77.1% CTRL: 48/65 = 73.8% Δ = +3.3pp 95% CI [−9.8, +16.4] |
Not significant on nominal rate. Becky (88%) and Denise (84%) match VERN task performance at face value. However, their clean completion rates collapse to 20% and 4% respectively when graceful exit and audit criteria are applied. Nominal goal rate is not a sufficient metric for task archetype comparison. | ⚠ Nominal only | ||||||||||||||||||||||||||||||||||||||||||||||||
| Group 5 — Per-Persona Binomial Tests (On-Goal Rate vs Chance) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Exact Binomial Tests
On-Goal Rate Per Persona Inferential · Binomial probability
|
For each major persona, tests whether on_goal=1.0 occurs significantly more often than chance (H₀: p=0.5). One-tailed. Wilson confidence intervals for proportions. Confirms each persona's behavioral containment individually. |
|
All personas significant at p<0.01. Controls also score high — confirming on_goal ceiling effect, not BCM specificity |
Every tested persona achieves on_goal=1.0 significantly above chance. Including controls. This confirms the ceiling effect: the 3-value on_goal scale lacks the resolution to distinguish VERN from controls at the between-group level. Useful for detecting individual failures; not useful for group comparison. | All sig. — but ceiling | |||||||||||||||||||||||||||||||||||||||||||||||||
| Group 6 — Within-VERN Variant Analysis | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Kruskal-Wallis H Test
Zeke Variant Comparison Inferential · Non-parametric ANOVA
|
Three Zeke variants (different PIDs, same persona name, different BCM configurations) compared on mood delta to detect configuration differences. Non-parametric one-way ANOVA. Each n≈25. | H = 0.695 df = 2 |
p = 0.706 | v1: Δ=+0.320, SD=0.557 v2: Δ=+0.480, SD=0.714 v3: Δ=+0.458, SD=0.833 n≈25 per variant |
No significant difference between Zeke variants despite raw mean differences (+0.32 to +0.48). Insufficient power (n≈25 per variant) to detect moderate effects. Study is underpowered for within-variant analysis; would require n≥80 per variant to detect a 0.2-point delta difference at 80% power. | ⚠ Underpowered | ||||||||||||||||||||||||||||||||||||||||||||||||
|
Spearman Correlation
Tangent Count → Mood Delta Descriptive · Non-parametric association
|
Tests whether conversations with more off-topic tangents show worse mood outcomes. Spearman rank correlation between tangent_count and delta_sent across all 668 annotated conversations. | ρ = −0.0549 n = 668 |
p = 0.157 | Near-zero effect No meaningful association |
Tangent frequency does not predict mood outcome. This is consistent with the revised rubric's finding that tangent-and-recovery = 1.0 goal_pursuit — tangents that resolve back to goal do not harm the user experience. The count metric may be insufficiently granular without the recovery flag. | ⚠ Not significant | ||||||||||||||||||||||||||||||||||||||||||||||||
| Group 7 — Special Case Tests | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Exact Binomial Test
Izzy: Goal Achievement Inferential · Binomial probability
|
Tests whether Izzy (autism social-skills training sim, VERN property) achieves goal_achieved=1 (practice scenario completed) significantly more often than chance. H₀: p=0.5. One-tailed. Wilson CI. | n=21, s=14 rate=66.7% |
p = 0.0946 | Wilson 95% CI [45.4%, 82.8%] on_goal=100% (21/21) |
Just misses significance (p=0.095) due to small n. Sessions coded 0 when user left before practice began — even when Izzy's behavior was exemplary. On_goal=100% and 100% safety event compliance are the stronger claims. Requires n≥30 for adequate power at this effect size. | ⚠ Underpowered (n=21) | ||||||||||||||||||||||||||||||||||||||||||||||||
| Group 8 — Measurement Quality & Regression (Annotated Sample) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Cronbach's Alpha
Rubric Internal Consistency Reliability · Scale analysis
|
Assesses internal consistency of the 4 rubric items (on_goal, engaged, normalised delta_sent, inverted tangent_count) as a composite scale. Alpha < 0.7 suggests items measure distinct constructs rather than a single latent variable. | α = 0.089 k = 4 items n = 668 |
SE = 0.058 95% CI [−0.024, 0.202] |
Item variances: on_goal: 0.018 engaged: 0.075 norm_delta: 0.007 inv_tang: 0.024 |
Alpha of 0.089 is well below the 0.70 threshold for acceptable consistency. This is not a flaw — it confirms that the rubric items measure different dimensions of conversation quality rather than a single latent trait. The rubric should be interpreted as a multi-dimensional profile, not a composite score. | Multidimensional | ||||||||||||||||||||||||||||||||||||||||||||||||
|
Split-Half Reliability
(Spearman-Brown Corrected) Reliability · Internal consistency estimate
|
Single-annotator data prevents true inter-rater reliability. Split-half method (odd/even conversation index) approximates internal consistency. Spearman-Brown correction applied for full-scale estimate. | on_goal: r=0.083 engaged: r=0.238 |
on_goal p=0.132 engaged p<0.001 |
SB-corrected: on_goal: ρ=0.153 engaged: ρ=0.385 |
Engaged shows modest but significant split-half reliability (SB=0.385, p<0.001), suggesting the annotator applied it somewhat consistently across conversations. On_goal split-half is non-significant (SB=0.153), consistent with ceiling clustering. Formal inter-rater reliability (target κ>0.70) was not possible with single-annotator data. | ⚠ Single annotator | ||||||||||||||||||||||||||||||||||||||||||||||||
|
Logistic Regression
Predictors of Full Engagement Inferential · Multivariate regression
|
Binary logistic regression predicting engaged=1.0 (full engagement) from 5 standardised predictors. Manual maximum likelihood estimation via BFGS optimiser. Standard errors from Hessian inverse. Outcome: AUC and per-predictor odds ratios. |
AUC=0.712, 95%CI=[0.674,0.751]
McFadden R²=0.105 n=668, converged=True |
AUC = 0.712 SE = 0.0196 95%CI [0.674, 0.751] McFadden R² = 0.105 |
Opening mood (OR=1.91) and mood lift (OR=1.95) are the strongest predictors of whether a human fully engages. Being on VERN vs control (is_vern, OR=1.09) and tangent frequency are not significant predictors once mood is controlled. The model explains ~10% of variance (McFadden R²=0.105) — adequate for a multi-construct behavioural outcome. | AUC=0.712 | |||||||||||||||||||||||||||||||||||||||||||||||||
|
Point-Biserial Correlation
Predictors vs Engagement Descriptive · Bivariate association
|
Pearson correlation between each continuous/binary predictor and the binary engaged=1.0 outcome (point-biserial interpretation). Supplements the logistic regression with marginal associations. |
|
Strongest: delta_sent r=0.223 open_sent r=0.205 Weakest: is_vern r=0.011 |
Mood-related variables (opening tone and mood trajectory) are the primary drivers of engagement. The VERN/control distinction has essentially zero marginal correlation with engagement (r=0.011) once mood is accounted for. This reinforces the conclusion that BCMs operate on session structure (graceful exit) more than on annotator-perceived conversation quality. | Mood drives engagement | |||||||||||||||||||||||||||||||||||||||||||||||||
Methodological Notes & Limitations
1
All tests are observational. No randomisation was used. Control and VERN personas differ by operator, user population, deployment channel, and time period. Causal inference is not supported.
2
Mood delta (delta_sent = close_sent − open_sent) is an annotator-perceived mood proxy on a 5-point ordinal scale. It is not a measure of VERN's utterance-level emotional signal system, which distinguishes discrete emotional states in real time. These are categorically different phenomena. Mood delta comparisons between groups are not reported as primary outcomes.
3
Cronbach's α = 0.089 indicates rubric items are multidimensional, not a single latent scale. This is the correct design for a behavioural quality instrument covering distinct constructs (goal adherence, human openness, character fidelity, outcome quality). Do not sum or average rubric items into a single score.
4
Binomial tests for on_goal rate are significant for both VERN and control personas, confirming a ceiling effect at the 3-value scale level. The metric is useful for detecting individual persona failures but cannot differentiate groups when both cluster at 1.0.
5
Craig's verified goal completion (est. 94–97%) is based on audit criteria (pitch content delivered + on-role + clean session end) rather than raw goal_achieved, which undercounts due to slide-navigation UX issues outside Craig's behavioral control. The verified rate is not formally tested due to the complexity of the criterion; the 100% on_goal rate (binomial p=0.008) and 97% graceful exit (production) are the reported statistics.
6
Single-annotator design prevents formal inter-rater reliability calculation (target κ > 0.70). Split-half reliability is reported as a construct estimate only. Future annotation rounds should include 10% double-annotation for κ measurement.
7
The logistic regression AUC of 0.712 indicates moderate discriminative ability for predicting full engagement. The is_vern predictor is not significant (OR=1.09, p=0.34) after controlling for mood variables, consistent with the finding that BCMs primarily affect session structure rather than annotated conversation quality in the current dataset.
8
Transcript annotation using VERN AI for utterance-level emotional state labelling is ongoing. That dataset, when complete, will enable direct comparison of VERN's affective system outputs against conversation outcomes — a categorically different and more appropriate test of VERN's emotional intelligence capability.
