Interpretable Multimodal Classification with Linear Discriminant Tree Ensembles

When models that combine text, audio, and video are asked to judge human affect or behavior, they face a trade‑off: deliver high predictive performance while also giving clinicians, teachers, or users a clear picture of why a particular decision was made. Recent work shows that large transformer‑based architectures excel at the former but obscure the latter, spreading information across dense, nonlinear layers that resist simple attribution. This post examines a recent proposal that attempts to reconcile the two goals by turning to a different family of learners.

What You Need to Know

The paper introduces Linear Discriminant Tree Ensembles (LDTE) as a framework for multimodal classification that remains both accurate and interpretable. Each base learner in the ensemble is a decision tree whose splits are guided by linear discriminant analysis, a technique that finds directions in feature space maximizing class separation while keeping the model’s internal representation linear and additive. Because the splits are based on simple linear combinations of the raw modalities (e.g., a weighted sum of certain TF‑IDF text features, Mel‑frequency cepstral coefficients, and facial action unit intensities), the resulting tree can be inspected to see which modality and which specific features contributed most to a decision.

Ensembling many such trees improves robustness and predictive power, much like traditional random forests or gradient‑boosted trees, but the linear discriminant constraint preserves a clear link between input features and the model’s logic. During training, the algorithm jointly optimizes the tree structure and the discriminant vectors, allowing the ensemble to capture interactions across modalities without resorting to opaque deep nets. Evaluation on several benchmark affect‑recognition datasets shows that LDTE achieves accuracy within a few points of state‑of‑the‑art transformer models while providing per‑instance feature importance scores that align with human judgments of relevance.

Why It Matters

In settings where decisions affect people’s wellbeing—such as monitoring depression symptoms in tele‑health sessions or assessing engagement in online learning environments—stakeholders need to trust the system’s output. Trust is not only a matter of raw accuracy; it also depends on the ability to audit, explain, and potentially contest a model’s recommendation. By delivering explanations that are directly traceable to measurable cues (e.g., a rise in vocal pitch combined with increased head movement), LDTE offers a practical path toward accountability without sacrificing competitive performance.

Moreover, the interpretability afforded by linear discriminants can facilitate model debugging and feature engineering. If a particular modality consistently receives low weight across trees, practitioners can investigate data quality or consider alternative representations. This transparency also supports regulatory compliance in domains where explainability is mandated, making the approach attractive for real‑world deployment beyond research prototypes.

Key Details

• Model core: Each tree uses linear discriminant analysis to determine split directions, preserving additive feature contributions.
• Ensemble strategy: Boosting or bagging of LDTEs improves accuracy while keeping individual tree explanations intact.
• Input handling: Raw modalities (text TF‑IDF, audio MFCCs, video AU intensities) are concatenated; no complex preprocessing like token‑level embeddings is required.
• Evaluation: Tested on CMU-MOSEI, IEMOCAP, and a classroom engagement benchmark; LDTE matches transformer F1 scores within 2‑3% and yields feature importance correlations >0.7 with human‑annotated saliency maps.
• Computation: Training time scales linearly with number of trees and features, offering a faster alternative to fine‑tuning large transformer networks.
• Interpretability output: For each prediction, the model returns a list of modality‑specific weights and the tree path taken, enabling straightforward visual explanations.

What’s Next

Future work could explore hybrid designs where LDTEs handle the early, interpretable fusion stages and a lightweight transformer refines residuals, aiming to push accuracy closer to the top tier while retaining explainable foundations. Additionally, extending the linear discriminant criterion to kernel‑based variants might capture nonlinear interactions without abandoning the ability to trace decisions back to original features. As multimodal affect sensing moves into real‑world applications, approaches like LDTE that balance performance with transparency will likely see increased adoption in both industry and clinical pipelines.

📌 Source: Arxiv Ai