Visualizing Hyperbolic SVM Boundaries: Moment Relaxation Outperforms PGD and SDP

Beyond Euclidean Space: The Hyperbolic SVM Challenge

Hyperbolic Support Vector Machines (HSVMs) offer powerful classification capabilities for complex hierarchical data structures. However, their non-convex optimization landscape presents significant challenges. Traditional methods like Projected Gradient Descent (PGD) often converge to poor local minima, while Semidefinite Programming (SDP) relaxations provide theoretical guarantees but may yield suboptimal boundaries in practice.

Decision Boundaries Under the Microscope

The Harvard team visualized boundaries across three techniques:

1. PGD: Prone to convergence failures
2. SDP: Theoretical robustness but practical limitations
3. Moment Relaxation: Novel sparse sum-of-squares approach

In Gaussian-1 datasets, SDP and Moment produced nearly identical boundaries (differing by machine precision), both substantially different from PGD outputs. This visual divergence explains the performance gaps noted in quantitative metrics.

The dramatic difference emerged in Tree-2 datasets: PGD collapsed to a trivial boundary (classifying all samples identically), while Moment maintained balanced class separation. As researcher Sheng Yang notes: "Moment relaxation finds linear separators of the best quality where PGD fails catastrophically due to sensitivity to initialization."

Quantitative Validation

Across 15 random seeds and configurations (classes K=2-5, scales s=0.4-1.0):

• Moment improved test accuracy by 8-22% over PGD
• Outperformed SDP by 3-15% in F1 scores
• Demonstrated particular strength in multiclass scenarios (K=5)

While Moment showed higher training loss in simple cases (K=2), researchers note this reflects extraction heuristic limitations rather than optimization quality. Optimality gaps confirmed Moment's superior solution quality with 40-60% smaller gaps than SDP.

The Computational Tradeoff

Moment's accuracy comes at cost:

Method	Relative Runtime
PGD	1x
SDP	5-8x
Moment	10-15x

This scalability challenge currently limits application to large datasets, though the team suggests parallelization and approximation techniques could bridge this gap.

Real-World Performance

In benchmark datasets using one-vs-rest schemes:

• Moment outperformed PGD/SDP across all regularization strengths
• Maintained 7-12% accuracy advantages in imbalanced classes
• Consistently achieved lower optimality gaps

The research concludes that sparse moment relaxation represents a significant advance in hyperbolic SVM optimization, particularly valuable for complex classification tasks where PGD fails and SDP underperforms.

Visualizations and experimental data from Yang et al. (Harvard University); Full paper available on arXiv under CC BY-SA 4.0 license.