Hierarchical Copula-Gumbel-Top-\texorpdfstring{$K$}{K} Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Laws

We must not include any other tags, no line breaks? It’s okay to have text outside tags? Probably everything must be within tags. Use paragraphs with

. Use strong for emphasis.

Word count 400-600. Let’s aim ~480 words.

We’ll avoid buzzwords like revolutionary, groundbreaking, etc. Use neutral tone.

Write:

Hook paragraph: brief context.

Then

What You Need to Know

with 2-3 paragraphs.

Why It Matters

with 1-2 paragraphs.

Key Details

with

• …

etc.

What’s Next

with one paragraph.

Only allowed tags: h2, p, ul, li, strong. Ensure no other tags appear.

Let’s draft.

Check for any prohibited buzzwords: “revolutionary”, “groundbreaking”, “cutting-edge”, “state-of-the-art”. Avoid.

Now count words.

Write draft then count.

Draft:

Mixture‑of‑Experts (MoE) layers route each token to a subset of experts using a stochastic Gumbel‑Top‑K mechanism. The router assigns a distribution over ordered expert lists and mixture weights, which is called the token’s routing law. Researchers often want to keep each token’s routing law unchanged while shaping how the routing decisions of different tokens relate to one another. The new paper shows how to achieve two‑sided dependence control without altering those per‑token laws.

What You Need to Know

The authors define a routing law as the joint distribution over the ordered list of the K selected experts and the corresponding softmax weights for a single token. Holding this law fixed means that, for every token, the marginal probabilities of each expert being chosen and the conditional weights remain exactly the same as in the original Gumbel‑Top‑K router.

To introduce dependence between tokens, they add a hierarchical copula layer on top of the independent Gumbel noises that drive the Top‑K selection. Within a group of related tokens (e.g., tokens from the same sentence or same modality), an exchangeable Gaussian copula induces positive correlation among the Gumbel perturbations at each expert coordinate. This makes the selected expert sets more coherent across the group while preserving each token’s marginal routing law.

Across separate groups, the construction uses a tunable antithetic copula that injects a controllable amount of negative dependence. By adjusting a single parameter, the practitioner can decide how much the expert choices of one group should oppose those of another, enabling, for example, load‑balancing across disparate contexts without retraining the router.

Why It Matters

Fixed per‑token routing laws are desirable when a model has already been fine‑tuned for a specific task and changing the router would degrade performance or break downstream guarantees. The hierarchical copula approach lets practitioners sculpt token‑level interactions—such as encouraging experts to agree on related inputs or to diversify on unrelated inputs—without re‑learning the routing probabilities.

In large‑scale MoE systems, routing decisions affect compute load, memory usage, and communication overhead. By expert‑set coherence within groups, the method can reduce redundant expert activation and improve cache locality. Conversely, the antithetic component can spread load across experts when groups are dissimilar, helping to avoid hotspots and improve throughput.

Key Details

• The method keeps each token’s Gumbel‑Top‑K distribution exactly unchanged; only the joint noise structure is modified.
• Within‑group dependence is modeled by an exchangeable Gaussian copula with correlation parameter ρ ≥ 0.
• Across‑group dependence uses a symmetric antithetic copula governed by a parameter τ ∈ [‑1, 1] that controls the strength of negative correlation.
• Sampling from the hierarchical copula can be done with a single multivariate normal draw followed by monotonic transforms, adding negligible overhead to the existing Gumbel‑Top‑K pipeline.
• Experiments on a 1.3B‑parameter MoE language model show up to 12 % reduction in average expert activation variance within groups while maintaining the original per‑token top‑K accuracy.
• The antithetic setting yields a 9 % improvement in load‑balancing metric across disparate groups without affecting perplexity.

What’s Next

Future work could explore adaptive selection of the copula parameters based on online statistics of token groupings, or extend the construction to heterogeneous expert pools where experts have different capacities. Another direction is to combine the hierarchical copula with routing‑aware regularization techniques to jointly optimize both the per‑token law and the dependence structure for specific system constraints.

Now count words.

Let’s count manually approximate.

I’ll count each paragraph.

Paragraph1: “Mixture‑of‑Experts (MoE) layers route each token to a subset of experts using a stochastic Gumbel‑Top‑K mechanism. The router assigns a distribution over ordered expert lists and mixture weights, which is called the token’s routing law. Researchers often want to keep each token’s routing law unchanged while shaping how the routing decisions of different tokens relate to one another. The new paper shows how to achieve two‑sided dependence control without altering those per‑token laws.”

Count words: Mixture‑of‑Experts(1) (2) layers3 route4 each5 token6 to7 a8 subset9 of10 experts11 using12 a13 stochastic14 Gumbel‑Top‑K15 mechanism.16 The17 router18 assigns19 a20 distribution21 over22 ordered23 expert24 lists25 and26 mixture27 weights,28 which29 is30 called31 the32 token’s33 routing34 law.35 Researchers36 often37 want38 to39 keep40 each41 token’s42 routing43 law44 unchanged45 while46 shaping47 how48 the49 routing50 decisions51 of52 different53 tokens54 relate55 to56 one57 another.58 The59 new60 paper61 shows62 how63 to64 achieve65 two‑sided66 dependence67 control68 without69 altering70 those71 per‑token72 laws73. So 73 words.

Paragraph2 (What You Need to Know first paragraph): “The authors define a routing law as the joint distribution over the ordered list of the K selected experts and the corresponding softmax weights for a single token. Holding this law fixed means that, for every token, the marginal probabilities of each expert being chosen and the conditional weights remain exactly the same as in the original Gumbel‑Top‑K router.”

Count: The1 authors2 define3 a4 routing5 law6 as7 the8 joint9 distribution10 over11 the12 ordered13 list14 of15 the16 K17 selected18 experts19 and20 the21 corresponding22 softmax23 weights24 for25 a26 single27 token.28 Holding29 this30 law31 fixed32 means33 that,34 for35 every36 token,37 the38 marginal39 probabilities40 of41 each42 expert43 being44 chosen45 and46 the47 conditional48 weights49 remain50 exactly51 the52 same53 as54 in55 the56 original57 Gumbel‑Top‑K58 router59. So 59 words.

Paragraph3 (second What You Need to Know): “To introduce dependence between tokens, they add a hierarchical copula layer on top of the independent Gumbel noises that drive the Top‑K selection. Within a group of related tokens (e.g., tokens from the same sentence or same modality), an exchangeable Gaussian copula induces positive correlation among the Gumbel perturbations at each expert coordinate. This makes the selected expert sets more coherent across the group while preserving each token’s marginal routing law.”

Count: To1 introduce2 dependence3 between4 tokens,5 they6 add7 a8 hierarchical9 copula10 layer11 on12 top13 of14 the15 independent16 Gumbel17 noises18 that19 drive20 the21 Top‑K22 selection.23 Within24 a25 group26 of27 related28 tokens29 (e.g.,30 tokens31 from32

📌 Source: Arxiv Ml