Blog

A controller that terminates 2,000 APs and 30,000 users is a large failure domain. Before AOS 8, Aruba networks protected it with VRRP and backup controllers, and a failover meant APs rebooting onto another controller and users re-authenticating — tens of seconds to minutes of disruption. AOS 8 clustering changed that: controllers share the load and keep standby copies of each other’s AP and user state, so a controller failure is a sub-second event that users do not notice. This post explains the older mechanisms briefly, then clustering in depth, followed by roaming and Live Upgrade, which both build on the cluster.

Before Clustering: VRRP, LMS and HA Groups

These mechanisms still matter on standalone controllers and on networks being migrated:

• VRRP — two controllers share a virtual IP. APs discover the VIP; if the primary fails, the backup assumes the VIP and APs re-establish their tunnels to it. APs reboot or re-bootstrap and clients re-authenticate.
• LMS / backup LMS — the AP system profile names the controller the AP should terminate on (lms-ip) and a fallback (bkup-lms-ip), with optional preemption back to the primary. Simple, but every failover is a re-bootstrap.
• HA groups (AP fast failover) — controllers are paired as active and standby (or dual-active); APs pre-build a standby tunnel to the standby controller, so AP failover is fast. User state is not synchronized, so clients still re-authenticate. HA groups and clustering cannot be used together.

Clustering Concepts

A cluster is a group of managed devices of the same family, running the same AOS version, that act as a single pool of capacity. Up to twelve 7200-series or virtual controllers can form a cluster (fewer for 7000-series branch controllers — four), and a cluster requires a Mobility Conductor.

AP Anchoring

When an AP comes up it contacts any cluster member (whatever DHCP option 43 or DNS points to), learns the full member list, and is assigned an AAC and an S-AAC by the leader. The AP keeps an active tunnel to the AAC and a standby tunnel to the S-AAC at all times. The leader spreads APs evenly across members, accounting for each platform’s capacity.

User Anchoring and the Bucket Map

The UAC is chosen independently of the AP: the client’s MAC address is hashed into one of 256 buckets, and the bucket map says which member is the UAC (and S-UAC) for that bucket. The AP builds tunnels to the other members as needed and forwards each client’s traffic directly to that client’s UAC.

This separation is the clever part of the design. The UAC is a function of the client’s MAC address only, so when the client roams from an AP anchored on MC1 to one anchored on MC3, its UAC does not change: the new AP simply sends that client’s traffic to the same UAC. The client keeps its IP address, VLAN, role, firewall sessions and encryption keys. From the network’s point of view, the roam never happened.

L2-Connected vs L3-Connected

When a cluster forms, members probe each other to check that every user VLAN exists on every member (the VLAN probe):

• L2-connected — all members share all user VLANs. Users can be placed on any member, and with redundancy enabled the cluster maintains S-UACs for hitless user failover. This is the design to aim for.
• L3-connected — members do not share all VLANs (for example, controllers in two buildings with different user subnets). AP load balancing and AP failover still work, but stateful user failover does not.

Configuring a Cluster

(MCR-1) [mynode] #cd /md/USA/HQ
(MCR-1) [HQ] #configure terminal

lc-cluster group-profile HQ-CLUSTER
    controller 10.1.10.21 priority 200 mcast-vlan 0 vrrp-ip 10.1.10.31 vrrp-vlan 10 group 0
    controller 10.1.10.22 priority 150 mcast-vlan 0 vrrp-ip 10.1.10.32 vrrp-vlan 10 group 0
    controller 10.1.10.23 priority 128 mcast-vlan 0 vrrp-ip 10.1.10.33 vrrp-vlan 10 group 0
    redundancy
!
lc-cluster group-membership HQ-CLUSTER
!
write memory

• Each controller line identifies a member by its controller-ip (Part 2).
• priority influences leader election.
• vrrp-ip/vrrp-vlan give each member an extra VRRP address that its standby can take over. These addresses are used as the NAS address for RADIUS, so that CoA messages from ClearPass always reach whichever member currently holds the user — even after a failover (see Part 5).
• redundancy enables standby UACs and S-AACs with full state synchronization.
• group-membership at the HQ node makes every controller beneath it a member. Membership can also be set on individual device nodes.

Verification

(HQ-MC1) #show lc-cluster group-membership

Cluster Enabled, Profile Name = "HQ-CLUSTER"
Redundancy Mode On
Active Client Rebalance Threshold = 50%
Standby Client Rebalance Threshold = 75%
Unbalance Threshold = 5%
AP Load Balancing: Enabled
...
Cluster Info Table
------------------
Type IPv4 Address    Priority Connection-Type STATUS
---- --------------- -------- --------------- ------
self      10.1.10.21      200             N/A CONNECTED (Leader)
peer      10.1.10.22      150    L2-Connected CONNECTED (Member, last HBT_RSP 10ms ago, RTD = 0.4 ms)
peer      10.1.10.23      128    L2-Connected CONNECTED (Member, last HBT_RSP 12ms ago, RTD = 0.5 ms)

(HQ-MC1) #show lc-cluster vlan-probe status         ! which VLANs failed the probe, if any
(HQ-MC1) #show lc-cluster load distribution ap      ! APs per member
(HQ-MC1) #show lc-cluster load distribution client  ! clients per member
(HQ-MC1) #show ap database long                     ! "Switch IP" = AAC, "Standby IP" = S-AAC

A member that shows L3-Connected when L2 was intended almost always has a user VLAN missing from its uplink trunk or from its VLAN configuration — the VLAN probe output names it.

Hitless Failover

When a member fails, the members that were S-AAC for its APs promote their standby tunnels, and the members that were S-UAC for its users promote their standby user entries. Because keys, roles and firewall sessions were already synchronized, clients keep their associations and sessions; long-lived flows such as voice calls and VPNs typically survive. The leader then recomputes the bucket map and assigns new standbys so the cluster is redundant again.

Capacity Planning: N-1

Redundancy means every AP and user is held twice, so usable capacity is roughly half of the raw total — or, put differently, the cluster must be able to carry the full load with one member down. A three-node cluster of 7220s (1,024 APs each) comfortably carries about 2,000 APs with redundancy; planning for 3,000 leaves nowhere to fail over to. Size clusters for N-1 at peak, and for maintenance windows with one member upgrading.

Load Balancing

The leader rebalances both APs and clients. AP load balancing redistributes AAC assignments when members join or leave. Client load balancing moves UAC assignments (by editing the bucket map) when a member’s active or standby client load crosses configurable thresholds relative to the others — the rebalance and unbalance thresholds shown in the output above. Rebalancing is designed to be hitless for clients, since it uses the same state-transfer machinery as failover.

Design Notes for Clusters

• All members must run the same AOS version and be the same family (do not mix 7000- and 7200-series).
• Members should be L2-connected: trunk every user VLAN to every member.
• Keep latency between members low; clustering is designed for controllers in the same data centre or campus core, not across a WAN.
• Point AP discovery (DHCP option 43 or DNS) at more than one member, or at a VIP, so APs can always find the cluster.
• Use the per-member VRRP addresses and define them as RADIUS clients on ClearPass, so authentication and CoA survive failover.
• Separate clusters for separate failure domains: a campus cluster and a separate cluster for Remote APs is common.

Roaming

Inside a Cluster

Thanks to the UAC, a roam between any two APs of the same cluster is a Layer 2 roam: no new DHCP, no new role, no firewall state lost. The remaining delay is the 802.11 reassociation and key exchange, which fast-roaming protocols minimize.

Fast Roaming: OKC and 802.11r

A full 802.1X authentication takes hundreds of milliseconds to several seconds — far too long for a voice call. Two mechanisms avoid repeating it on every roam:

• Opportunistic Key Caching (OKC) — the PMK from the first authentication is cached and reused on any AP of the same WLAN; the client only performs the 4-way handshake. Widely supported by Windows, less so elsewhere. Controlled by opp-key-caching in the 802.1X profile.
• 802.11r Fast BSS Transition (FT) — the standard method, supported by iOS, Android, macOS and Windows. Key derivation is folded into the authentication/reassociation frames, cutting roam time to tens of milliseconds.

wlan dot11r-profile CORP-11R
    dot11r
!
wlan ssid-profile CORP-SSID
    dot11r-profile CORP-11R

Some older clients misbehave when they see FT information in beacons. Test with the actual device fleet; where problems appear, newer releases offer an adaptive/mixed mode, or 802.11r can be enabled only on the voice WLAN.

802.11k and 802.11v

• 802.11k (radio resource management) — the AP gives clients a neighbour report, a list of nearby APs and channels, so clients scan fewer channels when they need to roam. Configured with a wlan dot11k-profile referenced by the VAP.
• 802.11v (BSS Transition Management) — the network can suggest that a client move to a better AP. ClientMatch (Part 7) uses 802.11v as its preferred steering method for clients that support it.

Between Clusters

A client moving between clusters (for example between two campuses) changes UAC. It re-authenticates (fast-roaming caches are per cluster) and, if the user VLANs differ, obtains a new IP address. Applications that cannot tolerate an IP change need the two areas served by the same cluster, or a common VLAN design. Older releases offered Layer 3 mobility (mobile IP between controllers) for this case; modern designs avoid needing it.

Live Upgrade

Upgrading a controller traditionally meant a maintenance window: the controller reloads, every AP downloads the new image and reboots, and the whole site is down for 10–20 minutes. AOS 8 Live Upgrade, orchestrated by the Conductor, upgrades a cluster with no coverage outage:

1. The Conductor is upgraded first (it may manage MDs on older versions, never newer).
2. The new image is copied to each cluster member, and APs pre-load the new AP image while still serving clients.
3. Using AirMatch’s RF neighbour data, APs are split into partitions such that neighbouring APs are in different partitions.
4. Cluster members are upgraded one at a time. APs move — partition by partition — to upgraded members and reboot onto the new image, while their neighbours in other partitions keep serving clients.
5. When all members and APs run the new version, the cluster rebalances.

Live Upgrade is started from the Conductor (WebUI under maintenance/software management, or the equivalent upgrade-profile CLI), pointing at an image server. Its prerequisites are a cluster with redundancy, AirMatch data and enough spare capacity to carry the load with one member out.

Loadable Service Modules

Some Conductor services — AirMatch, AppRF signatures, UCC (Unified Communications and Collaboration) and others — are packaged as Loadable Service Modules that can be upgraded independently of the main AOS image, without reloading the Conductor. Bug fixes and new application signatures can thus be deployed without a platform upgrade.

Exercises

1. A four-node cluster of 7210s (512 APs each) currently serves 1,500 APs with redundancy enabled. Is the design sound? What happens if one member fails?

2. show lc-cluster group-membership shows one peer as L3-Connected. Users on that member are not protected by hitless failover. What is the likely cause and how is it found?

3. A client’s MAC hashes to bucket 200. With the bucket map in Figure 2, which member is its UAC? It roams from an AP anchored on MC1 to one anchored on MC2 — what changes?

4. After a controller failure in a cluster, ClearPass CoA messages for some users time out. Users are otherwise fine. What was probably not configured?

5. A hospital’s voice handsets drop calls when roaming, although they stay in the same cluster. Authentication is EAP-PEAP. What should be checked and enabled?

Summary

AOS 8 clusters pool up to twelve controllers. The leader assigns each AP an anchor (AAC) and a standby (S-AAC) with pre-built tunnels, and assigns each client a user anchor (UAC) and standby (S-UAC) via a 256-bucket map keyed on the client’s MAC address. Because the UAC never changes as a client roams within the cluster, roams are seamless; because standby state is synchronized in an L2-connected cluster with redundancy, controller failures are hitless. Redundancy halves usable capacity, so clusters are sized for N-1. Per-member VRRP addresses keep RADIUS and CoA working through failover. 802.11r or OKC removes re-authentication from roams, 802.11k and 802.11v help clients pick the right AP, and Live Upgrade uses the cluster and AirMatch partitions to upgrade without a coverage outage.

Next: Part 7 closes the series with RF — ARM, AirMatch, ClientMatch, radio and regulatory profiles, spectrum and WIDS — and a structured troubleshooting toolkit, plus a look at how AOS 10 changes the picture.