4.1 Network Diagnostic Tools
Before telemetry platforms existed, engineers troubleshot with four tools: ping, traceroute, show, and debug. These remain the fastest way to validate reachability, path, and control-plane behavior. The exam tests all four plus conditional-debug filtering.
Ping and Traceroute
Ping sends ICMP Echo Request (Type 8) and expects an Echo Reply (Type 0). Traceroute incrementally increases the IP TTL, using the ICMP Time Exceeded (Type 11) replies from each hop to map the path. Cisco IOS traceroute uses UDP (ports 33434+) by default; Windows tracert uses ICMP Echo.
| Tool | Protocol | Purpose | Extended Options |
|---|---|---|---|
| ping | ICMP Echo | End-to-end reachability | Source interface, size, repeat count, DF bit, sweep |
| extended ping | ICMP Echo | Source-spoofed test from specific interface | Enter via ping then answer prompts |
| traceroute | UDP (IOS) / ICMP | Discover forwarding path per-hop | Timeout, probe count, TTL, source |
| pathping / mtr | ICMP | Continuous path + loss stats (non-Cisco) | — |
! = success, . = timeout (no reply), U = destination unreachable (ICMP Type 3), Q = source quench, M = MTU exceeded (DF bit set), ? = unknown. An all-timeout response may indicate an ACL blocking ICMP Echo Reply rather than the probe itself.Debug and Conditional Debug
debug enables real-time process output (adjacency events, route updates, auth attempts). It is CPU-expensive — an unfiltered debug ip packet on a busy router will crash it. Always direct output to the buffer via logging buffered, not the console.
Conditional debug scopes the debug to a specific interface, MAC, or IP to limit impact:
R1# debug condition interface GigabitEthernet0/1 R1# debug condition ip 10.1.1.10 R1# debug ip packet detail ! Only packets matching above conditions R1# show debug condition ! View active conditions R1# undebug all ! Stop all debugs (also: u all)
1) Set condition FIRST, then enable debug. 2) Redirect output to buffer:
logging buffered 16384 debugging + no logging console. 3) Collect with show log. 4) Always undebug all when done.4.2 SNMP and Syslog
SNMP (Simple Network Management Protocol)
SNMP is the traditional pull-based management protocol. Agents expose structured data through a MIB (Management Information Base) indexed by OIDs (Object Identifiers). Managers poll with GET, and agents push unsolicited TRAPs on events.
| Version | Auth | Encryption | Exam Note |
|---|---|---|---|
| SNMPv1 | Community string (clear-text) | None | Deprecated — insecure, 32-bit counters |
| SNMPv2c | Community string (clear-text) | None | 64-bit counters, GetBulk; still common but insecure |
| SNMPv3 | User-based (HMAC-MD5/SHA) | DES/3DES/AES | Three levels: noAuthNoPriv, authNoPriv, authPriv |
| Operation | Direction | Purpose |
|---|---|---|
| GET | Manager → Agent | Request single OID value |
| GETNEXT | Manager → Agent | Walk the MIB tree |
| GETBULK | Manager → Agent (v2c+) | Retrieve multiple OIDs efficiently |
| SET | Manager → Agent | Modify configuration |
| TRAP | Agent → Manager | Unsolicited event notification (fire-and-forget) |
| INFORM | Agent → Manager | Like TRAP but acknowledged (v2c+) |
noAuthNoPriv = username only. authNoPriv = HMAC-SHA/MD5 authenticates messages. authPriv = auth + AES/DES encryption. Always use authPriv in production; match security level in snmp-server user configuration and on the NMS.Syslog
Syslog is push-based, one-way logging over UDP/514 (or TCP/6514 with TLS). Messages include a facility (source subsystem) and severity (0–7). Lower number = more severe.
| Level | Keyword | Description | Example |
|---|---|---|---|
| 0 | emergencies | System unusable | Kernel panic |
| 1 | alerts | Immediate action needed | Temperature exceeded |
| 2 | critical | Critical conditions | Hardware failure |
| 3 | errors | Error conditions | Interface down |
| 4 | warnings | Warning conditions | Config change |
| 5 | notifications | Normal but significant | Line protocol up |
| 6 | informational | Info messages | ACL hits |
| 7 | debugging | Debug output | debug commands |
Syslog message format: <PRI>seq: timestamp: %FACILITY-SEVERITY-MNEMONIC: description
Example: *Apr 18 15:23:07.142: %LINEPROTO-5-UPDOWN: Line protocol on Interface Gi0/1, changed state to up
4.3 NetFlow and Flexible NetFlow
NetFlow is Cisco’s traffic telemetry technology. It identifies flows — unidirectional sequences of packets sharing the same 5/7-tuple (source/dest IP, source/dest port, protocol, ToS, ingress interface) — and exports aggregated records to a collector for analysis.
| Feature | Traditional NetFlow (v5/v9) | Flexible NetFlow | IPFIX (v10) |
|---|---|---|---|
| Fields | Fixed 7-tuple | User-configurable match/collect | Standards-based, extensible |
| IPv6 | Only v9 | Yes | Yes |
| Templates | v9 only | Yes | Yes |
| Standard | Cisco proprietary | Cisco | IETF RFC 7011 |
| Common Port | UDP 2055 / 9995 | UDP 2055 / 9995 | UDP 4739 |
Flexible NetFlow building blocks (in configuration order):
- Flow Record — defines which fields to
match(keys) andcollect(non-keys) - Flow Exporter — destination collector IP, UDP port, transport protocol
- Flow Monitor — binds record + exporter; applied to interface
- Flow Sampler (optional) — samples 1-in-N packets to reduce CPU
SNMP polls device counters (interface bytes, CPU). NetFlow exports per-flow details (who talked to whom, how much, when). Use both: SNMP for device health, NetFlow for traffic analysis and security (Stealthwatch detects anomalies from flow baselines).
4.4 SPAN / RSPAN / ERSPAN (Port Mirroring)
Port mirroring copies frames from source ports/VLANs to a destination port connected to a packet analyzer (Wireshark, IDS, DPI appliance). The three flavors differ by where the destination can be:
| Type | Destination Location | Transport | Use Case |
|---|---|---|---|
| SPAN (Local SPAN) | Same switch as source | Direct port copy | Analyzer physically attached |
| RSPAN | Different switch, same L2 domain | Dedicated VLAN on trunk | Central analyzer on another switch |
| ERSPAN | Anywhere reachable via IP | GRE encapsulation over Layer 3 | Remote site, data center, virtual |
A port cannot be both a SPAN source and SPAN destination in the same session. The destination port drops incoming packets (by default — use
ingress keyword to allow) and disables MAC learning, so it cannot carry regular traffic.4.5 IP SLA (IP Service Level Agreement)
IP SLA generates synthetic traffic on a schedule to measure network performance: reachability, jitter, one-way delay, loss, HTTP response, DNS, DHCP, UDP echo, and more. Combined with Track objects and EEM (Embedded Event Manager), IP SLA can trigger route changes, interface state actions, or custom scripts.
| Operation | Measures | Typical Use |
|---|---|---|
| icmp-echo | RTT, loss, reachability | Simple WAN reachability probe |
| udp-jitter | Jitter, one-way delay, loss, MOS | VoIP/video quality (requires IP SLA Responder) |
| tcp-connect | TCP 3-way handshake time | Server reachability |
| http | DNS + TCP + HTTP GET time | Web application monitoring |
| dns | Resolution time | DNS server health |
| dhcp | Lease acquisition time | DHCP server health |
IP SLA flow: probe → track → route policy
R1(config)# ip sla 10 R1(config-ip-sla)# icmp-echo 8.8.8.8 source-interface GigabitEthernet0/0 R1(config-ip-sla-echo)# frequency 5 ! Probe every 5 sec R1(config-ip-sla-echo)# threshold 200 ! RTT threshold in ms R1(config-ip-sla-echo)# timeout 1000 R1(config)# ip sla schedule 10 life forever start-time now R1(config)# track 1 ip sla 10 reachability R1(config-track)# delay down 10 up 5 ! Dampening timers R1(config)# ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 1 ! Primary ISP, tracked R1(config)# ip route 0.0.0.0 0.0.0.0 198.51.100.1 10 ! Floating static backup (AD 10)
Only
udp-jitter measures jitter and requires an IP SLA Responder on the far end (ip sla responder). ICMP-echo cannot measure jitter. MOS (Mean Opinion Score) is output by udp-jitter for VoIP quality scoring.4.6 Cisco Catalyst Center (DNA Center) Assurance
Cisco Catalyst Center (rebranded from DNA Center in 2023) is Cisco’s on-premises controller and analytics platform for Catalyst 9000 switches, routers, and WLCs. Its Assurance feature turns every network device into a telemetry sensor — streaming SNMP, NetFlow, Syslog, and model-driven telemetry into a big-data engine for correlation, baselining, and AI-driven root-cause analysis.
| Catalyst Center Pillar | Function | Example |
|---|---|---|
| Design | Define sites, IP pools, AAA servers, SSIDs, wireless RF profiles | Hierarchical site model (Global → Area → Building → Floor) |
| Policy | Define access policies (Cisco ISE / TrustSec / SGT) | Group-based access control |
| Provision | Push templated config to devices; claim devices from PnP | Zero-touch onboarding via PnP |
| Assurance | Real-time health scores, path trace, sensor tests, AI insights | Client 360, Device 360, Application 360 |
| Platform | Northbound REST APIs, webhooks, integrations | Intent API, Integration Hub to ServiceNow |
Key Assurance features to know:
- Health Scores (0–10) for Device, Client, Application, Network — color-coded (red <4, orange 4–7, green >7)
- Path Trace — end-to-end flow visualization with per-hop statistics, policy applied, and QoS marking
- Machine Reasoning Engine (MRE) — guided troubleshooting workflows that gather correlated evidence
- AI Network Analytics — cloud-based ML models compare local data against global baselines
- Sensor tests — Cisco Aironet 1800S sensors run synthetic wireless/app tests to validate user experience
- Issues & Insights — auto-correlated problem tickets with suggested remediations
Catalyst Center Assurance ingests: SNMP (device inventory/counters), syslog (events), NetFlow (traffic), Streaming Telemetry/gRPC (real-time state), Wireless Assurance data, and sensor test results. Model-driven telemetry is the preferred real-time feed — push-based, structured YANG data vs. SNMP’s polling model.
4.7 NETCONF, RESTCONF, and Model-Driven Telemetry
The automation/assurance sections of ENCOR overlap here. Modern IOS-XE exposes a YANG-modeled management plane accessible via three transports: NETCONF (SSH/XML), RESTCONF (HTTPS/JSON|XML), and gNMI (gRPC/Protobuf).
| Protocol | Transport | Encoding | Port | Operations |
|---|---|---|---|---|
| NETCONF | SSH | XML | 830 | get, get-config, edit-config, commit, lock, close-session |
| RESTCONF | HTTPS | JSON or XML | 443 | GET, POST, PUT, PATCH, DELETE |
| gNMI | gRPC over HTTP/2 | Protobuf | 57400 (default) | Get, Set, Subscribe, Capabilities |
YANG (Yet Another Next Generation, RFC 7950) is the data modeling language. It defines the schema — containers, leafs, lists, and data types — that NETCONF/RESTCONF/gNMI operate on. Two YANG flavors:
- Native models — Cisco-specific (
Cisco-IOS-XE-native) covering every IOS-XE CLI feature - OpenConfig — vendor-neutral community models (
openconfig-interfaces,openconfig-bgp) for portable automation - IETF models — standards track (
ietf-interfaces,ietf-routing)
Enabling NETCONF/RESTCONF on IOS-XE
R1(config)# netconf-yang ! Enable NETCONF on port 830 R1(config)# restconf ! Enable RESTCONF on HTTPS R1(config)# ip http secure-server ! Required for RESTCONF R1(config)# aaa new-model R1(config)# aaa authentication login default local R1(config)# aaa authorization exec default local R1(config)# username admin privilege 15 secret Cisco123 R1# show netconf-yang sessions ! Active NETCONF sessions R1# show platform software yang-management process ! Verify ncsshd, confd running
RESTCONF URL Structure
Format: https://<device>/restconf/data/<module>:<container>/<key>
Example — retrieve interface Gi0/1 config:
$ curl -k -u admin:Cisco123 \
-H "Accept: application/yang-data+json" \
https://10.1.1.1/restconf/data/ietf-interfaces:interfaces/interface=GigabitEthernet0%2F1
Model-Driven Telemetry (MDT)
Replaces SNMP polling with push-based streaming. Devices periodically publish YANG-encoded data to a collector. Two subscription styles:
- Cadence-based (periodic) — every N milliseconds
- On-change — only when a monitored leaf value changes (more efficient)
| Dial Style | Who Initiates | Encoding | Transport |
|---|---|---|---|
| Dial-out | Device → Collector | KVGPB (gRPC) or JSON/XML | gRPC, NETCONF |
| Dial-in | Collector → Device | KVGPB | gRPC (gNMI Subscribe) |
R1(config)# telemetry ietf subscription 101 R1(config-mdt-subs)# encoding encode-kvgpb R1(config-mdt-subs)# filter xpath /interfaces-ios-xe-oper:interfaces/interface/statistics R1(config-mdt-subs)# source-address 10.1.1.1 R1(config-mdt-subs)# stream yang-push R1(config-mdt-subs)# update-policy periodic 1000 ! Every 10 sec (centisec) R1(config-mdt-subs)# receiver ip address 10.1.1.100 57500 protocol grpc-tcp
SNMP is pull-based (NMS polls every N min) — stale data, high overhead on agents. MDT is push-based (device streams) — sub-second granularity, efficient binary encoding, and carries full YANG context. Catalyst Center and modern observability stacks prefer MDT.
Hands-On Labs
Lab 1 — Conditional Debug and Ping/Traceroute Troubleshooting
Redirect debug output to the buffer before enabling any debugs. Never run debug to the console on a production router.
R1(config)# logging buffered 65536 debugging R1(config)# no logging console R1(config)# service timestamps debug datetime msec R1(config)# service sequence-numbers
Troubleshoot OSPF adjacency flapping on Gi0/1 with a scoped debug.
R1# debug condition interface GigabitEthernet0/1 R1# debug ip ospf adj R1# show debug condition ! Verify active scope ! ... wait for events ... R1# show log | include OSPF R1# undebug all
Use extended ping to test with source-spoofing and DF-bit to find MTU black holes.
R1# ping Protocol [ip]: Target IP address: 10.2.2.2 Repeat count [5]: 100 Datagram size [100]: 1500 Extended commands [n]: y Source address or interface: Loopback0 Set DF bit in IP header? [no]: yes ! If replies come back with 1500 DF, path MTU ≥ 1500. "M" = MTU exceeded en route. R1# traceroute 10.2.2.2 source Loopback0 R1# traceroute 10.2.2.2 probe 1 ttl 1 15 ! One probe per hop, TTL 1–15
Lab 2 — SNMPv3 (authPriv) and Remote Syslog
Configure SNMPv3 with a user, group, view, and authPriv security.
R1(config)# snmp-server view OPS iso included R1(config)# snmp-server group NOC-GRP v3 priv read OPS R1(config)# snmp-server user nocadmin NOC-GRP v3 auth sha Cisco123! priv aes 256 Secret456! R1(config)# snmp-server host 10.1.1.100 version 3 priv nocadmin R1(config)# snmp-server enable traps config R1(config)# snmp-server enable traps ospf R1(config)# snmp-server enable traps syslog R1# show snmp user R1# show snmp group R1# show snmp host
Send syslog to a remote collector with a specific severity and timestamp format.
R1(config)# service timestamps log datetime msec localtime show-timezone R1(config)# logging host 10.1.1.100 transport tcp port 6514 R1(config)# logging trap informational ! Severity 6 and lower R1(config)# logging source-interface Loopback0 R1(config)# logging facility local6 R1# show logging ! Verify config + internal buffer
Lab 3 — Flexible NetFlow to Stealthwatch Collector
Define a flow record with IPv4 5-tuple + ToS + byte/packet counts.
R1(config)# flow record ENTERPRISE-RECORD R1(config-flow-record)# match ipv4 source address R1(config-flow-record)# match ipv4 destination address R1(config-flow-record)# match ipv4 protocol R1(config-flow-record)# match transport source-port R1(config-flow-record)# match transport destination-port R1(config-flow-record)# match ipv4 tos R1(config-flow-record)# collect counter bytes R1(config-flow-record)# collect counter packets R1(config-flow-record)# collect timestamp sys-uptime first R1(config-flow-record)# collect timestamp sys-uptime last R1(config-flow-record)# collect interface input R1(config-flow-record)# collect interface output
Define exporter and monitor, then apply to interface.
R1(config)# flow exporter SNA-COLLECTOR R1(config-flow-exporter)# destination 10.1.1.100 R1(config-flow-exporter)# source Loopback0 R1(config-flow-exporter)# transport udp 2055 R1(config-flow-exporter)# export-protocol netflow-v9 R1(config)# flow monitor ENTERPRISE-MON R1(config-flow-monitor)# record ENTERPRISE-RECORD R1(config-flow-monitor)# exporter SNA-COLLECTOR R1(config-flow-monitor)# cache timeout active 60 R1(config)# interface GigabitEthernet0/1 R1(config-if)# ip flow monitor ENTERPRISE-MON input R1(config-if)# ip flow monitor ENTERPRISE-MON output R1# show flow monitor ENTERPRISE-MON cache format table R1# show flow exporter SNA-COLLECTOR statistics
Lab 4 — Local SPAN, RSPAN, and ERSPAN
Configure Local SPAN — mirror Gi0/1 (ingress+egress) to Gi0/24 for a Wireshark capture.
SW1(config)# monitor session 1 source interface GigabitEthernet0/1 both SW1(config)# monitor session 1 destination interface GigabitEthernet0/24 SW1# show monitor session 1
Configure RSPAN — source on SW1, destination on SW2, using RSPAN VLAN 999.
! Both SW1 and SW2 — declare RSPAN VLAN SW1(config)# vlan 999 SW1(config-vlan)# name RSPAN-TRANSIT SW1(config-vlan)# remote-span ! SW1 — source side SW1(config)# monitor session 5 source interface GigabitEthernet0/1 both SW1(config)# monitor session 5 destination remote vlan 999 ! SW2 — destination side SW2(config)# vlan 999 SW2(config-vlan)# remote-span SW2(config)# monitor session 5 source remote vlan 999 SW2(config)# monitor session 5 destination interface GigabitEthernet0/24
Configure ERSPAN — GRE-encapsulated mirror over Layer 3 to a remote analyzer at 10.50.50.50.
! Source switch SW1(config)# monitor session 10 type erspan-source SW1(config-mon-erspan-src)# source interface GigabitEthernet0/1 both SW1(config-mon-erspan-src)# destination SW1(config-mon-erspan-src-dst)# erspan-id 100 SW1(config-mon-erspan-src-dst)# ip address 10.50.50.50 SW1(config-mon-erspan-src-dst)# origin ip address 10.1.1.1 ! Loopback IP recommended SW1(config-mon-erspan-src-dst)# no shutdown ! Destination switch (receives decapsulated copy) SW2(config)# monitor session 10 type erspan-destination SW2(config-mon-erspan-dst)# destination interface GigabitEthernet0/24 SW2(config-mon-erspan-dst)# source SW2(config-mon-erspan-dst-src)# erspan-id 100 SW2(config-mon-erspan-dst-src)# ip address 10.50.50.50 SW2(config-mon-erspan-dst-src)# no shutdown
Lab 5 — IP SLA with Object Tracking and EEM
Configure an ICMP-echo probe and track its reachability to drive failover.
R1(config)# ip sla 1 R1(config-ip-sla)# icmp-echo 203.0.113.1 source-interface GigabitEthernet0/0 R1(config-ip-sla-echo)# frequency 3 R1(config-ip-sla-echo)# timeout 1000 R1(config-ip-sla-echo)# threshold 500 R1(config)# ip sla schedule 1 life forever start-time now R1(config)# track 1 ip sla 1 reachability R1(config-track)# delay down 5 up 10
Configure a UDP-jitter probe (requires Responder on far end) to measure VoIP quality.
R2(config)# ip sla responder ! Far-end responder R1(config)# ip sla 2 R1(config-ip-sla)# udp-jitter 10.2.2.2 5000 codec g729a R1(config-ip-sla-jitter)# frequency 60 R1(config-ip-sla-jitter)# num-packets 100 R1(config)# ip sla schedule 2 life forever start-time now R1# show ip sla statistics 2 R1# show ip sla statistics 2 details | include MOS|Jitter|Loss
Tie it together with EEM — when track goes down, send an email and reroute.
R1(config)# event manager applet WAN-DOWN R1(config-applet)# event track 1 state down R1(config-applet)# action 1.0 syslog msg "WAN primary down — failing over to backup" R1(config-applet)# action 2.0 cli command "enable" R1(config-applet)# action 3.0 cli command "configure terminal" R1(config-applet)# action 4.0 cli command "ip route 0.0.0.0 0.0.0.0 198.51.100.1 5" R1(config-applet)# action 5.0 mail server "smtp.corp.local" to "[email protected]" from "[email protected]" subject "WAN failover"
Lab 6 — NETCONF/RESTCONF and Model-Driven Telemetry
Enable NETCONF and RESTCONF on IOS-XE.
R1(config)# netconf-yang R1(config)# restconf R1(config)# ip http secure-server R1(config)# aaa new-model R1(config)# aaa authentication login default local R1(config)# aaa authorization exec default local R1(config)# username netadmin privilege 15 secret NetOps123! R1# show netconf-yang status R1# show platform software yang-management process
Query interface state via RESTCONF from a Linux client.
# GET all interfaces (JSON) $ curl -sk -u netadmin:NetOps123! \ -H "Accept: application/yang-data+json" \ https://10.1.1.1/restconf/data/ietf-interfaces:interfaces | jq . # PATCH interface description $ curl -sk -u netadmin:NetOps123! -X PATCH \ -H "Content-Type: application/yang-data+json" \ -d '{"ietf-interfaces:interface":{"description":"Updated-via-RESTCONF"}}' \ https://10.1.1.1/restconf/data/ietf-interfaces:interfaces/interface=GigabitEthernet0%2F1
Configure dial-out model-driven telemetry streaming interface stats every 10 seconds to a gRPC collector.
R1(config)# telemetry ietf subscription 101 R1(config-mdt-subs)# encoding encode-kvgpb R1(config-mdt-subs)# filter xpath /if:interfaces-state/interface/statistics R1(config-mdt-subs)# stream yang-push R1(config-mdt-subs)# update-policy periodic 1000 ! 1000 centisec = 10 sec R1(config-mdt-subs)# receiver ip address 10.1.1.100 57500 protocol grpc-tcp R1# show telemetry ietf subscription 101 detail R1# show telemetry ietf subscription 101 receiver
Check Your Understanding
Twenty questions on this section. Each answer is explained as you go.
