Performance Routing Version 3 - PfRv3

Monday. July 06, 2020

Legacy

In the last few months, almost a year. I’ve been working deeply with a legacy technology that is one of the major pieces of Cisco’s iWAN and could be considered as the predecessor of SD-WAN. For that reason, the three previous posts were used to describe a transport independent WAN through DMVPN.

So, using a DMVPN together with the Intelligent Path Control configured through Performance Routing Version 3 (PfRv3), we can take forwarding decisions on the application requirements (i.e. bandwidth, delay, jitter, and packet loss) instead of only relying on information received from routing protocols. I will use this post to design a basic PfRv3 solution based on the following topology:

Topology

As PfRv3 and DMPVN are practically related to iWAN we have to remember the following characteristics and definitions:

iWAN Layers

Transport Independent Design based on DMVPN
- Branch spoke sites establish an IPsec tunnel to and register with the hub site.
- Data traffic flows over the DMVPN tunnels.
- WAN interface IP address used for the tunnel source address (Front VRF).
- One tunnel per VRF.
Overlay Routing
- BGP or EIGRP are typically used.
- IP routing exchanges prefix information for each site.
QoS
- Per-Tunnel QoS is used to prevent Hub site oversubscription to spokes sites.
- DSCP markings on traffic will be essential to determine policies on the Master Controller.
AVC
- Provides application awareness with deep packet inspection of traffic to identify and monitor applications performance.
- Visibility and control at the application level is provided through AVC technologies such as Network-Based Application Recognition 2 (NBAR2), NetFlow, Performance Monitoring.

Device Roles in PfRv3

Hub - Master Controller (MC): All policies are configured on the hub MC. It acts as master controller for the site and makes optimization decision.
Hub - Border Router (BR): WAN interface terminates in the Hub Border Routers. PfRv3 is enabled on the interfaces. Also, it is possible to configure more than one WAN interface on the same device.
Branch - Master Controller (Branch/MC): The Branch MC is the Master Controller at the branch site. There is no policy configuration. It receives policy from the Hub MC. Also, this device acts as Master Controller for that site for making optimization decision.
Branch - Border Router (Branch/BR): There is no configuration other than enabling of PfRv3 border MC on the device. The WAN interface that terminates on the device is detected automatically. The MC and BR is co-located on the same router and called branch MC/BR. Branch MC peer and get connected with t Hub MC to receive updates for policy and services.

Let’s assume that everything that involves routing and overlay with DMVPN already is configured.

MC1 - PfRv3 Domain

The PfR domain name is common for all of the routers participating in the PfR domain (MC/BR). All PfR related configuration will be made under the domain configuration. Since we don’t have any vrf on the infrastructure we set the vrf to default. The router is hub master and it is identified based on the Loopback0 interface, which has to be reachable for other domain elements BR/MC.

The ip prefix-list defines site local networks in this case DC-PREFIXES. It is mandatory on the HUB MC, without it, the status of MC will be down. I will use the following segments:

LAN - 10.192.168.0/24
VOICE - 172.16.192.0/24
VIDEO - 172.16.224.0/24
LOW_LATENCY - 172.16.240.0/24
BULK_DATA - 172.16.248.0/24

Also need to define enterprise networks with prefix-list in this case ENTERPRISE-PREFIXES. That will be all the networks that needs to be controlled by PfRv3 in PfR Domain.

The policy configuration is defined through traffic classes and this topology I will use four classes: VOICE, VIDEO, LOW_LATENCY and BULK_DATA.

!
interface Loopback0
 ip address 192.168.0.1 255.255.255.255
!
ip prefix-list DC-PREFIXES seq 10 permit 10.192.168.0/24
ip prefix-list DC-PREFIXES seq 20 permit 172.16.192.0/24
ip prefix-list DC-PREFIXES seq 30 permit 172.16.224.0/24
ip prefix-list DC-PREFIXES seq 40 permit 172.16.240.0/24
ip prefix-list DC-PREFIXES seq 50 permit 172.16.248.0/24
!
ip prefix-list ENTERPRISE-PREFIXES seq 10 permit 192.168.10.0/24
ip prefix-list ENTERPRISE-PREFIXES seq 20 permit 192.168.20.0/24
!
domain CERC
 vrf default
  master hub
   source-interface Loopback0
   site-prefixes prefix-list DC-PREFIXES
   load-balance advanced
    path-preference INET1 INET2 fallback routing
   enterprise-prefix  prefix-list ENTERPRISE-PREFIXES
   class VOICE sequence 10
    match dscp ef policy voice
    path-preference INET1 fallback INET2
   class VIDEO sequence 20
    match dscp af41 policy real-time-video
    path-preference INET2 fallback INET1
   class LOW_LATENCY sequence 30
    match dscp af21 policy low-latency-data
    path-preference INET1 fallback INET2
   class BULK_DATA sequence 40
    match dscp af11 policy bulk-data
    path-preference INET2 fallback INET1
!

Border Routers - PfRv3 Domain

The configuration of the PfR domain on both Border Routers at the hub site is similar to the configuration of the Master Controller. Will need to specify the domain name and vrf, the function defined as the border, the source interface will be Loopback0 and the site’s MC address (192.168.0.1). Path-name and IDs identifies the underlying transport network. These settings are configured under the tunnel interfaces.

So that the example does not become so extensive, we also have to assume that all the parts that involve QoS such as policy-maps and class-map are already defined. Where Per-Tunnel is extremely important because the branches will dynamically define a QoS profile based on the NHRP Group.

!
domain CERC
 vrf default
  border
   source-interface Loopback0
   master 192.168.0.1
!
! On BR_INET1
!
interface Tunnel100
 .
 .
 .
 nhrp map group GROUP-200MBPS service-policy output GROUP-200MBPS-POLICY
 nhrp map group GROUP-100MBPS service-policy output GROUP-100MBPS-POLICY
 nhrp map group GROUP-20MBPS service-policy output GROUP-20MBPS-POLICY
 nhrp map group GROUP-10MBPS service-policy output GROUP-10MBPS-POLICY
 domain CERC path INET1 path-id 10
!
! On BR_INET2
!
interface Tunnel200
 .
 .
 .
 nhrp map group GROUP-200MBPS service-policy output GROUP-200MBPS-POLICY
 nhrp map group GROUP-100MBPS service-policy output GROUP-100MBPS-POLICY
 nhrp map group GROUP-20MBPS service-policy output GROUP-20MBPS-POLICY
 nhrp map group GROUP-10MBPS service-policy output GROUP-10MBPS-POLICY
 domain CERC path INET2 path-id 20
!

Branch MC/BR - PfRv3 Domain

The configuration of the MC on Branch site is similar with Hub MC. The role will be set to master branch. Since Branch MC needs to communicate with the MC1, we need to define the ip address of MC1. And the Branch BR configuration will use its local MC instead of Hub Site MC, also there is no need for Path-name or IDs configuration since those are learned automatically from the Hub MC.

! On Branch_10
!
ip prefix-list SITE-PREFIX seq 10 permit 192.168.10.0/24
!
domain CERC
 master branch
 vrf default
  border
   source-interface Loopback0
   master local
  master branch
   source-interface Loopback0
   site-prefixes prefix-list SITE-PREFIX
   hub 192.168.0.1
!
!
interface Tunnel100
 nhrp group GROUP-10MBPS
!
interface Tunnel200
 nhrp group GROUP-100MBPS
!
! On Branch_20
!
ip prefix-list SITE-PREFIX seq 10 permit 192.168.20.0/24
!
domain CERC
 master branch
 vrf default
  border
   source-interface Loopback0
   master local
  master branch
   source-interface Loopback0
   site-prefixes prefix-list SITE-PREFIX
   hub 192.168.0.1
!
!
interface Tunnel100
 nhrp group GROUP-20MBPS
!
interface Tunnel200
 nhrp group GROUP-200MBPS
!

Verifying peering between the MC1 and BRs in PfRv3 Domain

From the MC1, it is possible to see that the operational status of MC1 is UP, the MC1 use Loopback 0 (192.168.0.1), connection status to both Hub site Border Routers is CONNECTED. Also there is one external interface per BR, BR_INET1 has external interface Tunnel100 (INET1) with Path-Id 10 and BR_INET2 has external interface Tunnel200 (INET2) with Path-id 20.

MC1# show domain CERC master status

  *** Domain MC Status ***

 Master VRF: Global
  Instance Type:    Hub
  Instance id:      0
  Operational status:  Up
  Configured status:  Up
  Loopback IP Address: 192.168.0.1
  Global Config Last Publish status: Peering Success
  Load Balancing:
   Admin Status: Enabled
   Operational Status: Up

  Borders:
    IP address: 192.168.0.10
    Version: 2
    Connection status: CONNECTED (Last Updated 10h ago )
    Interfaces configured:
      Name: Tunnel100 | type: external | Service Provider: INET1 path-id:10 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 8

    Tunnel if: Tunnel0

    IP address: 192.168.0.20
    Version: 2
    Connection status: CONNECTED (Last Updated 10h ago )
    Interfaces configured:
      Name: Tunnel200 | type: external | Service Provider: INET2 path-id:20 | Status: UP | Zero-SLA: NO | Path of Last Resort: Disabled
          Number of default Channels: 8

    Tunnel if: Tunnel0

From both Border Routers in the Hub site, the connection to MC1 is established and the external WAN interfaces are UP.

BR_INET1# show domain CERC border status

  **** Border Status ****

Instance Status: UP
Present status last updated: 10h ago
Loopback: Configured Loopback0 UP (192.168.0.10)
Master: 192.168.0.1
Master version: 2
Connection Status with Master: UP
MC connection info: CONNECTION SUCCESSFUL
Connected for: 10h
Minimum Requirement: Met
External Wan interfaces:
     Name: Tunnel100 Interface Index: 25 SNMP Index: 20 SP: INET path-id: 10 Status: UP Zero-SLA: NO Path of Last Resort: Disabled

Auto Tunnel information:

   Name:Tunnel0 if_index: 28
   Virtual Template: Not Configured
   Borders reachable via this tunnel:  192.168.0.20

BR_INET2# show domain CERC border status

  **** Border Status ****

Instance Status: UP
Present status last updated: 10h ago
Loopback: Configured Loopback0 UP (192.168.0.20)
Master: 192.168.0.1
Master version: 2
Connection Status with Master: UP
MC connection info: CONNECTION SUCCESSFUL
Connected for: 10h
Minimum Requirement: Met
External Wan interfaces:
     Name: Tunnel200 Interface Index: 25 SNMP Index: 20 SP: INET path-id: 20 Status: UP Zero-SLA: NO Path of Last Resort: Disabled

Auto Tunnel information:

   Name:Tunnel0 if_index: 28
   Virtual Template: Not Configured
   Borders reachable via this tunnel:  192.168.0.10

Verifying - Site Prefix database

In the background of the PfRv3 there is an EIGRP service address-family a.k.a. EIGRP SAF (But I don’t want to go too deep into this topic) used for the communication of the elements, and one of its functions is related to learning the prefixes.

It is possible to verify how the MC1 publish site-local prefixes. On the Branch_10 are prefix-lists related with local prefixes. If we take a look at the first site-prefix 192.168.10.0/24 which is the GigabitEthernet0/2 interface, we can see that it has C-flag attach to it. This means that the prefix is configured on the branch for the egress traffic.

Branch_10# show domain CERC master site-prefix 192.168.0.100
Site-id              Site-prefix          Last Updated         DC Bitmap  Flag
--------------------------------------------------------------------------------
192.168.0.100         192.168.0.100/32     01:12:39 ago         0x0         C
192.168.0.100         192.168.10.0/24      01:12:39 ago         0x0         C
--------------------------------------------------------------------------------

MC1# show domain CERC master site-prefix 192.168.0.100
Site-id              Site-prefix          Last Updated         DC Bitmap  Flag
--------------------------------------------------------------------------------
192.168.0.100         192.168.0.100/32     01:12:39 ago         0x0         S,C
192.168.0.100         192.168.10.0/24      01:12:39 ago         0x0         S,C
--------------------------------------------------------------------------------

Verifying - PfR Policy

Another function of the EIGRP SAF is to publish the PfR policies defined in the Master Controller and replicated for the branches.

MC1# show domain CERC master policy
 No Policy publish pending
 Last publish Status : Peering Success
 Total publish errors : 0

  class VOICE sequence 10
    path-preference INET1 fallback INET2
    class type: Dscp Based
      match dscp ef policy voice
        Number of Traffic classes using this policy: 8

  class VIDEO sequence 20
    path-preference INET2 fallback INET1
    class type: Dscp Based
      match dscp af41 policy real-time-video
        Number of Traffic classes using this policy: 8

  class LOW_LATENCY sequence 30
    path-preference INET1 fallback INET2
    class type: Dscp Based
      match dscp af21 policy low-latency-data
        Number of Traffic classes using this policy: 8

  class BULK_DATA sequence 40
    path-preference INET2 fallback INET1
    class type: Dscp Based
      match dscp af11 policy bulk-data
        Number of Traffic classes using this policy: 8

  class default
    path-preference INET1 INET2 fallback routing
      match dscp all
        Number of Traffic classes using this policy: 8

Branch_10# show domain CERC master policy

  class VOICE sequence 10
    path-preference INET1 fallback INET2
    class type: Dscp Based
      match dscp ef policy voice
        Number of Traffic classes using this policy: 2

  class VIDEO sequence 20
    path-preference INET2 fallback INET1
    class type: Dscp Based
      match dscp af41 policy real-time-video
        Number of Traffic classes using this policy: 2

  class LOW_LATENCY sequence 30
    path-preference INET1 fallback INET2
    class type: Dscp Based
      match dscp af21 policy low-latency-data
        Number of Traffic classes using this policy: 2

  class BULK_DATA sequence 40
    path-preference INET2 fallback INET1
    class type: Dscp Based
      match dscp af11 policy bulk-data
        Number of Traffic classes using this policy: 2

  class default
    path-preference INET1 INET2 fallback routing
      match dscp all
        Number of Traffic classes using this policy: 2

Verifying - Channels

Working with PfRv3, there is a topic called “Channels”, which are logical entities used to measure path performance for specific application or DSCP flows between sites.

Let’s assume that we want to check the low latency traffic marked with DSCP AF21 and that it is being sent between the branches and the 172.16.240.0/24 segment on the DC.

There are two channels from Branch site to Hub site for traffic marked with DSCP AF21, channel 169 over the link INET1 and Channel 168 over the link INET2. Based on destination site-id (IP address of the MC1), we know that both channels go to Hub site. Part of the pfr-Label field 0:10 (ch169 INET1) and 0:20 (ch168 INET2) shows both source site POP-ID (0) and PATH-IDs 10 and 20. Also, the pfr-Label field shows the same information about the destination site.

Branch_10# show domain CERC master channels summary | i Ch-ID|af21
Ch-ID - Channel ID, SP - Service Provider
Ch-ID      Dst-Site-ID     DSCP            SP    pfr-Label             Status TCA
168        192.168.0.1     af21[18]        INET2 0:20 | 0:0 [0x160000]       A      0/0/0
169        192.168.0.1     af21[18]        INET1 0:10 | 0:0 [0x150000]       A      0/0/0

Also, it is possible to see that INET1 link is currently the primary link for traffic-class DSCP AF21.

Branch_10#show domain CERC master traffic-classes dscp af21

 Dst-Site-Prefix: 172.16.240.0/24      DSCP: af21 [18] Traffic class id:1303430
  Clock Time:                 01:58:40 (VEN) 06/01/2020
  TC Learned:                 17:48:55 ago
  Present State:              CONTROLLED
  Current Performance Status: in-policy
  Current Service Provider:   INET1 since 01:58:03
  Previous Service Provider:  INET2 pfr-label: 0:20 | 0:0 [0xB0000] for 4968 sec
  BW Used:                    155 Kbps
  Present WAN interface:      Tunnel100 in Border 192.168.0.100
  Present Channel (primary):  168    INET1 pfr-label:0:10 | 0:0 [0x160000]
  Backup Channel:             169    INET2 pfr-label:0:20 | 0:0 [0x150000]
  Destination Site ID bitmap: 1
  Destination Site ID:        192.168.0.1 (Active)
  Class-Sequence in use:      30
  Class Name:                 LOW_LATENCY using policy low-latency-data
  BW Updated:                 00:00:01 ago
  Reason for Latest Route Change:    Next hop down
  Route Change History:
             Date and Time                   Previous Exit                                     Current Exit                                Reason

    1:  01:23:25 (VEN) 06/01/20   INET2(0:10|0:0)/192.168.0.100/Tu200 (Ch:167)         INET1(0:10|0:0)/192.168.0.100/Tu100 (Ch:168)        Next hop down
    2:  00:05:11 (VEN) 06/01/20   INET1(0:20|0:0)/192.168.0.100/Tu100 (Ch:166)         INET2(0:20|0:0)/192.168.0.100/Tu200 (Ch:169)        Next hop down

From MC1, there are two channels which terminate to Branch site.

MC1# show domain CERC master channels summary | i Ch-ID|af21
Ch-ID - Channel ID, SP - Service Provider
Ch-ID      Dst-Site-ID     DSCP            SP    pfr-Label             Status TCA
168        192.168.0.100   af21[18]        INET1 0:0 | 0:10 [0x15]           A      1/0/1
169        192.168.0.100   af21[18]        INET2 0:0 | 0:20 [0x15]           A      1/0/1

Note: It is extremely important to remember that all devices in an PfRv3/iWAN environment: Hub MC, Hub BRs and Branch Routers - must have the same version of IOS.

References