6-DC/Basic Layer 2 Operation and Features on Nexus switches Part 1.

In this post i will talk about some of basic features and technologies supported by Nexus switches that depend on layer 2 operation, so i will talk about their operation, configuration and verification commands from CLI point of view, while i will talk about the other features in the next post.

Layer 2 operation on Nexus switches:

All of us know that the basic purpose of the switch (basic Layer 2 switch) is to forward the frame it receive on its ports out the correct port so that the frame can reach the required destination, but how the switch operate ? or how the switch make its forwarding decision ? simply, the switch make its forwarding decision based on the Layer 2 information carried inside the Layer 2 header of the received Ethernet frame (as long as we talk about an Ethernet environment), this means that once the switch receive an Ethernet frame on one of its ports, it look at the destination field in its Layer 2 header, where the destination MAC address exist, then make lookup in its Layer 2 forwarding table, which is the MAC address table, and as you know the MAC address table is used to store the MAC addresses entries, at which each entry has multiple information, MAC address, associated switchport number, VLAN number to which this MAC address belongs and if this entry is dynamically or statically learned, and as we know that this MAC address table is stored at the Content Addressable Memory (CAM), then forward this Ethernet frame out the correct port. What happened when the Nexus switch receive an Ethernet frame ?  let’s see the following figure that shows the simple Layer 2 switched network, to explain what happened when the Nexus switch receive an Ethernet frame:

layer 2 switched network

The following represents the steps about when PC1 need to ping PC2, which means that it need to send an ICMP echo request packet(s) to PC2 to verify Layer 3 reachability:

layer 2 switched network -2

1-PC1  need to verify the Layer 3 reachability to PC2, and the simple method for this purpose is to use the “ping” tool, simply the ping work by sending an ICMP echo request packet from the source node to the destination node and wait for an ICMP echo reply packet to be received from this destination node, hence the Layer 3 reachability is verified, but before PC1 send an ICMP echo request packet, it needs to put or encapsulate this ICMP echo request payload inside both Layer3 header and Layer 2 header, it already know the IP address of the destination node (192.168.123.2), so it can encapsulate the ICMP echo request payload inside this Layer 3 header, but what about the Layer 2 header ? it needs the MAC address of the destination node (MAC address of PC2 at this case), so for this reason it needs to send an ARP request packet to know the MAC address mapped to the IP address 192.168.123.2 so that it can complete the Layer 2 header and send the entire Ethernet frame carrying the ICMP echo request payload.

2-Once the Nexus switch received this ARP request packet, it need to decide what it will do with this packet ? the answer is based on which switching mode the Nexus switch use, so we have to explain first the different switching modes supported by the Nexus switch. There are two switching modes provided by the Nexus switches:

  1. Store and Forward switching mode:
    The store and forward switching mode is designed to receive all the bits constructing the received Ethernet frame, this means that once it starts to receive an Ethernet frame, it waits for the entire Ethernet frame to be received, then it stores it at the ingress switchport buffer for further processing, hence the name “store” comes, after this it needs to check if there are any errors on this Ethernet frame or not, for this reason it needs to compute the Frame Check Sequence and compare this value with the value seen in the FCS field to check if any errors happened during the Ethernet frame transmission, if the switch detect any errors on the frame, it will drop the frame, if no errors found, hence it will consult the Layer 2 forwarding table (MAC address table) for forwarding purpose, hence the name “forward” comes, finally the switch has to wait for the entire frame to be received and stored at the ingress switchport buffer, then forward it out the correct egress port, hence the name “Store and Forward” comes. Store and Forward is used as well when the ingress switchport and the egress switchport have different physical characteristics, i mean that the ingress switchport is running at 10Gbps speed, while the egress switchport is running at 1Gbps, so at this case it needs to slowdown the rate of traffic entering the switch at high speed switchport, while is needed to be forwarded out lower speed switchport. Nexus 7K is one of the Nexus series use Store and Forward as its switching mode.
  2. Cut Through switching mode:
    The cut through switching mode is designed to enhance the Layer2 switching performance, at which it will forward the Ethernet frame once it see the Destination MAC address in the received Ethernet frame, this means that once it receive the first 48 bits of the actual Ethernet frame (as the destination MAC address is located at the first 48 bits within the Ethernet header of the received Ethernet frame) it will consult/lookup the Layer 2 forwarding table (MAC address table) to check on which port this destination MAC address is learned, once it found an entry in the MAC address table, it will forward the entire Ethernet frame out this switchport, or if this destination MAC address is unknown, the switch will perform unknown unicast flooding and flood it out all the ports (except the port received this Ethernet frame), this means that all the remaining bits (starting from bit number 49 till the last bit) of this frame will be forward out the same switchport, so by this action it didn’t store the entire Ethernet frame at the ingress switchport buffer, instead it stored only the first 48 bits just to know the destination MAC address, so by this action it enhance the Layer 2 switching performance, as once it receive any bit (starting from bit number 49) it will switch the bit immediately out the correct egress switchport and no latency exist here because of further processing.

Once the switch received the ARP request packet, it learn the MAC address seen on the source MAC address field at the Ethernet header, this means that it learn the MAC address of PC1 on switchport Eth2/1, hence it populate its MAC address table with this MAC address and the switchport Eth2/1.

3-Once the switch determine what it will do with the received Ethernet frame (based on the used switching mode), the switch need to make Layer 2 lookup inside the Layer 2 forwarding table (MAC address table) to know on which port this destination MAC address is learned, then will forward this Ethernet frame out this port. The received packet is an ARP request packet, and as you know the ARP request packet is used to know the MAC address mapped to certain IP address, this means that the destination MAC address for sure is not an unicast MAC address (as it is unknown at the moment), as well the ARP request packet should be sent to all the nodes connected to this subnet/network, so that the node configured with the requested IP address receive the ARP request packet and respond by sending an ARP reply packet, this means that the ARP request packet should be encapsulated inside a Broadcast Ethernet frame with destination MAC address of “ff:ff:ff:ff:ff:ff”. If the switch use store and forward switching mode, it will store the entire  broadcast Ethernet frame, then flood it out all the switchports within the same subnet/network (i.e switchports member on the same VLAN), while if it use cut through switching mode, it will store only the first 48 ones (as ff:ff:ff:ff:ff:ff = 11111111 11111111 11111111 11111111 11111111 11111111 = 48 ones), then it will flood those 48 bits and the next bits out all the switchports within the same subnet/network.

4-The Nexus switch forward the ARP request packet out the two switchports Eth2/2 and Eth2/3.

5-Both PC2 and PC3 received this ARP request packet, PC3 will ignore it as the target IP address mentioned in the ARP request is for another node rather than it, while PC2 process it and respond with an ARP reply packet to tell PC1 its MAC address, and encapsulate the ARP reply packet inside an Ethernet frame with destination MAC address of PC1

6-Once the switch received the ARP reply packet from PC2, it do the same steps mentioned at (2) and (3), at which it populate its MAC address table with the MAC address of PC2 and switchport Eth2/2, then forward it out Eth2/1 as the destination MAC address is the MAC address of PC1.

7-PC1 at this case know the MAC address of PC2, so it will encapsulate the ICMP echo request packet inside Layer 3 header of source IP address of PC1, destination IP address of PC2 and Layer 2 header of source MAC address of PC1 and destination MAC address of PC2.

8-Once the switch received the ICMP echo request packet from PC1, it do the same steps mentioned at (2) and (3), it already have an entry for PC2 MAC address at its MAC address table, so it will forward it out Eth2/2.

9-PC2 received the ICMP echo request packet, then it respond by sending an ICMP echo reply packet and encapsulate it inside Layer 3 header of source IP address of PC2, destination IP address of PC1 and Layer 2 header of source MAC address of PC2 and destination MAC address of PC1.

10-Once the switch received the ICMP echo reply packet from PC2, it do the same steps mentioned at (2) and (3), it already have an entry for PC1 MAC address at its MAC address table, so it will forward it out Eth2/1.

When the switch learn MAC address on one of its switchports, it will populate the MAC address table with that MAC address and this switchport, as you know the switchports are part of switching modules/I/O modules/line cards, this means that once the switch learn MAC address on one of its switchport, it will populate the MAC address table of the switching module/I/O modules/line cards of that switchport, as well it will synchronize the MAC address table of the other switching modules/I/O modules/line cards, so that the Nexus switch can provide consistent Layer 2 forwarding decisions among all the switching modules/I/O modules/line cards, as well once the entry of that MAC address expired, then the switch will remove this MAC address entry from the MAC address table of all the switching modules/I/O modules/line cards. The MAC address entry is expired based on the aging time, that is represented in seconds and is used to determine after how many seconds this idle entry will be removed from the MAC address table, the default value for the aging time is 1800 seconds.

Basic MAC address table configuration:

We can configure static entry for MAC address known via certain VLAN via certain switchport on NX-OS as the following:

static mac

In the previous figure, we configured static entry for the MAC address (aabb.cc00.0210) that is known via VLAN 1 out switchport (Eth2/1)

We can configure the aging time for the dynamically learned MAC addresses either for all VLANs or for specific VLAN on NX-OS as the following:

aging time

aging time vlan 1

In the previous figure, we configured the aging time for all VLANs to be 2000 seconds, while configure the aging time for only VLAN 1 to be 1900 seconds.

We can configure the MAC addresses learning mode on NX-OS, at which there are two main learning modes for the switch to learn the MAC addresses and hence populate its MAC address table with those addresses, “traditional learning” mode, which is the normal learning mode used with the Classic Ethernet deployment, at which the Nexus switch learn the MAC address and populate its MAC address table with that address once it receive an Ethernet frame, hence it will populate its MAC address table with the source MAC address seen on this Ethernet frame. The other mode is “conversational learning” mode, and it is basically used when Fabric Path feature is implemented on the Nexus switch, as well it can be used with the Classic Ethernet deployment , so i will mention it later in the Fabric Path post.

learning mode

In the previous figure, we configured the learning mode for VLAN 1 to be conversational.

To revert back and use the traditional learning mode, just use “no” keyword as the following:

learning mode1

 

Basic MAC address table verification:

We can verify that if certain MAC address exist in the MAC address table or not using the following command:

show mac address table

In the previous output, we can deduce that the MAC address (aabb.cc00.0210) exist in the MAC address table and it has the following:
1-This entry is considered as the primary entry, because of “*”.
2-This MAC address known via VLAN 1.
3-This MAC address is statically configured on the MAC address table, because of “static”.
4-This MAC address is learned on the switcport “Eth2/1”, hence it can be reachable via switchport “Eth2/1”.

We can check the aging time for certain VLAN using the following command:

show mac address table1

In the previous output, we can deduce that the aging time for MAC addresses known via VLAN 1 is 1800 seconds (default value).

We can check which learning mode is being used for certain VLAN using the following command:

show mac address table2

In the previous output, we can deduce that the learning mode for MAC addresses known via VLAN 1 is “Non-conversational-learning”, which is exactly “traditional learning”.

We can check how many MAC addresses are learned on the MAC address table for certain VLAN using the following command:

show mac address table3

In the previous output, we can deduce that the following:
1-Dynamic Address Count: 0, which means that no MAC addresses are dynamically learned.
2-Static Address (User-defined) Count: 1, which means that only 1 MAC address is statically configured by the user.
3-Secure Address Count: 0, which means that no MAC addresses are securely known.

We can check the MAC addresses learned on certain switchport using the following command:

show mac address table4

 

Virtual LAN (VLAN):

As we already know that VLAN is used to divide the broadcast domain of the switchports to multiple sub-broadcast domains, as by default the Layer 2 switched ports are member in VLAN1, this means that if any two end-hosts connected to those Layer 2 switched ports can communicate with each other as long as their IP addresses are within the same subnet/network, for this reason if we need to prevent those end-hosts from communicating with each other, we can configure each switched port to be member on different VLANs, this means that once an Ethernet frame is received on one switched port, the switch will not forward it out the other switched port (as they belong to different VLANs). For more information about VLAN operation, thanks to check this link 1-R&S/Virtual LAN (VLAN) and Spanning Tree Protocol (STP).

Basic VLAN configuration:

We can configure single VLAN on NX-OS as the following:

vlan

We can configure multiple VLANs on NX-OS at the same time as the following:

vlan1

We can define the VLAN state on NX-OS as the following:

vlan2

vlan3

The VLAN state maybe either “Active” or “Suspended”, Active means that the switch will pass the traffic for that VLAN, while Suspended means that the switch will not pass the traffic for that VLAN.

We can define the VLAN mode on NX-OS, at which the VLAN mode represents how this VLAN is used, there are two VLAN modes, “Classical Ethernet (CE)”, which means that the VLAN is used as normal Ethernet VLAN for the normal Layer2 operation. The other mode is “Fabric Path”, which means that this VLAN is used as Fabric Path VLAN, hence it is used with the Fabric Path deployment and i will explain it later in the Fabric Path post. So we can configure the VLAN mode on the NX-OS as the following:

vlan4

vlan5

We can configure the port to be switched (Layer 2) port, as well configure it to be member in certain VLAN as the following:

vlan6

In the previous figure, we configured the port Eth2/1 to be switched (Layer 2) port using the command “switchport”, as well we configured this switchport to be access (i.e it is member in only one VLAN) using the command “switchport mode access”, finally we configured this switchport to be member in VLAN 2 using the command “switchport access vlan 2”.

We can configure the port to be switched (Layer 2) port, as well configure it to be trunk port, hence it is member in multiple VLANs, or it is configured to carry traffic belong to multiple VLANs as the following:

vlan7

In the previous figure, we configured the port Eth2/2 to be switched (Layer 2) port using the command “switchport”, as well we configured this switchport to be trunk (i.e it is member in multiple VLANs) using the command “switchport mode trunk”, finally we configured this switchport to be member in VLANs from 1 to 5  using the command “switchport trunk allowed vlan 1-5”.

Basic VLAN verification:

We can check the VLANs configured on the Nexus switch using the following command:

vlan8

In the previous output, we can deduce that there are VLANs (1, 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 and 10), as well two switchports (Eth2/1, Eth2/2 and Eth2/3) are member in the VLAN 2.

We can check the number of configured VLANs on the Nexus switch using the following command:

vlan9

In the previous output, we can deduce the following:
1–Number of existing VLANs: 10, which means that there are 10 VLANs
2-Number of existing user VLANs: 10, which means that there are 10 user VLANs.
3-Number of existing extended VLANs: 0, which means that no extended VLANs are configured on the switch.

We can check the configuration about certain VLAN using the following command:

vlan10

In the previous output, we checked VLAN 2 configuration as the following:
1-VLAN ID: 2, which means that the VLAN ID of the VLAN we are checking is 2.
2-VLAN Name: VLAN0002, which means that the VLAN name is VLAN0002.
3-Status: active, which means that the status of VLAN ID 2 is “Active”.
4-Ports: Eth2/1, Eth2/2, Eth2/3, which means that the switchports member in this VLAN are Eth2/1, Eth2/2 and Eth2/3.
5-VLAN type: enet, which means that the type of the VLAN is Ethernet.
6-vlan-mode: CE, which means that this VLAN is working in Classical Ethernet deployment.

 

Private Virtual LAN (Private VLAN):

Private VLAN is a feature used by NX-OS so that it can divide the single VLAN into multiple sub-VLANS, at which we use the private VLAN feature so that we prevent most of the end-hosts connected to switchports member in such VLAN from communicating with each other for certain purpose. Assume that we have VLAN 10, and we have servers member in this VLAN, and assume that only two of those servers should communicate with each others via broadcast or multicast and they exchange many broadcast or multicast packets, while they are not allowed to communicate with other servers member in this VLAN from design perspective, for this reason we need these broadcast or multicast packets to not reach the other servers as no need for processing such packets, hence we need to use certain method to make such behavior, i mean we need certain method to allow the two servers to communicate normally with each others while not to communicate with the other servers and as well not to send the broadcast or multicast packets to the other servers so that not to utilize the links of other servers with such unwanted packets, hence the private VLAN is the perfect method to be used at such situation. Private VLAN feature is used to divide the VLAN into multiple VLANs to support its purpose, in the context of private VLAN there are different VLAN types as the following:

  1. Primary VLAN: This VLAN represents the actual VLAN needed to be divided into multiple sub-VLANs to provide the separation functionality, and it is allowed to communicate with all these sub-VLANs, this means that all the sub-VLANs can communicate only with the primary VLAN. Switchports member in the primary VLAN are called “Promiscuous ports” and are allowed to communicate with ports member in other sub-VLANs.
  2. Secondary VLAN: This VLAN represents the sub-VLAN that belongs to the primary VLAN. The secondary VLAN has two types:
    a-Community VLAN: Switchports member in this community VLAN are allowed to communicate with each others and promiscuous ports (switchports member in the primary VLAN) as well, but they are not allowed to communicate with ports member in another community VLAN.
    b-Isolated VLAN: Switchports member in this isolated VLAN are not allowed to communicate with each others, but are allowed only to communicate with promiscuous ports (switchports member in the primary VLAN), as well they are not allowed to communicate with any other ports either member in community or isolated VLANs.

This means that multiple sub-VLANs can be associated to the same primary VLAN, this means that multiple community and isolated VLANs can be associated to the same primary VLAN. The following figure represents the logical representation of the Private VLAN meaning:

private VLAN

From the previous figure, we can see that there are three VLANs
1-VLAN 11 is the primary VLAN and it has only one port member in this VLAN.
2-VLAN 12 is community VLAN and it has two ports member in this VLAN
3-VLAN 13 is isolated VLAN and it has two ports member in this VLAN
We can deduce the following from the figure:
1-The two ports member in VLAN 12 can communicate with each other and with the promiscuous port because of the logical connection via the switch fabric or switch backplane that connects the three ports so that the traffic can be exchanged between them.
2-The two ports member in VLAN 13 can’t communicate with each other, while they can communicate with the promiscuous port because of single logical connection via the switch fabric or switch backplane that connects each port to the promiscuous port so that the traffic can be exchanged only between each port and the promiscuous port and can’t communicate with each other.

All the end-hosts connected to switchports member in either primary, community or isolated VLANs have IP address within the same subnet, but not all of them can communicate with each other as it is determined based on which end-host belong to which VLAN. From the previous we can conclude the the private VLAN benefits as the following:

  1. Private VLAN provide Layer 2 security for the end-hosts, at which not all the end-hosts can communicate with each other as it is based on the which end-host belong to which VLAN, at which if we need certain end-host to not communicate with other end-hosts belong to the same subnet, so we assign its switchport to an isolated VLAN, hence it can communicate only with end-hosts connected to promiscuous ports, while if we need certain end-hosts to communicate with each other while not communicate with others, so we can assign their switchports to community VLAN, hence they can communicate with each other and with end-host member connected to promiscuous ports.
  2. Private VLAN provide IP addresses preservation, this means that the different end-hosts belong to different VLANs (either primary, community or isolated) have IP address within the same IP subnet, this means that if we move server from normal VLAN to another community or isolated VLAN, it still preserve its IP address without any changes.
  3. Private VLAN provide broadcast/multicast traffic suppression, this means that the end-hosts belong to different community VLANs or belong to isolated VLANs don’t bother other end-hosts with their broadcast/multicast traffic, hence prevent the unwanted broadcast/multicast from being flooded to all end-hosts.
  4. Private VLAN can be spanned across the Layer 2 switched network, this means that we can carry the primary, community and isolated VLAN over the trunk links connecting the switches, this means that the private VLAN concept can be used among the switches within the Layer 2 switched network.

Private VLAN Configuration:

Before we can configure private VLAN on NX-OS, we need to enable its feature first, so that we can configure it and all its parameters:

private VLAN1

Then we can configure private VLAN on NX-OS as the following:

private VLAN2

In the previous figure, we configured the following:
1-Community VLAN 12 with name “COMMUNITY_VLAN_12” using the commands:
“vlan 12”
“name COMMUNITY_VLAN_12”
“private-vlan community”
2-Isolated VLAN 13 with name “ISOLATED_VLAN_13” using the commands:
“vlan 13”
“name ISOLATED_VLAN_13”
“private-vlan isolated”
3-Primary VLAN 11 with name “PRIMARY_VLAN_11” and associate both the community and isolated VLANs with this primary VLAN using the commands:
“vlan 11”
“name PRIMARY_VLAN_11”
“private-vlan primary”
“private-vlan association add 12-13”

We can configure the switchport to be member in certain VLAN (either community or isolated) as the following:

private VLAN3

In the previous figure, we configured the following:
1-Configure the port Eth2/1 as switched port using the command:
“switchport”
2-Configure the switchport Eth2/1 as community port/isolated, which means that the port will be member in either community or isolated VLAN using the command:
“switchport mode private-vlan host”
3-Configure the community/isolated VLAN and the primary VLAN associated to it using the command:
“switchport private-vlan host-association 11 12”, which means that the primary VLAN is 11, while the community/isolated is 12.

We can configure the switchport to be member the primary VLAN as the following:

private VLAN4

In the previous figure, we configured the following:
1-Configure the port Eth2/2 as switched port using the command:
“switchport”
2-Configure the switchport Eth2/2 as promiscuous port, which means that the port will be member in the primary VLAN using the command:
“switchport mode private-vlan promiscuous”
3-Configure the primary VLAN and community/isolated VLANs associated to it using the command:
“switchport private-vlan mapping 11 12-13”, which means that the primary VLAN is 11, while the community or isolated VLANs are 12 and 13.

We can configure the trunk port to carry the traffic for primary and secondary VLANs as the following:

private VLAN5

In the previous figure, we configured the following:
1-Configure the port Eth2/3 as switched port using the command:
“switchport”
2-Configure the switchport Eth2/3 as promiscuous trunk port, which means that the port will carry traffic for primary and secondary VLANs using the command:
“switchport mode private-vlan trunk promiscuous”
3-Associate the primary VLAN with the secondary VLANs associated using the command:
“switchport private-vlan association trunk 11 12”
“switchport private-vlan association trunk 11 13”

We can configure SVI interface that is shared among all the VLANs (Primary and Secondary), at which the primary VLAN and its associated secondary VLANs can communicate it , but before we configure the SVI, we need first to enabled the “interface-vlan” feature, so that we can configure and all its parameters as the following:

private VLAN7

private VLAN8

In the previous figure, we enabled the “interface-vlan” feature, so that we can configure SVI interface for primary VLAN and its associated secondary VLANs, so we configured the following:
1-Enable the “interface-vlan” feature using the command:
“feature interface-vlan”
2-Create SVI interface for VLAN 11 using the command:
“interface Vlan11”
3-Enable the SVI interface as by default the SVI interface is “Admin down” using the command:
“no shutdown”
4-Configure an IP address “192.168.123.123/24” to the SVI for VLAN 11 using the command:
“ip address 192.168.123.123/24”
5-Map the secondary VLANs to that SVI interface, so that end-hosts member in secondary VLANs can communicate with this SVI interface using the command:
“private-vlan mapping 12-13”

Private VLAN verification:

We can check the private VLAN configuration using the following command:

private VLAN6

In the previous output, we can deduce that the primary VLAN ID is 11, while there are two secondary VLANs, 12 which is community VLAN and 13 which is isolated VLAN.

We can verify the private VLAN mapping for the SVI interface and the primary VLAN and its associated secondary VLANs using the following command:

private VLAN9

In the previous output, we can deduce that there is an interface VLAN 11, at which the primary VLAN is 11, and it is mapped to two secondary VLANs, VLAN 12 which is community VLAN and VLAN 13 is isolated VLAN.

 

I will talk about the other basic Layer 2 features and technologies in the next post.

Hope that the post is helpful.

Regards

Mostafa Hamza

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s