How to set up your device when connecting to AMS-IX? Here are some pointers to start with.
AMS-IX rules restrict the type of traffic and number of source MAC addresses that any member is allowed to send to the exchange. The AMS-IX platform is built around photonic cross connects, Layer 1 switches, which introduce short link flaps for the customers with 10GE connections when moving customer connections between Ethernet switches.
This article will tell you how to prevent those flaps from influencing your session and how to configure your interface towards AMS-IX to only send allowed traffic towards the exchange.
How to set up your device when connecting to AMS-IX? Here are some pointers to start with.
AMS-IX rules restrict the type of traffic and number of source MAC addresses that any member is allowed to send to the exchange. The AMS-IX platform is built around photonic cross connects, Layer 1 switches, which introduce short link flaps for the customers with 10GE connections when moving customer connections between Ethernet switches.
This article will tell you how to prevent those flaps from influencing your session and how to configure your interface towards AMS-IX to only send allowed traffic towards the exchange.
AMS-IX operates as a shared Layer 2 (L2) Ethernet infrastructure. Large Ethernet LANs require that more or less everyone plays by the same set of rules. In other words, they can be quite sensitive to misbehaviour.
In order to improve the stability of the Exchange, AMS-IX has defined a set of rules to which every member's connection must adhere.
It is not always easy to immediately grasp the subtleties of configuring equipment to adhere to the rules. Let us help you fill in some blanks and provide examples and hints for the most common equipment.
AMS-IX operates as a shared Layer 2 (L2) Ethernet infrastructure. Large Ethernet LANs require that more or less everyone plays by the same set of rules. In other words, they can be quite sensitive to misbehaviour.
In order to improve the stability of the Exchange, AMS-IX has defined a set of rules to which every member's connection must adhere.
It is not always easy to immediately grasp the subtleties of configuring equipment to adhere to the rules. Let us help you fill in some blanks and provide examples and hints for the most common equipment.
In this document we refer to terms like 'L2 device', 'L2/L3 hybrid', etc. Here are our definitions:
L2 Device
A device that functions as a Layer 2 (Ethernet) Bridge (a.k.a. 'switch', 'bridge', 'hub', etc).
L3 Device
A device that functions as a L3 (IP) router only. This means it does not bridge any Ethernet frames between its interfaces. Such a device is typically called a 'router'.
L2/L3 Hybrid
A device that functions both as a L2 bridge and a L3 router. This means it can both bridge Ethernet frames between its interfaces as well as route IP traffic and participate in IP routing protocols. Foundry/Brocade, Force10 and Extreme are common examples of this type of device.
In this document we refer to terms like 'L2 device', 'L2/L3 hybrid', etc. Here are our definitions:
L2 Device
A device that functions as a Layer 2 (Ethernet) Bridge (a.k.a. 'switch', 'bridge', 'hub', etc).
L3 Device
A device that functions as a L3 (IP) router only. This means it does not bridge any Ethernet frames between its interfaces. Such a device is typically called a 'router'.
L2/L3 Hybrid
A device that functions both as a L2 bridge and a L3 router. This means it can both bridge Ethernet frames between its interfaces as well as route IP traffic and participate in IP routing protocols. Foundry/Brocade, Force10 and Extreme are common examples of this type of device.
The AMS-IX network is built as a redundant (virtual) hub & spoke topology using Glimmerglass photonic cross-connects and Extreme Networks (formerly Brocade) switches.
Customers up to 1GE are directly connected to Brocade edge switches, available at each location. One can connect with 1 (or mutliple) GE via multimode or singlemode fiber. Fiber connections are supported using SX or LX optics, and in some cases also LH-A or LH-B.
10GE customers connect to the AMS-IX platform via Glimmerglass photonic cross-connects, Layer 1 switches. Those Layer 1 switches connect the customer to one of a pair of locally (in the datacenter) available Ethernet switches. The 10G Ethernet access switches are locally available at each location and one can connect with either ER or LR optics.
The AMS-IX network is built as a redundant (virtual) hub & spoke topology using Glimmerglass photonic cross-connects and Extreme Networks (formerly Brocade) switches.
Customers up to 1GE are directly connected to Brocade edge switches, available at each location. One can connect with 1 (or mutliple) GE via multimode or singlemode fiber. Fiber connections are supported using SX or LX optics, and in some cases also LH-A or LH-B.
10GE customers connect to the AMS-IX platform via Glimmerglass photonic cross-connects, Layer 1 switches. Those Layer 1 switches connect the customer to one of a pair of locally (in the datacenter) available Ethernet switches. The 10G Ethernet access switches are locally available at each location and one can connect with either ER or LR optics.
A photonic switch introduces less than 3 dB of attenuation between the AMS-IX patch panel and the Ethernet access switch. A switchover between the two topologies introduces a very short link flap (typically < 20 ms). In order to avoid BGP instability, you should configure your router to ignore such events.
Most vendors implement specific commands to ensure BGP ignores such events (see '10GE specifics' in the respective vendor sections for Cisco, Force10. Foundry/Brocade and Juniper configurations). If your router platform does not support such a feature, we advise you to configure the equivalent of:
A photonic switch introduces less than 3 dB of attenuation between the AMS-IX patch panel and the Ethernet access switch. A switchover between the two topologies introduces a very short link flap (typically < 20 ms). In order to avoid BGP instability, you should configure your router to ignore such events.
Most vendors implement specific commands to ensure BGP ignores such events (see '10GE specifics' in the respective vendor sections for Cisco, Force10. Foundry/Brocade and Juniper configurations). If your router platform does not support such a feature, we advise you to configure the equivalent of:
no bgp fast-external-fallover
no bgp fast-external-fallover
To ignore link flaps and wait for the BGP hold timers to expire before resetting sessions.
To ignore link flaps and wait for the BGP hold timers to expire before resetting sessions.
Here is what we recommend:
Here is what we recommend:
Each equipment vendor implements its own maximum ages for the IPv4 ARP and IPv6 neighbor caches. The values vary widely and in at least one case (Linux) it is not a constant.Low ARP timeouts can lead to excessive ARP traffic, especially if the values are lower than the BGP KEEPALIVE interval timers. On the other hand, long timeouts can theoretically lead to longer downtime if you change equipment (since your peers still have the old MAC address in their ARP cache). With BGP this is unlikely to happen because your router will start re-establishing BGP sessions as soon as it is back up, causing its peers to update their ARP cache as well.
We recommend setting the ARP cache timeout to at least two hours, preferably four (240 minutes). See the sections on specific equipment vendors for examples.
Each equipment vendor implements its own maximum ages for the IPv4 ARP and IPv6 neighbor caches. The values vary widely and in at least one case (Linux) it is not a constant.Low ARP timeouts can lead to excessive ARP traffic, especially if the values are lower than the BGP KEEPALIVE interval timers. On the other hand, long timeouts can theoretically lead to longer downtime if you change equipment (since your peers still have the old MAC address in their ARP cache). With BGP this is unlikely to happen because your router will start re-establishing BGP sessions as soon as it is back up, causing its peers to update their ARP cache as well.
We recommend setting the ARP cache timeout to at least two hours, preferably four (240 minutes). See the sections on specific equipment vendors for examples.
The IPv4 prefix for the AMS-IX peering LAN (206.108.115.0/24) is part of AS 62972, and is not supposed to be globally routable. This means the following:
In short, you can take the view that the Peering LAN is a link-local address range and you may decide to not even redistribute it internally (but in that case you may want to set a static route for management access so you can troubleshoot peering, etc.).
The IPv4 prefix for the AMS-IX peering LAN (80.249.208.0/21) is part of AS1200, and is not supposed to be globally routable. This means the following:
In short, you can take the view that the Peering LAN is a link-local address range and you may decide to not even redistribute it internally (but in that case you may want to set a static route for management access so you can troubleshoot peering, etc.).
Please exchange only unicast routes over your BGP sessions in the ISP peering LAN. Exchanging multicast routes is useless since multicast traffic is not allowed on the (unicast) ISP peering LAN.
Please exchange only unicast routes over your BGP sessions in the ISP peering LAN. Exchanging multicast routes is useless since multicast traffic is not allowed on the (unicast) ISP peering LAN.
The Technical Specifications state the following:
This implies IEEE 802.3 compliance, not 802.2, so no LLC encapsulation!
The Technical Specifications state the following:
This implies IEEE 802.3 compliance, not 802.2, so no LLC encapsulation!
The AMS-IX rules dictate that only one MAC address is allowed behind a port. This means that you have to be extremely careful when connecting a device that can act as a L2 device.
We allow only one MAC address because we allow no additional devices behind the AMS-IX ports. Extended L2 networks are not under the control of AMS-IX, but instabilities in a L2 network behind the AMS-IX switches can and typically do have a negative impact on the whole exchange. Forwarding loops and spanning tree topology changes are good examples of this. By enforcing the one-MAC-address-per-port rule, we effectively prevent forwarding loops and STP traffic from intermediate L2 devices.
In short, an intermediate L2 device may only bridge frames from the member's router to the AMS-IX port (so we see only one MAC address) and should otherwise be completely invisible. No connected device should bridge frames from other devices onto AMS-IX, or talk STP on its AMS-IX interface.
4.1.1 Connecting a L3 Device
The most preferred way of connecting to AMS-IX is directly through a L3 device (router), see the diagram below. This is your best chance of not leaking MAC addresses or STP traffic and it greatly increases the stability of the network.
The AMS-IX rules dictate that only one MAC address is allowed behind a port. This means that you have to be extremely careful when connecting a device that can act as a L2 device.
We allow only one MAC address because we allow no additional devices behind the AMS-IX ports. Extended L2 networks are not under the control of AMS-IX, but instabilities in a L2 network behind the AMS-IX switches can and typically do have a negative impact on the whole exchange. Forwarding loops and spanning tree topology changes are good examples of this. By enforcing the one-MAC-address-per-port rule, we effectively prevent forwarding loops and STP traffic from intermediate L2 devices.
In short, an intermediate L2 device may only bridge frames from the member's router to the AMS-IX port (so we see only one MAC address) and should otherwise be completely invisible. No connected device should bridge frames from other devices onto AMS-IX, or talk STP on its AMS-IX interface.
4.1.1 Connecting a L3 Device
The most preferred way of connecting to AMS-IX is directly through a L3 device (router), see the diagram below. This is your best chance of not leaking MAC addresses or STP traffic and it greatly increases the stability of the network.
4.1.2 Connecting Through a L2 Device
We neither recommend nor encourage connecting your router through a L2 device, but if you do so, keep the following in mind:
4.1.2 Connecting Through a L2 Device
We neither recommend nor encourage connecting your router through a L2 device, but if you do so, keep the following in mind:
On all intermediate L2 devices, consider using explicitly defined port-based VLANs for production ports. It forces you to understand your topology and reduces the chances of a nasty surprise further down the road. In particular, we strongly recommend using a dedicated VLAN for the path from your router to AMS-IX.
On all intermediate L2 devices, consider using explicitly defined port-based VLANs for production ports. It forces you to understand your topology and reduces the chances of a nasty surprise further down the road. In particular, we strongly recommend using a dedicated VLAN for the path from your router to AMS-IX.
4.1.3 Connecting a L2/L3 Hybrid
The L2/L3 hybrid switch/router requires careful configuration in order to prevent unwanted traffic from leaking onto the exchange. As with intermediate L2 devices, you need to keep the following in mind:
On a L2/L3 hybrid device, it is a good idea to put the AMS-IX connected interface (untagged) in a separate (non-default) port-based VLAN without spanning tree and with no other ports in it. This is the best way to ensure that no traffic from other ports will be bridged onto the AMS-IX port.
On a L2/L3 hybrid device, it is a good idea to put the AMS-IX connected interface (untagged) in a separate (non-default) port-based VLAN without spanning tree and with no other ports in it. This is the best way to ensure that no traffic from other ports will be bridged onto the AMS-IX port.
Any traffic other than the types mentioned in the previous section is deemed to be illegal traffic. In this section we will list some of the more common types of violations we see at AMS-IX and give some arguments as to why it is considered unwanted.
4.2.1 Multiple MAC addresses
Since AMS-IX operates on the principle of one router per port, there should be one MAC address visible behind each port. Some members connect through intermediate switches, or use a L2/L3 hybrid device. If these devices are not configured properly, they can cause forwarding loops, STP instabilites, and lots of unwanted traffic on the exchange. There is no excuse for these devices to leak traffic, and there is no necessity to talk STP on the link to AMS-IX. Hence, by enforcing the one-MAC-address rule, we also enforce these issues. Beware that this rule is enforced automatically, so if you leak traffic from another MAC address, your legitimate traffic may be blocked (depending on which MAC address the switch sees first)!
4.2.2 Spanning Tree (STP)
This point is closely related to the previous point. The device(s) connected to the AMS-IX port are not allowed to be visible as L2 bridges. This means that they should not speak STP (spanning tree) or any other (proprietary) L2 specific protocol.
4.2.3 Routing protocols: EIGRP, OSPF, RIP, IS-IS
The only routing protocol allowed on AMS-IX is BGP. There is no valid reason for interior routing protocols to appear on the shared medium. These protocols only cause unnecessary multicast and broadcast traffic.
4.2.4 (Cisco) Keepalive
By default Cisco routers and switches periodically test their (Fast) Ethernet links by sending out Loopback frames (ethertype 0x9000) addressed to themselves. Call it a 'L2 self-ping' if you will. In a switched environment it can be used to test the functionality of the switch and/or keep the router's MAC address in the switch's address table.
In the AMS-IX environment, this is not useful since we use MAC timeouts that are larger than the typical BGP and/or ARP timeouts. In fact, the keepalives a may actually cause port security violations if they are being sent by an intermediate switch.
4.2.5 Discovery protocols: CDP, EDP, LLDP
Various vendors (e.g. Extreme, Cisco) tend to ship their boxes as gregarious devices: by default they announce their existence out of all their interfaces and try to find family members. CDP (Cisco) and EDP (Extreme) are examples of this, but there are others.
The only reason for running discovery protocols is to support certain types of autoconfiguration. Autoconfiguration on an Internet Exchange is a very bad idea. Hence, there is absolutely no reason to run discovery protocols on your AMS-IX interface. Discovery protocols typically cause unwanted broadcast or multicast traffic.
4.2.6 Non-unicast IPv4: IGMP, DHCP, TFTP
On the ISP peering LAN, the only non-unicast traffic that is allowed is the ARP query.Sometimes we see equipment trying to get a configuration through broadcast TFTP, or configure themselves through DHCP. These options are unsafe and we strongly advise against them.Other equipment has IGMP turned on by default (or by accident). The Peering LAN is for unicast IP traffic only, so there is no point in configuring multicast on the AMS-IX interface.
4.2.7 Proxy ARP
Since traffic over AMS-IX is exchanged based on BGP routes, there is no reason to answer ARP queries for any other IP address(es) than those that are configured on your AMS-IX interface.Unfortunately, some vendors (e.g. Cisco) ship their products with proxy ARP enabled by default.Proxy ARP is not only sloppy, it can lead to unwanted traffic on your network. Consider that if you have it enabled at AMS-IX, it's likely to be enabled at other peering points, allowing parties on both sides to use you as a transit.Proxy ARP is not allowed.
4.2.8 Non-unicast IPv6: IPv6 ND-RA
IPv6 router advertisements are not allowed: they generate a lot of unnecessary traffic, since IPv6 hosts on AMS-IX are not autoconfigured and besides, you don't want to be the default router for AMS-IX as a whole.
4.2.9 Miscellaneous non-IP: DEC MOP, etc.
Some vendors enable protocols other than IP by default. Cisco, for example ships certain versions of IOS with DEC MOP enabled by default. This is non-IP traffic and has no place on AMS-IX.
Any traffic other than the types mentioned in the previous section is deemed to be illegal traffic. In this section we will list some of the more common types of violations we see at AMS-IX and give some arguments as to why it is considered unwanted.
4.2.1 Multiple MAC addresses
Since AMS-IX operates on the principle of one router per port, there should be one MAC address visible behind each port. Some members connect through intermediate switches, or use a L2/L3 hybrid device. If these devices are not configured properly, they can cause forwarding loops, STP instabilites, and lots of unwanted traffic on the exchange. There is no excuse for these devices to leak traffic, and there is no necessity to talk STP on the link to AMS-IX. Hence, by enforcing the one-MAC-address rule, we also enforce these issues. Beware that this rule is enforced automatically, so if you leak traffic from another MAC address, your legitimate traffic may be blocked (depending on which MAC address the switch sees first)!
4.2.2 Spanning Tree (STP)
This point is closely related to the previous point. The device(s) connected to the AMS-IX port are not allowed to be visible as L2 bridges. This means that they should not speak STP (spanning tree) or any other (proprietary) L2 specific protocol.
4.2.3 Routing protocols: EIGRP, OSPF, RIP, IS-IS
The only routing protocol allowed on AMS-IX is BGP. There is no valid reason for interior routing protocols to appear on the shared medium. These protocols only cause unnecessary multicast and broadcast traffic.
4.2.4 (Cisco) Keepalive
By default Cisco routers and switches periodically test their (Fast) Ethernet links by sending out Loopback frames (ethertype 0x9000) addressed to themselves. Call it a 'L2 self-ping' if you will. In a switched environment it can be used to test the functionality of the switch and/or keep the router's MAC address in the switch's address table.
In the AMS-IX environment, this is not useful since we use MAC timeouts that are larger than the typical BGP and/or ARP timeouts. In fact, the keepalives a may actually cause port security violations if they are being sent by an intermediate switch.
4.2.5 Discovery protocols: CDP, EDP, LLDP
Various vendors (e.g. Extreme, Cisco) tend to ship their boxes as gregarious devices: by default they announce their existence out of all their interfaces and try to find family members. CDP (Cisco) and EDP (Extreme) are examples of this, but there are others.
The only reason for running discovery protocols is to support certain types of autoconfiguration. Autoconfiguration on an Internet Exchange is a very bad idea. Hence, there is absolutely no reason to run discovery protocols on your AMS-IX interface. Discovery protocols typically cause unwanted broadcast or multicast traffic.
4.2.6 Non-unicast IPv4: IGMP, DHCP, TFTP
On the ISP peering LAN, the only non-unicast traffic that is allowed is the ARP query.Sometimes we see equipment trying to get a configuration through broadcast TFTP, or configure themselves through DHCP. These options are unsafe and we strongly advise against them.Other equipment has IGMP turned on by default (or by accident). The Peering LAN is for unicast IP traffic only, so there is no point in configuring multicast on the AMS-IX interface.
4.2.7 Proxy ARP
Since traffic over AMS-IX is exchanged based on BGP routes, there is no reason to answer ARP queries for any other IP address(es) than those that are configured on your AMS-IX interface.Unfortunately, some vendors (e.g. Cisco) ship their products with proxy ARP enabled by default.Proxy ARP is not only sloppy, it can lead to unwanted traffic on your network. Consider that if you have it enabled at AMS-IX, it's likely to be enabled at other peering points, allowing parties on both sides to use you as a transit.Proxy ARP is not allowed.
4.2.8 Non-unicast IPv6: IPv6 ND-RA
IPv6 router advertisements are not allowed: they generate a lot of unnecessary traffic, since IPv6 hosts on AMS-IX are not autoconfigured and besides, you don't want to be the default router for AMS-IX as a whole.
4.2.9 Miscellaneous non-IP: DEC MOP, etc.
Some vendors enable protocols other than IP by default. Cisco, for example ships certain versions of IOS with DEC MOP enabled by default. This is non-IP traffic and has no place on AMS-IX.
Cisco's philosophy seems to be similar to that of some PC OS vendors: enable as many protocols and features as possible by default, so the device works out-of-the-box in most situations. Unfortunately, this means that a lot of unnecessary features are turned on that, while harmless in LAN or corporate environments, can cause undesired traffic on an Internet Exchange.Typical things that need to be disabled are: autoconfiguration protocols (DHCP, BOOTP, TFTP config download over the AMS-IX interface), CDP, DEC MOP, IP redirects, IP directed broadcasts, proxy ARP, IPv6 Router Advertisements, keepalive.
Intermediate switches or hybrid devices will also need to disable VTP, STP, etc.
Cisco's philosophy seems to be similar to that of some PC OS vendors: enable as many protocols and features as possible by default, so the device works out-of-the-box in most situations. Unfortunately, this means that a lot of unnecessary features are turned on that, while harmless in LAN or corporate environments, can cause undesired traffic on an Internet Exchange.Typical things that need to be disabled are: autoconfiguration protocols (DHCP, BOOTP, TFTP config download over the AMS-IX interface), CDP, DEC MOP, IP redirects, IP directed broadcasts, proxy ARP, IPv6 Router Advertisements, keepalive.
Intermediate switches or hybrid devices will also need to disable VTP, STP, etc.
Global configuration
Global configuration
! Do not run a DHCP server/relay agent
no service dhcp
! Older IOS versions require this instead of the above.
no ip bootp server
! Do not download configs through TFTP
no service config
! Do not run CDP
no cdp run
! Do not run a DHCP server/relay agent
no service dhcp
! Older IOS versions require this instead of the above.
no ip bootp server
! Do not download configs through TFTP
no service config
! Do not run CDP
no cdp run
Interface configuration
Interface configuration
! Don't do redirects -- if they don't know
! how to route properly, tough luck!
no ip redirects
! Don't run proxy ARP on your AMS-IX interface
no ip proxy-arp
! Don't run CDP on your AMS-IX interface
no cdp enable
! Directed broadcasts are evil.
no ip directed-broadcast
! Disable the DEC drek if you haven't done so globally yet.
no mop enable
! For (Fast)Ethernet: no auto-negotiation on your connection.
! no negotiation auto
! duplex half
duplex full
! L2 keepalives are useless on the AMS-IX
no keepalive
! Don't do redirects -- if they don't know
! how to route properly, tough luck!
no ip redirects
! Don't run proxy ARP on your AMS-IX interface
no ip proxy-arp
! Don't run CDP on your AMS-IX interface
no cdp enable
! Directed broadcasts are evil.
no ip directed-broadcast
! Disable the DEC drek if you haven't done so globally yet.
no mop enable
! For (Fast)Ethernet: no auto-negotiation on your connection.
! no negotiation auto
! duplex half
duplex full
! L2 keepalives are useless on the AMS-IX
no keepalive
It is difficult to give a complete guide for Cisco products, because of the many different types of devices and (IOS) software versions. When in doubt, consult your documentation.
5.3.1 29xx and 35xx Series
If you use a Cisco Layer 2 device (such as the 2900 and 3500 series), you have to turn off VTP (VLAN Trunking Protocol), DTP (Dynamic Trunking Protocol), LLDP, and UDLD.
In global config mode:
It is difficult to give a complete guide for Cisco products, because of the many different types of devices and (IOS) software versions. When in doubt, consult your documentation.
5.3.1 29xx and 35xx Series
If you use a Cisco Layer 2 device (such as the 2900 and 3500 series), you have to turn off VTP (VLAN Trunking Protocol), DTP (Dynamic Trunking Protocol), LLDP, and UDLD.
In global config mode:
vtp mode transparent
!
no spanning-tree vlan 1200
! If you don't need LLDP, disable globally
no lldp run
! If you don't need CDP, disable globally
no cdp run
!
vlan 1200
name AMS-IX
!
interface /IfIdent/
description Interface to AMS-IX
switchport access vlan 1200
switchport mode access
switchport nonegotiate
no keepalive
speed nonegotiate
no udld enable
! If CDP has not been disabled globally:
no cdp enable
! If LLDP has not been disabled globally:
no lldp receive
no lldp transmit
! If you do not want to shut off STP:
spanning-tree bpdufilter enable
end
vtp mode transparent
!
no spanning-tree vlan 1200
! If you don't need LLDP, disable globally
no lldp run
! If you don't need CDP, disable globally
no cdp run
!
vlan 1200
name AMS-IX
!
interface /IfIdent/
description Interface to AMS-IX
switchport access vlan 1200
switchport mode access
switchport nonegotiate
no keepalive
speed nonegotiate
no udld enable
! If CDP has not been disabled globally:
no cdp enable
! If LLDP has not been disabled globally:
no lldp receive
no lldp transmit
! If you do not want to shut off STP:
spanning-tree bpdufilter enable
end
5.3.2 7600 Series
Members are advised not to run 12.2(33)SRC on their Cisco 7600's with a sup720. This software release does not always send or forward replies to solicit requests, even if it's acting as a pure Layer 2 switch between a member router and the AMS-IX fabric.
To make a Cisco 7600 switch 'silent' the following configuration seems to work:
5.3.2 7600 Series
Members are advised not to run 12.2(33)SRC on their Cisco 7600's with a sup720. This software release does not always send or forward replies to solicit requests, even if it's acting as a pure Layer 2 switch between a member router and the AMS-IX fabric.
To make a Cisco 7600 switch 'silent' the following configuration seems to work:
no service dhcp
no ip bootp server
vtp mode transparent
spanning-tree mode pvst
spanning-tree extend system-id
no spanning-tree vlan XX
!
vlan XX
name amsix
exit
!
interface GigabitEthernet6/0/0
description to-amsix switchport
switchport access vlan XX
switchport mode access
switchport nonegotiate
no mls qos trust
no cdp enable
spanning-tree bpdufilter enable
exit
!
no service dhcp
no ip bootp server
vtp mode transparent
spanning-tree mode pvst
spanning-tree extend system-id
no spanning-tree vlan XX
!
vlan XX
name amsix
exit
!
interface GigabitEthernet6/0/0
description to-amsix switchport
switchport access vlan XX
switchport mode access
switchport nonegotiate
no mls qos trust
no cdp enable
spanning-tree bpdufilter enable
exit
!
Vlan XX was also removed from the 'allow list' on all dot1q trunk ports not related to the setup, in this case every dot1q trunk port in the chassis.
5.3.3 Catalyst 6500 Series
CatOS and IOS are different beasts, so for Catalyst switches, the following applies:
Vlan XX was also removed from the 'allow list' on all dot1q trunk ports not related to the setup, in this case every dot1q trunk port in the chassis.
5.3.3 Catalyst 6500 Series
CatOS and IOS are different beasts, so for Catalyst switches, the following applies:
set vtp mode off
set port name /IfIdent/ My AMS-IX Port
set cdp disable /IfIdent/
set udld disable /IfIdent/
set trunk /IfIdent/ off dot1q
set spantree bpdu-filter /IfIdent/ enable
set vlan 1200 name My_AMS-IX_Vlan
set vlan 1200 /IfIdent
set vtp mode off
set port name /IfIdent/ My AMS-IX Port
set cdp disable /IfIdent/
set udld disable /IfIdent/
set trunk /IfIdent/ off dot1q
set spantree bpdu-filter /IfIdent/ enable
set vlan 1200 name My_AMS-IX_Vlan
set vlan 1200 /IfIdent
If, for some reason, you cannot afford to turn off VTP globally, the only way to turn it off on individual ports seems to be by using l2pt:
If, for some reason, you cannot afford to turn off VTP globally, the only way to turn it off on individual ports seems to be by using l2pt:
set port l2protocol-tunnel /IfIdent/ vtp enable
set port l2protocol-tunnel /IfIdent/ vtp enable
Depending on your CatOS platform, you may or may not be able to do this.
5.3.4 CRS (IOS-XR)
CDP, Proxy ARP, Directed Broadcast, Link Auto Negotiation, and ICMP redirects* are disabled by default in IOS-XR. ICMP redirect messages are disabled by default on the interface unless the Hot Standby Router Protocol (HSRP) is configured.
5.3.5 Other Devices
For other devices, some or all of the above may apply. Check your documentation for details.
Depending on your CatOS platform, you may or may not be able to do this.
5.3.4 CRS (IOS-XR)
CDP, Proxy ARP, Directed Broadcast, Link Auto Negotiation, and ICMP redirects* are disabled by default in IOS-XR. ICMP redirect messages are disabled by default on the interface unless the Hot Standby Router Protocol (HSRP) is configured.
5.3.5 Other Devices
For other devices, some or all of the above may apply. Check your documentation for details.
5.4.1 Catalyst 6500 Series
Configure the port-channel as on, or should you want LACP, as active. Please do not not configure any forms of negotiate or desirable as the AMS-IX switches do not speak PAgP.
Load-balancing over four ports may result in an unequal distribution due to bug CSCsg80948.
5.4.1 Catalyst 6500 Series
Configure the port-channel as on, or should you want LACP, as active. Please do not not configure any forms of negotiate or desirable as the AMS-IX switches do not speak PAgP.
Load-balancing over four ports may result in an unequal distribution due to bug CSCsg80948.
! Here is an example configuration:
interface GigabitEthernet1/1
description AMS-IX Link 1
no ip address
no ip redirects
no ip proxy-arp
no keepalive
no cdp enable
channel-group 1 mode on
!
interface GigabitEthernet1/2
description AMS-IX Link 2
no ip address
no ip redirects
no ip proxy-arp
no keepalive
no cdp enable
channel-group 1 mode on
!
interface Port-channel1
description AMS-IX aggregated link
ip address 80.249.20x.y 255.255.248.0
no ip redirects
no ip proxy-arp
no keepalive
!
! Here is an example configuration:
interface GigabitEthernet1/1
description AMS-IX Link 1
no ip address
no ip redirects
no ip proxy-arp
no keepalive
no cdp enable
channel-group 1 mode on
!
interface GigabitEthernet1/2
description AMS-IX Link 2
no ip address
no ip redirects
no ip proxy-arp
no keepalive
no cdp enable
channel-group 1 mode on
!
interface Port-channel1
description AMS-IX aggregated link
ip address 80.249.20x.y 255.255.248.0
no ip redirects
no ip proxy-arp
no keepalive
!
Here are examples of LACP configurations:
Here are examples of LACP configurations:
Cisco IOS 65xx/76xx:
interface GigabitEthernet1/1
description AMS-IX Link 1
channel-group 10 mode active
! (12.2(18)SXF2 or (12.2(33)SRC) upwards)
lacp rate fast
!
interface GigabitEthernet1/2
description AMS-IX Link 2
channel-group 10 mode active
!
interface Port-channel10
description AMS-IX aggregated link
no switchport
ip address 80.249.20x.y 255.255.248.0
!
Cisco IOS-XR:
interface Bundle-Ether 10
description AMS-IX aggregated link
ipv4 address 80.249.20x.y 255.255.248.0
!
interface GigabitEthernet 1/0/0/0
description AMS-IX Link 1
bundle-id 10 mode active
! (3.2 upwards)
lacp period short
!
interface GigabitEthernet 1/0/1/0
description AMS-IX Link 2
bundle-id 10 mode active
!
(don't forget to commit)
Cisco NX-OS:
feature lacp
!
interface ethernet 2/1
description AMS-IX Link 1
channel-group 10 mode active
lacp rate fast
!
interface ethernet 2/2
description AMS-IX Link 2
channel-group 10 mode active
!
interface port-channel 10
description AMS-IX aggregated link
ip address 80.249.20x.y 255.255.248.0
!
Cisco IOS 65xx/76xx:
interface GigabitEthernet1/1
description AMS-IX Link 1
channel-group 10 mode active
! (12.2(18)SXF2 or (12.2(33)SRC) upwards)
lacp rate fast
!
interface GigabitEthernet1/2
description AMS-IX Link 2
channel-group 10 mode active
!
interface Port-channel10
description AMS-IX aggregated link
no switchport
ip address 80.249.20x.y 255.255.248.0
!
Cisco IOS-XR:
interface Bundle-Ether 10
description AMS-IX aggregated link
ipv4 address 80.249.20x.y 255.255.248.0
!
interface GigabitEthernet 1/0/0/0
description AMS-IX Link 1
bundle-id 10 mode active
! (3.2 upwards)
lacp period short
!
interface GigabitEthernet 1/0/1/0
description AMS-IX Link 2
bundle-id 10 mode active
!
(don't forget to commit)
Cisco NX-OS:
feature lacp
!
interface ethernet 2/1
description AMS-IX Link 1
channel-group 10 mode active
lacp rate fast
!
interface ethernet 2/2
description AMS-IX Link 2
channel-group 10 mode active
!
interface port-channel 10
description AMS-IX aggregated link
ip address 80.249.20x.y 255.255.248.0
!
5.4.2 GSR Series
Do not set a static MAC address on the Port-channel interface. This causes CEF inconsistencies and other assorted failures.Link aggregation and IPv6 do not seem to play well together. Cisco advises against trying this.
Some changes will result in a different MAC address getting chosen for the aggregated link (likely such as reloading a linecard, if it contains the first port in the bundle). This will keep your ports dysfunctional due to port security on the AMS-IX switches and you will have to contact the AMS-IX NOC in such cases to fix this.
Some restrictions apply to what features are supported on link bundles (e.g. sampled NetFlow only on ISE/Engine4+; no uRPF). Also not all line cards support link bundling, and if traffic towards AMS-IX comes in on such an interface you will experience suboptimal load-balancing. Please see the Cisco documentation for more details.
Support for link bundling on Engine 5 linecards will come in 12.0(33)S.Cisco Engineering have a special train called 'Phase 3' (lb-eft-ph3) that is purported to also provide functionality such as MAC address accounting for Port-Channel interfaces. This seems to have been integrated into 12.0(32)S, but IPv6 does not seem to be supported yet.
Below follows a list of Cisco Bug IDs (ddts) related to link aggregation that you need to consider when choosing an appropriate IOS image
CSCee27396
present in 12.0(26)S1; fixed in 12.0(26)S3, 12.0(27)S2, 12.0(28)S1, 12.0(30)SSymptoms: Over 90% CPU usage by CEF Scanner on all linecards and %TFIB-7-SCANSABORTED errors occur when configuring a link bundle. Also, the router sends traffic to MAC addresses taken from its ARP table seemingly at random, instead of to the appropriate next-hop's MAC address.
CSCef12828
present in post-CSCee27396; fixed in 12.0(26)S4, 12.0(27)S3, 12.0(28)S1, 12.0(30)SSymptoms: When traffic passes through a router, the router blocks traffic for certain prefixes behind a port-channel link.
CSCdz33664
present in 12.0(25)S3, 12.0(26)S1, 12.0(27)S2, 12.0(28)S; fixed in 12.0(25)S4Symptoms: An HSRP state change on any Engine2 interface causes a microcode bundle flap on all other Engine2 linecards, preventing load balancing to work due to vanilla microcode getting loaded.
CSCee81071
present in 12.0(26)S3, 12.0(27)S2, 12.0(29)SSymptoms: Router sends Ethernet frames with a source MAC address of beef.f00d.beef and destination MAC address f00d.beef.f00d (which is the pattern scribbled in unallocated memory in linecards), with what looks to be a legitimate payload of transit traffic. This is one of the symptoms of CSCee27396
CSCeb38014
present in 12.0(26)S5; fixed in 12.0(26)S5, 12.0(27)SSymptoms: The BGP Router process flushes the BGP tables for each peer when you change one neighbor's description. This pegs the GRP CPU at 99% for quite a while.
CSCeg31951
present in 12.0(31)S; fixed in 12.0(31)S2 (CSCei53226)
Symptoms: IOS (at least in the PRP code) places each individual public peer in its own update-group if remove-private-as is configured on a peer. Needless to say, this scales badly for a router connected to an Internet exchange. (Try 'show ip bgp replication'.) A collection of hearsay follows for recent IOS images for the GSR PRP regarding link aggregation. AMS-IX does not run any GSRs. Please take this information with appropriately-sized grains of salt.
You can check for incorrect next-hops by attaching to the linecard and executing show controllers rewrite and show adjacency internal and comparing the two rewrite strings for a certain peer's IPv4 address (suffix the commands with | begin 80.249.20a.b). The first six bytes of the returned long hex string should be the peer's MAC address, and equal for all three occurrences.
5.4.2 GSR Series
Do not set a static MAC address on the Port-channel interface. This causes CEF inconsistencies and other assorted failures.Link aggregation and IPv6 do not seem to play well together. Cisco advises against trying this.
Some changes will result in a different MAC address getting chosen for the aggregated link (likely such as reloading a linecard, if it contains the first port in the bundle). This will keep your ports dysfunctional due to port security on the AMS-IX switches and you will have to contact the AMS-IX US NOC in such cases to fix this.
Some restrictions apply to what features are supported on link bundles (e.g. sampled NetFlow only on ISE/Engine4+; no uRPF). Also not all line cards support link bundling, and if traffic towards AMS-IX comes in on such an interface you will experience suboptimal load-balancing. Please see the Cisco documentation for more details.
Support for link bundling on Engine 5 linecards will come in 12.0(33)S.Cisco Engineering have a special train called 'Phase 3' (lb-eft-ph3) that is purported to also provide functionality such as MAC address accounting for Port-Channel interfaces. This seems to have been integrated into 12.0(32)S, but IPv6 does not seem to be supported yet.
Below follows a list of Cisco Bug IDs (ddts) related to link aggregation that you need to consider when choosing an appropriate IOS image
CSCee27396
present in 12.0(26)S1; fixed in 12.0(26)S3, 12.0(27)S2, 12.0(28)S1, 12.0(30)SSymptoms: Over 90% CPU usage by CEF Scanner on all linecards and %TFIB-7-SCANSABORTED errors occur when configuring a link bundle. Also, the router sends traffic to MAC addresses taken from its ARP table seemingly at random, instead of to the appropriate next-hop's MAC address.
CSCef12828
present in post-CSCee27396; fixed in 12.0(26)S4, 12.0(27)S3, 12.0(28)S1, 12.0(30)SSymptoms: When traffic passes through a router, the router blocks traffic for certain prefixes behind a port-channel link.
CSCdz33664
present in 12.0(25)S3, 12.0(26)S1, 12.0(27)S2, 12.0(28)S; fixed in 12.0(25)S4Symptoms: An HSRP state change on any Engine2 interface causes a microcode bundle flap on all other Engine2 linecards, preventing load balancing to work due to vanilla microcode getting loaded.
CSCee81071
present in 12.0(26)S3, 12.0(27)S2, 12.0(29)SSymptoms: Router sends Ethernet frames with a source MAC address of beef.f00d.beef and destination MAC address f00d.beef.f00d (which is the pattern scribbled in unallocated memory in linecards), with what looks to be a legitimate payload of transit traffic. This is one of the symptoms of CSCee27396
CSCeb38014
present in 12.0(26)S5; fixed in 12.0(26)S5, 12.0(27)SSymptoms: The BGP Router process flushes the BGP tables for each peer when you change one neighbor's description. This pegs the GRP CPU at 99% for quite a while.
CSCeg31951
present in 12.0(31)S; fixed in 12.0(31)S2 (CSCei53226)
Symptoms: IOS (at least in the PRP code) places each individual public peer in its own update-group if remove-private-as is configured on a peer. Needless to say, this scales badly for a router connected to an Internet exchange. (Try 'show ip bgp replication'.) A collection of hearsay follows for recent IOS images for the GSR PRP regarding link aggregation. AMS-IX does not run any GSRs. Please take this information with appropriately-sized grains of salt.
You can check for incorrect next-hops by attaching to the linecard and executing show controllers rewrite and show adjacency internal and comparing the two rewrite strings for a certain peer's IPv4 address (suffix the commands with | begin 80.249.20a.b). The first six bytes of the returned long hex string should be the peer's MAC address, and equal for all three occurrences.
! An example configuration follows:
!
interface Port-channel1
description AMS-IX Aggregated Link
ip address 80.249.20x.y 255.255.248.0
no ip redirects
no ip directed-broadcast
no ip proxy-arp
channel-group minimum active 1
no channel-group bandwidth control-propagation
hold-queue 150 in
!
interface GigabitEthernet1/2/1
no keepalive
no negotiation auto
channel-group 1
no cdp enable
!
interface GigabitEthernet1/2/2
no keepalive
no negotiation auto
channel-group 1
no cdp enable
!
! An example configuration follows:
!
interface Port-channel1
description AMS-IX Aggregated Link
ip address 80.249.20x.y 255.255.248.0
no ip redirects
no ip directed-broadcast
no ip proxy-arp
channel-group minimum active 1
no channel-group bandwidth control-propagation
hold-queue 150 in
!
interface GigabitEthernet1/2/1
no keepalive
no negotiation auto
channel-group 1
no cdp enable
!
interface GigabitEthernet1/2/2
no keepalive
no negotiation auto
channel-group 1
no cdp enable
!
Specifying a value is optional, but setting it to the amount of ports in an aggregated link multiplied by 75 is advised.show interfaces Port-channel 1 will display keepalives enabled even though they are not; also, the BIA (burnt-in address, shown as 0000.0000.0000) can be ignored.
Specifying a value is optional, but setting it to the amount of ports in an aggregated link multiplied by 75 is advised.show interfaces Port-channel 1 will display keepalives enabled even though they are not; also, the BIA (burnt-in address, shown as 0000.0000.0000) can be ignored.
5.4.3 CRS (IOS-XR)
5.4.3 CRS (IOS-XR)
interface Bundle-Ether1
description Aggregated interface to AMS-IX Peering LAN
ipv4 address 80.249.20x.y 255.255.248.0
ipv6 nd suppress-ra
ipv6 address 2001:07F8:1::A50a:bcde:1/64
ipv6 enable
bundle minimum-active links 1
!
interface TenGigE0/0/0/0
description interface to AMS-IX Peering LAN #1
bundle id 1 mode on
!
interface TenGigE0/0/0/1
description interface to AMS-IX Peering LAN #2
bundle id 1 mode on
!
interface Bundle-Ether1
description Aggregated interface to AMS-IX Peering LAN
ipv4 address 80.249.20x.y 255.255.248.0
ipv6 nd suppress-ra
ipv6 address 2001:07F8:1::A50a:bcde:1/64
ipv6 enable
bundle minimum-active links 1
!
interface TenGigE0/0/0/0
description interface to AMS-IX Peering LAN #1
bundle id 1 mode on
!
interface TenGigE0/0/0/1
description interface to AMS-IX Peering LAN #2
bundle id 1 mode on
!
IOS supports no bgp fast-external-fallover and event dampening . The no bgp fast external-fallover tells the device to not act immediately on link flaps but to wait for the BGP hold timers to expire before resetting sessions.
Newer versions of Cisco IOS even support ip bgp fast-external-fallover deny in a per-interface context.Note that in practice we have found that the previously advised carrier-delay does not work as expected on Cisco equipment. We suggest you disable fast-external-fallover instead.
In IOS-XR, to disable BGP Fast External Failover globally, add bgp fast-external-failover disable to your global bgp configuration.
IOS supports no bgp fast-external-fallover and event dampening . The no bgp fast external-fallover tells the device to not act immediately on link flaps but to wait for the BGP hold timers to expire before resetting sessions.
Newer versions of Cisco IOS even support ip bgp fast-external-fallover deny in a per-interface context.Note that in practice we have found that the previously advised carrier-delay does not work as expected on Cisco equipment. We suggest you disable fast-external-fallover instead.
In IOS-XR, to disable BGP Fast External Failover globally, add bgp fast-external-failover disable to your global bgp configuration.
Responses on a ICMPv6 multicast listener queries result in bursts of ICMPv6 multicast listener reports. To prevent this configure no ipv6 mld router in interface context. Some other per-interface commands we recommend on a Cisco device:
Responses on a ICMPv6 multicast listener queries result in bursts of ICMPv6 multicast listener reports. To prevent this configure no ipv6 mld router in interface context. Some other per-interface commands we recommend on a Cisco device:
! disable ICMPv6 multicast listener reports
no ipv6 mld router
! disable IPv6 multicast forwarding
no ipv6 mfib forwarding
! v6 ND-RA is unnecessary and undesired
ipv6 nd suppress-ra
! on IOS version 12.2(33)SRC it is the following syntax:
ipv6 nd ra suppress
! on even more later IOS/IOS-XE versions the "all" option is needed to also
! suppress responses to Router Solicitation messages besides periodic RAs:
ipv6 nd ra suppress all
! disable PIM on a specified interface
no ipv6 pim
! disable MLD snooping on hybrid devices and intermediate layer-2 devices
no ipv6 mld snooping
! disable ICMPv6 multicast listener reports
no ipv6 mld router
! disable IPv6 multicast forwarding
no ipv6 mfib forwarding
! v6 ND-RA is unnecessary and undesired
ipv6 nd suppress-ra
! on IOS version 12.2(33)SRC it is the following syntax:
ipv6 nd ra suppress
! on even more later IOS/IOS-XE versions the "all" option is needed to also
! suppress responses to Router Solicitation messages besides periodic RAs:
ipv6 nd ra suppress all
! disable PIM on a specified interface
no ipv6 pim
! disable MLD snooping on hybrid devices and intermediate layer-2 devices
no ipv6 mld snooping
On newer Cisco IOS/IOS-XR versions, the interface IP MTU is automatically set, based on the presence or absence of 802.1q tags. For more details, please consult this document.
On newer Cisco IOS/IOS-XR versions, the interface IP MTU is automatically set, based on the presence or absence of 802.1q tags. For more details, please consult this document.
CAUTION: Updating Firmware in an EAPS Environment
When updating firmware in an Extreme Networks EAPS environment, be sure to temporarily disable your AMS-IX port(s). TFTP file transfers may cause EAPS instabilities resulting in bogus traffic. This is likely to trip the port security on the AMS-IX switches, which may result in 10 minutes downtime.Most people who use Extreme equipment do not have problems with their AMS-IX connections, some do. We would appreciate feedback from people running Extreme equipment on how they configure their AMS-IX facing side.
CAUTION: Updating Firmware in an EAPS Environment
When updating firmware in an Extreme Networks EAPS environment, be sure to temporarily disable your AMS-IX port(s). TFTP file transfers may cause EAPS instabilities resulting in bogus traffic. This is likely to trip the port security on the AMS-IX switches, which may result in 10 minutes downtime.Most people who use Extreme equipment do not have problems with their AMS-IX connections, some do. We would appreciate feedback from people running Extreme equipment on how they configure their AMS-IX facing side.
The configuration fragment below shows how to configure an intermediate L2 switch, which is also part of an EAPS ring. Port 1 is connected to the AMS-IX switch. Ports 2 and 3 are in the ring. The router is somewhere in that ring, in the 'amsix' VLAN.
The configuration fragment below shows how to configure an intermediate L2 switch, which is also part of an EAPS ring. Port 1 is connected to the AMS-IX switch. Ports 2 and 3 are in the ring. The router is somewhere in that ring, in the 'amsix' VLAN.
create vlan "ring"
configure vlan "ring" tag 1200 # VLAN-ID=0x4b0 Global Tag 3
configure vlan "ring" qosprofile "QP8"
configure vlan "ring" add port 2 tagged
configure vlan "ring" add port 3 tagged
create vlan "amsix"
configure vlan "amsix" tag 1700 # VLAN-ID=0x6a4 Global Tag 9
configure vlan "amsix" add port 1 untagged
configure vlan "amsix" add port 2 tagged
configure vlan "amsix" add port 3 tagged
configure port 1 auto off speed 1000 duplex full
configure port 2 auto off speed 1000 duplex full
configure port 3 auto off speed 1000 duplex full
disable edp port 1
disable igmp snooping
disable igmp snooping with-proxy
create eaps "ring-eaps"
configure eaps "ring-eaps" mode transit
configure eaps "ring-eaps" primary port 2
configure eaps "ring-eaps" secondary port 3
configure eaps "ring-eaps" add control vlan "ring"
configure eaps "ring-eaps" add protect vlan "amsix"
enable eaps "ring-eaps"
create vlan "ring"
configure vlan "ring" tag 1200 # VLAN-ID=0x4b0 Global Tag 3
configure vlan "ring" qosprofile "QP8"
configure vlan "ring" add port 2 tagged
configure vlan "ring" add port 3 tagged
create vlan "amsix"
configure vlan "amsix" tag 1700 # VLAN-ID=0x6a4 Global Tag 9
configure vlan "amsix" add port 1 untagged
configure vlan "amsix" add port 2 tagged
configure vlan "amsix" add port 3 tagged
configure port 1 auto off speed 1000 duplex full
configure port 2 auto off speed 1000 duplex full
configure port 3 auto off speed 1000 duplex full
disable edp port 1
disable igmp snooping
disable igmp snooping with-proxy
create eaps "ring-eaps"
configure eaps "ring-eaps" mode transit
configure eaps "ring-eaps" primary port 2
configure eaps "ring-eaps" secondary port 3
configure eaps "ring-eaps" add control vlan "ring"
configure eaps "ring-eaps" add protect vlan "amsix"
enable eaps "ring-eaps"
The configuration fragment below shows the relevant configuration information for a L3-only device. As in the previous example, port 1 is connected to AMS-IX and is configured in the 'amsix' VLAN (untagged).
The configuration fragment below shows the relevant configuration information for a L3-only device. As in the previous example, port 1 is connected to AMS-IX and is configured in the 'amsix' VLAN (untagged).
#
# Config information for VLAN amsix.
#
create vlan "amsix"
configure vlan "amsix" tag 1200
configure vlan "amsix" protocol "IP"
configure vlan "amsix" ipaddress 80.249.20/X/./Y/ 255.255.248.0
configure vlan "amsix" add port 1 untagged
#
configure port 1 display-string "AMS-IX"
disable edp port 1
#
enable ipforwarding vlan "amsix"
disable ipforwarding broadcast vlan "amsix"
disable ipforwarding fast-direct-broadcast vlan "amsix"
disable ipforwarding ignore-broadcast vlan "amsix"
disable ipforwarding lpm-routing vlan "amsix"
disable isq vlan "amsix"
disable irdp vlan "amsix"
disable icmp unreachable vlan "amsix"
disable icmp redirects vlan "amsix"
disable icmp port-unreachables vlan "amsix"
disable icmp time-exceeded vlan "amsix"
disable icmp parameter-problem vlan "amsix"
disable icmp timestamp vlan "amsix"
disable icmp address-mask vlan "amsix"
disable subvlan-proxy-arp "amsix"
configure ip-mtu 1500 vlan "amsix"
#
# IP Route Configuration
#
configure iproute add blackhole default
disable icmpforwarding vlan "amsix"
disable igmp vlan "amsix"
#
# Config information for VLAN amsix.
#
create vlan "amsix"
configure vlan "amsix" tag 1200
configure vlan "amsix" protocol "IP"
configure vlan "amsix" ipaddress 80.249.20/X/./Y/ 255.255.248.0
configure vlan "amsix" add port 1 untagged
#
configure port 1 display-string "AMS-IX"
disable edp port 1
#
enable ipforwarding vlan "amsix"
disable ipforwarding broadcast vlan "amsix"
disable ipforwarding fast-direct-broadcast vlan "amsix"
disable ipforwarding ignore-broadcast vlan "amsix"
disable ipforwarding lpm-routing vlan "amsix"
disable isq vlan "amsix"
disable irdp vlan "amsix"
disable icmp unreachable vlan "amsix"
disable icmp redirects vlan "amsix"
disable icmp port-unreachables vlan "amsix"
disable icmp time-exceeded vlan "amsix"
disable icmp parameter-problem vlan "amsix"
disable icmp timestamp vlan "amsix"
disable icmp address-mask vlan "amsix"
disable subvlan-proxy-arp "amsix"
configure ip-mtu 1500 vlan "amsix"
#
# IP Route Configuration
#
configure iproute add blackhole default
disable icmpforwarding vlan "amsix"
disable igmp vlan "amsix"
There isn't much to configure on Force10 routers. The Network Operations Guide and various pages in the Team Cymru Document Collection provide useful information on Force10 router configuration and management.
There isn't much to configure on Force10 routers. The Network Operations Guide and various pages in the Team Cymru Document Collection provide useful information on Force10 router configuration and management.
! Disable proxy ARP on your AMS-IX interface
Force10(conf)#interface tengigabitethernet 0/0
Force10(conf-if-te-0/0)#no ip proxy-arp
! Disable IPv6 ND RAs
Force10(conf-if-te-0/0)#ipv6 nd suppress-ra
! The default ARP timeout is 4 hours, but can be changed with this command
Force10(conf)#interface tengigabitethernet 0/0
Force10(conf-if-te-0/0)#arp timeout /minutes/
! Disable proxy ARP on your AMS-IX interface
Force10(conf)#interface tengigabitethernet 0/0
Force10(conf-if-te-0/0)#no ip proxy-arp
! Disable IPv6 ND RAs
Force10(conf-if-te-0/0)#ipv6 nd suppress-ra
! The default ARP timeout is 4 hours, but can be changed with this command
Force10(conf)#interface tengigabitethernet 0/0
Force10(conf-if-te-0/0)#arp timeout /minutes/
Force10 E-Series switch/routers support no bgp fast-external-fallover, BGP Graceful Restart, and a link debounce timer to maintain BGP stability during topology switchovers.The recommended option is to use the /link debounce/ command to delay link change notifications on the interface. The default for fiber interfaces is 100 ms, which is a good value to use.
Force10 E-Series switch/routers support no bgp fast-external-fallover, BGP Graceful Restart, and a link debounce timer to maintain BGP stability during topology switchovers.The recommended option is to use the /link debounce/ command to delay link change notifications on the interface. The default for fiber interfaces is 100 ms, which is a good value to use.
The following fragment of configuration gives an idea of how to configure a Foundry (BigIron) device. Depending on the actual role of the device (router or switch between router and AMS-IX) and the type of code loaded into the device you may need to mix and match a little here.
The following fragment of configuration gives an idea of how to configure a Foundry (BigIron) device. Depending on the actual role of the device (router or switch between router and AMS-IX) and the type of code loaded into the device you may need to mix and match a little here.
! Define a single-port VLAN for the AMS-IX port
vlan number name "AMS-IX" by port
no spanning-tree
untagged ethernet if
! Configure the AMS-IX interface
interface ethernet if
port-name "AMS-IX"
! Behave as a router.
route-only
no spanning-tree
! Don't do IPv6 ND-RA (Router Advertisements)
ipv6 nd suppress-ra
! No weird discovery proto, please.
no vlan-dynamic-discovery
! IP address
ip address 80.249.20X.Y 255.255.248.0
! No redirects
no ip redirect
no ipv6 redirect
! AMS-IX recommends 2 hour ARP timeouts
ip arp-age 120
! For fast-ethernet: no autoconfig.
speed-duplex 100-full
! Define a single-port VLAN for the AMS-IX port
vlan number name "AMS-IX" by port
no spanning-tree
untagged ethernet if
! Configure the AMS-IX interface
interface ethernet if
port-name "AMS-IX"
! Behave as a router.
route-only
no spanning-tree
! Don't do IPv6 ND-RA (Router Advertisements)
ipv6 nd suppress-ra
! No weird discovery proto, please.
no vlan-dynamic-discovery
! IP address
ip address 80.249.20X.Y 255.255.248.0
! No redirects
no ip redirect
no ipv6 redirect
! AMS-IX recommends 2 hour ARP timeouts
ip arp-age 120
! For fast-ethernet: no autoconfig.
speed-duplex 100-full
On a Foundry BigIron RX, software version < 2.4, we noticed together with a customer that his device had a very aggressive default setting for ICMPv6 ND queries for known MAC addresses. It retransmitted them every second. The retransmit interval can be altered in interface context as follows:
On a Foundry BigIron RX, software version < 2.4, we noticed together with a customer that his device had a very aggressive default setting for ICMPv6 ND queries for known MAC addresses. It retransmitted them every second. The retransmit interval can be altered in interface context as follows:
! Set the retransmit timer to 1 hour
ipv6 nd ns-retransmit 3600
! Set the retransmit timer to 1 hour
ipv6 nd ns-retransmit 3600
Note: This command should not be confused with 'ipv6 nd ns-interval', which applies to ND queries for unresolved MAC addresses.
Note: This command should not be confused with 'ipv6 nd ns-interval', which applies to ND queries for unresolved MAC addresses.
BigIron JetCore-based switches support link aggregation only on adjacent ports. The first port must be oddly numbered, and the other port must directly follow the first one. The same goes for any additional pairs of ports in an aggregated link.
CAUTION: On BigIron 15000 switches you cannot build trunks with ports on blade 8, or spanning ports on both sides of slot 8!
BigIron JetCore-based switches support link aggregation only on adjacent ports. The first port must be oddly numbered, and the other port must directly follow the first one. The same goes for any additional pairs of ports in an aggregated link.
CAUTION: On BigIron 15000 switches you cannot build trunks with ports on blade 8, or spanning ports on both sides of slot 8!
! Create an aggregate on a Jet-Core based switch
trunk server ethernet slot/port to slot/port+1
! Create an aggregate on a Jet-Core based switch
trunk server ethernet slot/port to slot/port+1
BigIron RX or NetIron MLX/XMR switches don't have limits to port placement for aggregated links. Ports can be non-adjacent or even distributed over multiple blades. BigIron RX has a limit of 8 ports per aggregated link, NetIron MLX/XMR raise this to 16 in software 3.5.0, 32 in 3.8.0
BigIron RX or NetIron MLX/XMR switches don't have limits to port placement for aggregated links. Ports can be non-adjacent or even distributed over multiple blades. BigIron RX has a limit of 8 ports per aggregated link, NetIron MLX/XMR raise this to 16 in software 3.5.0, 32 in 3.8.0
! Create an aggregate on a RX/MLX/XMR switch
trunk ethe slot/port to slot/port ethe otherslot/otherport to otherslot/otherport
! Create an aggregate on a RX/MLX/XMR switch
trunk ethe slot/port to slot/port ethe otherslot/otherport to otherslot/otherport
As of RX software release 2.5.0 and MLX/XMR software release 3.9.0 the link aggregation syntax changed. The configuration now looks like:
As of RX software release 2.5.0 and MLX/XMR software release 3.9.0 the link aggregation syntax changed. The configuration now looks like:
! Create a LAG on a RX/MLX/XMR switch
lag "" static
ports ethernet #/# ethernet #/#
primary-port #/#
deploy
!
! Create a LAG on a RX/MLX/XMR switch
lag "" static
ports ethernet #/# ethernet #/#
primary-port #/#
deploy
!
The primary-port is used as a single point of configuration. All configuration changes to the primary-port are propagated to the other ports in the lag group.
The keyword 'static' designates a standard aggregated link. For an LACP-enabled link, use:
The primary-port is used as a single point of configuration. All configuration changes to the primary-port are propagated to the other ports in the lag group.
The keyword 'static' designates a standard aggregated link. For an LACP-enabled link, use:
! Create a dynamic LAG on a RX/MLX/XMR switch
lag "" dynamic
ports ethernet #/# ethernet #/#
primary-port #/#
lacp-timeout short
deploy
!
! Create a dynamic LAG on a RX/MLX/XMR switch
lag "" dynamic
ports ethernet #/# ethernet #/#
primary-port #/#
lacp-timeout short
deploy
!
We recommend setting the LACP timeout to 'short' to reduce the service interruption time during photonic failovers.
We recommend setting the LACP timeout to 'short' to reduce the service interruption time during photonic failovers.
Foundry/Brocade supports a feature called BGP Graceful Restart that, if all peers support it, will reduce the impact of prefix flaps but the CPU will still have to re-establish any flapped BGP session before the configured interval passes.The command delay-link-event can make the router ignore short link flaps (for example, in the case of a photonic switch swap). We recommend setting this to 20 which equals to 1000 msecs. Consequently, the flap will be logged in syslog, but higher level protocols (BGP in this case) will be unaffected. We suggest to leave fast-external-fallover in its default state.
Foundry/Brocade supports a feature called BGP Graceful Restart that, if all peers support it, will reduce the impact of prefix flaps but the CPU will still have to re-establish any flapped BGP session before the configured interval passes.The command delay-link-event can make the router ignore short link flaps (for example, in the case of a photonic switch swap). We recommend setting this to 20 which equals to 1000 msecs. Consequently, the flap will be logged in syslog, but higher level protocols (BGP in this case) will be unaffected. We suggest to leave fast-external-fallover in its default state.
Recommendations we received for HP ProCurve devices:
spanning-tree ifname bpdu-filter spanning-tree ifname tcn-guard lldp admin-status ifname disable
Recommendations we received for HP ProCurve devices:
spanning-tree ifname bpdu-filter spanning-tree ifname tcn-guard lldp admin-status ifname disable
For Juniper routers, there isn't much to disable. The Juniper Documents contain useful hints on how to set up your Juniper router.
CAUTION: IGMP Bug (PR/20343) in Junos OS versions 5.3R4 !
There's a bug in Junos OS versions up to 5.3R4, that will cause a Juniper router to emit IGMP packets on all its interfaces, even when IGMP is disabled. The only way to stop your router from transmitting IGMP is to configure outgoing packet filters on your AMS-IX interface(s).
For Juniper routers, there isn't much to disable. The Juniper Documents contain useful hints on how to set up your Juniper router.
CAUTION: IGMP Bug (PR/20343) in Junos OS versions 5.3R4 !
There's a bug in Junos OS versions up to 5.3R4, that will cause a Juniper router to emit IGMP packets on all its interfaces, even when IGMP is disabled. The only way to stop your router from transmitting IGMP is to configure outgoing packet filters on your AMS-IX interface(s).
Make sure to exchange only unicast routes in the unicast ISP peering LAN by explicitly adding the following statement to ,em>all neighbors, groups and prefix-limits:
Make sure to exchange only unicast routes in the unicast ISP peering LAN by explicitly adding the following statement to ,em>all neighbors, groups and prefix-limits:
set family inet unicast
set family inet unicast
Be thorough with family inet unicast
If even one of the neighbours, groups or prefix-limits is defined with a family inet "any", you'll enable multicast and turn on MBGP.
Be thorough with family inet unicast
If even one of the neighbours, groups or prefix-limits is defined with a family inet "any", you'll enable multicast and turn on MBGP.
Increasing interface hold-time (1200ms) to preserve BGP sessions during 10/100GE interface swapping
AMS-IX connects 10/100GE members via photonic switch (Glimmerglass), so we can redirect optic signal to our primary and backup switch in case of failure or for maintenance. The signal redirect takes around 20ms, enough to trigger port state change advertisement within the router; because of this, BGP sessions will be torn down as the result. Therefore, we recommend to configure a higher hold-time value on 10/100GE interface to preserve BGP sessions during interface swapping.
Increasing interface hold-time (1200ms) to preserve BGP sessions during 10/100GE interface swapping
AMS-IX connects 10/100GE members via photonic switch (Glimmerglass), so we can redirect optic signal to our primary and backup switch in case of failure or for maintenance. The signal redirect takes around 20ms, enough to trigger port state change advertisement within the router; because of this, BGP sessions will be torn down as the result. Therefore, we recommend to configure a higher hold-time value on 10/100GE interface to preserve BGP sessions during interface swapping.
---
user@router# show interfaces xe-0/1/0
description "interface to AMS-IX Peering LAN";
hold-time up 1200 down 1200
---
---
user@router# show interfaces xe-0/1/0
description "interface to AMS-IX Peering LAN";
hold-time up 1200 down 1200
---
Juniper's default ARP cache timeout is 20 minutes (by comparison: Cisco's default ARP cache timeout is 4 hours, which fits AMS-IX's relatively static environment much better).
To reduce the amount of unnecessary broadcast traffic, we recommend setting the ARP cache timeout on Juniper routers to 4 hours. A recipe for this follows:
Juniper's default ARP cache timeout is 20 minutes (by comparison: Cisco's default ARP cache timeout is 4 hours, which fits AMS-IX's relatively static environment much better).
To reduce the amount of unnecessary broadcast traffic, we recommend setting the ARP cache timeout on Juniper routers to 4 hours. A recipe for this follows:
> configure
Entering configuration mode
[edit]
you@juniper# edit system arp
[edit system arp]
you@juniper# set aging-timer 240
[edit system arp]
you@juniper# show | compare
[edit system arp]
+ aging-timer 240;
[edit system arp]
you@juniper# commit and-quit
commit complete
Exiting configuration mode
> configure
Entering configuration mode
[edit]
you@juniper# edit system arp
[edit system arp]
you@juniper# set aging-timer 240
[edit system arp]
you@juniper# show | compare
[edit system arp]
+ aging-timer 240;
[edit system arp]
you@juniper# commit and-quit
commit complete
Exiting configuration mode
Since Junos 9.4 the ARP cache timeout is also configurable on an interface level:
Since Junos 9.4 the ARP cache timeout is also configurable on an interface level:
[edit system arp aging-timer interface interface-name] aging-timer-minutes;
[edit system arp aging-timer interface interface-name] aging-timer-minutes;
and on more recent versions of Junos that syntax has changed to:
and on more recent versions of Junos that syntax has changed to:
[edit system arp interface interface-name] aging-timer aging-timer-minutes;
[edit system arp interface interface-name] aging-timer aging-timer-minutes;
10.3.1 M-Series
We have encountered no issues with aggregated links and Jun OS (M40, M160, T320). Junos releases prior to 6.0 required VLAN tagging on aggregated interfaces. This limitation has since been removed. An example configuration follows:
10.3.1 M-Series
We have encountered no issues with aggregated links and Jun OS (M40, M160, T320). Junos releases prior to 6.0 required VLAN tagging on aggregated interfaces. This limitation has since been removed. An example configuration follows:
[edit]
niels@junix# show chassis
aggregated-devices {
ethernet {
device-count 1;
}
}
---
[edit]
niels@junix# show interfaces ge-2/1/0
gigether-options {
802.3ad ae0;
}
[edit]
niels@junix# show interfaces ge-3/1/0
gigether-options {
802.3ad ae0;
}
---
[edit]
niels@junix# show interfaces ae0
description "AMS-IX";
unit 0 {
family inet {
filter {
input AMSIX-in;
output AMSIX-out;
}
address 80.249.20x.y/21;
}
family inet6 {
address 2001:07F8:1::A50a:bcde:1/64;
}
}
[edit]
niels@junix# show chassis
aggregated-devices {
ethernet {
device-count 1;
}
}
---
[edit]
niels@junix# show interfaces ge-2/1/0
gigether-options {
802.3ad ae0;
}
[edit]
niels@junix# show interfaces ge-3/1/0
gigether-options {
802.3ad ae0;
}
---
[edit]
niels@junix# show interfaces ae0
description "AMS-IX";
unit 0 {
family inet {
filter {
input AMSIX-in;
output AMSIX-out;
}
address 80.249.20x.y/21;
}
family inet6 {
address 2001:07F8:1::A50a:bcde:1/64;
}
}
Additionally and optionally you can configure more granular load balancing:
Additionally and optionally you can configure more granular load balancing:
#
---
routing-options {
autonomous-system abcde;
forwarding-table {
export [ load-balance ];
}
}
policy-options {
policy-statement load-balance {
then {
load-balance per-packet;
}
}
}
forwarding-options {
hash-key {
family inet {
layer-3;
layer-4;
}
}
}
---
#
---
routing-options {
autonomous-system abcde;
forwarding-table {
export [ load-balance ];
}
}
policy-options {
policy-statement load-balance {
then {
load-balance per-packet;
}
}
}
forwarding-options {
hash-key {
family inet {
layer-3;
layer-4;
}
}
}
---
In case that is not granular enough, you can modify the hash-key algorithm with some undocumented options in Junos OS 7.x and up:
In case that is not granular enough, you can modify the hash-key algorithm with some undocumented options in Junos OS 7.x and up:
---
hash-key {
family inet {
layer-3 {
destination-address;
protocol;
source-address;
}
layer-4 {
destination-port;
source-port;
type-of-service;
}
}
}
---
---
hash-key {
family inet {
layer-3 {
destination-address;
protocol;
source-address;
}
layer-4 {
destination-port;
source-port;
type-of-service;
}
}
}
---
Also, you can set your aggregated min-links to a value that will cause the bundle to drop in the event that your links can no longer support the amount of traffic you plan on shoving down the pipe. Thus, 2-port aggregated link, pushing 1.2 Gbps sustained across, drop bundle if n == 1;
Also, you can set your aggregated min-links to a value that will cause the bundle to drop in the event that your links can no longer support the amount of traffic you plan on shoving down the pipe. Thus, 2-port aggregated link, pushing 1.2 Gbps sustained across, drop bundle if n == 1;
---
aggregated-ether-options {
minimum-links 2;
link-speed 1g;
}
---
---
aggregated-ether-options {
minimum-links 2;
link-speed 1g;
}
---
In a situation with load-balancing over multiple IP interfaces (not AMS-IX), the final statement will make traceroute more confusing to novices as packets may seem to 'bounce' between interfaces by also including TCP/UDP port numbers and ICMP checksums in the algorithm.On an IP1 load-balance per-packet really means per-packet; on an IP2 it actually works per flow, which is preferable.
In a situation with load-balancing over multiple IP interfaces (not AMS-IX), the final statement will make traceroute more confusing to novices as packets may seem to 'bounce' between interfaces by also including TCP/UDP port numbers and ICMP checksums in the algorithm.On an IP1 load-balance per-packet really means per-packet; on an IP2 it actually works per flow, which is preferable.
The link flap introduced by the PXCs make that you have to damp interface transitions. Junos supports a configurable hold-time . A good value would be 1200 ms.
The link flap introduced by the PXCs make that you have to damp interface transitions. Junos supports a configurable hold-time . A good value would be 1200 ms.
[edit]
arien@router# show interfaces xe-0/1/0
description " interface to AMS-IX Peering LAN";
hold-time up 1200 down 1200
[edit]
arien@router# show interfaces xe-0/1/0
description " interface to AMS-IX Peering LAN";
hold-time up 1200 down 1200
Aggregated interfaces require hold timers on all physical interfaces and on the logical aggregated interface. Respectively xe-0/1/0 and ae0 in the example below:
Aggregated interfaces require hold timers on all physical interfaces and on the logical aggregated interface. Respectively xe-0/1/0 and ae0 in the example below:
[edit]
arien@router# show interfaces xe-0/1/0
description "10GE LinkAgg #1";
hold-time up 1200 down 1200;
gigether-options {
802.3ad ae0;
}
[edit]
arien@router# show interfaces xe-0/1/0
description "10GE LinkAgg #1";
hold-time up 1200 down 1200;
gigether-options {
802.3ad ae0;
}
[edit]
arien@router# show interfaces ae0
description "Aggregated interface to AMS-IX Peering LAN";
hold-time up 1200 down 1200;
aggregated-ether-options {
minimum-links 1;
link-speed 10g;
}
unit 0 {
description "Aggregated interface to AMS-IX Peering LAN";
bandwidth 20g;
family inet {
address 80.249.20x.y/21;
}
}
[edit]
arien@router# show interfaces ae0
description "Aggregated interface to AMS-IX Peering LAN";
hold-time up 1200 down 1200;
aggregated-ether-options {
minimum-links 1;
link-speed 10g;
}
unit 0 {
description "Aggregated interface to AMS-IX Peering LAN";
bandwidth 20g;
family inet {
address 80.249.20x.y/21;
}
}
The configured MTU should be 1514 (this includes Ethernet headers but not the FCS), or 1518 when tagged.
The configured MTU should be 1514 (this includes Ethernet headers but not the FCS), or 1518 when tagged.
Recommendations we received for Arista routers.
Recommendations we received for Arista routers.
Configure the interface facing the Peering LAN as a routed port, disable IPv6 router advertisements and disable LLDP:
Configure the interface facing the Peering LAN as a routed port, disable IPv6 router advertisements and disable LLDP:
interface Ethernet1
description AMS-IX
no switchport
ip address ...
ipv6 address ...
ipv6 nd ra disabled
no lldp transmit
interface Ethernet1
description AMS-IX
no switchport
ip address ...
ipv6 address ...
ipv6 nd ra disabled
no lldp transmit
If you do decide to configure the port as a switched port with a VLAN-interface, make sure STP is disabled:
If you do decide to configure the port as a switched port with a VLAN-interface, make sure STP is disabled:
router(config-if-Vl1)#no spanning-tree
router(config-if-Vl1)#no spanning-tree
To ignore short link-flaps, configure the link-debounce setting:
To ignore short link-flaps, configure the link-debounce setting:
router(config-if-Et1)#link-debounce time 1200
router(config-if-Et1)#link-debounce time 1200
To create an LACP-bundle, configure the ports in a channel-group. This will create a virtual port-channel interface on which you configure the Peering LAN IP address and other settings:
To create an LACP-bundle, configure the ports in a channel-group. This will create a virtual port-channel interface on which you configure the Peering LAN IP address and other settings:
interface Ethernet1
description AMS-IX port 1
channel-group 1 mode active
interface Ethernet2
description AMS-IX port 2
channel-group 1 mode active
interface Port-Channel1
description AMS-IX
ip address ...
ipv6 address ...
interface Ethernet1
description AMS-IX port 1
channel-group 1 mode active
interface Ethernet2
description AMS-IX port 2
channel-group 1 mode active
interface Port-Channel1
description AMS-IX
ip address ...
ipv6 address ...
The default ARP timeout on Arista is 4 hours, which is acceptable for the Peering LAN. Should you wish to change it, you can do so as follows:
The default ARP timeout on Arista is 4 hours, which is acceptable for the Peering LAN. Should you wish to change it, you can do so as follows:
router(config-if-Et1)#arp aging timeout
router(config-if-Et1)#arp aging timeout
We are not aware of any major issues with Linux boxes used as routers, and they seem to be pretty rare on the Exchange. Having said that, there are a few parameters that can (and usually should) be tuned:
For more information on tuning your Linux system for routing, see the Linux Advanced Routing & Traffic Control HOWTO. NOTE: Please be aware while configuring sysctl parameters, that interface specific entries override global ones. For instance, proxy-arp will be enabled (which is undesirable) if both of these are set:
We are not aware of any major issues with Linux boxes used as routers, and they seem to be pretty rare on the Exchange. Having said that, there are a few parameters that can (and usually should) be tuned:
For more information on tuning your Linux system for routing, see the Linux Advanced Routing & Traffic Control HOWTO . NOTE: Please be aware while configuring sysctl parameters, that interface specific entries override global ones. For instance, proxy-arp will be enabled (which is undesirable) if both of these are set:
net.ipv4.conf.eth0.proxy_arp = 1
net.ipv4.conf.eth0.proxy_arp = 1
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.proxy_arp = 0
The Linux approach to IP addresses is that they belong to the system, not any single interface. As a result, Linux hosts have a default behaviour that is different from most other systems: interfaces semi-promiscuously answer for all IP addresses of all other interfaces. Example:
The Linux approach to IP addresses is that they belong to the system, not any single interface. As a result, Linux hosts have a default behaviour that is different from most other systems: interfaces semi-promiscuously answer for all IP addresses of all other interfaces. Example:
In this example, host tuxco is a Linux box with a peering connection on eth0 (192.168.1.1/24) and a backbone link on eth1 (10.0.0.1/24).When host kannix (192.168.1.2) sends an ARP query for 10.0.0.1 it will get a reply from tuxco's eth0 interface!
In other words, a Linux host will answer to ARP queries coming in on any interface if the queried address is configured on any of its interfaces. The idea behind this is that an IP address belongs to the system, not just to a single interface. Although this may work well for server or desktop systems, it is not desirable behaviour in a router system. One reason is that it is a limited version of proxy-arp, which is forbidden on the AMS-IX peering LAN. Another reason is that two separate routers could potentially answer ARP queries for the same RFC1918 address.
12.1.1 Fixing ARP
The ARP behaviour can be fixed by using arp_ignore and arp_announce on the WAN interface:
In this example, host tuxco is a Linux box with a peering connection on eth0 (192.168.1.1/24) and a backbone link on eth1 (10.0.0.1/24).When host kannix (192.168.1.2) sends an ARP query for 10.0.0.1 it will get a reply from tuxco's eth0 interface!
In other words, a Linux host will answer to ARP queries coming in on any interface if the queried address is configured on any of its interfaces. The idea behind this is that an IP address belongs to the system, not just to a single interface. Although this may work well for server or desktop systems, it is not desirable behaviour in a router system. One reason is that it is a limited version of proxy-arp, which is forbidden on the AMS-IX peering LAN. Another reason is that two separate routers could potentially answer ARP queries for the same RFC1918 address.
11.1.1 Fixing ARP
The ARP behaviour can be fixed by using arp_ignore and arp_announce on the WAN interface:
tuxco# sysctl -w net.ipv4.conf.
ifname
.arp_ignore=1
tuxco# sysctl -w net.ipv4.conf.
ifname
.arp_ignore=1
tuxco# sysctl -w net.ipv4.conf.
ifname
.arp_announce=1
tuxco# sysctl -w net.ipv4.conf.
ifname
.arp_announce=1
11.1.2 Multiple Interfaces on One Subnet
If you have multiple interfaces on the same subnet, you may also want to enable arp_filter:
This prevents the ARP entry for an interface to fluctuate between two or more MAC addresses. However, you need to use source routing to make this work correctly. From the Documentation/networking/ip-sysctl-2.6.txt file in the kernel source:
[...]
arp_filter - BOOLEAN
1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP'd IP out that interface (therefore you must use source based routing for this to work). In other words it allows control of which cards (usually 1) will respond to an arp request.
[...]
12.1.2 Multiple Interfaces on One Subnet
If you have multiple interfaces on the same subnet, you may also want to enable arp_filter:
This prevents the ARP entry for an interface to fluctuate between two or more MAC addresses. However, you need to use source routing to make this work correctly. From the Documentation/networking/ip-sysctl-2.6.txt file in the kernel source:
[...]
arp_filter - BOOLEAN
1 - Allows you to have multiple network interfaces on the same subnet, and have the ARPs for each interface be answered based on whether or not the kernel would route a packet from the ARP'd IP out that interface (therefore you must use source based routing for this to work). In other words it allows control of which cards (usually 1) will respond to an arp request.
[...]
The ARP cache timeout on Linux-based routers should be changed from the default, especially if you have a large number of peers. This parameter can be tuned by setting the appropriate procfs variable through the*sysctl* interface. The Linux arp(7) manual says:
[...]
SYSCTLS
ARP supports a sysctl interface to configure parameters on a global or per-interface basis. The sysctls can be accessed by reading or writing the /proc/sys/net/ipv4/neigh/*/* files or with the *sysctl*(2) interface. Each interface in the system has its own directory in /proc/sys/net/ipv4/neigh/. The setting in the default directory is used for all newly created devices. Unless otherwise specified time related sysctls are specified in seconds.
[...]
base_reachable_time
Once a neighbour has been found, the entry is considered to be valid for at least a random value between base_reachable_time/2 and 3*base_reachable_time/2. An entry's validity will be extended if it receives positive feedback from higher level protocols. Defaults to 30 seconds.
This means that Linux systems keep ARP entries in their cache for some time between 15 and 45 seconds (and yes, the average works out to 3 seconds). This is not very high. In fact, it is lower than the typical BGP keepalive interval and may thus result in excessive ARPs.
We suggest a timeout of at least two hours for ARP entries on your AMS-IX interface, so you'd have to set the base_reachable_time to 2 x 2hrs = 4 hours.
The ARP cache timeout on Linux-based routers should be changed from the default, especially if you have a large number of peers. This parameter can be tuned by setting the appropriate procfs variable through the*sysctl* interface. The Linux arp(7) manual says:
[...]
SYSCTLS
ARP supports a sysctl interface to configure parameters on a global or per-interface basis. The sysctls can be accessed by reading or writing the /proc/sys/net/ipv4/neigh/*/* files or with the *sysctl*(2) interface. Each interface in the system has its own directory in /proc/sys/net/ipv4/neigh/. The setting in the default directory is used for all newly created devices. Unless otherwise specified time related sysctls are specified in seconds.
[...]
base_reachable_time
Once a neighbour has been found, the entry is considered to be valid for at least a random value between base_reachable_time/2 and 3*base_reachable_time/2. An entry's validity will be extended if it receives positive feedback from higher level protocols. Defaults to 30 seconds.
This means that Linux systems keep ARP entries in their cache for some time between 15 and 45 seconds (and yes, the average works out to 3 seconds). This is not very high. In fact, it is lower than the typical BGP keepalive interval and may thus result in excessive ARPs.
We suggest a timeout of at least two hours for ARP entries on your AMS-IX interface, so you'd have to set the base_reachable_time to 2 x 2hrs = 4 hours.
tuxco1# sysctl net.ipv4.neigh.ifname.base_reachable_time
net.ipv4.neigh.ifname.base_reachable_time = 30
tuxco1# sysctl net.ipv4.neigh.ifname.base_reachable_time
net.ipv4.neigh.ifname.base_reachable_time = 30
The above command tells you that the ARP cache timeout is 30 seconds average. To change it so it's between 2 and 6 hours, use the following command:
The above command tells you that the ARP cache timeout is 30 seconds average. To change it so it's between 2 and 6 hours, use the following command:
tuxco1# sysctl -w net.ipv4.neigh.ifname.base_reachable_time=14400
net.ipv4.neigh.ifname.base_reachable_time = 14400
tuxco1# sysctl -w net.ipv4.neigh.ifname.base_reachable_time=14400
net.ipv4.neigh.ifname.base_reachable_time = 14400
Here ifname is the name of the interface that connects to AMS-IX. You can also use "default" here, but that may have undesired side-effects for your other interfaces.
Here ifname is the name of the interface that connects to AMS-IX. You can also use "default" here, but that may have undesired side-effects for your other interfaces.
As with the IPv4 ARP cache, Linux systems tend to set the lifetime of the IPv6 neighbor cache quite short as well. The lifetime is controlled in a similar way as for IPv4 ARP.
As with the IPv4 ARP cache, Linux systems tend to set the lifetime of the IPv6 neighbor cache quite short as well. The lifetime is controlled in a similar way as for IPv4 ARP.
Disable proxy-arp using sysctl:
Disable proxy-arp using sysctl:
sysctl -w net.ipv4.conf..proxy_arp =
sysctl -w net.ipv4.conf..proxy_arp =
router# sysctl -w net.ipv4.conf.
ifname
.proxy_arp=0
router# sysctl -w net.ipv4.conf.
ifname
.proxy_arp=0
IPv6 stateless autoconfiguration must be disabled:
IPv6 stateless autoconfiguration must be disabled:
router# sysctl -w net.ipv6.conf.
ifname
.autoconf=0net.ipv6.conf.ifname.autoconf = 0
router# sysctl -w net.ipv6.conf.
ifname
.autoconf=0net.ipv6.conf.ifname.autoconf = 0
You may need to turn off the Reverse Path Filter (rp_filter) functionality on a Linux-based router to allow asymmetric routing, particularly on your WAN interface.To disable the RP filter:
You may need to turn off the Reverse Path Filter (rp_filter) functionality on a Linux-based router to allow asymmetric routing, particularly on your WAN interface.To disable the RP filter:
tuxco1# sysctl -w net.ipv4.conf.
ifname
.rp_filter=0net.ipv4.conf.ifname.rp_filter = 0
tuxco1# sysctl -w net.ipv4.conf.
ifname
.rp_filter=0net.ipv4.conf.ifname.rp_filter = 0
The various system parameters discussed above can be set at boot time by adding it to a file such as /etc/sysctl.conf. The exact name, location and very existence of this file typically depends on the Linux distribution in use, but both Debian and Red Hat/Fedora use /etc/sysctl.conf:
The various system parameters discussed above can be set at boot time by adding it to a file such as /etc/sysctl.conf. The exact name, location and very existence of this file typically depends on the Linux distribution in use, but both Debian and Red Hat/Fedora use /etc/sysctl.conf:
# file: /etc/sysctl.conf
# These settings should be duplicated for all interfaces that are
# on a peering LAN.
### Typical stuff you really want on a router
# Fix the "promiscuous ARP" thing...
net.ipv4.conf.ifname.arp_ignore=1
net.ipv4.conf.ifname.arp_announce=1
# Turn off RP filtering to allow asymmetric routing:
net/ipv4/conf/ifname/rp_filter=0
# Multiple (non-aggregated) interfaces on the same peering LAN.
# READ THE MANUAL FIRST!
#net.ipv4.conf.ifname.arp_filter=1
### Keep the AMS-IX ARP Police happy. :-)
net.ipv4.neigh.ifname.base_reachable_time=14400
net.ipv6.neigh.ifname.base_reachable_time=14400
# file: /etc/sysctl.conf
# These settings should be duplicated for all interfaces that are
# on a peering LAN.
### Typical stuff you really want on a router
# Fix the "promiscuous ARP" thing...
net.ipv4.conf.ifname.arp_ignore=1
net.ipv4.conf.ifname.arp_announce=1
# Turn off RP filtering to allow asymmetric routing:
net/ipv4/conf/ifname/rp_filter=0
# Multiple (non-aggregated) interfaces on the same peering LAN.
# READ THE MANUAL FIRST!
#net.ipv4.conf.ifname.arp_filter=1
### Keep the AMS-IX ARP Police happy. :-)
net.ipv4.neigh.ifname.base_reachable_time=14400
net.ipv6.neigh.ifname.base_reachable_time=14400
CAUTION: Modules must be loaded before sysctl is executed
On Debian systems, kernel modules for some network interfaces (e.g. 10GE cards) are not loaded before the init process executes the script thatruns the sysctl commands. In those cases, it is necessary to force the module to be loaded earlier. The same goes for the IPv6 settings; the ipv6 module is usually not loaded until the network interfaces are brought up, which is typically after the sysctl variables are set by the procps.sh script.(On Red Hat/Fedora systems no action needs to be taken; the /etc/init.d/network script automatically (re-)sets the sysctl variables before and after bringing up the interfaces.)There are a few ways around this:
On Debian-based systems, this can be done by creating a symbolic link in /etc/rc2.d to re-run procps.sh after the network is brought up:
CAUTION: Modules must be loaded before sysctl is executed
On Debian systems, kernel modules for some network interfaces (e.g. 10GE cards) are not loaded before the init process executes the script thatruns the sysctl commands. In those cases, it is necessary to force the module to be loaded earlier. The same goes for the IPv6 settings; the ipv6 module is usually not loaded until the network interfaces are brought up, which is typically after the sysctl variables are set by the procps.sh script.(On Red Hat/Fedora systems no action needs to be taken; the /etc/init.d/network script automatically (re-)sets the sysctl variables before and after bringing up the interfaces.)There are a few ways around this:
On Debian-based systems, this can be done by creating a symbolic link in /etc/rc2.d to re-run procps.sh after the network is brought up:
root@tuxco# ln -s ../init.d/procps.sh /etc/rc2.d/S20procps.sh
root@tuxco# ln -s ../init.d/procps.sh /etc/rc2.d/S20procps.sh
Enable bonding driver support in the kernel (CONFIG_BONDING=m)Edit /etc/modules to load the bonding driver on boot:
Enable bonding driver support in the kernel (CONFIG_BONDING=m)Edit /etc/modules to load the bonding driver on boot:
bonding miimon=100
bonding miimon=100
The miimon parameter specifies the frequency for link-monitoring, measured in ms.Install the ifenslave package (apt-get install ifenslave). This package provides the /sbin/ifenslave tool, which is used to attach physical interfaces to the bonding interface.Add the bonding interface to /etc/network/interfaces:
The miimon parameter specifies the frequency for link-monitoring, measured in ms.Install the ifenslave package (apt-get install ifenslave). This package provides the /sbin/ifenslave tool, which is used to attach physical interfaces to the bonding interface.Add the bonding interface to /etc/network/interfaces:
# Ams-IX side
auto bond0
iface bond0 inet static
address 80.249.20x.y
netmask 255.255.248.0
post-up /sbin/ifenslave bond0 eth0 eth1
# Ams-IX side
auto bond0
iface bond0 inet static
address 80.249.20x.y
netmask 255.255.248.0
post-up /sbin/ifenslave bond0 eth0 eth1
The above example creates a bonding interface with two physical interfaces.For more information see the file Documentation/networking/bonding.txt in the kernel source tree.
The above example creates a bonding interface with two physical interfaces.For more information see the file Documentation/networking/bonding.txt in the kernel source tree.
Modern kernels have MLDv2 on by default and there is no sysctl parameter to switch it off. The only known way by now is to drop it with an outgoing filter:
Modern kernels have MLDv2 on by default and there is no sysctl parameter to switch it off. The only known way by now is to drop it with an outgoing filter:
ip6tables -A OUTPUT -p icmpv6 --icmpv6-type 143 -j DROP
ip6tables-save
ip6tables -A OUTPUT -p icmpv6 --icmpv6-type 143 -j DROP
ip6tables-save
Miktrotik hints are listed below.
Miktrotik hints are listed below.
/interface/ set sfp28-1 name="AMS-IX_P1"
/interface/ set sfp28-1 name="AMS-IX_P1"
/interface/bonding/ add name=LAG2 mode=802.3ad mtu=1514 slaves=AMS-IX_P1 comment=LAGtoAMS-IX
/interface/bonding/ add name=LAG2 mode=802.3ad mtu=1514 slaves=AMS-IX_P1 comment=LAGtoAMS-IX
/ip/address/ add address=[Peeering-IP/Mask] interface=LAG2 comment=Peering
/ip/address/ add address=[Peeering-IP/Mask] interface=LAG2 comment=Peering
By default Mikrotik routers have their own proprietary Mikrotik Discovery Protocol and CDP enabled. To turn these discovery protocols off, in the Web UI go to IP > Neighbors > Discovery Interfaces and disable the protocols on the AMS-IX-facing interface.
By default Mikrotik routers have their own proprietary Mikrotik Discovery Protocol and CDP enabled. To turn these discovery protocols off, in the Web UI go to IP > Neighbors > Discovery Interfaces and disable the protocols on the AMS-IX-facing interface.
/interface bridge port print
/interface bridge port print
/ip/neighbor/discovery-settings/ set lldp-med-net-policy-vlan=disabled protocol=lldp,cdp,mndp discover-interface-list=static mode=rx-only
/ip/neighbor/discovery-settings/ set lldp-med-net-policy-vlan=disabled protocol=lldp,cdp,mndp discover-interface-list=static mode=rx-only
/routing/ospf/instance/ print
/routing/ospf/instance/ print
/routing/rip/interface/ print
/routing/rip/interface/ print
/interface bridge print detail
/interface bridge print detail
/ip/dhcp-server/ print
/ip/dhcp-client/ print
/ip/dhcp-relay/ print
/ip/dhcp-server/ print
/ip/dhcp-client/ print
/ip/dhcp-relay/ print
/interface/bonding/ set LAG2 arp=proxy-arp disabled=yes
/interface/bonding/ set LAG2 arp=proxy-arp disabled=yes
ipv6/nd/ print
ipv6/nd/ set interface=LAG2 disabled=yes
ipv6/nd/ print
ipv6/nd/ set interface=LAG2 disabled=yes
The below CLI configuration details the basic BGP configuration to establish an IPv4 BGP session towards one of our Route Servers. The customer can use a similar approach for IPv6:
The below CLI configuration details the basic BGP configuration to establish an IPv4 BGP session towards one of our Route Servers. The customer can use a similar approach for IPv6:
/routing/bgp/connection/ set RS1-IPv4 remote.as=6777 remote.address=80.249.208.255 local.address=80.249.210.202 vrf=ISP501 routing-table=ISP501 local.role=ebgp output.filter-chain=to-rs-no-export
/routing/bgp/connection/ set RS1-IPv4 remote.as=6777 remote.address=80.249.208.255 local.address=80.249.210.202 vrf=ISP501 routing-table=ISP501 local.role=ebgp output.filter-chain=to-rs-no-export
To configure link aggregation on Redback SMS routers you need to do the following.
To configure link aggregation on Redback SMS routers you need to do the following.
!Create the link group interface and assign an IP address to it
[local]Redback(config)#context local
[local]Redback(config-ctx)#interface AMS-IX
[local]Redback(config-if)#ip address 80.249.20x.x/21
[local]Redback(config-if)#exit
!Create the link group and bind it to its interface
[local]Redback(config)#link-group AMS-IX ether
[local]Redback(config-link-group)#bind interface AMS-IX local
!Configure an ethernet port and add it to the link group
[local]Redback(config-config)#port ethernet 1/1
[local]Redback(config-port)#no shutdown
[local]Redback(config-port)#link-group AMS-IX
[local]Redback(config-port)#exit
!Configure another ethernet port and add it to the link group
[local]Redback(config-config)#port ethernet 1/2
[local]Redback(config-port)#no shutdown
[local]Redback(config-port)#link-group AMS-IX
[local]Redback(config-port)#exit
!To match the AMS-IX arp timeout (4 hours) you need to configure this
under the interface
[local]Redback(config)#context local
[local]Redback(config-ctx)#int AMS-IX
[local]Redback(config-if)#ip arp timeout 14400
[local]Redback(config-port)#exit
!Also, you can set your aggregated min-links to a value that will
cause the bundle to drop in the event that your links can no longer
support the amount of traffic you move trough the link-group.Thus, 2-
port aggregated link, pushing 1.2 Gbps sustained across, drop bundle
if n == 1;
[local]Redback(config)#link-group AMS-IX ether
[local]Redback(config-link-group)#minimum-links 2
[local]Redback(config-link-group)#exit
!Create the link group interface and assign an IP address to it
[local]Redback(config)#context local
[local]Redback(config-ctx)#interface AMS-IX
[local]Redback(config-if)#ip address 80.249.20x.x/21
[local]Redback(config-if)#exit
!Create the link group and bind it to its interface
[local]Redback(config)#link-group AMS-IX ether
[local]Redback(config-link-group)#bind interface AMS-IX local
!Configure an ethernet port and add it to the link group
[local]Redback(config-config)#port ethernet 1/1
[local]Redback(config-port)#no shutdown
[local]Redback(config-port)#link-group AMS-IX
[local]Redback(config-port)#exit
!Configure another ethernet port and add it to the link group
[local]Redback(config-config)#port ethernet 1/2
[local]Redback(config-port)#no shutdown
[local]Redback(config-port)#link-group AMS-IX
[local]Redback(config-port)#exit
!To match the AMS-IX arp timeout (4 hours) you need to configure this
under the interface
[local]Redback(config)#context local
[local]Redback(config-ctx)#int AMS-IX
[local]Redback(config-if)#ip arp timeout 14400
[local]Redback(config-port)#exit
!Also, you can set your aggregated min-links to a value that will
cause the bundle to drop in the event that your links can no longer
support the amount of traffic you move trough the link-group.Thus, 2-
port aggregated link, pushing 1.2 Gbps sustained across, drop bundle
if n == 1;
[local]Redback(config)#link-group AMS-IX ether
[local]Redback(config-link-group)#minimum-links 2
[local]Redback(config-link-group)#exit
On Riverstone equipment, proxy ARP seems to be enabled by default, so you will need to disable it:
On Riverstone equipment, proxy ARP seems to be enabled by default, so you will need to disable it:
ip disable proxy-arp interface ifname
ip disable proxy-arp interface ifname
Here, ifname refers to your interface towards AMS-IX, or the string 'all'
Here, ifname refers to your interface towards AMS-IX, or the string 'all'
Various people contributed to this document. We received configuration info from:
Thanks to all those who contributed.
Various people contributed to this document. We received configuration info from:
Thanks to all those who contributed.
© 2024 - Amsterdam Internet Exchange Responsible Disclosure Policy Mailing list code of conduct Privacy Policy Email Disclaimer Cookie policy
Trade register: 34128666