AMS-IX currently offers link aggregation on any 1G/10G/100G physical connection. Aggregating links acquired from different partners or resellers, however, is not supported. The port prices for aggregated links are identical to the normal port prices.

Due to technical limitations of the switches used by AMS-IX it may be necessary to relocate your existing port. If this turns out to be the case, AMS-IX will inform you and advise you of any additional steps necessary for this process.

AMS-IX can deliver aggregated links at all co-locations. Although a strict reading of the spec forbids it, we can offer aggregated links over different media types of the same speed.

The load-balancing algorithms used in our NetIron MLX switches uses a modulo operation, leading to the best distribution over links with the entropy available in source and destination IPv4 or IPv6 address, TCP or UDP source and destination port number, as applicable.

LACP is supported at AMS-IX for all connection types. When used with 10Gbit/s and 100Gbit/s links (where available) use of LACP does introduce an inconvenient side effect. After a topology failover ports with LACP enabled will stay in blocking mode until the first LACP frame is received. Because this may take several tens of second (depending on vendor implementation) this can cause BGP sessions to flap.

When enabling LACP we advise to configure the LACP timeout to short, to limit the maximum failover time to 3 seconds.

We have collected information about link aggregation for several router platforms in our configuration guide.

A quarantine VLAN is a VPLS instance on the AMS-IX switch containing the following:

• (new) customer ports
• AMS-IX monitoring system

The monitoring system sniffs all broadcast, multicast and unknown unicast in the quarantine VLAN.

AMS-IX defines a fairly strict set of allowed traffic types on the peering LANs. Not all routers (and intermediate L2 devices) adhere to these guidelines; they typically have various protocols turned on by default such as CDP, EDP, STP, DEC MOP, etc., or they present more than one MAC address to the platform. These misbehaving/misconfigured devices potentially endanger the stability of the peers and/or switching platform. Hence, we cannot allow them on the peering LANs.Rather than act reactively once a customer port is in production, we prefer to detect and fix these issues beforehand. Therefore, we introduced the concept of a quarantine VLAN. Once a customers router is connected and the port is up, we can quickly see if it adheres to the rules. If not, the violating traffic does not harm the rest of the platform.

New ports are always put into a quarantine VLAN first. This also is the case for upgrades, downgrades and relocations.

In addition to the above, existing customer ports may be moved into quarantine VLAN if they violate the allowed traffic types. Please note that this is only done in extreme cases.

If your port is moved into Quarantine the AMS-IX NOC will notify you for this. If the reason is because you are sending illegal traffic, your configuration should be updated accordingly. Once you are confident the port adheres to the rules please contact the AMS-IX NOC and request the port be put back in production. The NOC will check the port's behaviour again. If all is fine, the port will be moved (back) into production. If not, we will notify you with details of the problem.

sFlow is a standard to capture traffic data in switched or routed networks. It uses a sampling technology to collect statistics from the device and is for that reason applicable to gigabit speeds or higher. Due to it being an open standard, described in RFC 3176, it is implemented on a wide range of devices, like the AMS-IX Brocade/Extreme switches.

The sFlow agent (Brocade/Extreme switch) supports two forms of operation:

• time-based sampling of counters
• packet-based sampling of ethernet frames

The counter samples provide exactly the same information AMS-IX uses for its traffic statistics now, therefore the sFlow implementation at AMS-IX makes use of the packet-based samples (called flowsamples) to provide additional analysis of the exchanged traffic.

Based on a defined sampling rate, one out of N frames from the incoming traffic for each interface gets sampled and sent to a central server which is statistically analyzing the traffic. If we see one packet out of N, we assume that all the N-1 packets we haven't seen are the same type and size.

Without sampling technology, packet analysis on a network with a throughput like AMS-IX would not be possible. For more detailed information about the accuracy of packed based sampling see the documents on the official sFlow website.

The sFlow samples on the server get analysed by software developed at AMS-IX. The software package is written in PERL and based on the sFlow decoding module Net::sFlow.

The IPv6 set-up on the AMS-IX ISP peering LAN is as follows:

• The prefix in use is: 2001:7f8:1::/64
• The prefix is sourced from AS1200.
• The prefix is tagged with the no-export community and should not be announced outside of your AS.

The suffix (‘allocation’) scheme for 16 bits ASNs is as follows:

A50 x:xxxx:n

The suffix (‘allocation’) scheme for 32 bits ASNs is as follows:

A500 :xx:xxxx:n

Here, "x:xxxx" or “xx:xxxx” is your (zero-padded) AS number in decimals and “n” is a serial number depending on the number of interfaces you are using for IPv6 peerings (starting from 1 for the first interface, 2 for the second interface and so on).

Examples:

AMS-IX uses AS1200 (zero-padded: 001200), so its IPv6 peering addresses are:

2001:7f8:1::A500:1200:1/64
2001:7f8:1::A500:1200:2/64
…

A member with a 16-bit ASN of 64523 would use:

2001:7f8:1::A506:4523:1/64
2001:7f8:1::A506:4523:2/64
...

A member with a 32-bit ASN of 195000 would use:

2001:7f8:1::A500:19:5000:1/64
2001:7f8:1::A500:19:5000:2/64
...

The IPv6 set-up on the AMS-IX GRX peering LAN is as follows:

• The prefix in use is: 2001:7f8:86:1::/64
• The prefix is sourced from AS64523.
• The prefix is tagged with the no-export community and should not be announced outside of your AS.

The suffix (‘allocation’) scheme for 16 bits ASNs is as follows:

A50 x:xxxx:n

The suffix (‘allocation’) scheme for 32 bits ASNs is as follows:

A500 :xx:xxxx:n

Here, "x:xxxx" or “xx:xxxx” is your (zero-padded) AS number in decimals and “n” is a serial number depending on the number of interfaces you are using for IPv6 peerings (starting from 1 for the first interface, 2 for the second interface and so on).

Examples:

AMS-IX uses AS1200 (zero-padded: 001200), so its IPv6 peering addresses are:

2001:7f8:86:1::A500:1200:1/64
2001:7f8:86:1::A500:1200:2/64
…

A member with a 16-bit ASN of 64523 would use:

2001:7f8:86:1::A506:4523:1/64
2001:7f8:86:1::A506:4523:2/64
...

A member with a 32-bit ASN of 195000 would use:

2001:7f8:86:1:A500:19:5000:1/64
2001:7f8:86:1:A500:19:5000:2/64
...

Relevant IRR objects for peers that hold BGP sessions with AS1200 border routers:

• AS1200:AS-AMS-IX-PEERS (List of networks peering directly with AS1200)

ARP (Address Resolution Protocol) is the Layer-2 protocol used by AMS-IX member's router to associate IPv4 address with the MAC address of peers interfaces. Learn more about ARP here.

On Ethernet networks, the Address Resolution Protocol (ARP) is used to find the MAC-address for a given IPv4 address. ARP uses Ethertype 0x0806 together with Ethernet broadcasting. A node will broadcast an ARP Request packet to ask for the MAC address of an unknown IPv4 address. The node using the requested IP address replies (using regular unicast) with an ARP Reply packet, which includes its MAC address. In order to work, it is important that all nodes using IPv4 listen for ARP packets and reply to them if necessary.

The nodes therefore need to process all Ethernet broadcast messages with Ethertype 0x0806. For each ARP packet, they must decide whether or not to reply. Processing ARP packets can take a lot of processing power. Because all ARP packets need to be examined in order for ARP to work, processing ARP packets may take precedence over other activities, depending on the Operating System. As such, when there is a lot of ARP traffic, routers may be unable to do other processing tasks like maintaining BGP sessions.

This problem was noticed on AMS-IX when the ISP peering LAN was renumbered to new IPv4 addresses. Members in the new IPv4 range were trying to reach members in the old IPv4range and vice versa. Larger amounts of ARP packets than usual crossed the network, consuming all available processing power on some customer routers, not leaving enough resources to process BGP in a timely manner, resulting in lost BGP sessions. Also, routers trying to re-establish old BGP sessions started sending ARP packets, resulting in an ARP storm that caused even more problems on customer equipment.

To help routers survive heavy ARP traffic, AMS-IX tries to keep the amount of ARP traffic to a minimum. For this purpose, AMS-IX developed a daemon, written in Perl, called ARP Sponge. The ARP Sponge daemon listens on the ISP peering LAN for ARP traffic. When the number of ARP Requests for a certain IP address exceeds a threshold, the ARP Sponge sends out an ARP Reply for that IP address using its own MAC address. From that moment, the IP address is sponged: all traffic to that node is sent to the ARP Sponge. This prevents ARP storms because it keeps the amount of ARP traffic limited.

When the interface of a sponged IP address comes up again, it generally sends out a gratuitous ARP request packet. This is an ARP packet with both source and destination IP address set to the IP address of the node sending the packet. It is used mostly in case the MAC-address changed, so that other nodes can update their ARP caches. When the ARP Sponge receives any traffic from a sponged IP address (including but not limited to gratuitous ARP requests, ARP requests for other nodes, BGP peering initiations, etc.), it ceases sponging the IP address, thus no longer sending out ARP replies for that IP address.

Current AMS-IX ARP Sponge MAC address is : c6:17:44:26:e8:87

* Unable to exchange traffic with AMS-IX peers when IPv4 address comes up after being inactive for a period of time

If a IPv4 is sponged, it means that in the members ARP tables, the ARP entry for this IP is registered with the ARP sponge MAC address. After the IPv4 is again reachable again and being "un-sponged", the ARP table of peers might not be updated fast enough with the customer’s MAC address, result in traffic from these peers toward the recovered IP still being forward to the sponged MAC address.

For instance, if the IP 80.249.208.1 of member A with MAC address AAAA.AAAA.AAA is sponged with the sponged MAC address EEEE.EEEE.EEEE, then member B ARP entry for the address will be 80.249.208.1 - EEEE.EEEE.EEE. After the member A recovers, it send traffics toward member B, but member B ARP entry is not yet updated with the original address AAAA.AAAA.AAAA, then traffic will be ended up sending to EEEE.EEEE.EEEE, until member B updates the ARP entry.

This issue should be automatically resolved after a certain period of time, after the daemon stop replying to ARP reply for this IP and let the un-sponged IP and peers update ARP entries themselves.

The issue is more noticeable with members that only have peering sessions with the AMS-IX route-servers. If members do not have peering sessions with route-servers, BGP sessions with peers must be brought up one-by-one and ARP entries are sure to be updated through the BGP initialisation process. Subsequently, traffic will be properly forwarded and received from each peers. However, if the newly "un-sponged" member only has peering sessions with route-servers, and after recovery establishes BGP sessions and receives AMS-IX peers prefixes from there, there could be a case that traffic is forwarded to the next hop IP of peers that still have the spoofed ARP entries.

Therefore, AMS-IX NOC recommend members that have their IPv4 address being unreachable for prolonged period of time (so it is certainly sponged), to temporary shutdown peering with route-servers and send gratuitous ARP request to update peer's ARP tables.