15 September 2016

Next Generation Switch Hardware for the AMS-IX Platform

Blogpost-image_paper-3

Because of the continuing growth of traffic on our platform, AMS-IX always needs to be prepared for the future. Therefore, we continuously look into possibilities to further increase our growth capabilities in line with our high standards and requirements. Currently, our most pressing requirement is to scale up the (100GE) port density of the switches that make up the AMS-IX platform. We have seen that the connected customer capacity to the platform on 100GE exceeds that of 10GE and we foresee further growth in this. Next to plain port density, a strong decrease in power usage and cost is a requirement. Compared to the original 100GE implementations based on SFP, a reduction by a factor of 7 is realizable for every 100GE port by the use of 100GE QSFP28 optics.

The solution as well as upgrade of the core of the AMS-IX network, which starts in the November 2016 timeframe, will be provided by the Brocade SLX platform. The main reasons for having selected Brocade’s SLX9850 platform is that the specifications fit our requirements very well and could match or were better than those of other vendors. Our long and very good relationship with Brocade also contributed to this decision.

The evolution of the AMS-IX platform in the 100GE era

Back in 2012, AMS-IX installed the first 100GE customer facing interface. In the meantime we had already started upgrading the inter-switch links in the Amsterdam metro area from 10GE lags to 100GE or 100GE lags. At the time, the platform we used was the Brocade MLXe-16 or MLXe-32 and the optics used were CFP-LR4 or CFP-LR10.

Since then the evolution of the 100GE optics has gone from the CFP form factor with typical 24W of power dissipation for LR4 type to CFP2-LR4 (8W) to now QSFP28-LR4 (3.5W). A reduction in power dissipation by a factor of almost 7. In the same time the port density increased by a factor of 2 when migrating from CFP to the CFP2 form factor (using the same chassis). Now using the QSFP28 form factor port density typical increases by a factor of 8 in the same amount of physical space, or with a factor of 4 in a chassis half the size of an MLXe-32. And again in the same timeframe, pricing of a 100GE interface plus optics decreased by almost a factor of 10.

As mentioned above we needed 100GE interfaces on the IX platform to support both customer connections and the inter-switch links between the access and the core switches very early after 100GE based products entered the market. Because we struggled with the low 100GE density per chassis we implemented intermediate solutions to overcome the last two years:
• We introduced more core locations and switches to lower the load per core switch.
• We introduced more access switches to handle the increasing requirement on 100GE access ports.

Both intermediate steps introduced inefficiencies in that we used more power and space than we liked.

During this time, we looked into various products that were coming to market or were on the roadmap from different vendors. Based on our evaluation of this investigation we decided back in 2014 that we would stay with Brocade for a next generation platform. The main reasons for this being that the Brocade SLX platform specifications could match or were better than the others and very well fitted our requirements. On top of this, our very good relationship with Brocade added to this decision. From the beginning of the development of the SLX platform we discussed our requirements with Brocade. Many sessions between Brocade and AMS-IX engineers have taken place to ensure that our requirements are available in the first release of the product so we can use it from day one. AMS-IX engineers have been testing the SLX in the Brocade CAT lab in June and August 2016. We also could use an early test system in the AMS-IX labs in Amsterdam. Based on the results of these tests we will implement the first new chassis in the November timeframe.

Implementation

Our first use of the SLX9850 platform will be in the core of the network. Because of the density limitations mentioned above, three years ago we added two additional core nodes in two separate locations. Now with the SLX9850 platform we will bring this back to the original two locations where we use 2 chassis in each for redundancy. The first implementation of the SLX8950 we expect in the September to November 2016 timeframe. After further extensive testing, we will replace one of the current core switches with an SLX9850 chassis and migrate backbone links to this new switch, meanwhile load balancing traffic over the remaining inters-switch links. If after a defined period we see no issues with the new SLX9850 in production we will replace the remaining core switches after which any 100GE port density issues in the backbone is gone for a long time. This whole operation will result in a considerable operational expense reduction as we need lesser space for the core and fewer dark fibers through the metro.

In the second half of 2017 we expect some of the access switches, specifically on locations where there is a high density of 100GE customer ports, to migrate to the SLX9850 platform.

AMS-IX lab

Next to this, we have the availability of a 4 slot SLX chassis in the AMS-IX lab in Amsterdam since June. We use this for local testing and programming our management software against the new platform. The latter being necessary as not only the hardware is completely new but also the operating system environment.

Henk Steenman, CTO AMS-IX