Core Move: Flipping a bowl of spaghetti without losing the sauce

After making the decision and getting the contracts up and running it was time to sort out the connections between all the PoP locations and the new core node. This job always takes an enormous amount of time and focus as we need to make sure the connections to the two core nodes are not crossed anywhere in the network, otherwise we would end up with a single point of failure. The task to get this done right requires an enormous amount of time and effort from both our engineers as well as the potential supplier. Just like starting out with multiple suppliers for the location, we also ended up in talks with multiple suppliers for the darkfiber network and shortened the list very fast, but it took months to finally find routes that would provide the redundancy we needed and just a single supplier was able to provide this redundancy, Relined.

On top of the redundancy, Relined also made sure that they would splice every single fiber as much as possible, taking the maximum amount of connectors out of the network, which resulted in a serious drop in light losses and the need for amplification on the darkfibers. Again, less active equipment which could fail, making the network more resilient.

Once the darkfibers were ordered and the first ones started to get delivered, a next challenge started, cross-connects. Ordering the required cross-connects sounds like a simple task, but somehow this always ends up in a large amount of work and additional tasks as positions are always occupied by previous customers or they get delivered with tx/rx issues. All this caused many delays in our planning, but there was a bigger challenge just waiting to happen, which nobody could have seen coming and would cause the biggest delay.

During our challenges with getting the darkfibers and cross-connects in place, Covid-19 hit and, soon after, the lockdowns started to be put in place. It goes without saying, this had a serious effect on the migration, and nearly every part of the migration had to be delayed. Suppliers were having issues sending out engineers to perform their jobs and the increases in traffic made them busier than ever before. That same increase in traffic caused for many additional backbone upgrades for the AMS-IX engineers as well, taking away resources that were planned for the migration itself. A curfew was put in place, limiting our ability to travel at night, basically making the migration impossible and on top of that, the different datacenters added a maximum number of people allowed inside the datacenter, limiting our options even more. As you can imagine, a core switch is quite heavy and large so it takes multiple people to put it in place and cable it up. We also saw an increase in sickness leave, from people taking additional precautions when having signs of Covid and staying away from other coworkers because of it. All in all, Covid was the main driver to postpone the migration itself until a later stage.

Another challenge we ran into, was the announcement of our DCI supplier that the systems we use were no longer available for ordering. We knew it would go out of sale at some point, but due to miscommunication, we never received the news that the end of sale date was pushed forward until we did our last order. This became a challenge because we had planned to build a 4 Tbit link between the old and the new core, so we could make sure that all the PoP locations had a redundant connection to our core network at all times.

For this we increased the amount of modules we had in storage, but as traffic increased at amazing amounts, we used a lot more DCI modules to increase the backbone capacity, up to a point where we also needed to take equipment away from the additional stock. In the end, we did not have enough equipment anymore to build the link between the core nodes. This basically meant we needed to create new plans, each of them ending up in either lower redundancy during the migrations or a very much extended timeline, which would take a toll on our resources.

During the week, we kept on monitoring closely and no strange behaviour was noticed. No errors, no packet loss or flapping links. In short, the next migration would be a GO. During that same week of monitoring, the onsite engineers started to prepare for the next migrations. They reused the DCI equipment that was released from the first migration and started to prepare as many connections as possible for the next migration. This way they minimised the amount of work needed during the nights, giving us the ability to migrate multiple sites per night.

On top of this, it was already known that the next sites to be migrated would be Nikhef and Interxion Science Park. Both of which would be connected with point-to-point connections via the campus fibers, releasing an even higher amount of DCI, giving the ability to prepare even more connections for the later migrations. It was for this reason that we added one night without any migrations after we would be done with Nikhef and Interxion.

On Monday, June 14th at 00.00 AM, we started the migrations of Nikhef and Interxion Science Park. These two locations were to be connected via campus fibers, for which we arranged a total of 56 fibers and cross-connects. During the preparation and testing of these connections we came across a lot of issues and weird mappings that needed to be resolved and even though we were sure we had it all sorted out, this still created some additional worries during the migrations. Luckily, all went as expected and because of the countless hours of testing and preparations, there were no issues with the connections at all.

The amount of DCI that was released by the three migrations that were now finalised, was nearly enough to prepare all the migrations to come, and the onsite engineers working the day shift used the extra day we had in our planning to install and cable everything they could.

From this point forward, the physical parts of the nightly migrations would be a lot faster, but at the same time, any potential troubleshooting would be more sophisticated. We prepared the network in a way that only the darkfiber connected to the multiplexer at the PoP location, which was currently connected to EUNetworks, would need to be swapped by the new darkfiber going to Digital Realty. All the cabling was prepared and tested, and the new connection was ready to be used, but given it is a new darkfiber with a different length, these simple steps could easily turn out to become a headache if troubleshooting was needed.

These findings in the night were communicated with the engineers that performed the preparations during the day, and they adjusted the cabling for the other 2 locations which also had active multiplexers installed.

The 4th night of migrations went ultra smooth with just over 2 hours of work performed to migrate 4 locations, including transit times between the locations and after this night we felt good. One more night to go and to migrate the remaining 3 locations and then start our well deserved weekend after which the cleanup could start.

Unfortunately, the 5th and last night did not go as smooth. The 1st location during this night caused for many issues with modules or links that did not want to come online and it took over 4 hours to get this 1st location migrated. This meant that we still had 2 locations to do with less then 3 hours to do so and the knowledge that 1 more of these location might prove troublesome as it had some additional connections to be migrated.

Luckily the next location went ultra smooth again, leaving us with about 2,5 hours for the last migration, and this last one did give some minor tx/rx issues and an optic that did not feel like working anymore, but at 05:16 AM we received the best message in our chat, “All wdm lights here are green !!!!!”, marking the end of the migrations.