I just saw a blog post today that trumpets improvements that Linode made to their datacenter networks. It looks pretty good, bandwidth is improved, etc. I would never know any better, except that a month ago I was at a presentation by Yihua He of Yahoo. And, for a layman like me, it seemed that he was quite a bit ahead.
Link-wise, the He-style Yahoo networking is very similar to Linode's Nextgen. It uses a so-called "leaf-spine" architecture with Top-of-Rack (ToR) switches being the leaves. Linode's post did not articulate clearly the advantages that He architure brings: increased bisection that accomodates growing east-west traffic and reduction in oversubscription, but I presume Linode customers will reap improvements in that area too. However, there's one curious difference: Linode allowed a vendor to provide the balancing that leaf-spine requires, whereas He went with a COTS gear.
He accomplishes this by using routers as spines where Linode uses Cisco Nexus 5000. As it turns out, using iBGP works in place of whatever proprietary magic Cisco employs. He simply gives each ToR a private ASN.
Once you go with COTS gear, you are free from the yoke of Cisco. You can mix and match as you with, build datacenters incrementally, and run any kind of monitoring apps you like. And the whole thing is much, much cheaper.
To be sure, there are issues as well. For example, depending on how magical Cisco is, you might be able to migrate VMs at will throughout the datacenter. In case of He, you can't. Although, using OpenStack with floating IPs probably accomplishes a useful result in such a case anyway.
But in the end, if you can use an open solution, is it ever worth using the vendor solution? And I definitely didn't expect Linode and Yahoo feature in this ironic role reversal. Which one of them is the scrappy startup and which one is a bureaucratic monster courted by lock-in vendors?