Do you know the data center network architecture?

Whether we are talking about SDN, NFV or other virtual network technologies, one thing needs to be clear. Network packets end up running on physical networks. The characteristics of the physical network, such as bandwidth, MTU, delay, etc., ultimately determine the characteristics of the virtual network directly or indirectly. It can be said that the physical network determines the “ceiling” of the virtual network.

In the performance test report of the OpenStack Neutron by Mirantis, it can be seen that the upgrade and adjustment of the network equipment, such as the use of a high-speed network card and the configuration of the MTU9000, can significantly improve the transmission efficiency of the virtual network. When optimizing network performance, some physical network features can be improved by upgrading devices or lines, but some are related to the network architecture. The risks and costs of upgrading or changing the network architecture are enormous, so the initial choice of the data center, the choice, and design of the network architecture, especially need to be cautious. On the other hand, when designing a virtual network, it is inevitable to consider the actual physical network architecture. Understanding the physical network architecture is indispensable for the final understanding of the virtual network.

Next, I will talk about my own understanding of the data center network architecture.

Traditional data center network architecture

In traditional large data centers, the network is usually a three-tier structure. Cisco calls it: a hierarchical inter-networking model. This model contains the following three layers:

• Access Layer: Sometimes called Edge Layer. Access switches are usually located at the top of the rack, so they are also known as ToR (Top of Rack) switches, which are physically connected to the server.

• Aggregation Layer: Sometimes called the Distribution Layer. The aggregation switch connects to the Access switch and provides other services such as firewall, SSL offload, intrusion detection, and network analysis.

• Core Layer: The core switch provides high-speed forwarding for packets entering and leaving the data center, providing connectivity for multiple aggregation layers. The core switch provides a flexible L3 routing network for the entire network.

A three-layer network architecture diagram is as follows:

Generally, the aggregation switch is the demarcation point between the L2 and L3 networks. The aggregation switch is below the L2 network. The above is the L3 network. Each group of aggregation switches manages a POD (Point Of Delivery), and each POD is an independent VLAN network. The server does not have to modify the IP address and default gateway to migrate within the POD because one POD corresponds to one L2 broadcast domain.

STP (Spanning Tree Protocol) is usually used between the aggregation switch and the access switch. STP makes only one aggregation layer switch available for a VLAN network, and other aggregation layer switches are used in the event of a failure (dashed line in the figure above). That is to say, the aggregation layer is an active-passive HA mode. In this way, at the aggregation layer, horizontal expansion is not possible, because even if multiple aggregation switches are added, only one is still working. Some proprietary protocols, such as Cisco’s vPC (Virtual Port Channel), can improve the utilization of the aggregation layer switch, but on the one hand, this is a proprietary protocol, on the other hand, vPC cannot really achieve full horizontal expansion. The figure below shows a convergence layer as the L2/L3 boundary line and uses the vPC network architecture.

With the development of cloud computing, computing resources are pooled. In order to make computing resources arbitrarily allocated, a large two-tier network architecture is needed. That is, the entire data center network is an L2 broadcast domain so that the server can be created and migrated anywhere, without modifying the IP address or the default gateway. The second-tier network architecture, the L2/L3 boundary is in the core switch, the core switch, that is, the entire data center is the L2 network (of course, it can contain multiple VLANs, and the VLANs are connected through the core switch for communication). The network architecture of the second and second tiers is shown below:

Although the large two-tier network architecture enables the virtual machine network to be flexibly created, the problems brought about are obvious. The BUM (Broadcast, Unknown Unicast, Multicast) storm brought by the shared L2 broadcast domain increases significantly with the increase of the network size, which will eventually affect the normal network traffic.

Traditional three-tier network architectures have existed for decades, and are still used in some data centers. What was the original reason for this architecture? On the one hand, because early L3 routing devices are much more expensive than L2 bridge devices. Even now, core switches are much more expensive than aggregation access layer devices. With this architecture, multiple aggregation layer PODs can be connected using a core set of switches. For example, in the above figure, a pair of core switches are connected to multiple aggregation layer PODs. On the other hand, in the early data centers, most of the traffic was north-south traffic. For example, a web application is deployed on a server for use by clients outside the data center. Using this architecture, you can control the inflow and outflow of data in the core switch, add a load balancer, and load balance data traffic.

The impact of technology development on network architecture

The data center is for data services. With the development of technology, the content and form of data have also changed.

• The popularity of virtualization. In a traditional data center, server utilization is not high. A three-layer network architecture with a certain oversubscription can effectively share the performance of the core switch and some other network devices. However, the popularity of virtualization makes server utilization higher. A physical server can virtualize multiple virtual machines and run their own tasks to take their own network paths. Therefore, high server utilization requires a smaller over-proportion. A Gartner report: Forecast: x86 Server Virtualization, Worldwide, 2012-2018, 2014 Update states that in 2018, 82% of servers will be virtual servers. The impact of virtualization on the data center network architecture is enormous.

• Decoupling of software architecture. Traditional software architectures are deployed in a dedicated mode. Software systems typically run on a physical server and are physically isolated from other systems. However, modular, layered software architecture design has become the mainstream now. Multiple components of a system are typically distributed across multiple virtual machines/containers. The most typical is the three-tier WEB application, which includes Client/Application/DB. A request is no longer done by a virtual machine/physical machine, but by multiple servers. The impact of this on the network is that the east-west traffic has increased.

• The rise of new applications. Traditional data centers are designed for .com applications, and this traffic is mostly between the client and server. With distributed computing, big data is emerging, and these applications generate a lot of traffic between servers in the data center. For example, Hadoop distributes data in hundreds of servers in a data center for parallel computing. It is said that a Hadoop cluster on Facebook has more than 100 petabytes of data. It can be seen that for some applications, the east-west traffic of the data center is huge.

• The proposed Software Defined Data Center (SDDC). SDDC proposes a software-defined data center that requires the data center’s computer storage network to be software-definable. Corresponding to the network, it is SDN. The traditional three-tier network architecture was not designed with SDN in mind.

To sum up, the technology development requires that the new data center has a smaller over-proportion or even no over-capture; higher east-west traffic bandwidth; support SDN.

In these requirements, higher east-west traffic support is especially important. The front talks about the north-south flow, east-west flow, what are these Dongdong? Data center traffic can be divided into the following general categories:

• North-South traffic: Traffic between clients outside the data center to the data center server, or traffic from the data center server to the Internet.

• East-West traffic: Traffic between servers in a data center.

• Cross-data center traffic: Traffic across data centers, such as disaster recovery between data centers, communication between private and public clouds.

According to the Cisco Global Cloud Index: Forecast and Methodology, 2015– 2020, by 2020, 77% of data center traffic will be internal to the data center, that is, east-west traffic, which is related to the impact of the above technology development on the network architecture. This is also why the east-west traffic is especially important.

What is the east-west traffic under the traditional three-tier network architecture?

As mentioned earlier, the traditional three-layer network architecture was born in the .com era, mainly designed for north-south traffic. However, the traditional network architecture does not support east-west traffic. Let’s analyze the east-west traffic trend in the traditional three-tier network architecture.

First, east-west traffic is divided into L2 and L3 traffic.

East-West L2 traffic, if both the source and destination hosts are under the same access layer switch, then full speed can be achieved because the access switch can complete the forwarding.

If you need to cross the rack, but still in an aggregation layer POD, you need to forward it through the aggregation layer switch. The bandwidth depends on the forwarding rate of the aggregation layer switch, the port bandwidth, and how many access layer switches share the aggregation layer switch at the same time. As mentioned earlier, STP is generally used between the aggregation layer and the access layer, which allows only one aggregation layer switch to work in one aggregation layer POD. In order to meet the L2 forwarding across the rack, the performance of the aggregation layer switch, such as bandwidth, forwarding rate must be greater than the access layer switch.

If the L2 traffic needs to span the aggregation layer POD (larger layer architecture), it must go through the core switch. The same problem persists and the requirements for core switches will be higher.

The east-west L3 traffic, whether it is under an access layer switch, needs to go to the core switch with the L3 function to complete the forwarding. As shown below:

This is a hair-pin traffic that not only wastes valuable core switch resources, but multi-layer forwarding increases network transmission latency. Similarly, due to the existence of over-proportion, it does not guarantee full-speed L3 traffic.

In general, in order to guarantee any east-west traffic bandwidth, higher performance aggregation switches and core switches are bound to be required. On the other hand, you can also carefully design, and try to put the server with east-west traffic under the same access switch. In any case, this adds cost and reduces usability.

The impact of changes in market demand on network architecture

Due to cost and operation and maintenance factors, data centers are generally large enterprises capable of deploying. But with the development of technology, some small and medium-sized enterprises also need to deploy data centers. The difference is that the needs of small and medium-sized enterprises are generally started on a small scale, and gradually expand the data center as their business grows. The size of the data center depends largely on the size of the network. The corresponding network needs are started with a low-cost, small-scale network architecture, but can be scaled to a larger scale.

The size of a traditional three-tier network architecture depends on the performance and scale of the core layer devices, depending on the port density of the switch. The largest data center corresponds to the largest and highest performance network equipment. This type of equipment is not available to all network equipment vendors, and the corresponding capital cost and operation and maintenance cost are also high. With a traditional three-tier network architecture, enterprises will face the dilemma of cost and scalability.

Conclusion

The traditional three-tier network architecture will not disappear in the short term, but due to the development of technology and market, its shortcomings are becoming more and more obvious. It is very necessary to improve based on the existing network architecture. The new network architecture is preferably composed of relatively small-scale switches, which can be easily extended horizontally. It supports HA (active-active mode) and supports full speed. East-West traffic, not purchasing high-performance core switches can also remove super-capacity, support SDN and so on.

Original article from https://zhuanlan.zhihu.com/p/29881248