72 Line Infiniband Switch

Case Study: Engineering a Super Computer.

72 Port Infiniband Switch The primary purpose of 72 Port Switch Card is to reduce cable count and complexity in highly integrated Infiniband switch systems. This approach was initially utilized to create a switching matrix as the heart of a supercomputer. The 72 Port Switch Card is a subset of the full design and was meant to support localized nodes with a moderate concentration of local I/O requirements. This might serve as a separate building or a separate group within the overall network. The Primary protocol is Infiniband with other protocol and interface translation accomplished externally as needed. Most communications are expected to utilize Infiniband in order to maximize speed and throughput of the overall system. It works by routing external data to a fabric card that is able to accomplish N by N switching between any channels in the chassis. In quad data rate (QDR) mode, 10gb/s transmission rates per channel are typical. The switch element ASIC is the InfiniScaleIV (I4) from Mellanox. This device provides 36 4x QDR ports. To obtain 72 ports of IB data, six chips were employed per assembly.

The main challenge in dealing with a design of this magnitude is the routing of high speed signals. To accommodate this challenge, routing groups had to be placed on different PCB layers separated by internal ground planes. This method would guarantee the reduction of crosstalk between signal groups. Signal groups consist of signal pairs coming to and from (Tx, Rx) the outside world (IPASS connectors), and signals coming to and from (Tx, Rx) the Fabric Card. Each layer has its own unique routing format to keep signal pairs from overlapping. Utilizing a Port Allocation Table facilitated a signal naming convention that enabled a more “route friendly” method for layout.

Back to Electrical Engineering Overview