Exadata Cell(Storage) Nodes

Exadata之存储节点

1个以太网口(eth0用于管理网络，连接自带的CISCO交换机)

2个Infiniband口(分别连接自带的两个infiniband叶交换机。完成与计算节点间的传输数据)

1个HBA磁盘控制器(连接本地磁盘)

12块本地磁盘(可选高性能磁盘15000RPM 600GB，X5被极速闪存(Extreme Flash)替代)，或高容量磁盘7200RPM 2T-4T，版本不同对应的配置也不同)

4个PCIe Flash卡(每个卡上有4FDOM，从操作系统上可以看到16Flash磁盘，不同版本的型号，容量也不同，有PCIe 2.0和PCIe3.0接口，型号F20，F40，F80，容量也在不断扩大)

1个USB disk (保存系统镜像，当操作系统遇到问题时，可以通过USB快速恢复)

2个电源模块(PSU)

1个ILOM模块

6个PCIe插槽，一个用做LSI MegaRAID HBA卡，一个InfinibandHCA卡，剩余4插槽为闪存卡所用

Disks

存储节点上有12块盘，配置了一块LSI MegaRAID卡，级别为RAID0。前2块盘划分出部分空间做操作系统（操作系统存放在0号和1号磁盘的分区中,约占29G），剩余空间为数据库所用。本地磁盘分两种类型高性能和高容量。
HP：12 * 600GB 15K RPM High Performance SAS (hot-swappable)
HC：12 * 3TB 7,2K RPM High Capacity SAS (hot-swappable)

客户一般用高容量的多一些，因为除了这些盘之外，还有性能更强悍的闪存卡。从X5开始，存储节点的配置发生了重大变化，不再提供配有高性能磁盘的存储节点，而以更高速度的EF (Extreme Flash)替代，大小为1U。  每个存储节点配备了8个1.6T的Flash卡作为数据存储(共12.8T)，代替了从前14.4TB(HP 600G)的SAS硬盘。

磁盘编号也有改变

[root@cel01 ~]# fdisk -l /dev/sd[a-b]

Disk /dev/sda: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          15      120456   fd  Linux raid autodetect
/dev/sda2              16          16        8032+  83  Linux
/dev/sda3              17       69039   554427247+  83  Linux
/dev/sda4           69040       72824    30403012+   f  W95 Ext’d (LBA)
/dev/sda5           69040       70344    10482381   fd  Linux raid autodetect
/dev/sda6           70345       71649    10482381   fd  Linux raid autodetect
/dev/sda7           71650       71910     2096451   fd  Linux raid autodetect
/dev/sda8           71911       72171     2096451   fd  Linux raid autodetect
/dev/sda9           72172       72432     2096451   fd  Linux raid autodetect
/dev/sda10          72433       72521      714861   fd  Linux raid autodetect
/dev/sda11          72522       72824     2433816   fd  Linux raid autodetect

Disk /dev/sdb: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          15      120456   fd  Linux raid autodetect
/dev/sdb2              16          16        8032+  83  Linux
/dev/sdb3              17       69039   554427247+  83  Linux
/dev/sdb4           69040       72824    30403012+   f  W95 Ext’d (LBA)
/dev/sdb5           69040       70344    10482381   fd  Linux raid autodetect
/dev/sdb6           70345       71649    10482381   fd  Linux raid autodetect
/dev/sdb7           71650       71910     2096451   fd  Linux raid autodetect
/dev/sdb8           71911       72171     2096451   fd  Linux raid autodetect
/dev/sdb9           72172       72432     2096451   fd  Linux raid autodetect
/dev/sdb10          72433       72521      714861   fd  Linux raid autodetect
/dev/sdb11          72522       72824     2433816   fd  Linux raid autodetect 

/dev/sda and /dev/sdb each contain the same partitions with the same sizes. But lets look at the other 10 disks and see what they look like:

[root@ cel01 ~]# fdisk -l /dev/sd[c-l]

Disk /dev/sdc: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdc doesn’t contain a valid partition table

Disk /dev/sdd: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdd doesn’t contain a valid partition table

Disk /dev/sde: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sde doesn’t contain a valid partition table

Disk /dev/sdf: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdf doesn’t contain a valid partition table

Disk /dev/sdg: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdg doesn’t contain a valid partition table

Disk /dev/sdh: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdh doesn’t contain a valid partition table

Disk /dev/sdi: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdi doesn’t contain a valid partition table

Disk /dev/sdj: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdj doesn’t contain a valid partition table

Disk /dev/sdk: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdk doesn’t contain a valid partition table

Disk /dev/sdl: 598.9 GB, 598999040000 bytes
255 heads, 63 sectors/track, 72824 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdl doesn’t contain a valid partition table 

[root@cel01 ~]# mount

/dev/md6 on / type ext3 (rw,usrquota,grpquota)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/md8 on /opt/oracle type ext3 (rw,nodev)
/dev/md4 on /boot type ext3 (rw,nodev)
/dev/md11 on /var/log/oracle type ext3 (rw,nodev)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/sdm1 on /bla type ext3 (rw)
/dev/sdm1 on /mnt/usb type ext3 (rw)

The mounts show no evidence of partitions on /dev/sda or /dev/sdb. Instead we see various references to /dev/md partitions. We can also see this in /etc/fstab: 

[root@cel01 ~]# cat /etc/fstab

/dev/md6                /                       ext3    defaults,usrquota,grpquota        1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/md2                swap                    swap    defaults        0 0
/dev/md8                /opt/oracle             ext3    defaults,nodev  1 1
/dev/md4                /boot                   ext3    defaults,nodev  1 1
/dev/md11               /var/log/oracle         ext3    defaults,nodev  1 1 

/dev/md4:
        Version : 0.90
  Creation Time : Thu Feb  4 06:03:28 2010
     Raid Level : raid1
     Array Size : 120384 (117.58 MiB 123.27 MB)
  Used Dev Size : 120384 (117.58 MiB 123.27 MB)
   Raid Devices : 2
  Total Devices : 2

Preferred Minor : 4
    Persistence : Superblock is persistent
    Update Time : Fri Oct 14 06:44:52 2011
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
 Spare Devices : 0
           UUID :
         Events : 0.64
    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
We can see that the boot file system is also mirrored across /dev/sda1 and /dev/sdb1. 

FLASH卡

FLASH卡后面后面有对应的卡的编号，X2和之前的PCIe使用了Energy Storage Module (ESM) 电池，电池必须每3年更换一次，X3之后使用了电容
X2 F20 PCIe card              ,battery     ,96G
X3 F40 PCIe card              ,Capacitor  ,400G
X4 F80 PCIe card              ,Capacitor  ,800G
X5 F160 NVMePCIe card   ,Capacitor  ,1.6T

Sun Flash Accelerator F20 PCIe Card

Sun Flash Accelerator F40 PCIe Card

Sun Flash Accelerator F80 PCIe Card

内置USB

存储节点的扩展柜

存储扩展柜(Storage expansion rack)，当计算能力需要扩展时，Exadata支持将多个机柜进行连接，添加一个spine switch ，将infiniband网络级联。早期如果想扩展存储，计算节点也得买，存储扩展柜是后来推出的，可以增加只包含存储节点的扩展柜。扩展柜的配置同样分为1/4配，半配，和全配，分别包含4个，9个，和18个存储节点。 在X5的弹性配置中，存储节点的数量为4~19可选的。