Infrastructure

the android tinderbox (libreoffice2.dh.bytemark.co.uk) currently uses 4 disks in raid10 on a 3ware 9690SA-4I card.

two of the four disks have over 30000 reallocated sectors/are dying...
libreoffice2:/home/android# /usr/sbin/tw_cli /c0/p0 show all
/c0/p0 Status = OK
/c0/p0 Model = ST500DM002-1BD142
/c0/p0 Firmware Version = KC45
/c0/p0 Serial = Z2AKT1TE
/c0/p0 Capacity = 465.76 GB (976773168 Blocks)
/c0/p0 WWN = 5000c5003fc23438
/c0/p0 Drive Type = SATA
/c0/p0 Interface Type = Direct
/c0/p0 Drive Ports = 1
/c0/p0 Drive Connections = 1
/c0/p0 Link Speed Supported = 1.5 Gbps and 3.0 Gbps
/c0/p0 Link Speed = 3.0 Gbps
/c0/p0 Queuing Supported = Yes
/c0/p0 Queuing Enabled = Yes
/c0/p0 Reallocated Sectors = 0
/c0/p0 Power On Hours = 5387
/c0/p0 Temperature = 29 deg C
/c0/p0 Spindle Speed = 7200 RPM
/c0/p0 Identify Status = N/A
/c0/p0 Belongs to Unit = u0

Segmentation fault
libreoffice2:/home/android# /usr/sbin/tw_cli /c0/p1 show all
/c0/p1 Status = OK
/c0/p1 Model = ST500DM002-1BD142
/c0/p1 Firmware Version = KC45
/c0/p1 Serial = Z2AKXQ7D
/c0/p1 Capacity = 465.76 GB (976773168 Blocks)
/c0/p1 WWN = 5000c5003fc23625
/c0/p1 Drive Type = SATA
/c0/p1 Interface Type = Direct
/c0/p1 Drive Ports = 1
/c0/p1 Drive Connections = 1
/c0/p1 Link Speed Supported = 1.5 Gbps and 3.0 Gbps
/c0/p1 Link Speed = 3.0 Gbps
/c0/p1 Queuing Supported = Yes
/c0/p1 Queuing Enabled = Yes
/c0/p1 Reallocated Sectors = 496
/c0/p1 Power On Hours = 5380
/c0/p1 Temperature = 26 deg C
/c0/p1 Spindle Speed = 7200 RPM
/c0/p1 Identify Status = N/A
/c0/p1 Belongs to Unit = u0

Segmentation fault
libreoffice2:/home/android# /usr/sbin/tw_cli /c0/p2 show all
/c0/p2 Status = OK
/c0/p2 Model = ST500DM002-1BD142
/c0/p2 Firmware Version = KC45
/c0/p2 Serial = Z2AKWM9W
/c0/p2 Capacity = 465.76 GB (976773168 Blocks)
/c0/p2 WWN = 5000c5003fc21af3
/c0/p2 Drive Type = SATA
/c0/p2 Interface Type = Direct
/c0/p2 Drive Ports = 1
/c0/p2 Drive Connections = 1
/c0/p2 Link Speed Supported = 1.5 Gbps and 3.0 Gbps
/c0/p2 Link Speed = 3.0 Gbps
/c0/p2 Queuing Supported = Yes
/c0/p2 Queuing Enabled = Yes
/c0/p2 Reallocated Sectors = 32280
/c0/p2 Power On Hours = 4712
/c0/p2 Temperature = 27 deg C
/c0/p2 Spindle Speed = 7200 RPM
/c0/p2 Identify Status = N/A
/c0/p2 Belongs to Unit = u0

Segmentation fault
libreoffice2:/home/android# /usr/sbin/tw_cli /c0/p3 show all
/c0/p3 Status = OK
/c0/p3 Model = ST500DM002-1BD142
/c0/p3 Firmware Version = KC45
/c0/p3 Serial = Z2AKXQX2
/c0/p3 Capacity = 465.76 GB (976773168 Blocks)
/c0/p3 WWN = 5000c5003fc26d91
/c0/p3 Drive Type = SATA
/c0/p3 Interface Type = Direct
/c0/p3 Drive Ports = 1
/c0/p3 Drive Connections = 1
/c0/p3 Link Speed Supported = 1.5 Gbps and 3.0 Gbps
/c0/p3 Link Speed = 3.0 Gbps
/c0/p3 Queuing Supported = Yes
/c0/p3 Queuing Enabled = Yes
/c0/p3 Reallocated Sectors = 34936
/c0/p3 Power On Hours = 5387
/c0/p3 Temperature = 28 deg C
/c0/p3 Spindle Speed = 7200 RPM
/c0/p3 Identify Status = N/A
/c0/p3 Belongs to Unit = N/A

The box refused to boot, but I managed to get it back up - but rebuilding the degraded array fails/drive p3 gets kicked out/won't allow a rebuild to finish.

libreoffice2:/home/android# /usr/sbin/tw_cli /c0 show alarms

Ctl Date Severity Alarm Message
------------------------------------------------------------------------------
c0 [Thu Apr 23 15:20:54 2020] WARNING Primary DCB read error occurred: phy=3, error=0x202
c0 [Thu Apr 23 15:20:54 2020] INFO Battery capacity test is overdue
c0 [Thu Apr 23 15:20:54 2020] WARNING Incomplete unit detected: unit=0
c0 [Thu Apr 23 15:20:54 2020] WARNING Incomplete unit detected: unit=1
c0 [Thu Apr 23 15:44:31 2020] WARNING SMART threshold exceeded: phy=3
c0 [Thu Apr 23 15:44:37 2020] WARNING Primary DCB read error occurred: phy=3, error=0x202
c0 [Thu Apr 23 15:54:54 2020] INFO Rebuild started: unit=0, subunit=1
c0 [Thu Apr 23 16:00:57 2020] INFO Battery capacity test started
c0 [Thu Apr 23 16:00:57 2020] INFO Battery charging started
c0 [Thu Apr 23 16:00:59 2020] INFO Battery charging completed
c0 [Thu Apr 23 16:02:27 2020] WARNING Drive removed: phy=3
c0 [Thu Apr 23 16:02:27 2020] ERROR Degraded unit: unit=0, vport=3
c0 [Thu Apr 23 16:02:29 2020] WARNING SMART threshold exceeded: phy=3
c0 [Thu Apr 23 16:02:29 2020] INFO Drive inserted: phy=3
c0 [Thu Apr 23 16:02:29 2020] INFO Unit operational: unit=0
c0 [Thu Apr 23 16:03:55 2020] INFO Rebuild started: unit=0, subunit=1
c0 [Thu Apr 23 16:12:05 2020] WARNING SMART threshold exceeded: phy=2
c0 [Thu Apr 23 16:12:05 2020] WARNING SMART threshold exceeded: phy=2
c0 [Thu Apr 23 16:12:10 2020] WARNING SMART threshold exceeded: phy=3
c0 [Thu Apr 23 16:12:10 2020] WARNING SMART threshold exceeded: phy=3
c0 [Thu Apr 23 16:12:15 2020] INFO Rebuild paused: unit=0, subunit=1
c0 [Thu Apr 23 16:12:30 2020] INFO Rebuild started: unit=0, subunit=1
c0 [Thu Apr 23 16:13:47 2020] WARNING Drive removed: phy=3
c0 [Thu Apr 23 16:13:47 2020] ERROR Degraded unit: unit=0, vport=3
c0 [Thu Apr 23 16:18:22 2020] WARNING SMART threshold exceeded: phy=3
c0 [Thu Apr 23 16:18:22 2020] INFO Drive inserted: phy=3
c0 [Thu Apr 23 16:18:22 2020] INFO Unit operational: unit=0
c0 [Thu Apr 23 16:19:49 2020] WARNING Drive removed: phy=3
c0 [Thu Apr 23 16:19:49 2020] ERROR Degraded unit: unit=0, vport=3
c0 [Thu Apr 23 16:23:32 2020] WARNING SMART threshold exceeded: phy=3
c0 [Thu Apr 23 16:23:32 2020] INFO Drive inserted: phy=3
c0 [Thu Apr 23 16:23:33 2020] INFO Unit operational: unit=0
c0 [Thu Apr 23 16:24:56 2020] INFO Rebuild started: unit=0, subunit=1
c0 [Thu Apr 23 16:25:00 2020] WARNING Drive removed: phy=3
c0 [Thu Apr 23 16:25:00 2020] ERROR Degraded unit: unit=0, vport=3
c0 [Thu Apr 23 16:27:02 2020] WARNING SMART threshold exceeded: phy=3
c0 [Thu Apr 23 16:27:02 2020] INFO Drive inserted: phy=3
c0 [Thu Apr 23 16:27:02 2020] INFO Unit operational: unit=0
c0 [Thu Apr 23 16:28:29 2020] WARNING Drive removed: phy=3
c0 [Thu Apr 23 16:28:29 2020] ERROR Degraded unit: unit=0, vport=3