Procedure

This section demonstrates the resizing procedure in detail. 

1. Ensure prerequisites are met 

  • FREE_MB > REQUIRED_MIRROR_FREE_MB in V$ASM_DISKGROUP for the disk group that will shrink  (typically RECO); ideally, FREE_MB > 2 * REQUIRED_MIRROR_FREE_MB [USED_MB may be more realistic, need to check] to cut number of rebalances in half by doing and ADD/DROP operation simultaneously for each cell, except the first and last cells.  On the first cell you will do a DROP disk operation and on the last one you will do an ADD disk operation. 
  • If the RECO disk group is configured as high redundancy, then you can do this procedure on a Half Rack or greater. You will not be able to do this procedure on a Quarter Rack because ASM will not allow you to drop a failure group such that only one copy of the data remains. 
  • All disk groups are backed up and you have tested the recovery strategy. 
  • The load on the system will not inhibit rebalancing activity at the desired power level. 
  • All DCLI commands should use the celladmin user on the cells as a best practice.  This may require establishing equivalency to the celladmin user from the oracle user on DB nodes. 
  • Run the Exachk script ( MOS note 1070954.1) one or two weeks before doing this procedure to ensure the system is healthy. 

2.  Calculate RECO and DATA disk group and grid disk sizing changes 

In this step we assume that you have properly estimated the space needed for objects to store  in RECO (without considering redundancy), e.g., number of archivelogs you want to store, etc.  The system in the examples below was a quarter-rack installation containing three cells for a total of 36 grid disks in each of the DATA_A and RECO_A disk groups.  We are using DATA_A and RECO_A disk groups instead of the customary DATA and RECO disk groups to make our testing flexible and not interfere with the other disk groups (this also demonstrates that the procedure will work for any two disk groups). 

This value is known as USABLE_MB and is expressed in megabytes.
We will first obtain information about the current disk group sizes:

a.  Query 1: Find the amount of free space in the disk groups: 

select group_number, name, type,total_mb,free_mb,required_mirror_free_mb,usable_file_mb,(total_mb-free_mb) used_mb

from v$asm_diskgroup

order by group_number;

  

GROUP_NUMBER NAME                           TYPE     TOTAL_MB    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB    USED_MB

———— —————————— —— ———- ———- ———————– ————– ———-

           4 DATA_A_DM01                  NORMAL    6659712    6577052                 2219904        2178574      82660

           5 RECO_A_DM01                  NORMAL   10742976   10721816                 3580992        3570412      21160 

For our example here, we will choose to resize DATA_A_DM01 and RECO_A_DM01.  Other disk groups won’t be affected and hence aren’t shown. 

We need to determine the number of cells and disks that are being used; this is accomplished by the following queries:

b. Query 2: Find miscellaneous information about the ASM disks: 

SQL> SELECT dg.name, count(1) “Num Disks”
FROM v$asm_disk d, v$asm_diskgroup dg
WHERE d.group_number = dg.group_number
GROUP BY dg.name;
  2    3    4  
NAME                            Num Disks
—————————— ———-
DATA_A_DM01                          36
DATA_DM01                            36
DBFS_DG                                30
RECO_A_DM01                          36
RECO_DM01                            36

SQL> SELECT d.group_number, dg.name, d.os_mb
FROM  v$asm_disk d, v$asm_diskgroup dg
WHERE d.group_number = dg.group_number
AND dg.name IN (‘DATA_A_DM01′,’RECO_A_DM01’)
ORDER BY dg.name

           4 DATA_A_DM01                      184992
           4 DATA_A_DM01                      184992
           5 RECO_A_DM01                      298416
           5 RECO_A_DM01                      298416
           5 RECO_A_DM01                      298416    

c. Query 3: Find the number of failure groups (cells): 

SELECT dg.name DG_NAME, count(distinct failgroup) NUM_FAILGROUPS

FROM v$asm_disk d, v$asm_diskgroup dg

WHERE d.group_number = dg.group_number

AND dg.name like ‘RECO%’

GROUP BY dg.name;

 

DG_NAME                        NUM_FAILGROUPS

—————————— ————–

RECO_A_DM01                               3 

For this specific example, based on our needs, we have determined that RECO_A requires 1 TB of usable file space (instead of the 3.5 TB it currently has) and will resize disk groups DATA_A and RECO_A accordingly.  This should give DATA_A an increase of about 2.5 TB for a total of about 4.6 TB of usable free space. 

We decide to reserve space in case of cell failure instead of disk failure, so we will use this formula to calculate the total size of the RECO disk group we will need (see the discussion in the Exadata Database Machine Owners Guide, Chapter 7, Resizing Storage Grid Disks: 

 RECO_TOTAL_MB = 2 * C /(C-1) * USABLE_MB  

Using the above queries, we know the following:

C = NUM_FAILGROUPS (in query 3) = 3

D = DISK_COUNT (in query 2) in the RECO and DATA disk groups = 36 

We apply the formulas, with our chosen value of USABLE_MB = 1 TB: 

RECO_TOTAL_MB = 2 * C /(C-1) * USABLE_MB  = 2 * 3/2 * 1048576 = 3145728 MB

RECO_GD_MB = RECO_TOTAL_MB / D   = CEIL(3145728 / 36/16) * 16 = 87392 MB

Note: The CEIL() function returns values to the next largest integer; this is used to round up to the next 16MB boundary. 

DATA GD_MB = (DATA_OS_MB + RECO_OS_MB) – RECO GD_MB 

                            = (184992 + 298416) – 87392 = 396016 MB

New DATA disk group total size = DATA GD_MB * D = 396016* 36 = 14256576 MB ~ 14 TB 

As a sanity check, we look at the existing grid disk definitions for RECO_A: 

$ dcli -g cell_group -l celladmincellcli -e list griddisk attributes name,size where name like \’RECO_A.*\'”

dm01cel04: RECO_A_CD_01_dm01cel04    291.421875G

dm01cel06: RECO_A_CD_12_dm01cel06    291.421875G

 

The current size of the RECO_A grid disks agree with the value of OS_MB in query 2 (291.421875G = 298416 MB).  These will be re-created with the size of 87392 MB we calculated above.

This command will be used to find the DBFS_DG starting offset to use when we recreate the RECO grid disks (this will likely be different in your environment): 

 dcli -g cell_group -l celladmincellcli -e list griddisk attributes name,offset | grepi dbfs_dg

 dm01cel04: DBFS_DG_CD_03_dm01cel04           854.9375G

dm01cel06: DBFS_DG_CD_12_dm01cel06           854.9375G 

We also look at the existing grid disk definitions for DATA_A:

$ dcli -g cell_group -l celladmincellcli -e list griddisk attributes name,size where name like \’DATA_A.*\'”

dm01cel04: DATA_A_CD_01_dm01cel04         180.65625G

dm01cel06: DATA_A_CD_12_dm01cel06         180.65625G

The current size of the DATA_A  grid disks agree with the value of OS_MB in query 2.  These grid disks will be resized to the value of 396026 MB.  We sanity-check our calculations here to ensure we are growing and not shrinking the DATA_A grid disks (e.g., 396026 > 184992). 

3.  DROP ASM disks from RECO Disk group on the first cell

First, ensure no rebalancing operations are occurring by running this query (and all subsequent SQL statements shown in the examples below in an ASM instance); no rows should return from this query: 

SQL> SELECT * FROM gv$asm_operation;

If no rebalancing operations are occurring then, drop the ASM disks from the RECO disk group for one cell (in this example we have cells dm01cel04 – dm01cel06 in a split-rack):     

SQL> alter diskgroup RECO_A_DM01 drop disks in failgroup DM01CEL04 rebalance  power 32 NOWAIT; 

 In the example above we use a power level of 32 with the assumption that ASM is running version 11.2.0.2 or higher.  Your choice of power level depends on how much workload is on the system at the time of the rebalance operation, how quickly you wish the rebalance to finish, and how much higher power levels will impact the workload. We do not recommend rebalance power levels higher than 32.  If there is application activity that is sensitive to I/O performance, you may want to start with a small rebalance power and increase it (using the ALTER DISKGROUP command) to higher power levels until the application begins to be impacted. 

4. Wait for Rebalance to Finish

First, ensure no rebalancing operations are occurring by running this query; no rows should return:           

SQL> SELECT * FROM gv$asm_operation; 

Note: Rebalancing involves a “compaction” phase after the file extents are rebalanced and redundancy is established.  This compaction phase may take some time and its progress is not reported in gv$asm_operation.  Please be patient while this activity proceeds.  To monitor progress, please see MOS note 1274322.1 for the script in entry, “Check ASM rebalance forward progress if you suspect a problem” 

5. Drop the RECO grid disks on the cell

a. Ensure that ASM disks were properly dropped: 

column header_status heading HSTATUS format A15

column name format A35

column path format A55

set lines 200

set pages 100 

SQL> SELECT group_number,path,failgroup,header_status,mount_status

   FROM v$asm_disk

   WHERE failgroup = ‘DM01CEL04’

   ORDER BY header_status,path

PATH                                                    FAILGROUP                      HSTATUS         MOUNT_S

——————————————————- —————————— ————— ——-

o/192.168.227.203/RECO_A_CD_01_dm01cel04                DM01CEL04                      FORMER          CLOSED

o/192.168.227.203/RECO_A_CD_02_dm01cel04                DM01CEL04                      FORMER          CLOSED

o/192.168.227.203/RECO_A_CD_11_dm01cel04                DM01CEL04                      FORMER          CLOSED

o/192.168.227.203/RECO_A_CD_12_dm01cel04                DM01CEL04                      FORMER          CLOSED 

 The GROUP_NUMBER should be zero if the disks were successfully dropped (usually you will also see the HEADER_STATUS as “FORMER” and the MOUNT_STATUS as “CLOSED” ), otherwise do not proceed until you investigate the state of the disks further. 

b. Verify the grid disks are in the proper status to proceed

$ dcli –c dm01cel04 –l celladmincellcli –e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome

         RECO_A_CD_01_dm01cel04        UNUSED  Yes

         RECO_A_CD_12_dm01cel04        UNUSED  Yes 

If the ASMMODESTATUS is “UNUSED” and the ASMDEACTIVATIONOUTCOME is “Yes”, then it is safe to proceed.  Otherwise, investigate further and correct before continuin

c. Drop the grid disks from the cell

From the first DB node, run the following DCLI command as the oracle user (be sure the cell name is correct!):

 $ dcli –c dm01cel04 –l celladmincellcli -e drop griddisk all harddisk prefix=RECO_A”

NOTE: you may need to add the FORCE option to the “drop griddisk” command if the cell doesn’t allow this command to succeed and you are sure that the ASM disks on the specific cell were successfully dropped in steps 3 and 4 above. 

6. Recreate RECO grid disks on the cell to a smaller size

We will need the size and offset calculations that were done in step 2 here.  For this particular example:

  • The DBFS_DG disk group has its starting offset at 854.9375 GB = 875456 MB
  • We calculated the new size for the RECO_A grid disks as  87392 MB

 Therefore, the starting offset for the RECO_A grid disk will be:  875456 – 87392 = 788064 MB.  This will make the RECO_A grid disks contiguous with the DBFS_DG grid disks on the inner part of the celldisk.

 From the first DB node, run the following DCLI command as the oracle user to create the RECO_A grid disks:

 $ dcli -c dm01cel04 -l celladmincellcli -e CREATE GRIDDISK ALL HARDDISK PREFIX=’RECO_A’, offset=788064M, size=87392M

 Verify the grid disks were created properly and contiguous with DBFS_DG:

 $ dcli -c dm01cel04 -l celladmincellcli -e list griddisk attributes name, offset, size”

 Verify free space available now in the celldisk to use for the DATA_A grid disks in a later step (there should only be one “offset”, “size” pair listed per celldisk): 

 $  dcli -c dm01cel04 -l celladmincellcli -e list celldisk attributes name,freespace,freespacemap

 dm01cel04: CD_01_dm01cel04   291.421875G ((offset=769.59375G,size=291.421875G))  

7.  Add RECO ASM disks from the cell to the RECO Disk group and drop RECO ASM disks from the next cell 

If sufficient free space exists in the RECO DG (twice the amount of REQUIRED_MIRROR_FREE_MB), then you can run this command:

SQL> alter diskgroup RECO_A_DM01 drop disks in failgroup DM01CEL05 add disk ‘o/*/RECO_A*dm01cel04′ rebalance power 32 NOWAIT; 

Otherwise, you’ll need to run the add and drop commands separately:

SQL> alter diskgroup RECO_A_DM01 add disk ‘o/*/RECO_A*dm01cel04′ rebalance power 32 NOWAIT;

Wait for rebalance to finish successfully: 

SQL> SELECT * FROM gv$asm_operation;

SQL> alter diskgroup RECO_A_DM01 drop disks in failgroup DM01CEL05 rebalance power 32 NOWAIT; 

8. Wait for Rebalance to Finish

Ensure no rebalancing operations are occurring by running this query; no rows should return:

SQL> SELECT * FROM gv$asm_operation;

 Verify ASM disks were dropped and grid disks are ready to be dropped as in steps 5a and 5b above.

 Ensure dm01cel04 RECO_A ASM disks are added back successfully with HEADER_STATUS=”MEMBER” and MOUNT_STATUS=”CACHED”; and dm01cel05 RECO ASM disks are now unused: 

 PATH                                         FAILGROUP     HSTATUS  MOUNT_S

——————————————– ————- ——– ——-

o/192.168.227.203/RECO_A_CD_01_dm01cel04   DM01CEL04   MEMBER   CACHED

o/192.168.227.203/RECO_A_CD_02_dm01cel04   DM01CEL04   MEMBER   CACHED

o/192.168.227.204/RECO_A_CD_01_dm01cel05   DM01CEL05   FORMER   CLOSED

o/192.168.227.204/RECO_A_CD_02_dm01cel05   DM01CEL05   FORMER   CLOSED

9. Repeat steps 5 through 8 for the remaining cells

Be careful to change the name of the cells and failure groups as you copy and execute commands on the various cells.

Note: On the final cell, we only need to add the ASM disks back into the disk group with the command:

 SQL> alter diskgroup RECO_A_DM01 add disk ‘o/*/RECO_A*dm01cel06′ rebalance power 32 NOWAIT; 

10. Wait for Rebalance to Finish

Ensure no rebalancing operations are occurring by running this query; no rows should return:

SQL> SELECT * FROM gv$asm_operation;

Verify ASM disks were added properly using the query in 5a above (be sure to change the failgroup name); the HEADER_STATUS should be “MEMBER” and the MOUNT_STATUS should be “CACHED”: 

PATH                                                                          FAILGROUP         HSTATUS  MOUNT_S

—————————————————- —————————— ————— ——-

o/192.168.227.205/RECO_A_CD_01_dm01cel06         DM01CEL06   MEMBER       CACHED

o/192.168.227.205/RECO_A_CD_02_dm01cel06         DM01CEL06   MEMBER       CACHED

 When rebalancing is done, you may proceed to the next step. 

11. Resize all DATA grid disks up to the desired size

Now that all RECO_A grid disks have been recreated to a smaller size, there should be sufficient free space in every celldisk to extend the DATA_A grid disks.

 a. Confirm available space 

$  dcli –g cell_group -l celladmincellcli -e list celldisk attributes name,freespace

dm01cel04: CD_01_dm01cel04   206.078125G  

 

        b. Confirm current size of the DATA_A grid disks:  

$  dcli –g cell_group -l celladmincellcli -e list griddisk attributes name,size

dm01cel04: DATA_A_CD_01_dm01cel04            180.65625G

       c. Calculate the new size for the DATA_A grid disks:

Current size = 180.65625G

Free space = 206.078125G

New size = current size + free space = 180.65625G + 206.078125G = 386.734375G = 396016 MB

 We note that this is exactly what we calculated in step 2 for “DATA GD_MB”, so we will proceed; any discrepancies should be investigated. 

      d. Execute the DATA_A grid disk resizing commands:

Hint: You can obtain a list of disks to resize using this command:

SQL>  select name||’,’ from v$asm_disk where group_number = order by path

 

$ dcli -c dm01cel04 -l celladmincellcli -e alter grid disk DATA_A_CD_01_DM01CEL04, DATA_A_CD_02_DM01CEL04, DATA_A_CD_03_DM01CEL04, DATA_A_CD_04_DM01CEL04, DATA_A_CD_05_DM01CEL04, DATA_A_CD_06_DM01CEL04,DATA_A_CD_07_DM01CEL04, DATA_A_CD_08_DM01CEL04, DATA_A_CD_09_DM01CEL04,DATA_A_CD_10_DM01CEL04,DATA_A_CD_11_DM01CEL04,DATA_A_CD_12_DM01CEL04 size=396016M 

This operation is immediate and can be done online.  After running this on all cells (change the –c parameter and grid disk names accordingly), verify the DATA_A  grid disks are indeed larger on all cells:

$ dcli –g cell_group -l root “cellcli -e list griddisk attributes name,offset,size 

Verify that all free space on the celldisks are allocated to the grid disks: 

$ dcli -g cell_group -l celladmincellcli -e list celldisk attributes name,freespace,freespacemap

dm01cel06: CD_01_dm01cel06   0

dm01cel06: CD_02_dm01cel06   0 

 e. Verify the ASM disk sizes match the grid disk sizes for DATA_A 

SELECT name, failgroup, os_mb FROM v$asm_disk

WHERE group_number = (SELECT group_number FROM v$asm_diskgroup

                     WHERE name = ‘DATA_A_DM01’)

ORDER BY failgroup; 

Note: Executing this query will refresh ASM’s cache and ensure the next step succeeds.  

12. Resize all DATA ASM disks up to the same size as grid disks in previous step 

ALTER DISKGROUP DATA_A_DM01 RESIZE ALL SIZE 396016M REBALANCE POWER 32 NOWAIT;

 This will trigger a rebalance.

CAUTION: Be sure to set the size for the ASM disks to EXACTLY the same size as the griddisks created and verified in step 11.c ; failure to match the sizes could corrupt the diskgroup

13. Wait for Rebalance to Finish

Ensure no rebalancing operations are occurring by running this query; no rows should return:

SQL> SELECT * FROM gv$asm_operation; 

14. Verify all sizes

When the rebalance is finished, verify that the disk group sizes match what was calculated in step 2:

GROUP_NUMBER NAME                                TYPE     TOTAL_MB    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB    USED_MB

———— ———————————– —— ———- ———- ———————– ————– ———-

…            4 DATA_A_DM01                         NORMAL   14256576   14173916                 4752192        4710862      82660

           5 RECO_A_DM01                         NORMAL    3146112    3124952                 1048704        1038124      21160

We confirm that DATA_A_DM01 disk group grew to 14 TB and RECO shrank to about 3 TB. 

15. End of Procedure