1.1 Oracle block devices

Due to the fact that RH will not be supporting raw devices after enterprise linux 5 Oracle now recommend using block devices instead of raw devices.

The following instructions for setting up the ASM devices is taken from the Oracle technical note  number 602952.1

 

1.1.1 Partition devices

Identify the sd devices to be used by asm.

“cat /proc/partitions” will list all devices, look at the “sd“  devices.

e.g.

# cat /proc/partitions

major minor  #blocks  name

8     0    20971520 sda

8     0        104391 sda1

8     0    20860402 sda2

8    16   12582912 sdb

8    32   12582912 sdc

8    48     1048576 sdd

8    64     1048576 sde

8    64     1048576 sdf

8    64     1048576 sdg

8    64     1048576 sdh

Here the ASM volumes are sdb and sdc (each 12Gb)

The OCR and Voting disks are sdd to sdh (each 1Gb)

Now we have identified the disks we need to create partitions on them.

For each device enter:

$ fdisk /dev/sd-??

Command (m for help): u

Command (m for help): p

Command (m for help): n

Command (m for help): p

1

32768

w

(Where ?? are the characters used to identify the device.)

The reason we are using 32768 as the start block for the device is because of the way scsi/san volume mapping uses the first few blocks of a device and subsequently causes excessive write I/O. This is better explained in the following link:

http://www.pythian.com/news/411/aligning-asm-disks-on-linux

e.g.

# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel. Changes will remain in memory only,

until you decide to write them. After that, of course, the previous

content won’t be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): u

Changing display/entry units to sectors

Command (m for help): p

Disk /dev/dm-5: 5394 MB, 5394923520 bytes

255 heads, 63 sectors/track, 655 cylinders, total 10536960 sectors

Units = sectors of 1 * 512 = 512 bytes

Device Boot      Start         End      Blocks   Id  System

Command (m for help): n

Command action

e   extended

p   primary partition (1-4)

p

Partition number (1-4): 1

First sector (63-10536959, default 63): 32768

Last sector or +size or +sizeM or +sizeK (32768-25165823, default 25165823):

Using default value 25165823

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

#

1.2 Configure device protection

Create or edit the file /etc/udev/rules.d/98-oracle-perm.rules and add the following:

#OCR DISKS

KERNEL==”sdb*”, GROUP=”oinstall”, MODE=”640”

KERNEL==”sdc*”, GROUP=”oinstall”, MODE=”640”

#VOTE DISKS

KERNEL==”sdd*”, OWNER=”oracle”, GROUP=”oinstall”, MODE=”660”

KERNEL==”sde*”, OWNER=”oracle”, GROUP=”oinstall”, MODE=”660”

#ASM DISKS

KERNEL==”sdf*”, OWNER=”oracle”, GROUP=”oinstall”, MODE=”660”

KERNEL==”sdg*”, OWNER=”oracle”, GROUP=”oinstall”, MODE=”660”

KERNEL==”sdh*”, OWNER=”oracle”, GROUP=”oinstall”, MODE=”660”

Changing the sd device name where necessary.

 

Create or edit the file /etc/udev/rules.d/99-oracle-naming.rules and add the following:

KERNEL==”sdb1”, NAME=”ocr1”

KERNEL==”sdc1”, NAME=”ocr2”

KERNEL==”sdd1”, NAME=”vote1”

KERNEL==”sde1”, NAME=”vote2”

KERNEL==”sdf1”, NAME=”vote3”

KERNEL==”sdg1”, NAME=”asm2”

KERNEL==”sdh1”, NAME=”asm1”

Changing the sd device name where necessary.

 

Reboot the system.

After the reboot we should see the asm, ocr and vote volumes in /dev.

 

1.2.1 Map the asm volumes

For each of the asm volumes execute the command:

/etc/init.d/oracleasm createdisk ASMn /dev/asmn

Replacing the “n” with the relevant numbers.

Perform the following on all nodes in the cluster:

# /etc/init.d/oracleasm scandisks

# /etc/init.d/oracleasm listdisks

The ASM volumes should now be listed in the directory /dev/oracleasm/disks

# ls /dev/oracleasm/disks

ASM1     ASM2    ASM3 etc

 

ALTERNATIVE COMMANDS

================

You have to add the disk information to the below rules file on all nodes

# view /etc/udev/rules.d/99-oracle-asmdevices.rules

Ex entry

KERNEL==”dm-[0-9]*”, ENV{DM_UUID}==”mpath-360060e801526af00000126af00001c2b”, NAME=”ISP_A_ARCH_10″, OWNER=”oracle”, GROUP=”asmdba”, MODE=”0660″

Below information for the above line will be available in /etc/multipath.conf file

360060e801526af00000126af00001c2b

ISP_A_ARCH_10

Reload and Restart the udev rules on all nodes

# udevadm control –reload-rules

# start_udev

 

When you look the /dev/mapper directory; you can see the new disk added and you can see the disk the asmca