December 08, 2000
Election count blues and VxVM [ Software ]
It's been a month since my wife and I woke up, expecting to hear the results of the election on WETA 90.9 FM on the morning of November 8th 2000. No such luck.
I voted for Bush, despite being a liberal partisan during high school and college, because I can't get past Al Gore's personality. If there was a way to vote for the Democratic platform without electing Gore, I'd have done it.
Partisan opinions don't get much done when your root disk is incorrectly set up in Veritas Volume Manager, though. After the nice guy from Veritas came by to install FirstWatch and Volume Manager on our new firewall cluster, I looked closer at the configuration he left behind on the disks.
Turns out, he encapsulated the root disk but didn't mirror it like he claimed to. Whoops. Not a big problem, though, as long as you catch it before your drive sucks up that Mountain Dew you spilled last night and starts making sounds like a compressed duck while your manager asks you why the new server isn't up.
So now I get to learn HOW TO FIX UNMIRRORED DISKS IN VERITAS VOLUME MANAGER.
First step - check and see if your configuration's set up right. We have two internal 9GB disks per machine used for the operating system. I installed the operating system before we got the license for Volume Manager, so when VxVM was installed, we had to encapsulate the disk to preserve the existing data.
'vxprint' will show a disk group (labeled as "dg"), disk media (labeled as "dm"), and volumes (what Veritas calls a filesystem, labeled as "v"). Each volume should include at least one subdisk (labeled as "sd") and plex (what Veritas calls a logical data device within a volume, labeled as "pl") linked to that volume.
If you're mirroring a disk or volumes on that disk, vxprint output should look something like:
# vxprint -g rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg rootdg rootdg - - - - - -
dm rootdg01 c0t0d0s2 - 17678493 - - - -
dm rootdg02 c0t8d0s2 - 17678493 - - - -
v rootvol root ENABLED 4197879 - ACTIVE - -
pl rootvol-01 rootvol ENABLED 4197879 - ACTIVE - -
sd rootdg01-01 rootvol-01 ENABLED 4197879 0 - - -
pl rootvol-02 rootvol ENABLED 4197879 - ACTIVE - -
sd rootdg02-01 rootvol-02 ENABLED 4197879 0 - - -
[remaining output snipped for brevity]
This shows the "rootdg" disk group, the "rootdg01" and "rootdg02" disk media devices, and the "rootvol" volume (where the / filesystem is mounted). One half of the "rootvol" mirror comes from the "rootvol-01" plex, which is bound to the "rootvol" volume, using the "rootdg01-01" subdisk. The other half of the mirror comes from the "rootvol-02" plex, which is bound to the "rootvol" volume, using the "rootdg02-01" subdisk.
So if a volume is mirrored on two disks, there will be two disk devices (sd) and logical data devices (plexes) associated with that volume. No such luck here. 'vxprint' on the machine showed:
# vxprint -g rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg rootdg rootdg - - - - - -
dm rootdisk c0t0d0s2 - 17678493 - - - -
dm rootmirror c0t8d0s2 - 17678493 - - - -
v rootvol root ENABLED 4197879 - ACTIVE - -
pl rootvol-01 rootvol ENABLED 4197879 - ACTIVE - -
sd rootdisk-02 rootvol-01 ENABLED 4197879 0 - - -
Hmmm, got the disk group, disk media devices, the volume and one plex/subdisk pair. Nothing's being used from "rootmirror", only from "rootdisk". Guess it's not really mirrored after all.
Second step - pull off and remirror one disk. I used vxdiskadm and vxassist to do this. There's probably a more studly way to go about this with vxmake & company, but this worked for me.
To remove a disk from a mirror, run 'vxdiskadm' and select option 3 to remove the disk.
# vxdiskadm
Volume Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
list List disk information
? Display help about menu
?? Display help about the menuing system
q Exit from menus
Select an operation to perform: 3
Remove a disk
Menu: VolumeManager/Disk/RemoveDisk
Use this operation to remove a disk from a disk group. This
operation takes, as input, a disk name. This is the same name
that you gave to the disk when you added the disk to the disk
group.
Enter disk name [,list,q,?] list
Disk group: rootdg
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
dm rootdisk c0t0d0s2 sliced 3590 17678493 -
dm rootmirror c0t8d0s2 sliced 3590 17678493 -
Enter disk name [,list,q,?] rootmirror
Requested operation is to remove disk rootmirror from group rootdg.
Continue with operation? [y,n,q,?] (default: y) y
Removal of disk rootmirror is complete.
Remove another disk? [y,n,q,?] (default: n)
If there's running volumes on this disk, the vxdiskadm will complain:
Enter disk name [,list,q,?] rootdisk
The following volumes currently use part of disk rootdisk:rootvol swapvol tmp usr var
Volumes must be moved from rootdisk before it can be removed.
Move volumes to other disks? [y,n,q,?] (default: n) n
Remove a different disk? [y,n,q,?] (default: n)
You'll have to use 'vxassist remove mirror
# vxassist remove volume rootvol
vxvm:vxassist: ERROR: Volume rootvol is opened, cannot remove
In this case, go back into vxdiskadm and attempt to remove the disk again. Then run 'vxassist remove mirror
Once the disk is removed, add it back to the same disk group and re-initialize it by selecting option 1 in vxdiskadm.
Volume Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
list List disk information
? Display help about menu
?? Display help about the menuing system
q Exit from menus
Select an operation to perform: 1
Add or initialize disks
Menu: VolumeManager/Disk/AddDisks
Use this operation to add one or more disks to a disk group. You can
add the selected disks to an existing disk group or to a new disk group
that will be created as a part of the operation. The selected disks may
also be added to a disk group as spares. Or they may be added as
nohotuses to be excluded from hot-relocation use. The selected
disks may also be initialized without adding them to a disk group
leaving the disks available for use as replacement disks.
More than one disk or pattern may be entered at the prompt. Here are
some disk selection examples:
all: all disks
c3 c4t2: all disks on both controller 3 and controller 4, target 2
c3t4d2: a single disk
Select disk devices to add: [,all,list,q,?] list
DEVICE DISK GROUP STATUS
c0t0d0 rootdisk rootdg online
c0t8d0 - - online
Select disk devices to add: [,all,list,q,?] c0t8d0
Here is the disk selected. Output format: [Device_Name]
c0t8d0
Continue operation? [y,n,q,?] (default: y)
You can choose to add this disk to an existing disk group, a
new disk group, or leave the disk available for use by future
add or replacement operations. To create a new disk group,
select a disk group name that does not yet exist. To leave
the disk available for future use, specify a disk group name
of "none".
Which disk group [,none,list,q,?] (default: rootdg)
Use a default disk name for the disk? [y,n,q,?] (default: y) n
Add disk as a spare disk for rootdg? [y,n,q,?] (default: n)
Exclude disk from hot-relocation use? [y,n,q,?] (default: n)
The selected disks will be added to the disk group rootdg with
disk names that you will specify interactively.
c0t8d0
Continue with operation? [y,n,q,?] (default: y)
The following disk device appears to have been initialized already.
The disk is currently available as a replacement disk.
Output format: [Device_Name]
c0t8d0
Use this device? [y,n,q,?] (default: y)
The following disk you selected for use appears to already have
been initialized for the Volume Manager. If you are certain the
disk has already been initialized for the Volume Manager, then you
do not need to reinitialize the disk device.
Output format: [Device_Name]
c0t8d0
Reinitialize this device? [y,n,q,?] (default: y)
Initializing device c0t8d0.
Enter disk name for c0t8d0 [,q,?] (default: disk01) rootdg02
Adding disk device c0t8d0 to disk group rootdg with disk
name rootdg02.Add or initialize other disks? [y,n,q,?] (default: n)
Now that it's added to the disk group, mirror the volumes from the other disk by selecting option 6 from vxdiskadm. This may take a while, but once it's completed, you'll have your encapsulated disk properly mirrored.
Volume Manager Support Operations
Menu: VolumeManager/Disk
1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
list List disk information
? Display help about menu
?? Display help about the menuing system
q Exit from menus
Select an operation to perform: 6
Mirror volumes on a disk
Menu: VolumeManager/Disk/Mirror
This operation can be used to mirror volumes on a disk. These
volumes can be be mirrored onto another disk or onto any
available disk space. Volumes will not be mirrored if they are
already mirrored. Also, volumes that are comprised of more than
one subdisk will not be mirrored.
Mirroring volumes from the boot disk will produce a disk that
can be used as an alternate boot disk.
At the prompt below, supply the name of the disk containing the
volumes to be mirrored.
Enter disk name [,list,q,?] list
Disk group: rootdg
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
dm rootdg02 c0t8d0s2 sliced 3590 17678493 -
dm rootdisk c0t0d0s2 sliced 3590 17678493 -
Enter disk name [,list,q,?] rootdisk
You can choose to mirror volumes from disk rootdisk onto any
available disk space, or you can choose to mirror onto a specific
disk. To mirror to a specific disk, select the name of that disk.
To mirror to any available disk space, select "any".
Enter destination disk [,list,q,?] (default: any) rootdg02
The requested operation is to mirror all volumes on disk rootdisk
in disk group rootdg onto available disk space on disk rootdg02.
NOTE: This operation can take a long time to complete.
Continue with operation? [y,n,q,?] (default: y)
Mirror volume rootvol ...
Mirror volume swapvol ...
Mirror volume tmp ...
Mirror volume usr ...
Mirror volume var ...
Mirroring of disk rootdisk is complete.
Mirror volumes on another disk? [y,n,q,?] (default: n)
You can repeat this procedure with the other disk (remove, add back, mirror volumes from the remaining disk), if you don't care for the naming convention used by the Veritas installation tech, like myself. This also gets rid of some of the problems with encapsulated disks, because the removal & re-mirroring will remove any original disk layout in favor of what Veritas likes.
Third step - check your work. 'vxprint' should now show two disk devices (sd) and two logical data devices (plexes) associated with each volume on your root disk.
# vxprint -g rootdg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg rootdg rootdg - - - - - -
dm rootdg02 c0t8d0s2 - 17678493 - - - -
dm rootdisk c0t0d0s2 - 17678493 - - - -
v rootvol root ENABLED 4197879 - ACTIVE - -
pl rootvol-01 rootvol ENABLED 4197879 - ACTIVE - -
sd rootdisk-02 rootvol-01 ENABLED 4197879 0 - - -
pl rootvol-02 rootvol ENABLED 4197879 - ACTIVE - -
sd rootdg02-01 rootvol-02 ENABLED 4197879 0 - - -
There we go! Reboot and make sure everything's happy.
Remember, test this first on your configuration and equipment before trying it - like the Florida presidential recounts, your mileage may vary.
Original content copyright ©1995-2006 Eric Dobbs, except where otherwise noted.
