Vxdisksetup and encapsulating a disk via vxdiskadm are two very different things.
vxdisksetup:
Vxdisksetup configures the disk for use by Veritas Volume Manager (VxVM). vxdisksetup is called by the vxdiskadd command as part of initial disk configuration. It is not typically necessary to call this command directly.
I believe you are sellecting the root disk encapsulate & mirror in vxdiskadm. Is this for rootdg?
See the guide, page 114:
http://sfdoccentral.symantec.com/sf/5.0/solaris/pdf/vxvm_admin.pdf
From the vxencap man page:
"In the case of partitions that contain file systems, the entries in /etc/vfstab are modified to use the new volume devices. ... A reboot may be necessary to complete the process of encapsulation. "
Encapsulation is only disk/volumes where you are saving data. There are challenges to growing and shrinking encapsulated data. (until 5.0mp3). So, if it's not the boot disks, it's probably better to create a new volume & filesystem, copy the data, umount the old, remount the new in it's place, and change the mount device in /etc/vfstab.