"Failed to Create Shared Memory" error when installing NBU in a Solaris 10 zone

I persume you run Solaris.  You may check the Netbackup installation guide - there are some required kernel tuneables regarding shared memory and Message Queue.

This is a cut & past from the installation guide page 15:

Message Queue parameters: It may be necessary to increase the system’s
message queue resources to avoid having NetBackup daemons hang.
For example, you may need to edit the /etc/system file to include or
change the following parameters:

set msgsys:msginfo_msgmap=512
set msgsys:msginfo_msgmax=8192
set msgsys:msginfo_msgmnb=65536
set msgsys:msginfo_msgmni=256
set msgsys:msginfo_msgssz=16
set msgsys:msginfo_msgtql=512
set msgsys:msginfo_msgseg=8192

After editing the file, you must reboot the NetBackup server.
■     Semaphore parameters: You must allocate enough semaphores on UNIX
systems, or errors may occur. System requirements vary, but the following
changes to the /etc/system file should be sufficient for an average
system:
set semsys:seminfo_semmap=64
set semsys:seminfo_semmni=1024
set semsys:seminfo_semmns=1024
set semsys:seminfo_semmnu=1024
set semsys:seminfo_semmsl=300
set semsys:seminfo_semopm=32
set semsys:seminfo_semume=64

After editing the file, you must reboot the NetBackup server.