Forum Discussion

Ziggy1941's avatar
Ziggy1941
Level 4
10 years ago

Problem with HotAdd transport

I have NBU 7.6.1 master standalone server in VMware 5.1 virtual infrastructure (VM with SUSE Linux 11 SP3). When I try to use hotadd transport for backup anyone VM, only the first job (first job after reboot NBU master server) is successful. The following tasks are completed with an error: "ERR - Error opening the snapshot disks using given transport mode: hotadd Status 23".

Example "good" job:

05.03.2015 12:52:05 - Info nbjm(pid=4171) starting backup job (jobid=1541) for client 1c-app.ussc.ru, policy 1c-app, schedule Full  
05.03.2015 12:52:05 - estimated 31373924 Kbytes needed
05.03.2015 12:52:05 - Info nbjm(pid=4171) started backup (backupid=1c-app.ussc.ru_1425540858) job for client 1c-app.ussc.ru, policy 1c-app, schedule Full on storage unit dedup-stu using backup host nbu-s2.ussc.ru
05.03.2015 12:52:06 - Info bpbrm(pid=10823) 1c-app.ussc.ru is the host to backup data from     
05.03.2015 12:52:06 - Info bpbrm(pid=10823) reading file list for client        
05.03.2015 12:52:06 - Info bpbrm(pid=10823) accelerator enabled           
05.03.2015 12:52:06 - started process bpbrm (10823)
05.03.2015 12:52:08 - Info bpbrm(pid=10823) starting bpbkar on client         
05.03.2015 12:52:08 - Info bpbkar(pid=10861) Backup started           
05.03.2015 12:52:08 - Info bpbrm(pid=10823) bptm pid: 10865          
05.03.2015 12:52:08 - Info bptm(pid=10865) start            
05.03.2015 12:52:08 - connecting
05.03.2015 12:52:08 - connected; connect time: 0:00:00
05.03.2015 12:52:09 - Info bptm(pid=10865) using 524288 data buffer size        
05.03.2015 12:52:09 - Info bptm(pid=10865) using 256 data buffers         
05.03.2015 12:52:09 - Info bptm(pid=10865) start backup           
05.03.2015 12:52:20 - begin writing
05.03.2015 12:52:45 - Info bpbkar(pid=10861) 0 entries sent to bpdbm        
05.03.2015 12:52:59 - Info bpbkar(pid=10861) 95000 entries sent to bpdbm        
05.03.2015 12:53:11 - Info bpbkar(pid=10861) 190000 entries sent to bpdbm        
05.03.2015 12:53:21 - Info bpbkar(pid=10861) 285000 entries sent to bpdbm        
05.03.2015 12:53:23 - Info bpbkar(pid=10861) 313599 entries sent to bpdbm        
05.03.2015 12:53:23 - Info bpbkar(pid=10861) 313600 entries sent to bpdbm        
05.03.2015 12:53:23 - Info bpbkar(pid=10861) 313871 entries sent to bpdbm        
05.03.2015 12:53:23 - Info bpbkar(pid=10861) INF - Transport Type =  hotadd      
05.03.2015 12:53:23 - Info bpbkar(pid=10861) 313919 entries sent to bpdbm        
05.03.2015 12:54:01 - Info bpbkar(pid=10861) accelerator sent 382283264 bytes out of 26681522176 bytes to server, optimization 98.6% 
05.03.2015 12:54:01 - Info bpbkar(pid=10861) bpbkar waited 5 times for empty buffer, delayed 1081 times   
05.03.2015 12:54:42 - Info bptm(pid=10865) waited for full buffer 274 times, delayed 4015 times    
05.03.2015 12:54:54 - Info bptm(pid=10865) EXITING with status 0 <----------        
05.03.2015 12:54:54 - Info nbu-s2.ussc.ru(pid=10865) StorageServer=PureDisk:nbu-s2.ussc.ru; Report=PDDO Stats for (nbu-s2.ussc.ru): scanned: 26086164 KB, CR sent: 165799 KB, CR sent over FC: 0 KB, dedup: 99.4%, cache disabled
05.03.2015 12:54:54 - Info bpbrm(pid=10823) validating image for client 1c-app.ussc.ru        
05.03.2015 12:54:54 - Info bpbkar(pid=10861) done. status: 0: the requested operation was successfully completed    
05.03.2015 12:54:54 - end writing; write time: 0:02:34
the requested operation was successfully completed (0)

Exapmle "bad" job:

05.03.2015 12:56:29 - Info nbjm(pid=4171) starting backup job (jobid=1549) for client 1c-app.ussc.ru, policy 1c-app, schedule Full  
05.03.2015 12:56:29 - estimated 31384672 Kbytes needed
05.03.2015 12:56:29 - Info nbjm(pid=4171) started backup (backupid=1c-app.ussc.ru_1425541122) job for client 1c-app.ussc.ru, policy 1c-app, schedule Full on storage unit dedup-stu using backup host nbu-s2.ussc.ru
05.03.2015 12:56:29 - started process bpbrm (13786)
05.03.2015 12:56:29 - Info bpbrm(pid=13786) 1c-app.ussc.ru is the host to backup data from     
05.03.2015 12:56:29 - Info bpbrm(pid=13786) reading file list for client        
05.03.2015 12:56:29 - Info bpbrm(pid=13786) accelerator enabled           
05.03.2015 12:56:30 - Info bpbrm(pid=13786) starting bpbkar on client         
05.03.2015 12:56:30 - Info bpbkar(pid=13824) Backup started           
05.03.2015 12:56:30 - Info bpbrm(pid=13786) bptm pid: 13828          
05.03.2015 12:56:30 - Info bptm(pid=13828) start            
05.03.2015 12:56:30 - connecting
05.03.2015 12:56:30 - connected; connect time: 0:00:00
05.03.2015 12:56:31 - Info bptm(pid=13828) using 524288 data buffer size        
05.03.2015 12:56:31 - Info bptm(pid=13828) using 256 data buffers         
05.03.2015 12:56:31 - Info bptm(pid=13828) start backup           
05.03.2015 12:56:34 - begin writing
05.03.2015 12:58:48 - Error bpbrm(pid=13786) from client 1c-app.ussc.ru: ERR - Error opening the snapshot disks using given transport mode: hotadd Status 23
05.03.2015 12:58:50 - Critical bpbrm(pid=13786) from client 1c-app.ussc.ru: FTL - cleanup() failed, status 6    
05.03.2015 12:58:52 - Error bptm(pid=13828) media manager terminated by parent process       
05.03.2015 12:59:01 - Info nbu-s2.ussc.ru(pid=13828) StorageServer=PureDisk:nbu-s2.ussc.ru; Report=PDDO Stats for (nbu-s2.ussc.ru): scanned: 2 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%, cache disabled
05.03.2015 12:59:01 - Info bpbkar(pid=0) done. status: 6: the backup failed to back up the requested files 
05.03.2015 12:59:01 - end writing; write time: 0:02:27
the backup failed to back up the requested files (6)

I can't understand the reason for this behavior. Please tell me which direction to look for a solution...

7.6
Backing Up
Backup and Recovery
Error messages
Linux
NetBackup
  • mnolan's avatar
    mnolan
    10 years ago

    The reason I was asking was so that you could ensure that you are setup correctly for hotadd.

    If you feel you are setup correctly for HotAdd, you may need to log a support call.

     

    " HotAdd: When running VMware Backup Host on a Virtual Machine, vStorage APIs can take advantage of the SCSI Hot-add capability of the ESX/ESXi server to attach the VMDKs of a Virtual Machine being backed up to the VMware Backup Host. This is referred to as HotAdd transport mode.

    Running the VMware Backup server on a virtual machine has two advantages: it is easy to move a virtual machine around and it can also back up local storage without using the LAN, although this incurs more overhead on the physical ESX/ESXi host than when using SAN transport mode.

    Best practices when using HotAdd:

    • HotAdd works only with virtual machines with SCSI disks and is not supported for backing up virtual machines with IDE disks.
    • A single SCSI controller can have a maximum of 15 disks attached. To run multiple concurrent jobs totally more than 15 disks it is necessary to add more SCSI controllers to the HotAdd host.  The maximum number of 4 SCSI controllers can be added to a HotAdd host, so a total of 60 devices are supported at the maximum.
    • HotAdd requires the VMware Backup Host to have access to datastores where the Virtual Machine being backed up resides. This essentially means:
      • ESX where the VMware backup host is running should have access to datastores where the Virtual Machine being backed up resides. 
      • Both the VMware backup host and Virtual Machine being backed up should be under the same datacenter.
    • HotAdd cannot be used if the VMFS block size of the datastore containing the virtual machine folder for the target virtual machine does not match the VMFS block size of the datastore containing the VMware Backup Host virtual machine. For example, if you back up virtual disk on a datastore with 1MB blocks, the VMware Backup Host must also be on a datastore with 1MB blocks.
    • Restores using HotAdd on a Windows Server 2008 proxy require setting the SAN policy to onlineAll
    • If you are converting a physical machine to a virtual machine with the intention of using hottadd to back up the virtual machine, do not use IDE controllers for any disks that are used during the conversion process.
    • The VMware Backup Host will need the ability to connect to TCP port 902 on ESX/ESXi hosts while using HotAdd for backup/restores."

    VMware Transport Modes: Best practices and troubleshooting

    Article:TECH183072  |  Created: 2012-03-06  |  Updated: 2014-10-08  |  Article URL http://www.symantec.com/docs/TECH183072
  • RiaanBadenhorst's avatar
    RiaanBadenhorst
    Level 6
    10 years ago

    Hi,

     

    You'll have to enable vxms logging on your backup host to see exactly what is causing the 23.

     

    See the note for logging

    http://www.symantec.com/docs/HOWTO70905

  • RiaanBadenhorst's avatar
    RiaanBadenhorst
    Level 6
    10 years ago

    One question, while I look at the logs.... Why do you want to use hotadd? Why not NBD or SAN?
     

  • RiaanBadenhorst's avatar
    RiaanBadenhorst
    Level 6
    10 years ago

    Hi found this in the logs. And these notes that might apply?

    http://www.symantec.com/docs/TECH175235

    http://www.symantec.com/docs/TECH178670 (<<<relates to French but maybe some issue with russian if you're using it)

     

      03/05/2015 17:15:52 : g_vixInterfaceLogger:libvix.cpp:1844 <DEBUG> : [VFM_ESINFO] 2015-03-05T17:15:52.259+05:00 [7FB251508720 verbose 'transport'] Mounting VM vim.VirtualMachine:vm-14476 using transport hotadd failed : .

       03/05/2015 17:15:52 : g_vixInterfaceLogger:libvix.cpp:1844 <DEBUG> : [VFM_ESINFO] 2015-03-05T17:15:52.259+05:00 [7FB251508720 info 'transport'] Deleted file /tmp/vmware-root/421d001f-c0bf-2eb3-d0de-acf2b4046a04-vm-14476/hotadd/unmount.dat

       03/05/2015 17:15:52 : g_vixInterfaceLogger:libvix.cpp:1844 <DEBUG> : [VFM_ESINFO] 2015-03-05T17:15:52.259+05:00 [7FB251508720 info 'transport'] Deleted directory /tmp/vmware-root/421d001f-c0bf-2eb3-d0de-acf2b4046a04-vm-14476/hotadd

       03/05/2015 17:15:52 : g_vixInterfaceLogger:libvix.cpp:1844 <DEBUG> : [VFM_ESINFO] 2015-03-05T17:15:52.259+05:00 [7FB251508720 verbose 'transport'] Cannot use mode hotadd to access [prod-sw7000] 1c-app.ussc.ru/1C Application Server-000002.vmdk: Cannot mount using this method.

       03/05/2015 17:15:52 : vdOpen:VixInterface.cpp:474 <DEBUG> : Done with VixDiskLib_Open(): 0
       03/05/2015 17:15:52 : vdOpen:VixInterface.cpp:495 <ERROR> : VixDiskLib_Open() error: 13
       03/05/2015 17:15:52 : openLeafSnapshotDisks:VixGuest.cpp:478 <DEBUG> : vdOpen() error = 13. Calling closeLeafSnapshotDisks()
       03/05/2015 17:15:52 : log:Error.cpp:249 <WARN> : Error: 0x00000017 occured in file ../VixGuest.cpp, at line 624
       03/05/2015 17:15:52 : log:Error.cpp:249 <WARN> : Error: 0x00000017 occured in file ../VixGuest.cpp, at line 624
       03/05/2015 17:15:52 : vixMapObjCtl:VixCoordinator.cpp:990 <ERROR> : Returning: 23
       03/05/2015 17:15:52 : vix_map_objctl:libvix.cpp:1275 <ERROR> : Returning: 23

  • Ziggy1941's avatar
    Ziggy1941
    Level 4
    10 years ago

    All is simple ) NBD is too slow for our infrastructure - about 20-30 mb/sec. And SAN is impossible because NBU Master deployed on the VMware VM not on the physical server with dedicated FC HBA.

  • Ziggy1941's avatar
    Ziggy1941
    Level 4
    10 years ago

    We use Cisco UCS blade system for virtualization with VMware vSphere.

    I will try to create dedicated FCoE vHBA interfaces and provide them as passthrough devices into NBU VM. Perhaps it will help to use SAN transport. But I 'm not sure about FCoE (

  • Ziggy1941's avatar
    Ziggy1941
    Level 4
    10 years ago

    We leave a total reinstall as a last resort ))

    I thought here, I can manually recreate new VM and mount the same VMDK and RDM disks from old VM. I will try... 

  • Marianne's avatar
    Marianne
    Level 6
    10 years ago

    Are you aware that RDMs cannot be backed up with VMware policy?