VMware backups are failing with status 23 and 6 after Vcenter server upgrade from 5.1 to 5.5 U3B

[kams_240]

Issue:-

VMware backups are failing with status 23 and 6 after Vcenter server upgrade from 5.5 U 3B

As per the Netbackup VMware comatibility Matrix, we upgraded our Master and VMware backup host (Windows) from 7.5.0.4 to 7.6.0.4 to support Vsphere 5.5 U 3B

https://sort.veritas.com/checklist/install/nbu_compatibilityListUpdates

Description of Change	Date	NetBackup Version Start of Support
Added support for vSphere 5.5 U3	10/22/2015	NetBackup 7.6.0.2

After the upgrade to Netbackup 7.6.0.4 - VADP backups were started failing with status 6 (23) and later we came to know that Vsphere 5.5 U3b is not supported with Netbackup 7.6.0.2 and there is no official annoncement yet from Veritas related to the support for 5.5 U3b

VxMS Logs:-

 04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [51500 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:000000000246f1a0, TCP:server-vc-01:443>; cnx: (null), error: class Vmacore::Ssl::SSLException(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)

   04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [51500 trivia 'Default'] Setting error in state 1 : class Vmacore::Ssl::SSLException(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)

   04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [51500 trivia 'ThreadPool'] HandleWork() leaving

   04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [51500 trivia 'HttpConnectionPool-000000'] [DecConnectionCount] Number of connections to <cs p:000000000246f1a0, TCP:pru-swi-vc-01:443> decremented to 0

   04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [49468 trivia 'transport'] Initializing SSL context

   04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [59120 trivia 'ThreadPool'] ThreadPool[idle:4, busy_io:0, busy_long:0] HandleWork(type: 0, fun: class boost::_bi::bind_t<void,class boost::_mfi::mf3<void,class Vmacore::System::ThreadPoolAsio,class boost::shared_ptr<class Vmacore::Exception> const & __ptr64,int,class Vmacore::Functor<void,class Vmacore::Exception * __ptr64,int,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil> const & __ptr64>,class boost::_bi::list4<class boost::_bi::value<class Vmacore::Ref<class Vmacore::System::ThreadPoolAsio> >,class boost::_bi::value<class boost::shared_ptr<class Vmacore::Exception> >,class boost::_bi::value<int>,class boost::_bi::value<class Vmacore::Functor<void,class Vmacore::Exception * __ptr64,int,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil> > > >)

   04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [49468 info 'transport'] Connecting to host server-vc-01 on port 443 using protocol https via vim.version.version3

Log from the KM article:- https://www.veritas.com/support/en_US/article.TECH228283 (w.r.t. this article fix for this issue is available from 7.6.1.1, but there is no such document from Veritas saying that Vsphere 5.5 U3b suite is supported from 7.6.1.1

    14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:0000000002854f00, TCP:vcenter5:443>; cnx: (null), error: class Vmacore::Ssl::SSLException(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)

           14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 trivia 'Default'] Setting error in state 1 : class Vmacore::Ssl::SSLException(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)

           14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 trivia 'vmomi.soapStub[1]'] Request completed [class Vmacore::Http::UserAgentImpl::AsyncSendRequestHelper:0000000003562E98]

           14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 trivia 'ThreadPool'] HandleWork() leaving

           14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 trivia 'HttpConnectionPool-000000'] [DecConnectionCount] Number of connections to <cs p:0000000002854f00, TCP:vcenter5:443> decremented to 0

           14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [07260 trivia 'ThreadPool'] ThreadPool[idle:4, busy_io:0, busy_long:0] HandleWork(type: 0, fun: class boost::_bi::bind_t<void,class boost::_mfi::mf3<void,class Vmacore::System::ThreadPoolAsio,class boost::shared_ptr<class Vmacore::Exception> const & __ptr64,int,class Vmacore::Functor<void,class Vmacore::Exception * __ptr64,int,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil> const & __ptr64>,class boost::_bi::list4<class boost::_bi::value<class Vmacore::Ref<class Vmacore::System::ThreadPoolAsio> >,class boost::_bi::value<class boost::shared_ptr<class Vmacore::Exception> >,class boost::_bi::value<int>,class boost::_bi::value<class Vmacore::Functor<void,class Vmacore::Exception * __ptr64,int,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil> > > >)

Update from Veritas Support:-

1. Customers quite often misinterpret our compatibility lists that show 5.5 U3 as supported from 7.6.0.2 onwards. After releasing U3B VMware decided to rename U3 to U3A which our support statement applies to – but not 3B.

2. Currently Netbackup is not supporting Vsphere 5.5 3b version. With the vSphere 5.5 update 3b release, the support for SSLv3 is disabled by default for all services and ports as a protection against the POODLE vulnerability.  Update 3b disables SSLv3 and breaks VDDK communications to vCenter and ESXi. There is a workaround for this problem which might work, but it is still unsupported. You would have to manually re-enable SSLv3 protocol on port 902 on ESXi and port 443 on vCenter. Please refer to TN below for more information. I would have to reiterate, the workaround may be considered as an option instead of rolling back Vcenter upgrade but it is still unsupported by Netbackup.
 

Workaround/Solution:-

1. Do not disable SSLv3 in vCenter until all VMware backup hosts are upgraded to NetBackup version 7.6.1.1 includes VDDK 5.5.4 or above.

2. Upgraded our Master and VMware backup host to 7.6.1.2 and backups are working fine (Our Solution)

Knowledge Base Articles Referred:-

https://www.veritas.com/support/en_US/article.TECH228283

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2139396

https://www.veritas.com/support/en_US/article.000025159

[Marianne]

It seems that you have received good advice from Support. Are you looking for alternative suggestions here or just sharing information? My advice is not to waste time with any version up to 7.6.x. Upgrade straight to 7.7.1. All versions up to 7.6.x will reach EOSL in a year from now.

[sdo]

Great post.  Thank you Kams.

[kams_240]

@Mariaane - Thanks - At present we can't upgrade our backup enviornment to 7.7.1 due to the DPA compatibility with NBU 7.7.1. Once we upgrade our DPA to the latest version 6.1.x we we will plan the upgrade of our backup infrastructure to 7.7.1

Issue Statement:- I have already asked Veritas support and Businness Account Manager to update the Netbackup 7.x VMware comatiblity document to clearly mention the support for Vsphere 5.5 U3A and Vsphere 5.5 U3B with 7.6.0.2 and 7.6.0.1 onward.