VMware backups are failing with status 23 and 6 after Vcenter server upgrade from 5.1 to 5.5 U3B
Issue:-
VMware backups are failing with status 23 and 6 after Vcenter server upgrade from 5.5 U 3B
As per the Netbackup VMware comatibility Matrix, we upgraded our Master and VMware backup host (Windows) from 7.5.0.4 to 7.6.0.4 to support Vsphere 5.5 U 3B
https://sort.veritas.com/checklist/install/nbu_compatibilityListUpdates
| Description of Change | Date | NetBackup Version Start of Support |
| Added support for vSphere 5.5 U3 | 10/22/2015 | NetBackup 7.6.0.2 |
After the upgrade to Netbackup 7.6.0.4 - VADP backups were started failing with status 6 (23) and later we came to know that Vsphere 5.5 U3b is not supported with Netbackup 7.6.0.2 and there is no official annoncement yet from Veritas related to the support for 5.5 U3b
VxMS Logs:-
04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [51500 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:000000000246f1a0, TCP:server-vc-01:443>; cnx: (null), error: class Vmacore::Ssl::SSLException(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [51500 trivia 'Default'] Setting error in state 1 : class Vmacore::Ssl::SSLException(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [51500 trivia 'ThreadPool'] HandleWork() leaving
04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [51500 trivia 'HttpConnectionPool-000000'] [DecConnectionCount] Number of connections to <cs p:000000000246f1a0, TCP:pru-swi-vc-01:443> decremented to 0
04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [49468 trivia 'transport'] Initializing SSL context
04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [59120 trivia 'ThreadPool'] ThreadPool[idle:4, busy_io:0, busy_long:0] HandleWork(type: 0, fun: class boost::_bi::bind_t<void,class boost::_mfi::mf3<void,class Vmacore::System::ThreadPoolAsio,class boost::shared_ptr<class Vmacore::Exception> const & __ptr64,int,class Vmacore::Functor<void,class Vmacore::Exception * __ptr64,int,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil> const & __ptr64>,class boost::_bi::list4<class boost::_bi::value<class Vmacore::Ref<class Vmacore::System::ThreadPoolAsio> >,class boost::_bi::value<class boost::shared_ptr<class Vmacore::Exception> >,class boost::_bi::value<int>,class boost::_bi::value<class Vmacore::Functor<void,class Vmacore::Exception * __ptr64,int,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil> > > >)
04:45:29.0441 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2016-01-19T04:45:29.441Z [49468 info 'transport'] Connecting to host server-vc-01 on port 443 using protocol https via vim.version.version3
Log from the KM article:- https://www.veritas.com/support/en_US/article.TECH228283 (w.r.t. this article fix for this issue is available from 7.6.1.1, but there is no such document from Veritas saying that Vsphere 5.5 U3b suite is supported from 7.6.1.1
14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 error 'HttpConnectionPool-000000'] [ConnectComplete] Connect failed to <cs p:0000000002854f00, TCP:vcenter5:443>; cnx: (null), error: class Vmacore::Ssl::SSLException(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 trivia 'Default'] Setting error in state 1 : class Vmacore::Ssl::SSLException(SSL Exception: error:140000DB:SSL routines:SSL routines:short read)
14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 trivia 'vmomi.soapStub[1]'] Request completed [class Vmacore::Http::UserAgentImpl::AsyncSendRequestHelper:0000000003562E98]
14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 trivia 'ThreadPool'] HandleWork() leaving
14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [06836 trivia 'HttpConnectionPool-000000'] [DecConnectionCount] Number of connections to <cs p:0000000002854f00, TCP:vcenter5:443> decremented to 0
14:58:36.0093 : g_vixInterfaceLogger:libvix.cpp:1825 <DEBUG> : [VFM_ESINFO] 2015-02-11T14:58:36.093-05:00 [07260 trivia 'ThreadPool'] ThreadPool[idle:4, busy_io:0, busy_long:0] HandleWork(type: 0, fun: class boost::_bi::bind_t<void,class boost::_mfi::mf3<void,class Vmacore::System::ThreadPoolAsio,class boost::shared_ptr<class Vmacore::Exception> const & __ptr64,int,class Vmacore::Functor<void,class Vmacore::Exception * __ptr64,int,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil> const & __ptr64>,class boost::_bi::list4<class boost::_bi::value<class Vmacore::Ref<class Vmacore::System::ThreadPoolAsio> >,class boost::_bi::value<class boost::shared_ptr<class Vmacore::Exception> >,class boost::_bi::value<int>,class boost::_bi::value<class Vmacore::Functor<void,class Vmacore::Exception * __ptr64,int,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil,class Vmacore::Nil> > > >)
Update from Veritas Support:-
1. Customers quite often misinterpret our compatibility lists that show 5.5 U3 as supported from 7.6.0.2 onwards. After releasing U3B VMware decided to rename U3 to U3A which our support statement applies to – but not 3B.
2. Currently Netbackup is not supporting Vsphere 5.5 3b version. With the vSphere 5.5 update 3b release, the support for SSLv3 is disabled by default for all services and ports as a protection against the POODLE vulnerability. Update 3b disables SSLv3 and breaks VDDK communications to vCenter and ESXi. There is a workaround for this problem which might work, but it is still unsupported. You would have to manually re-enable SSLv3 protocol on port 902 on ESXi and port 443 on vCenter. Please refer to TN below for more information. I would have to reiterate, the workaround may be considered as an option instead of rolling back Vcenter upgrade but it is still unsupported by Netbackup.
Workaround/Solution:-
1. Do not disable SSLv3 in vCenter until all VMware backup hosts are upgraded to NetBackup version 7.6.1.1 includes VDDK 5.5.4 or above.
2. Upgraded our Master and VMware backup host to 7.6.1.2 and backups are working fine (Our Solution)
Knowledge Base Articles Referred:-
https://www.veritas.com/support/en_US/article.TECH228283
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2139396
https://www.veritas.com/support/en_US/article.000025159
