NBA upgrade to 5.3 passes AURA, but fails to upgrade

[TimWillingham]

I've got a case opened with Veritas for this, just wanted to see if anyone else is experiencing this.  Based on the telemetry log output, it appears the logic for the upgrade check passed is reversed.

appliance.Software> Install VRTS_NBAPP_update-5.3-1.x86_64.rpm
>> Before you upgrade this appliance, make sure you have performed the required tasks described in the pre-upgrade cheklist article: https://www.veritas.com/support/en_US/article.100052181.html
Have you performed the required pre-upgrade tasks? [yes, no] (no)yes
>> Before you upgrade this appliance, you must first download and run the latest version of the Appliance Upgrade Readness Analyzer. The analyzer tool results must show that the appliance is ready for an upgrade. Refer to the following aticle to download the latest version of Upgrade Readiness Analyzer: https://www.veritas.com/content/support/en_US/downlads/update.UPD773233.
Have you run the latest version of the Upgrade Readiness Analyzer tool on this appliance? [yes, no] (no)yes
- [Info] Call Home connectivity state check passed.
- [Warning] Do not restart or power off the appliance manually or change any of the appliance configurations and servic states while the upgrade is in-progress.

You are about to begin the upgrade to NetBackup appliance software version 5.3.
Before you continue, be aware of the following changes and behaviors that occur during and after the upgrade:
-During the upgrade, NetBackup services and the Appliance Web Console are unavailable.
-After the upgrade, any previously existing user checkpoints are deleted.
-Any appliance configuration changes made by elevating to the OS (root) prompt are not preserved.
-During the upgrade, the SDCS mode is set to the 'unmanaged' mode. If your previous SDCS configuration was set to the anaged mode, make sure to change SDCS to the managed mode and apply the latest IDS and IPS policies after the upgrade hs completed.
-The appliance restarts automatically as part of the upgrade process.
- [Info] Checking free inode number.............................................. [OK]
- [Info] Checking system variable max_user_watches............................... [OK]
- [Info] Checking system space................................................... [OK]
- [Info] Checking MSDP space..................................................... [OK]
- [Info] Checking appliance storage status....................................... [OK]
- [Info] Checking appliance infrastructure services status....................... [OK]
- [Info] Checking last run of Appliance Upgrade Readiness Analyzer............... [OK]
To complete the upgrade, this media server requires a CA certificate and a host ID-based certificate. To deploy the host ID-based certificate, a token may also be required based on the security level of the associated primary server. Refer to the NetBackup security certificate topics for instructions.
>> The upgrade preflight checks are complete and the system is ready to begin the upgrade.
Do you want to continue? [yes, no] (yes)
>> Before you can proceed with the upgrade, you must answer the following questions to determine how you want the upgrade process to respond if errors occur during the upgrade. You can select to have the upgrade process start a rollback immediately after detecting errors (yes), or select to keep the upgrade process paused while you investigate the errors (no).
If an error occurs during the upgrade, do you want to immediately enforce an automatic rollback? [yes, no] (yes)
To continue with the upgrade, verify the following CA certificate detail and enter "yes" to trust the CA certificate.
CA Certificate Details:
Subject Name : /CN=nbatd/OU=root@primary.domain.com/O=vx
Start Date : Jan 14 19:54:39 2014 GMT
Expiry Date : Jan 09 21:09:39 2034 GMT
SHA-1 Fingerprint : <cleared for security>
SHA-256 Fingerprint : <cleared for security>
Key Strength : 1024
Subject Key Identifier : <cleared for security>

>> Do you want to trust the CA certificate? [yes, no](yes)
An unknown error has occurred. Contact Veritas Support to resolve the issue.
Cannot deploy the host ID-based certificate on the appliance
- [Info] The software upgrade has been aborted!

Excerpt from upgrade_telemetry.log:

2024-01-04T16:12:42.949-05:00 (UTC-5): INFO : aim_status_json_str: {"steps_def": [{"PRE_FLIGHT_CHECK": "Performing the preflight check"}, {"PRE_UPGRADE_SELFTEST": "Performing the pre-upgrade self-test"}, {"CREATE_PRE_UPGRADE_CHECKPOINT": "Creating the pre-upgrade checkpoint"}, {"PRE_UPGRADE_PROCESSING": "Performing the pre-upgrade tasks"}, {"OS_INSTALLATION": "Upgrading the operating system"}, {"UPGRADE_NBU": "Upgrading NetBackup"}, {"POST_UPGRADE_PROCESSING": "Performing the post-upgrade tasks"}, {"POST_UPGRADE_SELFTEST": "Performing the post-upgrade self-test"}, {"COMPLETING_UPGRADE": "Completing the upgrade"}], "status": [{"step": "PRE_FLIGHT_CHECK", "step_status": "FAILED", "cur_time": "Thu Jan 4 16:10:16 2024", "estimated_time": "3", "actual_time": "0:02:14.119378", "details": [{"short_msg": "Appliance upgrade preflight check started.", "UMI": "V-409-776-30068", "severity": "Info", "long_msg": "Appliance upgrade preflight check started."}, {"short_msg": "rpm name is VRTS_NBAPP_update-5.3-1.x86_64.rpm", "UMI": "V-409-777-1860", "severity": "Info", "long_msg": "rpm name is VRTS_NBAPP_update-5.3-1.x86_64.rpm"}, {"short_msg": "START FROM 5.1.1MR2 TO 5.3", "UMI": "V-409-777-1870", "severity": "Info", "long_msg": "START FROM 5.1.1MR2 TO 5.3"}, {"short_msg": "Checking free inode number passed.", "UMI": "V-409-776-30100", "severity": "Info", "long_msg": "Checking free inode number passed."}, {"short_msg": "Checking system variable max_user_watches passed.", "UMI": "V-409-777-9250", "severity": "Info", "long_msg": "Checking system variable max_user_watches passed."}, {"short_msg": "System free space check passed.", "UMI": "V-409-777-1720", "severity": "Info", "long_msg": "System free space check passed."}, {"short_msg": "MSDP free space check passed.", "UMI": "V-409-777-1080", "severity": "Info", "long_msg": "MSDP free space check passed."}, {"short_msg": "Catalog free space check passed.", "UMI": "V-409-777-1840", "severity": "Info", "long_msg": "Catalog free space check passed."}, {"short_msg": "Appliance storage status check passed.", "UMI": "V-409-777-1120", "severity": "Info", "long_msg": "Appliance storage status check passed."}, {"short_msg": "Checking appliance infrastructure services status passed.", "UMI": "V-409-777-9180", "severity": "Info", "long_msg": "Checking appliance infrastructure services status passed."}, {"short_msg": "Checking last run of Appliance Upgrade Readiness Analyzer passed.", "UMI": "V-409-777-9210", "severity": "Info", "long_msg": "Checking last run of Appliance Upgrade Readiness Analyzer passed."}, {"short_msg": "Appliance upgrade preflight check failed.", "UMI": "V-409-777-1002", "severity": "Error", "long_msg": "Appliance upgrade preflight check failed."}]}], "summary": {"steps_total_time": 105, "model": "5240", "host_name": "appliance_name", "serial_id": "VTASNUM", "time_zone": "EST", "target_vxos_version": "2.0", "cur_nbu_version": "10.1.1", "cur_vxos_version": "8.6.21", "target_nbu_version": "10.3", "elapsed_time": 0, "percentage_completed": "5%", "target_nba_version": "5.3", "target_sp_version": "", "cur_nba_version": "5.1.1", "cur_sp_version": "2"}}
2024-01-04T16:12:42.953-05:00 (UTC-5): INFO : (cmd=/bin/date +%Y-%m-%dT%H:%M:%S.%N%z,status_code=0)
2024-01-04T16:12:42.953-05:00 (UTC-5): INFO : Command output in run_command is b'2024-01-04T16:12:42.953146376-0500\n'
2024-01-04T16:12:42.954-05:00 (UTC-5): INFO : event: {"eventType": "TRACE_EVENT", "eventDetails": {"state": "ERROR", "step": "PRE_FLIGHT_CHECK", "status": "FAILED", "upgrade_id": "1704402615", "umiCode": "V-409-777-1002", "umiMessage": "[Error] V-409-777-1002: Appliance upgrade preflight check failed."}, "collectorComponent": {"serialNumber": "VTASNUM", "timestamp": "2024-01-04T16:12:42.953146376-0500", "componentId": "PRE_FLIGHT_CHECK", "type": "NBA Upgrade", "properties": {"step": "PRE_FLIGHT_CHECK", "status": "FAILED", "upgrade_id": "1704402615", "umiMessage": "[Error] V-409-777-1002: Appliance upgrade preflight check failed."}}}
2024-01-04T16:12:42.963-05:00 (UTC-5): INFO : (cmd=/inst/patch/appliance/asc-telemetry/asc-telemetry/asc config -g upgradetelemetrylocation,status_code=0)
2024-01-04T16:12:42.963-05:00 (UTC-5): INFO : Command output in run_command is b'/log/NetBackupAppliance_upgrade_20240104160934\n'
2024-01-04T16:12:42.964-05:00 (UTC-5): INFO : write event to file /log/NetBackupAppliance_upgrade_20240104160934/events/event_1704402762.9639933.json.tmp
2024-01-04T16:12:43.161-05:00 (UTC-5): INFO : in store_event()
2024-01-04T16:12:43.161-05:00 (UTC-5): INFO : LOG_PATH: /inst/patch/appliance/upgrading/aim_status_log.json
2024-01-04T16:12:43.171-05:00 (UTC-5): INFO : (cmd=/inst/patch/appliance/asc-telemetry/asc-telemetry/asc config -g upgradetelemetrylocation,status_code=0)
2024-01-04T16:12:43.171-05:00 (UTC-5): INFO : Command output in run_command is b'/log/NetBackupAppliance_upgrade_20240104160934\n'
2024-01-04T16:12:43.171-05:00 (UTC-5): INFO : no act_req_id_file, skip to read: [Errno 2] No such file or directory: '/log/NetBackupAppliance_upgrade_20240104160934/act_req_id'
2024-01-04T16:12:43.172-05:00 (UTC-5): INFO : aim_status_json_str: {"steps_def": [{"PRE_FLIGHT_CHECK": "Performing the preflight check"}, {"PRE_UPGRADE_SELFTEST": "Performing the pre-upgrade self-test"}, {"CREATE_PRE_UPGRADE_CHECKPOINT": "Creating the pre-upgrade checkpoint"}, {"PRE_UPGRADE_PROCESSING": "Performing the pre-upgrade tasks"}, {"OS_INSTALLATION": "Upgrading the operating system"}, {"UPGRADE_NBU": "Upgrading NetBackup"}, {"POST_UPGRADE_PROCESSING": "Performing the post-upgrade tasks"}, {"POST_UPGRADE_SELFTEST": "Performing the post-upgrade self-test"}, {"COMPLETING_UPGRADE": "Completing the upgrade"}], "status": [{"step": "PRE_FLIGHT_CHECK", "step_status": "FAILED", "cur_time": "Thu Jan 4 16:10:16 2024", "estimated_time": "3", "actual_time": "0:02:14.119378", "details": [{"short_msg": "Appliance upgrade preflight check started.", "UMI": "V-409-776-30068", "severity": "Info", "long_msg": "Appliance upgrade preflight check started."}, {"short_msg": "rpm name is VRTS_NBAPP_update-5.3-1.x86_64.rpm", "UMI": "V-409-777-1860", "severity": "Info", "long_msg": "rpm name is VRTS_NBAPP_update-5.3-1.x86_64.rpm"}, {"short_msg": "START FROM 5.1.1MR2 TO 5.3", "UMI": "V-409-777-1870", "severity": "Info", "long_msg": "START FROM 5.1.1MR2 TO 5.3"}, {"short_msg": "Checking free inode number passed.", "UMI": "V-409-776-30100", "severity": "Info", "long_msg": "Checking free inode number passed."}, {"short_msg": "Checking system variable max_user_watches passed.", "UMI": "V-409-777-9250", "severity": "Info", "long_msg": "Checking system variable max_user_watches passed."}, {"short_msg": "System free space check passed.", "UMI": "V-409-777-1720", "severity": "Info", "long_msg": "System free space check passed."}, {"short_msg": "MSDP free space check passed.", "UMI": "V-409-777-1080", "severity": "Info", "long_msg": "MSDP free space check passed."}, {"short_msg": "Catalog free space check passed.", "UMI": "V-409-777-1840", "severity": "Info", "long_msg": "Catalog free space check passed."}, {"short_msg": "Appliance storage status check passed.", "UMI": "V-409-777-1120", "severity": "Info", "long_msg": "Appliance storage status check passed."}, {"short_msg": "Checking appliance infrastructure services status passed.", "UMI": "V-409-777-9180", "severity": "Info", "long_msg": "Checking appliance infrastructure services status passed."}, {"short_msg": "Checking last run of Appliance Upgrade Readiness Analyzer passed.", "UMI": "V-409-777-9210", "severity": "Info", "long_msg": "Checking last run of Appliance Upgrade Readiness Analyzer passed."}, {"short_msg": "Appliance upgrade preflight check failed.", "UMI": "V-409-777-1002", "severity": "Error", "long_msg": "Appliance upgrade preflight check failed."}]}], "summary": {"steps_total_time": 105, "model": "5240", "host_name": "appliance_name", "serial_id": "VTASNUM", "time_zone": "EST", "target_vxos_version": "2.0", "cur_nbu_version": "10.1.1", "cur_vxos_version": "8.6.21", "target_nbu_version": "10.3", "elapsed_time": 0, "percentage_completed": "5%", "target_nba_version": "5.3", "target_sp_version": "", "cur_nba_version": "5.1.1", "cur_sp_version": "2"}}

[jeanpaul_douay]

Hi,

I have same error, i found this erros into /log/nbapp_seccom.log

[exec_sys_cmd] /inst/patch/incoming/nbcertcmdtool/NB_10.3_0062/linuxR_x86_3.10.0/nbcertcmdtool -atLibPath /inst/patch/incoming/nbcertcmdtool/NB_10.3_0062/linuxR_x86_3.10.0 -getCertificate
2024-02-19 15:19:41.239981
{
"Command": "/inst/patch/incoming/nbcertcmdtool/NB_10.3_0062/linuxR_x86_3.10.0/nbcertcmdtool -atLibPath /inst/patch/incoming/nbcertcmdtool/NB_10.3_0062/linuxR_x86_3.10.0 -getCertificate",
"stdout": "Host certificate and certificate revocation list already exist for master server [csm-ntb-lp01]\nUse the -force option to overwrite the existing certificate.\n",
"stderr": "double free or corruption (fasttop)\n",
"returncode": -6
}
2024-02-19 15:19:41.240499
token_message=An unknown error has occurred. Contact Veritas Support to resolve the issue.

[TimWillingham]

The workaround is to open another CLISH window and manually stop the NBU services before answering the final question to start the update.  I also upgraded from 9.2.0.1 to 10.3 without issues, so it appears to be a problem with 10.1.1 only.