12-17-2014 04:03 AM
Hi Friends,
My name is Priyeranjan and today I am facing an upgrade issue with Netbackup Appliance 5230 media from version 2.6.0.1 to 2.6.0.4.
I started upgrade from Manage / Software / Install pkg name
Upgrade process was submittd properly with all required information, later on when i am checking the "Upgrade Status" I am getting bellow message and errors.
dc-nbmedia01.Software> UpgradeStatus
The appliance is creating a pre-upgrade checkpoint!
dc-nbmedia01.Software> UpgradeStatus
The appliance is creating a pre-upgrade checkpoint!
dc-nbmedia01.Software> UpgradeStatus
The appliance is creating a pre-upgrade checkpoint!
dc-nbmedia01.Software> UpgradeStatus
Failed to create the pre-upgrade checkpoint, please resolve this issue first!
dc-nbmedia01.Software>
Can you please help me to verify full logging for the upgrade as I am unable to find.
Or if you have already faced similar kind of issue please help me with your recomendation..
Looking forward your kind help.
Thank you.
Regards,
Priyeranjan
12-17-2014 06:26 AM
This is interesting. Quick question. Have you ever upgraded this appliance before? Only one of each checkpoint can exist on an appliance at a time so you will have pre, post, user and factory checkpoints when doing an upgrade. I suppose it could be possible there is an existing checkpoint out there however unlikely it may seem. You may need to contact support to go through the checkpoint logs
12-17-2014 07:32 AM
Hi Jeff,
Many thanks for reverting back. I have checked the existing check point through Support / Checkpoint on the Appliance CLISH login and i did not find any previous checkpoints there then i tried checkpoint rollback and rollback also did not find previous created files.
One question i want to ask you - How can i verify the path where checkpoint will store the information during upgrade - I am suspectig might be space issue is causing this upgrade failure.
I tried to create manual checkpoint and have not seen any progress on that.
Looking forward your recomendation in this regard.
Thank you.
Regards,
Priyeranjan
12-17-2014 12:15 PM
Hi Priyeranjan... I am inclined to agree with you regarding the space issue. I wouldn't think there would have been any checkpoints on your appliance but it was worth a look.. I do not know for certain the location of the checkpoints. I will research that as soon as I can find time but if I had to guess the checkpoints would be on the mirrored root drive. Again until I find out for sure that is only a guess. I'll get back to you.
12-17-2014 02:06 PM
Hi Priyeranjan .. The checkpoints would be written to the following path:
/repository/checkpoints
12-17-2014 04:38 PM
To see the log for checkpoints, you can run this from root:
vxlogview -p 409 -o NBAPP_CKPT -d all
My guess is that there are sparse *.vmdk* files left over in /tmp that are making the checkpoint much larger than it should be. I would elevate to root then check /tmp
nbmaster:~ # ls -lash /tmp | less
The 6th field is the on-disk size of the file. For example:
0 -rw-r--r-- 1 root root 100G Mar 23 23:24 VM-name_1-000004.vmdk_1_1750811607
I would delete the sparse files and re-attempt the upgrade.
nbmaster:~ # rm /tmp/*.vmdk*
12-17-2014 11:48 PM
Hi Jeff,
I tried to create manual checkpoint and it has thrown an error. Can you have a look on below checkpoint errors and provide your fedback..
dc-nbmedia01.Support> Checkpoint
Create Rollback
dc-nbmedia01.Support> Checkpoint Rollback
- [Info] Disable interrupt (CTRL-C), quit, and suspend signals
+-----------------------+
| Appliance Rollback |
+-----------------------+
Rolling back to an Appliance Checkpoint will restore the system back to the checkpoint's point-in-time. This can help undo any misconfiguration or system failures that might have occurred.
Rolling back to an Appliance Checkpoint will revert the following components:
1) Appliance Operating System
2) Appliance Software
3) NetBackup Software
4) Networking Configuration
5) Any previously applied patches
6) Backup data is not reverted
- [Info] Running rollback validation...please wait.
- [Info] Rollback validation successful.
There are no Appliance Checkpoints available for rollback. Checkpoints can be created via the appliance shell menu's "Support->Checkpoint create" command.
dc-nbmedia01.Support> Checkpoint Rollback
- [Info] Disable interrupt (CTRL-C), quit, and suspend signals
+-----------------------+
| Appliance Rollback |
+-----------------------+
Rolling back to an Appliance Checkpoint will restore the system back to the checkpoint's point-in-time. This can help undo any misconfiguration or system failures that might have occurred.
Rolling back to an Appliance Checkpoint will revert the following components:
1) Appliance Operating System
2) Appliance Software
3) NetBackup Software
4) Networking Configuration
5) Any previously applied patches
6) Backup data is not reverted
- [Info] Running rollback validation...please wait.
- [Info] Rollback validation successful.
There are no Appliance Checkpoints available for rollback. Checkpoints can be created via the appliance shell menu's "Support->Checkpoint create" command.
dc-nbmedia01.Support>
dc-nbmedia01.Support> Checkpoint
Create Rollback
dc-nbmedia01.Support> Checkpoint Create
- [Info] Disable interrupt (CTRL-C), quit, and suspend signals
+-------------------------------+
| Create Appliance Checkpoint |
+-------------------------------+
Creating an Appliance Checkpoint allows the user to easily rollback the entire s that might have occurred. An Appliance Checkpoint captures the following compone
1) Appliance Operating System
2) Appliance Software
3) NetBackup Software
4) Networking Configuration
5) Any previously applied patches
6) Backup data is not reverted
- [Info] Running create checkpoint validation...please wait
- [Info] Create checkpoint validation successful.
There are no checkpoints in the system.
There is no user checkpoint. Please continue to create a user checkpoint
>> Would you like to proceed? (yes/no) yes
>> Checkpoint description (OPTIONAL): Manual before upgrade
>> Create checkpoint? (yes/no) yes
- [Info] No existing USER checkpoint was detected.
- [Info] CREATING USER CHECKPOINT
- [Info] Creating checkpoint. This operation can take up to 15 minutes.
Please wait...
^C
^C^C^C^C^C^C^C^C^C^C
- [Info] Checkpoint creation failed. Performing cleanup...
- [Error] On node dc-nbmedia01:
- [Error] Error check pointing volume '/dev/system/root'. Failed to create tarball.
- [Error] Error creating snapshot volume /dev/system/root_SNAP_USER_2014_12_17_21_07_36.
- [Error] On node dc-nbmedia01:
- [Error] Error check pointing volume '/dev/sda2'. Failed to create tarball.
- [Error] Error creating snapshot volume .
- [Error] On node dc-nbmedia01:
- [Error] Failed to create checkpoint. Removing partial snapshots..
- [Error] Appliance Checkpoint creation failed. Retry again once errors are resolved. Please see the Symantec NetBackup Appliance Administrator's Guide for mre troubleshooting information.
dc-nbmedia01.Support>
12-18-2014 02:12 AM
Hi Jeff,
I have verified the avilable space, Can you please have a look on below output.
dc-nbmedia01:/home/maintenance # df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root
92G 6.2G 81G 8% /
udev 63G 592K 63G 1% /dev
tmpfs 63G 76K 63G 1% /dev/shm
/dev/sda2 198M 28M 160M 15% /boot
/dev/mapper/system-inst
50G 26G 21G 56% /inst
/dev/mapper/system-log
367G 56G 293G 16% /log
/dev/mapper/system-rep
131G 5.6G 119G 5% /repository
tmpfs 4.0K 0 4.0K 0% /dev/vx
/dev/vx/dsk/nbuapp/pdvol
25T 14T 12T 55% /disk
/dev/vx/dsk/nbuapp/cfgvol
25G 3.8G 22G 16% /config
/dev/vx/dsk/nbuapp/advol
15T 262G 15T 2% /advanceddisk/dp1/advol
/dev/mapper/system-rescue
9.7G 5.6G 3.6G 62% /repository/checkpoints/appliance_checkpoint_kr3Vbs
/dev/mapper/system-rescue
9.7G 5.6G 3.6G 62% /repository/checkpoints/appliance_checkpoint_g1Ob2C
/dev/mapper/system-rescue
9.7G 5.6G 3.6G 62% /repository/checkpoints/appliance_checkpoint_bTChqL
dc-nbmedia01:/home/maintenance #
Rgards,
Priyeranjan
12-18-2014 02:14 AM
Hi Jeff,
Can you please have a look on below output.
dc-nbmedia01:/home/maintenance # df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root
92G 6.2G 81G 8% /
udev 63G 592K 63G 1% /dev
tmpfs 63G 76K 63G 1% /dev/shm
/dev/sda2 198M 28M 160M 15% /boot
/dev/mapper/system-inst
50G 26G 21G 56% /inst
/dev/mapper/system-log
367G 56G 293G 16% /log
/dev/mapper/system-rep
131G 5.6G 119G 5% /repository
tmpfs 4.0K 0 4.0K 0% /dev/vx
/dev/vx/dsk/nbuapp/pdvol
25T 14T 12T 55% /disk
/dev/vx/dsk/nbuapp/cfgvol
25G 3.8G 22G 16% /config
/dev/vx/dsk/nbuapp/advol
15T 262G 15T 2% /advanceddisk/dp1/advol
/dev/mapper/system-rescue
9.7G 5.6G 3.6G 62% /repository/checkpoints/appliance_checkpoint_kr3Vbs
/dev/mapper/system-rescue
9.7G 5.6G 3.6G 62% /repository/checkpoints/appliance_checkpoint_g1Ob2C
/dev/mapper/system-rescue
9.7G 5.6G 3.6G 62% /repository/checkpoints/appliance_checkpoint_bTChqL
dc-nbmedia01:/home/maintenance #
12-18-2014 09:14 AM
Hi Priyeranjan ... Sorry.. been a bit busy this week. Thanks for the "df" output. /repository is a tmpfs filesystem which is where "/checkpoints" will be mounted on. I see you're attempt at creating a checkpoint but it looks to me like you control-c'd out of it which caused the resulting error messages. What has me curious is in the checkpoint mounts from the "df" output. It's showing that over 5GB is currently used and only 3.6GB is available. If there are no existing checkpoints then what is taking up that space? See if you can navigate to these respective directories and run an "ls -al" and a "du -k" for each then post the output here.
/repository/checkpoints/appliance_checkpoint_kr3Vbs
/repository/checkpoints/appliance_checkpoint_g1Ob2C
/repository/checkpoints/appliance_checkpoint_bTChqL
01-12-2015 06:44 PM
Hi Jeff,
Sorry for late response on this opened forum..
Issue has been solved by taking below steps.
Increased the size of /repository till 230 GIG from System volume and and after that rebooted the appiance.
Performed the self test and started the upgrade and it went fine.
Thank you for your kind help and support during this activity.
Regards,
Priyeranjan