cancel
Showing results for 
Search instead for 
Did you mean: 

Facing an issue in 5230 Appliance upgrade - Failed to create the pre-upgrade checkpoint

Priyeranjan
Level 4
Partner Accredited

Hi Friends,

 

My name is Priyeranjan and today I am facing an upgrade issue with Netbackup Appliance 5230 media from version 2.6.0.1 to 2.6.0.4.

I started upgrade from Manage / Software / Install pkg name 

Upgrade process was submittd properly with all required information, later on when i am checking the "Upgrade Status" I am getting bellow message and errors.

dc-nbmedia01.Software> UpgradeStatus

The appliance is creating a pre-upgrade checkpoint!
dc-nbmedia01.Software> UpgradeStatus

The appliance is creating a pre-upgrade checkpoint!
dc-nbmedia01.Software> UpgradeStatus

The appliance is creating a pre-upgrade checkpoint!
dc-nbmedia01.Software> UpgradeStatus

Failed to create the pre-upgrade checkpoint, please resolve this issue first!
dc-nbmedia01.Software>

 

Can you please help me to verify full logging for the upgrade as I am unable to find.

Or if you have already faced similar kind of issue please help me with your recomendation..

 

Looking forward your kind help.

 

Thank you.

Regards,

Priyeranjan

 

 

10 REPLIES 10

Jeff_Foglietta
Level 5
Partner Accredited Certified

This is interesting. Quick question. Have you ever upgraded this appliance before? Only one of each checkpoint can exist on an appliance at a time so you will have pre, post, user and factory checkpoints when doing an upgrade. I suppose it could be possible there is an existing checkpoint out there however unlikely it may seem. You may need to contact support to go through the checkpoint logs

Priyeranjan
Level 4
Partner Accredited

Hi Jeff,

Many thanks for reverting back. I have checked the existing check point through Support / Checkpoint on the Appliance CLISH login and i did not find any previous checkpoints there then i tried checkpoint rollback and rollback also did not find previous created files.

One question i want to ask you - How can i verify the path where checkpoint will store the information during upgrade - I am suspectig might be space issue is causing this upgrade failure.

I tried to create manual checkpoint and have not seen any progress on that.

Looking forward your recomendation in this regard. 

Thank you.

 

 

Regards,

Priyeranjan

 

 

Jeff_Foglietta
Level 5
Partner Accredited Certified

Hi Priyeranjan... I am inclined to agree with you regarding the space issue. I wouldn't think there would have been any checkpoints on your appliance but it was worth a look.. I do not know for certain the location of the checkpoints. I will research that as soon as I can find time but if I had to guess the checkpoints would be on the mirrored root drive. Again until I find out for sure that is only a guess. I'll get back to you.

Jeff_Foglietta
Level 5
Partner Accredited Certified

Hi Priyeranjan .. The checkpoints would be written to the following path:

/repository/checkpoints

 

David_Affinito
Level 2
Accredited Certified

To see the log for checkpoints, you can run this from root:

vxlogview -p 409 -o NBAPP_CKPT -d all 

My guess is that there are sparse *.vmdk* files left over in /tmp that are making the checkpoint much larger than it should be. I would elevate to root then check /tmp

nbmaster:~ # ls -lash /tmp | less

The 6th field is the on-disk size of the file.  For example:

0 -rw-r--r-- 1 root root 100G Mar 23 23:24 VM-name_1-000004.vmdk_1_1750811607

I would delete the sparse files and re-attempt the upgrade.

nbmaster:~ # rm /tmp/*.vmdk*

Priyeranjan
Level 4
Partner Accredited

Hi Jeff,

 

I tried to create manual checkpoint and it has thrown an error. Can you have a look on below checkpoint errors and provide your fedback..

dc-nbmedia01.Support> Checkpoint
Create   Rollback
dc-nbmedia01.Support> Checkpoint Rollback
- [Info] Disable interrupt (CTRL-C), quit, and suspend signals
+-----------------------+
|  Appliance Rollback   |
+-----------------------+
Rolling back to an Appliance Checkpoint will restore the system back to the checkpoint's point-in-time. This can help undo any misconfiguration or system failures that might have occurred.
Rolling back to an Appliance Checkpoint will revert the following components:
        1) Appliance Operating System
        2) Appliance Software
        3) NetBackup Software
        4) Networking Configuration
        5) Any previously applied patches
        6) Backup data is not reverted

- [Info] Running rollback validation...please wait.
- [Info] Rollback validation successful.
There are no Appliance Checkpoints available for rollback. Checkpoints can be created via the appliance shell menu's "Support->Checkpoint create" command.
dc-nbmedia01.Support> Checkpoint Rollback
- [Info] Disable interrupt (CTRL-C), quit, and suspend signals
+-----------------------+
|  Appliance Rollback   |
+-----------------------+
Rolling back to an Appliance Checkpoint will restore the system back to the checkpoint's point-in-time. This can help undo any misconfiguration or system failures that might have occurred.
Rolling back to an Appliance Checkpoint will revert the following components:
        1) Appliance Operating System
        2) Appliance Software
        3) NetBackup Software
        4) Networking Configuration
        5) Any previously applied patches
        6) Backup data is not reverted

- [Info] Running rollback validation...please wait.
- [Info] Rollback validation successful.
There are no Appliance Checkpoints available for rollback. Checkpoints can be created via the appliance shell menu's "Support->Checkpoint create" command.
dc-nbmedia01.Support>
dc-nbmedia01.Support> Checkpoint
Create   Rollback
dc-nbmedia01.Support> Checkpoint Create
- [Info] Disable interrupt (CTRL-C), quit, and suspend signals
+-------------------------------+
|  Create Appliance Checkpoint  |
+-------------------------------+
Creating an Appliance Checkpoint allows the user to easily rollback the entire s                                                                             that might have occurred. An Appliance Checkpoint captures the following compone
        1) Appliance Operating System
        2) Appliance Software
        3) NetBackup Software
        4) Networking Configuration
        5) Any previously applied patches
        6) Backup data is not reverted

- [Info] Running create checkpoint validation...please wait
- [Info] Create checkpoint validation successful.
There are no checkpoints in the system.
There is no user checkpoint. Please continue to create a user checkpoint
 >> Would you like to proceed? (yes/no) yes

>> Checkpoint description (OPTIONAL): Manual before upgrade

 >> Create checkpoint? (yes/no) yes
- [Info] No existing USER checkpoint was detected.
- [Info] CREATING USER CHECKPOINT
- [Info] Creating checkpoint. This operation can take up to 15 minutes.
     Please wait...

^C


^C^C^C^C^C^C^C^C^C^C

- [Info] Checkpoint creation failed. Performing cleanup...
- [Error] On node dc-nbmedia01:
- [Error] Error check pointing volume '/dev/system/root'. Failed to create tarball.
- [Error] Error creating snapshot volume /dev/system/root_SNAP_USER_2014_12_17_21_07_36.
- [Error] On node dc-nbmedia01:
- [Error] Error check pointing volume '/dev/sda2'. Failed to create tarball.
- [Error] Error creating snapshot volume .
- [Error] On node dc-nbmedia01:
- [Error] Failed to create checkpoint. Removing partial snapshots..
- [Error] Appliance Checkpoint creation failed. Retry again once errors are resolved. Please see the Symantec NetBackup Appliance Administrator's Guide for mre troubleshooting information.
dc-nbmedia01.Support>

 

 

Priyeranjan
Level 4
Partner Accredited

Hi Jeff,

I have verified the avilable space, Can you please have a look on below output.

 

dc-nbmedia01:/home/maintenance # df -kh
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/system-root
                       92G  6.2G   81G   8% /
udev                   63G  592K   63G   1% /dev
tmpfs                  63G   76K   63G   1% /dev/shm
/dev/sda2             198M   28M  160M  15% /boot
/dev/mapper/system-inst
                       50G   26G   21G  56% /inst
/dev/mapper/system-log
                      367G   56G  293G  16% /log
/dev/mapper/system-rep
                      131G  5.6G  119G   5% /repository
tmpfs                 4.0K     0  4.0K   0% /dev/vx
/dev/vx/dsk/nbuapp/pdvol
                       25T   14T   12T  55% /disk
/dev/vx/dsk/nbuapp/cfgvol
                       25G  3.8G   22G  16% /config
/dev/vx/dsk/nbuapp/advol
                       15T  262G   15T   2% /advanceddisk/dp1/advol
/dev/mapper/system-rescue
                      9.7G  5.6G  3.6G  62% /repository/checkpoints/appliance_checkpoint_kr3Vbs
/dev/mapper/system-rescue
                      9.7G  5.6G  3.6G  62% /repository/checkpoints/appliance_checkpoint_g1Ob2C
/dev/mapper/system-rescue
                      9.7G  5.6G  3.6G  62% /repository/checkpoints/appliance_checkpoint_bTChqL
dc-nbmedia01:/home/maintenance #

 

 

Rgards,

Priyeranjan

 

Priyeranjan
Level 4
Partner Accredited

Hi Jeff,

Can you please have a look on below output.

 

dc-nbmedia01:/home/maintenance # df -kh
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/system-root
                       92G  6.2G   81G   8% /
udev                   63G  592K   63G   1% /dev
tmpfs                  63G   76K   63G   1% /dev/shm
/dev/sda2             198M   28M  160M  15% /boot
/dev/mapper/system-inst
                       50G   26G   21G  56% /inst
/dev/mapper/system-log
                      367G   56G  293G  16% /log
/dev/mapper/system-rep
                      131G  5.6G  119G   5% /repository
tmpfs                 4.0K     0  4.0K   0% /dev/vx
/dev/vx/dsk/nbuapp/pdvol
                       25T   14T   12T  55% /disk
/dev/vx/dsk/nbuapp/cfgvol
                       25G  3.8G   22G  16% /config
/dev/vx/dsk/nbuapp/advol
                       15T  262G   15T   2% /advanceddisk/dp1/advol
/dev/mapper/system-rescue
                      9.7G  5.6G  3.6G  62% /repository/checkpoints/appliance_checkpoint_kr3Vbs
/dev/mapper/system-rescue
                      9.7G  5.6G  3.6G  62% /repository/checkpoints/appliance_checkpoint_g1Ob2C
/dev/mapper/system-rescue
                      9.7G  5.6G  3.6G  62% /repository/checkpoints/appliance_checkpoint_bTChqL
dc-nbmedia01:/home/maintenance #

 

Jeff_Foglietta
Level 5
Partner Accredited Certified

Hi Priyeranjan ... Sorry.. been a bit busy this week. Thanks for the "df" output. /repository is a tmpfs filesystem which is where "/checkpoints" will be mounted on. I see you're attempt at creating a checkpoint but it looks to me like you control-c'd out of it which caused the resulting error messages. What has me curious is in the checkpoint mounts from the "df" output. It's showing that over 5GB is currently used and only 3.6GB is available. If there are no existing checkpoints then what is taking up that space? See if you can navigate to these respective directories and run an "ls -al" and a "du -k" for each then post the output here.

 

/repository/checkpoints/appliance_checkpoint_kr3Vbs

/repository/checkpoints/appliance_checkpoint_g1Ob2C

/repository/checkpoints/appliance_checkpoint_bTChqL

Priyeranjan
Level 4
Partner Accredited

Hi Jeff,

Sorry for late response on this opened forum..

Issue has been solved by taking below steps.

Increased the size of /repository till 230 GIG from System volume and and after that rebooted the appiance.
Performed the self test and started the upgrade and it went fine.

Thank you for your kind help and support during this activity.

Regards,
Priyeranjan