2.6.0.1 Upgrade keeps rolling back?

[Stanleyj]

Im in the process of trying to upgrade two 5200 appliances from 2.5.3 to 2.6.0.1 and they keep rolling back about an hour into it.  One is set as a media server only and one is master/media and they both are rolling back at the same spot with the same error.

/opt/PDOS/install/preparepdos.sh: Line 674: /opt/NBUAppliance/scripts/config_pp_factory.sh: No such file or directory

Has anyone seen this and found a way to fix?

I have had a ticket with support but not much is going on there for me.

[Mark_Solutions]

Nothing in the release notes about this, although it does say you cannot image a 5200 using a 2.6 image so there are clearly differences - but it does say that you can upgrade one

Have you has a look to see what the script calls and if the file in question does actually exist? If so has it got restricted rights on the file?

[Stanleyj]

Mark,

Dont know why something as simple as "look to see if the file exist" makes so much sense.  HAHA.

I checked the directory and sure enough config_pp_factory.sh does not exist.  There is a config_nbapp_factory.sh.

I havent come across any one upgrading a 5200 so maybe there is a bug with the upgrade on 5200 considering you cant image one with the newest version.

[Mark_Solutions]

Makes you wonder if that file name changed between versions - could do with someone that has got a 5220 / 5230 checking for us to see what the contents of that file are - if they are the same as your file then i would imagine the preparedos.sh script could be edited to suit (or the config_nbapp_factory.sh could be renamed to config_pp_factory.sh - which is more likely as it may reference that file name in the future)

I guess we need to wait for support!

I will see if i can find a 5230 and get that file - could you post your config_nbapp_factory.sh file on here (as an attachment please)

[Mark_Solutions]

Just had a look at a 5220 and that file doesnt exist on that either

Sounds like the installer script is wrong / corrupt and it should actually say config_nbapp_factory.sh

[Stanleyj]

Mark,

Im sorry but the file does exist.

-rwxr-xr-x  1 root root    23779 Jun 27  2013 config_nbapp_factory.sh

The permissions are the same on all files in this directory.  Im not really sure what to change the to?

[Mark_Solutions]

Yes - the config_nbapp_factory.sh exists - same as on the 5220 i checked - but the script is calling config_pp_factory.sh which it cannot find

Looks like the script is missing the "nba" in its text (config_nbapp_factory.sh)

[Mark_Solutions]

I believe that /opt/PDOS/install/preparepdos.sh is a built in script and not part of the package itself (I have extracted it to the n'th degree and cannot see it anywhere!)

Guess you really need support to confirm all this

[SymTerry]

Hello,

You can infact upgrade a 5200 to 2.6.0.1 according to TECH205913. Please note the depndencys.

I checked with our Appliance SME's about the installer script being wrong / corrupt and they have not heard of this issue. There is no Etracks for it as of yet. If you could please open a case with support for this, we can see if this is indeed a bug.

[Stanleyj]

Mark,

Im very sorry but wasnt paying close enough attention to the error message.  it is actually looking for config_nbapp_factory.sh which does exist from what I can tell.  I have attached a screen shot of the error message and the files directory.

After speaking with support few minutes ago they want me to download the upgrade again and see if that helps first considering im getting the same error two different appliances and Im using the same upgrade package on both.  Im about to run it again to see what happens.

[Stanleyj]

Downloading a new upgrade package did not work.  Same error message.  Will have to wait on support at this point.

SymTerry,

Here is the ticket # i have open incase you want to take a look at it.

Case # 05792911 - Assistance to upgrade to a newer version has been created [ref:00D30jPy.5005QB    [ ref:_00D30jPy._50050QB3CN:ref ]

[SymTerry]

I will contact the tech and make sure he has all the resources he needs. It looks like we may need to get further logging from /logs.

[Mark_Solutions]

According to the installer it should log everything to /var/log/iso_postinstall.log

Nothing special about the /opt/PDOS/install/preparepdos.sh as far as i can see - just a set of routines to go through

Not sure that a re-download would have helped as the preparedos.sh is a built in script and not part of the package

I can only think that there is a hidden gap in the command or something - unless the installer has unmounted something too early during the upgrade?

Worth a look at the log but i am guessing it may just confirm what the installer says

[Stanleyj]

Mark im sifting thru the iso_postinstall.log and it basically consist of the error below a couple thousand times and some network information.

Cannot determine dependencies of module megasr. Is modules.dep up to date?
Cannot determine dependencies of module megasr. Is modules.dep up to date?
Driver modules:    scsi_mod sd_mod megaraid_sas processor thermal fan edd libata ata_piix scsi_transport_fc firmware_class qla2xxx
Filesystem modules:    jbd ext3
Including:    initramfs fsck.ext3
20674 blocks

I did find this message kind of interesting.

preparepdos.sh already ran, delete /opt/PDOS/install/.installed and rerun this script to force a rerun
Root device:    /dev/disk/by-id/scsi-3600605b00213ac001428f60e08624b32-part2 (/dev/sda2) (mounted on / as ext3)
Module list:     megasr megaraid_sas processor thermal fan jbd ext3 edd ata_piix qla2xxx (xennet xenblk)

[Mark_Solutions]

The .installed is interesting - that does get mentioned in the script - maybe something has been left over from a previous upgrade the very very first issue with this one

Worth taking a look to see if it does exist and if so delete it and try again?

Anything back from support yet? - dont want to suggest you do something they are not happy with!!

[Stanleyj]

Nothing from support just yet.  They are looking into some logs i send on friday and will contact me this afternoon.

[Mark_Solutions]

OK - keep us updated - best wait on them - do add the word "escalate" when you mail or phone them - get it moving!

[Stanleyj]

Here is what support sent me yesterday.

Hi Jason,

Here is an entry in the patch log that we are trying to find out what is the resolution

[2014-01-09 10:30:41]   Mounting Catalog volume...

UX:vxfs mount.vxfs: ERROR: V-3-21264: /dev/vx/dsk/nbuapp/advol is = already mounted, /cat is busy,

                                 or the allowable number of mount points has been exceeded.

It appears that the upgrade see's the volume as being already mounted and begins rolling back the upgrade.  I also have one of my Backline engineers looking at this as well.

[SymTerry]

Here are the logs generated. The assigned tech on the case should, or will soon be, requesting some of these from you soon: 

Pre-reimage logs

   /log/patch_log_2.6<timestamp>

   /log/patch_output_2.6<timestamp>

   moved to /log/iso before disk format

Reimage logs YaST and ISO

   /var/log/YaST2

   /var/adm/autoinstall

   /var/log/iso_postinstall.log

   /var/log/config_nb_factory.log

Post-reimage

   /log/patch_log_2.6_<timestamp>

   /log/patch_output_2.6<timestamp>

Rollback

   logs preserved in /log/patch_log_2.6.0.1_<datestamp>_ROLLBACK

Mostly the focus is on the rollback. We will want to get that whole directory for the latest rollback, ie. /log/patch_log_2.6.0.1_012014100952_ROLLBACK/

You can try having a look at it for errors.

[Stanleyj]

I was able to get all the files you mentioned except for the either the post or pre-remage logs because there the same name.  I dont see a /log/iso directory.

All of the files are 281mb compressed so i will have to wait for suppor to open me an ftp connection.  Thanks for the heads up.