Forum Discussion

Tonaco_pt's avatar
Tonaco_pt
Moderator
9 years ago

Vault items "Awaiting Backup" not cleairing

HI, in the passed I open the https://www.veritas.com/community/forums/vault-items-awaiting-backup-not-cleairing-down-below-3414, 1 year later the value is up to 4497. 

AndrewB said that "the key is the "unsecured SIS parts in other partitions" so you'll have to look at your other partitions to find which one is the true culprit." I check other post and found out that I should be using EVSVR.EXE command to find the error, after reading the manual I understand how does the tools work but I don't know what options I should use to figure out my issue?

can someone help me?

I'm going to recieve more volume to add to EV server but has I don't have more driveletter I'm going to convert to mountpoint, in the aritcle http://www.veritas.com/docs/000028294 one point is to "Confirm all archived items have been backed up (SQL Server)", I really need to correct this error, We still use Microsoft DPM to backup all drivers.

  • Following the notes we did get from Veritas in an older case: 

    1)  Run the query
     
    use <VaultStoreDB>
    select * from watchfile where itemsecured = 1
     
    2)  Copy the IDTransaction value for the first entry into the clipboard (should be ArchiveTransactionID now)
     
    3)  Run the query
     
    use <VaultStoreDB>
    select savesetidentity from saveset where idtransaction = '[idtransaction from step 2]'
     
    4)  Copy the SavesetIdentity value into the clipboard
     
    5)  Run the query
     
    use <VaultStoreDB>
    select * from view_savesetsisparts
    where savesetidentity = '[savesetidentity from step 3]'
     
    6)  Make a note of the FPDistinctionByte value and copy the ParentTransactionID value into the clipboard.  The FPDistinctionByte value will correspond with the member table number in the query below.
     
    7)  Run the query
     
    use <FingerprintDB>
    select * from  MemberTable_[FPDistinctionByte] where ParentTransactionID = '[ParentTransactionID from Step 5]'
     
    8)  Confirm that the record returned has a Secured value of 0
     
    9)  Run the query
     
    Select * from saveset where idtransaction = '[parenttransactionid from step 5]'
     
    You will have to run this query against all Vault Stores until you find the item as it could be in any of the Vault Stores under that particular vault store group.  Once you have found it make a note of the archived date of the item.  
     
    10)  Open the partition for the Vault Store in which this item exists and confirm that files for this item exist.  The folder structure is:
     
    Year\Month-Day\First Character of the Transaction ID\Following 3 characters of the transaction ID
     
    11)  If the item exists and that partition has definitely been backed up, run the query below to set the associated SISpart to secured
     
    use <FingerprintDB>
    update MemberTable_[FPDistinctionByte]
    set Secured = 1
    where parenttransactionID = '[parenttransactionid from step 5]'
     
    12)  Rename the ignorearchivebittrigger.old file to ignorearchivetriggerbit.txt on the partition location for the VaultStoreMBX2 Vault Store.
     
    13)  Put the specific vaultstore into backup mode and then take it out again.  This will trigger EV into checking if the items are secure or not.  
     
    14)  Re-run the query from Step 1.  The number of items should have reduced.  Pick the top record again and run through the steps again.  Repeat until the query from step 1 returns no records.

    Used this successfully for EV 10.0.4 CHF3. 

    Please note that changing the secured flag for 1 item may cause a massiv decrease of your count as many items may be referenced to just a few unsecured ones. This was the case in my situation. 

  • You could change all closed partitions to look for a trigger and then place a trigger file manually in them. Have you looked in SQL at all?
  • Hi DcVAST,

    I Have a schedule tasks that run's 3x a day, that set EV in Backupmode (and take it off 30 min later), that creates the trigger files and restart the EnterpriseVaultTaskControllerService.

    From SQL I can get the files that are not secure, with a powershell script I validated that all file are present. 

    I don't know what else to do.

  • Hi, I run DTRACE, SHOWCONTIG and some SQL Query

    SQL Query for 1 of the unsecure iten.

    USE [EVVSVaultStore_1] SELECT SavesetIdentity,ItemSecured FROM WatchFile WHERE SavesetIdentity = '25985876'
    USE [EVVSVaultStore_1] SELECT SavesetIdentity,FPDistinctionByte,ParentTransactionId FROM WatchSISPartFile WHERE SavesetIdentity = '25985876'
    USE [EVVSGVaultStore_1_1] SELECT ParentTransactionId,Secured FROM MemberTable_101 WHERE ParentTransactionId = '{90f25c96-1c4a-1633-3ee9-86586c7c2ac1}'

    SavesetIdentity
    ItemSecured
    FPDistinctionByte
    ParentTransactionId
    Secured
    25985876
    1 101
    {90f25c96-1c4a-1633-3ee9-86586c7c2ac1}
    0

     

    DBCC SHOWCONTIG scanning 'spt_fallback_db' table...
    Table: 'spt_fallback_db' (117575457); index ID: 0, database ID: 1
    TABLE level scan performed.
    - Pages Scanned................................: 0
    - Extents Scanned..............................: 0
    - Extent Switches..............................: 0
    - Avg. Pages per Extent........................: 0.0
    - Scan Density [Best Count:Actual Count].......: 100.00% [0:0]
    - Extent Scan Fragmentation ...................: 0.00%
    - Avg. Bytes Free per Page.....................: 0.0
    - Avg. Page Density (full).....................: 0.00%
    DBCC SHOWCONTIG scanning 'spt_fallback_dev' table...
    Table: 'spt_fallback_dev' (133575514); index ID: 0, database ID: 1
    TABLE level scan performed.
    - Pages Scanned................................: 0
    - Extents Scanned..............................: 0
    - Extent Switches..............................: 0
    - Avg. Pages per Extent........................: 0.0
    - Scan Density [Best Count:Actual Count].......: 100.00% [0:0]
    - Extent Scan Fragmentation ...................: 0.00%
    - Avg. Bytes Free per Page.....................: 0.0
    - Avg. Page Density (full).....................: 0.00%
    DBCC SHOWCONTIG scanning 'spt_fallback_usg' table...
    Table: 'spt_fallback_usg' (149575571); index ID: 0, database ID: 1
    TABLE level scan performed.
    - Pages Scanned................................: 0
    - Extents Scanned..............................: 0
    - Extent Switches..............................: 0
    - Avg. Pages per Extent........................: 0.0
    - Scan Density [Best Count:Actual Count].......: 100.00% [0:0]
    - Extent Scan Fragmentation ...................: 0.00%
    - Avg. Bytes Free per Page.....................: 0.0
    - Avg. Page Density (full).....................: 0.00%
    DBCC SHOWCONTIG scanning 'spt_monitor' table...
    Table: 'spt_monitor' (1115151018); index ID: 0, database ID: 1
    TABLE level scan performed.
    - Pages Scanned................................: 1
    - Extents Scanned..............................: 1
    - Extent Switches..............................: 0
    - Avg. Pages per Extent........................: 1.0
    - Scan Density [Best Count:Actual Count].......: 100.00% [1:1]
    - Extent Scan Fragmentation ...................: 0.00%
    - Avg. Bytes Free per Page.....................: 8024.0
    - Avg. Page Density (full).....................: 0.86%
    DBCC SHOWCONTIG scanning 'spt_values' table...
    Table: 'spt_values' (1131151075); index ID: 1, database ID: 1
    TABLE level scan performed.
    - Pages Scanned................................: 12
    - Extents Scanned..............................: 3
    - Extent Switches..............................: 2
    - Avg. Pages per Extent........................: 4.0
    - Scan Density [Best Count:Actual Count].......: 66.67% [2:3]
    - Logical Scan Fragmentation ..................: 25.00%
    - Extent Scan Fragmentation ...................: 33.33%
    - Avg. Bytes Free per Page.....................: 534.3
    - Avg. Page Density (full).....................: 93.40%
    DBCC SHOWCONTIG scanning 'spt_values' table...
    Table: 'spt_values' (1131151075); index ID: 2, database ID: 1
    LEAF level scan performed.
    - Pages Scanned................................: 7
    - Extents Scanned..............................: 2
    - Extent Switches..............................: 1
    - Avg. Pages per Extent........................: 3.5
    - Scan Density [Best Count:Actual Count].......: 50.00% [1:2]
    - Logical Scan Fragmentation ..................: 28.57%
    - Extent Scan Fragmentation ...................: 50.00%
    - Avg. Bytes Free per Page.....................: 507.4
    - Avg. Page Density (full).....................: 93.73%
    DBCC SHOWCONTIG scanning 'MSreplication_options' table...
    Table: 'MSreplication_options' (1163151189); index ID: 0, database ID: 1
    TABLE level scan performed.
    - Pages Scanned................................: 1
    - Extents Scanned..............................: 1
    - Extent Switches..............................: 0
    - Avg. Pages per Extent........................: 1.0
    - Scan Density [Best Count:Actual Count].......: 100.00% [1:1]
    - Extent Scan Fragmentation ...................: 0.00%
    - Avg. Bytes Free per Page.....................: 7900.0
    - Avg. Page Density (full).....................: 2.40%
    DBCC execution completed. If DBCC printed error messages, contact your system administrator.

    Dtrace

    DT> set StorageFileWatch v y 

    With the following these steps: (1) Run a dtrace (2) Set backup mode and  make sure that the .old file is removed, (3) take a new backup, (4) clear backup mode createthe .txt file (5) wait for the .txt where renamed to .old (6) stop Dtrace 

     

    From this forum post:

    https://www.veritas.com/community/forums/unsecured-sis-parts-other-partitions

    https://www.veritas.com/community/forums/vault-items-awaiting-backup-not-cleairing-down

  • Had a similar situation in the past. 

    I described my situation under https://www.veritas.com/community/forums/tracking-sis-parts-databases

    In this case we had the problem that entrys in the journlarchive table had backupcomplete = 0 but the corresponding dvs part in the watchfile had itemsecured = 1.

    I took one of these entries, did a dumpsaveset and verified if the used sis parts for the entry in the journalarchive table was recognized as secured (verified it over the sis database) 

    Can you verify is this is the case for you?

    1) Pick up on of the 4497 entrys in the journalarchive table where backupcomplete is 0

    2) Verify the fitting enty in the watchfile table with the help of the transactionid from the journalarchive table. Do you see an entry with itemsecured = 1?

     

    You say that ev backup takes place three times a day. Do you secure all partitions at this times? (also the closed ones?) Do you see a current date for the last scan? 

    Could you provide more details about the sharing settings and the number of vaultstoregroups, vaultstores and partitions? 

     

    I also had many customers that forget to backup old partitions lets say in the journaling vaultstore. Sharing was configured "Share within group" and some entries in a mailbox vaulstore were referenced to unsecured entries in the old journaling partition. I think its worth a try to check the status of all partitions in your environment. 

     

     

     

     

     

     

     

  • HI Marcde09, 

    Now I have a better grasp of the problem, and yes I believe that the problem are the same.

    First the Database maintenance issued was not related.

    I already came to the conclusion that this problem is not repeating anymore (http://www.veritas.com/community/forums/unsecured-sis-parts-other-partitions), as you said I have some sis parts in the journalarchive table and EVVSGVaultStore_1_1\MemberTable_000 marked has unsecured, so all new message archived that use some of this sis part will appear in the watchfile (that why it still growing), furthermore now I know that all the unsecured saveset are from 11/3/2013 to 25/07/2014.

    To answer you question:

    1) Yes all of them have journalarchive table where backupcomplete is 0

    2) Yes all of them have entry with itemsecured = 1

    We backup all partitions even closed one, every time a full backup run, we have a Powershell script that active/deactivate EV Backup Mode, delete .old trigger files and creates new ones and restart the EnterpriseVaultTaskController Service. Partitions are detected with a sql query. 

    Yes all partitions have  "Last scan started" with today date.

    We have 1 VaultStoreGroup, 1 VaultStore and 16 vaultpartition.

    I never user the "EVSVR DumpSaveset", so going to learn how and see if I get the same result.

    Can you tell me what is the "internal document from technical support which shows a way to clear these entrys."?

     

    I have a SQL scritpt ready but I don't if the correct way,

  • Hi, 

    Run the EVSVR DumpSaveset, and if I understand you right in the xml file  VSDBRecords.xml found the following entries that proves what you said:

     

    <JournalArchive xmlns:..

      <IndexCommited>1</IndexCommited>

      <BackupComplete>0</BackupComplete>

    <WatchFile xmlns:..

      <ItemSecured>1</ItemSecured>

     

    Can you tell me with is the document you referring in this statement "Found an internal document from technical support which shows a way to clear these entry's. " 

    I think I found a way to do that but I would like to be sure.

    I have not thanks you, your post is very useful, it help to collect my thoughts, and learn a few things, thank you so much.

  • Following the notes we did get from Veritas in an older case: 

    1)  Run the query
     
    use <VaultStoreDB>
    select * from watchfile where itemsecured = 1
     
    2)  Copy the IDTransaction value for the first entry into the clipboard (should be ArchiveTransactionID now)
     
    3)  Run the query
     
    use <VaultStoreDB>
    select savesetidentity from saveset where idtransaction = '[idtransaction from step 2]'
     
    4)  Copy the SavesetIdentity value into the clipboard
     
    5)  Run the query
     
    use <VaultStoreDB>
    select * from view_savesetsisparts
    where savesetidentity = '[savesetidentity from step 3]'
     
    6)  Make a note of the FPDistinctionByte value and copy the ParentTransactionID value into the clipboard.  The FPDistinctionByte value will correspond with the member table number in the query below.
     
    7)  Run the query
     
    use <FingerprintDB>
    select * from  MemberTable_[FPDistinctionByte] where ParentTransactionID = '[ParentTransactionID from Step 5]'
     
    8)  Confirm that the record returned has a Secured value of 0
     
    9)  Run the query
     
    Select * from saveset where idtransaction = '[parenttransactionid from step 5]'
     
    You will have to run this query against all Vault Stores until you find the item as it could be in any of the Vault Stores under that particular vault store group.  Once you have found it make a note of the archived date of the item.  
     
    10)  Open the partition for the Vault Store in which this item exists and confirm that files for this item exist.  The folder structure is:
     
    Year\Month-Day\First Character of the Transaction ID\Following 3 characters of the transaction ID
     
    11)  If the item exists and that partition has definitely been backed up, run the query below to set the associated SISpart to secured
     
    use <FingerprintDB>
    update MemberTable_[FPDistinctionByte]
    set Secured = 1
    where parenttransactionID = '[parenttransactionid from step 5]'
     
    12)  Rename the ignorearchivebittrigger.old file to ignorearchivetriggerbit.txt on the partition location for the VaultStoreMBX2 Vault Store.
     
    13)  Put the specific vaultstore into backup mode and then take it out again.  This will trigger EV into checking if the items are secure or not.  
     
    14)  Re-run the query from Step 1.  The number of items should have reduced.  Pick the top record again and run through the steps again.  Repeat until the query from step 1 returns no records.

    Used this successfully for EV 10.0.4 CHF3. 

    Please note that changing the secured flag for 1 item may cause a massiv decrease of your count as many items may be referenced to just a few unsecured ones. This was the case in my situation. 

  • Hi Marcde09,

    Thank you very much. 

    I use this Sql to find FPDistinctionByte and ParentTransactionId

    SELECT
        WatchFile.ArchiveTransactionId
      , WatchFile.[ItemName]
      , WatchFile.ItemSecured
      , view_SavesetSISParts.FPDistinctionByte
      , view_SavesetSISParts.ParentTransactionId

    FROM
        [dbo].[WatchFile]
    INNER JOIN [dbo].[Saveset]
    ON ArchiveTransactionId = IdTransaction

    INNER JOIN [dbo].[view_SavesetSISParts]
    ON Saveset.SavesetIdentity = view_SavesetSISParts.SavesetIdentity

    WHERE [ItemSecured] = 1
    ORDER BY [FPDistinctionByte]

    Remove duplicate 

    With the help of Powershell (foreach) got the unsecure sispart 

    SELECT ParentTransactionId,ArchivedDateUTC,Secured FROM MemberTable_999 WHERE ParentTransactionId  = '{ParentTransactionId}' " AND [Secured] = '0'"

    With this result with the help of excel I run the following query for each unsecured part. (Step 11)

    update MemberTable_999 set Secured = 1 where parenttransactionID = '{ParentTransactionId}'

    Follow your step 12 and 13, when I check the watch file again it was clear (only new itens)