Forum Discussion

Sarah_Seftel1's avatar
Sarah_Seftel1
Level 6
12 years ago

Archive Rate / Averages?

Hi

 

any known table that show averages of archiving? ie - how much data can be archived in 1 hour? how much time will it take to archive 100 GB? 

I have a customer that is interested to know what will be the average time for him to archive 100GB.
Although it depends on file size, index, amount of items etc - he wants to know if there is a general archive rate \ average so he can calculate the time it will take him to archive 100GB (more or less) 

Enterprise Vault
Information Governance
Tip-How to
  • JesusWept3's avatar
    JesusWept3
    12 years ago

    First off, sorry for the really really long post

    Yeah its just such a difficult thing to calculate, the performance guide is obviously going off of a best case scenario, you may have slightly slower disks on sql/exchange/vault store partitions, you may have network saturation at the point of archiving, you might have exchange maintenance going on etc etc, all of which can slow down the archiving rate

    you might come across items that have 10,000 recipients, extremely large items etc, you might have a GC that is slow to respond and it takes time to do security lookups and address lookups etc

    You might end up scanning a mailbox with 50,000 folders that has 40GB of mail that takes more time to scan than it does to actually archive anything

    The best thing you can do is a pilot against a couple of dozen users, run archiving overnight, or a couple of nights, calculate how many items were archived and the size and then judge it from there

    So you could say enable 100 users, run archiving in report mode and look at how many items are eligible for archiving, so it might say that 15,000 items are eligible for archiving, then let the schedule kick in

    Then do a query like


     

    USE EVVSYourVaultStore_1
SELECT COUNT(S.IdTransaction) "Item Count",
       SUM(S.ItemSize)/1024 "Size After Compression (MB)",
       SUM(SP.OriginalSize)/1024/1024 "Original Exchange Size (MB)"
FROM   Saveset S,
       SavesetProperty SP
WHERE  S.SavesetIdentity = SP.SavesetIdentity
  AND  S.ArchivedDate > '2013-20-01 23:00:00.000'
  AND  S.ArchivedDate < '2013-21-01 03:00:00.000'

     


    This will give you the item count and the total size in MB of items stored in EV , and the size of the items gotten from Exchange with a run of 4 hours (11pm to 3am)

    So you may get a result like

     

    Archived Items Size After Compression (MB) Original Exchange Size (MB)
    180,000 5091 5625

     


    So based on that you've archived 5,625MB on Exchange and 180,000 items in 4 hours.

    To calculate how many items and how much time it took for 1GB we do the following
    1. Archived Items / Original Size   (i.e. 180,000 items / 5,625MB) = 32
    2. Original Size - 1024 (1GB)  (i.e. 1,242 - 1024GB) = 4,601MB
    3. Result of Step 1 * Result of Step 2 (i.e. 32 * 4,601) = 147,232
    4. Archived Items - Result of Step 3 (i.e. 180,000 - 147,232) = 32,768 Items in 1GB (1024MB)

    So now you know that you have 32,768 items equate to 1GB

    Then you need to find out how much time that took
    1. Hours to run * 60 * 60 ...(i.e. 4 * 60 * 60) = 14,400 seconds
    2. Result of step 1 / Archived Items  (i.e. 14,400 / 180,000) = 0.08
    3. Result of Items in 1GB * Result of Step 2 (i.e. 32,768 * 0.08) = 2,621 seconds
    4. Result of Step 3 / 60 / 60 (i.e. 2,621 / 60 / 60) = 0.72 (i.e 44 minutes)

    So now you have the following results for 1GB of items

     

    Size Of Items Time Taken To Archive Number of Items Archived
    1,024MB (1GB) 44 minutes 32,768 items

     

    Then multiply that all by 100 to get the 100GB
    And thats 100GB, 73 hours 20 minutes and 3,276,800 items archived
    So if you have a run time of 4 hours per night, that would take 18 nights of archiving to achieve that 100GB goal


     

  • Sarah_Seftel1's avatar
    Sarah_Seftel1
    Level 6
    12 years ago

    isn't there a tool where we can put the data and get results?
    Ho do we calculate the archive rate of 100GB from the tables in the link?

    I remember there was a sizing tool before...

  • Rob_Wilcox1's avatar
    Rob_Wilcox1
    Level 6
    12 years ago

    Partners that do Enterprise Vault installations may have a spreadsheet, but, the question 'how long will take to archive 100Gb' depends on a whole range of factors, as you have indicated in your original post.

     

    That being said at the top of page 20 in the PDF is an example that says an 8 core server can archive around 60,000 70 Kb items per hour.  That's 4 Gb by my reckoning, so that's 25 hours for your 100 Gb example.

  • JesusWept3's avatar
    JesusWept3
    Level 6
    12 years ago

    First off, sorry for the really really long post

    Yeah its just such a difficult thing to calculate, the performance guide is obviously going off of a best case scenario, you may have slightly slower disks on sql/exchange/vault store partitions, you may have network saturation at the point of archiving, you might have exchange maintenance going on etc etc, all of which can slow down the archiving rate

    you might come across items that have 10,000 recipients, extremely large items etc, you might have a GC that is slow to respond and it takes time to do security lookups and address lookups etc

    You might end up scanning a mailbox with 50,000 folders that has 40GB of mail that takes more time to scan than it does to actually archive anything

    The best thing you can do is a pilot against a couple of dozen users, run archiving overnight, or a couple of nights, calculate how many items were archived and the size and then judge it from there

    So you could say enable 100 users, run archiving in report mode and look at how many items are eligible for archiving, so it might say that 15,000 items are eligible for archiving, then let the schedule kick in

    Then do a query like


     

    USE EVVSYourVaultStore_1
SELECT COUNT(S.IdTransaction) "Item Count",
       SUM(S.ItemSize)/1024 "Size After Compression (MB)",
       SUM(SP.OriginalSize)/1024/1024 "Original Exchange Size (MB)"
FROM   Saveset S,
       SavesetProperty SP
WHERE  S.SavesetIdentity = SP.SavesetIdentity
  AND  S.ArchivedDate > '2013-20-01 23:00:00.000'
  AND  S.ArchivedDate < '2013-21-01 03:00:00.000'

     


    This will give you the item count and the total size in MB of items stored in EV , and the size of the items gotten from Exchange with a run of 4 hours (11pm to 3am)

    So you may get a result like

     

    Archived Items Size After Compression (MB) Original Exchange Size (MB)
    180,000 5091 5625

     


    So based on that you've archived 5,625MB on Exchange and 180,000 items in 4 hours.

    To calculate how many items and how much time it took for 1GB we do the following
    1. Archived Items / Original Size   (i.e. 180,000 items / 5,625MB) = 32
    2. Original Size - 1024 (1GB)  (i.e. 1,242 - 1024GB) = 4,601MB
    3. Result of Step 1 * Result of Step 2 (i.e. 32 * 4,601) = 147,232
    4. Archived Items - Result of Step 3 (i.e. 180,000 - 147,232) = 32,768 Items in 1GB (1024MB)

    So now you know that you have 32,768 items equate to 1GB

    Then you need to find out how much time that took
    1. Hours to run * 60 * 60 ...(i.e. 4 * 60 * 60) = 14,400 seconds
    2. Result of step 1 / Archived Items  (i.e. 14,400 / 180,000) = 0.08
    3. Result of Items in 1GB * Result of Step 2 (i.e. 32,768 * 0.08) = 2,621 seconds
    4. Result of Step 3 / 60 / 60 (i.e. 2,621 / 60 / 60) = 0.72 (i.e 44 minutes)

    So now you have the following results for 1GB of items

     

    Size Of Items Time Taken To Archive Number of Items Archived
    1,024MB (1GB) 44 minutes 32,768 items

     

    Then multiply that all by 100 to get the 100GB
    And thats 100GB, 73 hours 20 minutes and 3,276,800 items archived
    So if you have a run time of 4 hours per night, that would take 18 nights of archiving to achieve that 100GB goal


     

  • GertjanA's avatar
    GertjanA
    Moderator
    12 years ago

    Excellent explanation JW! This is worth an article! Thumbs up from me.

  • John_Santana's avatar
    John_Santana
    Level 6
    12 years ago

    Wow that's cool to know.

    But when I executed the SQL script I got this:

     

    Msg 242, Level 16, State 3, Line 2
    The conversion of a char data type to a datetime data type resulted in an out-of-range datetime value.
  • JesusWept3's avatar
    JesusWept3
    Level 6
    12 years ago

    Sorry, schoolboy error

    Change
     AND  S.ArchivedDate > '2013-20-01 23:00:00.000'
      AND  S.ArchivedDate < '2013-21-01 03:00:00.000'

    To
     AND  S.ArchivedDate > '2013-01-20 23:00:00.000'
      AND  S.ArchivedDate < '2013-01-21 03:00:00.000'

    the issue was that i was doing yyy-dd-mm and it should be yyyy-mm-dd