BE2012 Disk Storage Pool Behavior

[melview1]

I have 4 external USB3.0 drives attached to my BE2012 server.  I have configured the backup jobs for my 6 servers to these disks.  After running for a couple weeks to get a handle on my storage requirements, I believe my storage capacity is sufficient, but barely.  Rather than replace 3 and 4TB disks with larger ones or a NAS or something, I have decided to create storage pools.  Not revolutionary, I know.

Storage Pool 1 includes Disk A and D

Storage Pool 2 includes Disk B and D

Storage Pool 3 includes Disk C and D

As mentioned, Disk A, B and C are sufficient for the backup jobs to run without issue when not using pools, however 2 of them are up to 90% full at any one time.  What I would like to happen with the pools is BE to continue to backup as normal to Disks A, B and C and then use Disk D as an overflow disk when it warrants due to a full disk.  Basically, Disk D would be empty until a backup "overflows" to it due to Disk A, B or C becoming full.  I have reconfigured my backup jobs to point to the pools instead of the individual disks.  None of my backup job windows overlap.

My question is what order does BE2012 choose to use the disks in the pool?  Will it operate as I desire, always using the first disk in the pool unless the device is busy?  Or some other way based on utilizing all available resources for performance reasons?

Thanks.

--Mark

[Colin_Weaver]

It's chosen in Alphanumeric order based on the names of the disk storage devices (inlcuding which  are online, non-full and, for parallel starting jobs, the number of concurrent writes configured on the disk device) within the pool.

BE aware that GRT enabled jobs cannot span between disk storage devices should the device fill up in the middle of a job, the job will sit asking for media and fail.

Also BE 2014 will have the ability to use devices based on most free space, least free space  or round robin order (but not alphanumeric order other than within the round robin). This ability will be set by BEMCLI command and not within the GUI. BE 2010 R3 and earlier did round robin only.

[melview1]

Thank you Colin.

I'd understand if it wasn't able to accomodate the disk filling up due to other simulatneous jobs while it is running and not being able to dynamically span disks, but let's pretend it's always a single job to a single disk and never running at the same time.  Does the job check the available space before beginning?  Or is there a "pre-scan" setting that I can enable that would do this?  For instance, check the first disk in alphanumeric order, if enough space, continue, otherwise, check available space on the next disk? 

If there isn't some sort of check, I'd think that every job would fail when a disk gets full and would never go to the next available disk in a pool.

Thanks.

[Spencer_Simpson]

I recently had a support call about this, and the answer is a decided NO.   The only check is done in the middle of a job, when it's time to create a new B2D file in the volume. If there isn't enough space, BE checks the next device in the pool, then the next one until it finds a device with enough space.  

I'm not sure what Colin meant by "alphanumeric order".  He lists a bunch of criteria but it's unclear how they result in a name.

I have a pool with 3 devices and round robin appears to operate in the order the devices were created and/or originally added to the pool.  It's definitely not the names of the devices, but it might be the REVERSE order of the drive letter names (in my case G:\,F:\, E:\)

[melview1]

I have 6 backup jobs that target 3 pools.  Each pool contains a unique disk, plus an "overflow" disk, and this disk is the same disk for all pools.  Since I implemented pools early last week, every backup job appeared to target the Overflow disk, rather than it's unique disk despite having terabytes free on each unique disk.  All disks were online and there were no parallel jobs running at any time.  On Sunday, the Overflow disk became full mid-job and transitioned to its pool's unique disk without issue.  Although, this happened to be a job that wasn't using GRT.

I can come up with 2 explanations of this behavior: the Overflow drive was the newest drive added to BE or that alphanumerically, 'O' is last in all pools.  That would mean that it is either choosing the newest drive to that was added to BE or it is doing a reverse-alphanumeric order.

Pool/Alphanumeric Begginging Letter/Drive Letter:

Pool A:  D(F:\); Overflow(G:\)

Pool B:  Overflow(G:\); F(H:\)

Pool C:  B(J:\); Overflow(G:\)

If I could get it to do exactly the opposite of what it's doing right now, target the unique disks first, I would be happy.  If a GRT job happen to fail due to a full disk mid-job and doesn't transition to the next disk in the pool as Colin suggests it won't, I guess I'll have to deal with that if it happens.

Thoughts?

[Spencer_Simpson]

Was the job that worked targeting pool B? Anyway, GRT technology should only be used to backup resources that benefit from it (such as System State) and not mixed with other resources (database and file system). Isolate the different types of resources in different sets of jobs. If volumes are indeed targeted in reverse drive letter order you might be able to get BE to use backup pools the way you want by changing drive letters. If you target your GRT backups to a specific device, you might be able to ferret out what the best arrangement is for your other jobs.

[Spencer_Simpson]

Was the job that worked targeting pool B? Anyway, GRT technology should only be used to backup resources that benefit from it (such as System State) and not mixed with other resources (database and file system). Isolate the different types of resources in different sets of jobs. If volumes are indeed targeted in reverse drive letter order you might be able to get BE to use backup pools the way you want by changing drive letters. If you target your GRT backups to a specific device, you might be able to ferret out what the best arrangement is for your other jobs.

[melview1]

Pool C was the job that successfully switched disks mid-job.  It was not a GRT job though.

I'm thinking that the disk is chosen by either:

a. Newest Disk Added to BE

          -  to test, I would have to remove disks and re-add, but add the overflow disk first

b. Reverse Alphanumeric Order

          -  to test, I would rename the Overflow disk to something like ~Overflow or aOverflow

I probably won't be able to test until this weekend since the overflow disk is full and contains the latest backup sets.  I'm open to other suggestions and or theories of how BE actually chooses which disk is first in a pool.

Thanks thus far Colin and Spencer.

--Mark

[Spencer_Simpson]

Well, the GRT jobs won't ever switch drives, no matter what you do.

That's one of the reasons why you want to do your file system, database, and System State backups all in separate jobs.

Anyway, if the job targeting Pool C started with drive J then successfuly switched to drive G then it might still be reverse drive letter or folder path.

I can guarantee it's not the device friendly names in any order (what you apear to want to do); My 3-device pool has

R0 - F:\BEData\
R1 - E:\BEData\
S - G:\BEData\

and BE always chooses S, R0, R1.  Note that the last one is in the middle of the friendly-naming sequence.

[Colin_Weaver]

OK I just did some testing with 3 B2D folders

1st Job appeared to go to alphanumerically first disk storage device (based on the name inside the BE console)  - this also happened to be the lowest local drive letter as well

I then removed that device from the pool and ran another backup - this job did use what has been the second drive alphanumerically (by both BE console name and drive letter) Note: I mainly did this step so I could then put the first drive back in and see if the order added to the pool makes a difference (as all the devices were added to the pool at the same time initially)

I then added the first drive back into the pool and ran another backup - this jobs went back to the first device (which actually proves that it is not the order added to the pool, or the last drive used by a job as long as that drive is online)

I then renamed inside the BE storage tab the first drive so that it starts with a 'z' and ran another backup - which still used that drive and kind of means:

a) It is not an alphanumeric order choice (and I apologise for the wrong detail provided earlier in this thread)

b) it is possibly chosen by order added to the database (internal identifier) and previously most of us have just tested in ways where the order added to BE and the alphanumerical order were the same and then made an assumption based on what the operation looked like at first glance.

I then noticed that when you look inside the details of the pool and go to the properties section that the drive I renamed to start with a z still shows as listed first. Which does potentially give a way to identify what the order of drives is - especially as the same order is seen when you create a new pool as well. This screen may confirm that we do base it on the DeviceID/Order of first creating the disk device inside Backup Exec. Note: with care you could look at the order of "DeviceID" against "DeviceName" inside the "Device" table of the Backup Exec database just to confirm it, however DO NOT be tempted to manually edit inside the database, as being a relational database identifiers in this table are referenced elsewhere.

Of course all this information helps with is the initial choice within a job, it won't change the behaviour of spanning or make it choose the drive with most free space

[Spencer_Simpson]

I did a direct SQL query on BEDB and got the following for my storage pool:

DeviceID DeviceName TargetID FolderPath CreationDateTime
-------- ---------- -------- ---------- -----------------------
1012     S          1        G:\BEData\ 2012-03-22 20:26:44.420
1024     R0         7        F:\BEData\ 2012-12-14 16:46:17.897
1025     R1         8        E:\BEData\ 2012-12-14 17:06:37.160

DeviceID and TargetID,values are most likely generated in the same order of creation, so those and CreationDateTime are all possibilities. But I agree it's probably DeviceID. 

[Spencer_Simpson]

Just FYI, those results were pared down from the original query:

use bedb
go
select PoolID, PoolName, b2d.DeviceID, b2d.DeviceName, b2d.TargetID, b2d.FolderPath, b2d.CreationDateTime
       from (select * from dbo.device d 
             right outer join dbo.BackupToDiskFolder b2df
              on DeviceID=b2df.BackupToDiskID) b2d
       right outer join (select dpd.DeviceID, dp.DeviceID PoolID, dp.DeviceName PoolName
                         from dbo.DevicePool_Device dpd
                              left outer join dbo.device dp
                              on dpd.DevicePoolID=dp.DeviceID) dj
             on dj.DeviceID=b2d.DeviceID

[Colin_Weaver]

Nice query

[melview1]

Update:

I have not been able to get this to function as I would like, so with my limited time and resources to address it further, I chose to abandon storage pools.  Perhaps I'll revisit this another time.

Thank you to everyone for your insight and discussion.

--Mark

[Colin_Weaver]

FYI BE 2014 does not work like BE 2012 with respect to disk devcie pool choices.

2014 Default is round robin (as it used to be in BE 2010)

other two choices are

- Most full disk first

- Least full disk first

You change this behaviour using BEMCLI (the choice is not visible in the BE admin console)

[melview1]

Thanks Colin.

I presume it will chose the disk with the most/least disk space, but still wouldn't check if the available space is adequate for the amount of data being backed up, correct?  That's just something it can't do.

Can you define round robin?  Does each subsequent job choose the next disk in the round robin order based on what the last job last used?  Or does it choose the next disk in the round robin order based on what the job last used?

[Colin_Weaver]

1) No we won't pre-calculate the data as that would require an extra process before the job actually starts moving data which would impact your overall backup window

2) Round robon is first job uses first disk, 2nd job uses 2nd, 3rd job uses 3rd but if only 3 disks then 4th job would go back to 1st etc with exact order based on how you added the disks to Backup Exec in the first place. This is how it worked in BE 2010 and we have gone back to the same method. EDIT: although I haven't checked this yet by actually trying it myself in BE 2014 as the last test I did was re-running the same job and not running different jobs.