We're experiencing very slow indexing speed on our 12.3.2 EV platform.
We're using ECS.
The archiving speed is extremely fast (more than 30 million items per day), however, the indexing only manages to process 4.5 million items per day. Our EV servers (we have 18 of them) are not struggling at all, their CPU usage never goes over 15%, and the RAM usage is on par with that : 15% to 20% usage.
Each server is able to process around 250k items a day (4.5 million items in total for our platform).
We checked with the network team, there is no bottleneck on their side, same thing with the storage team : ECS is far from being overloaded.
The EVTemp folder is located on a local disk, and its IOPS is very low at the moment, between 1MB and 5MB/sec.
What could be the bottleneck here ? Is it really on EV side ?
What is the backlog of indexes per vault store? You can enable the option under Monitoring in site properties and run a check.
With that kind of numbers on archiving, I would check the SQL database fragementation levels.
Look at the configurations for Indexing on the site and indexing servers properties. Maybe you can fine tune that.
Maybe a Dtrace of the index admin service would also help.
The backlog on indexing go from 500k to several millions, depending on the vault store.
It seems like only 200k items per day per server is indexed.
SQL defrag is ran twice a week, however, how can we be sure it's processing the right tables, and running the right jobs ?
Is there a best practice specific to SQL defrag to speed up indexing ?
We tried to modify several options on indexing (advanced properties on the servers), and it doesn't seem to change anything.
I ran a few Dtraces on those services, but no error comes up, it looks like regular process.
I feel like the fetching of data from ECS to EV is very slow and inconsistent, and the CPU/RAM resources on EV servers are not used at all.
Access archived emails, performing search on your client and see how long it takes to fetch the data. Not very accurate but atleast gives a general idea.
The SQL databases tend to have high fragmentation if there are large number of read/write operations. You can ask your DBA's to check the fragmentation levels and confirm if the maintenance jobs are completing on time. We have rebuild Index and stats jobs and are run daily at night to keep the frag level under 30%.
For optimal performance, logical fragmentation needs to be below 10% and Extent below 70%
With that level of ingestion, you do need to keep your database in shape. As you probably know, there is a document describing what maintenance to do on the SQL databases. Our DBA's use scripts from below source. We've had a healthcheck done by VRTs, they mention the same source.
We were using 12 archiving servers, to archive approximately 3.2 million messages per day, from Journal Mailboxes, and had an Index Server group consisting of 4 servers, which could keep up reasonably.
We currently have 5 SMTP Archiving servers, which do their own indexing, who keep up fine. And (due to killing the overhead when using mbx archiving) the items archived drops to 2.3 mill, indexes is faster, and lesser footprint.
IOPS are usually not measured in MB/s. It is the count of in and output operations per second.
The index should reside on relatively fast storage. You could run an IOPS Test.
An SQL index defragmentation could also increase the speed.
I would assume the bottleneck is either the storage or the SQL Server.