Forum Discussion

Choucroute's avatar
Choucroute
Level 3
6 years ago

Very slow indexing

Hi,

We're experiencing very slow indexing speed on our 12.3.2 EV platform.

We're using ECS.

The archiving speed is extremely fast (more than 30 million items per day), however, the indexing only manages to process 4.5 million items per day. Our EV servers (we have 18 of them) are not struggling at all, their CPU usage never goes over 15%, and the RAM usage is on par with that : 15% to 20% usage.

Each server is able to process around 250k items a day (4.5 million items in total for our platform).

We checked with the network team, there is no bottleneck on their side, same thing with the storage team : ECS is far from being overloaded.

The EVTemp folder is located on a local disk, and its IOPS is very low at the moment, between 1MB and 5MB/sec.

What could be the bottleneck here ? Is it really on EV side ?

Thanks.

Best regards,

 

  • OldEmployee's avatar
    OldEmployee
    Level 5
    6 years ago

    What is the backlog of indexes per vault store? You can enable the option under Monitoring in site properties and run a check.

    With that kind of numbers on archiving, I would check the SQL database fragementation levels.

    Look at the configurations for Indexing on the site and indexing servers properties. Maybe you can fine tune that.

    Maybe a Dtrace of the index admin service would also help.

    • Choucroute's avatar
      Choucroute
      Level 3
      6 years ago

      The backlog on indexing go from 500k to several millions, depending on the vault store.

      It seems like only 200k items per day per server is indexed.

      SQL defrag is ran twice a week, however, how can we be sure it's processing the right tables, and running the right jobs ?

      Is there a best practice specific to SQL defrag to speed up indexing ?

      We tried to modify several options on indexing (advanced properties on the servers), and it doesn't seem to change anything.

      I ran a few Dtraces on those services, but no error comes up, it looks like regular process.

      I feel like the fetching of data from ECS to EV is very slow and inconsistent, and the CPU/RAM resources on EV servers are not used at all.

      Thanks.

      • OldEmployee's avatar
        OldEmployee
        Level 5
        6 years ago

        Access archived emails, performing search on your client and see how long it takes to fetch the data. Not very accurate but atleast gives a general idea.

        The SQL databases tend to have high fragmentation if there are large number of read/write operations. You can ask your DBA's to check the fragmentation levels and confirm if the maintenance jobs are completing on time. We have rebuild Index and stats jobs and are run daily at night to keep the frag level under 30%.

        For optimal performance, logical fragmentation needs to be below 10% and Extent below 70%

        https://www.veritas.com/support/en_US/article.100031020

  • CConsult's avatar
    CConsult
    Moderator
    6 years ago

    Hi,

    IOPS are usually not measured in MB/s. It is the count of in and output operations per second.

    The index should reside on relatively fast storage. You could run an IOPS Test.

    An SQL index defragmentation could also increase the speed.

    I would assume the bottleneck is either the storage or the SQL Server.