Index Engines System Can Index a Billion Objects

Index Engines, providers of enterprise discovery technology, announced support for index and search capability up to one billion data objects in a single engine. Full text and metadata is indexed in 250 million object segments and through a centralized index, allowing a billion objects to be fully queried and extracted within one appliance. Index Engines unmatched discovery capabilities can handle data management for extremely large environments. This technology allows information management professionals to intelligently manage, store and retrieve electronic information from vast enterprise or government information networks.

"Our customers have told us that speed and scale are the critical factors when considering an information management solution," said Jim McGann, Vice President of Marketing for Index Engines. "At Index Engines we have taken this message to heart and have built our indexing platform to keep pace with the volume of data created in very large, global corporations, some storing more data than exists on the Internet. Our flexible platform is compatible with any retention program, and also powerful enough to execute these same policies against enterprise class data centers."

Indexing capacity of the Index Engines system of a billion objects is based on a 2U Engine with 6TB of storage. This metric assumes data with an average object size of 20K (mostly email) and an index footprint of 8%. The average Index Engines index footprint ranges from 4 to 8%. The system’s actual indexing capability far exceeds one billion, but to maintain deterministic query and extraction speeds, only one billion is currently supported.

Index Engines uses either high speed Network Data Management Protocols (NDMP) or multi-threaded NFS/CIFS scanning to index network data. At speeds several orders of magnitude faster than other indexing approaches, the non-invasive NDMP method pushes incremental data to the Index Engines system for indexing, without making a cache copy of the content or adding any load onto the server. The resulting index is generated at speeds up to 70 MB per second, with a footprint that is only 4 to 8% of the original files. Multi-threaded crawling allows for the efficient processing of data that is not centrally stored on an NDMP enabled file server.  With either approach, or using a combination of the two methods, Index Engine system, is optimized to index large data repositories quickly and efficiently in order to improve access and management of network content.