Package com.broadleafcommerce.search.index.core.queue

Interface QueueLoader<T>

  • Type Parameters:
    T - The type of Object the QueueLoader will load.
    All Known Implementing Classes:
    AbstractBatchIdQueueLoader, AbstractQueueLoader
    public interface QueueLoader<T>
    The purpose of this component is to coordinate or gather values from a data source or web service to put on a ReindexQueue for processing.

    While this is not necessarily the case, these values are most commonly IDs for the items that need to be processed. This allows a single thread to coordinate reading IDs, and delegate those IDs for other threads to process. With respect to multi-threaded processing, each thread needs to know which data to process. Each thread can begin with the first record and read a batch, but more than one thread would duplicate the work. On the other hand, you could allow each thread to read batches of records (e.g. 1-100). There are cases where that might not work without having a data source or some mechanism to .

    This component provides more control by allowing a single, master thread on a single node to read the data that need to be processed, and then delegates batches of data (typically IDs) to individual workers by queuing them up. The workers will then grab a batch of data off of the queue and augment/process that data by calling additional data sources or services. This allows the workers to share the I/O and read time, rather than having a single reader do the reading and create a bottleneck or starve the workers.

    Author:
    Kelly Tisdell (ktisdell)

    • Method Detail

      • populateQueue

        long populateQueue​(ReindexQueue<T> queue,
                   ProcessStateHolder processStateHolder)
            throws com.broadleafcommerce.search.api.SearchIndexException
        Reads T from a source (e.g. REST service, DB, etc.) and puts them on a queue for processing by another thread.
        Parameters:
        queue - the queue to populate
        processStateHolder - the state holder tracking this process
        Returns:
        the number of IDs that were read and put on the queue.
        Throws:
        com.broadleafcommerce.search.api.SearchIndexException - if the queue was unable to be populated