BatcherTo facilitate long-running write jobs, batches documents added
by many external threads and coordinates internal threads to send
the batches round-robin to all appropriate hosts in the cluster.
Appropriate hosts are those containing a forest associated with the
database for the DatabaseClient provided to DataMovementManager.
Many external threads (threads not managed by WriteBatcher) can
concurrently add documents by calling WriteBatcher
add or addAs.
Each time enough documents are added to make a batch, the batch is
added to an internal queue where the first available internal
thread will pick it up and write it to the server. Since batches
are not written until they are full, you should always call
flushAsync() or flushAndWait() when no more
documents will be written to ensure that any partial batch is
written.
WriteBatcher whb = dataMovementManager.newWriteBatcher()
.withBatchSize(100)
.withThreadCount(20)
.onBatchSuccess(batch -> {
logger.debug("batch # {}, so far: {}", batch.getJobBatchNumber(), batch.getJobWritesSoFar());
})
.onBatchFailure((batch,throwable) -> throwable.printStackTrace() );
JobTicket ticket = dataMovementManager.startJob(whb);
whb.add ("doc1.txt", new StringHandle("doc1 contents"));
whb.addAs("doc2.txt", "doc2 contents");
whb.flushAndWait(); // send the two docs even though they're not a full batch
dataMovementManager.stopJob(ticket);
Note: All Closeable content or metadata handles passed to
add methods will be closed as soon as possible
(after the batch is written). This is to avoid IO resource leakage.
This differs from the normal usage of the Java Client API because
WriteBatcher is asynchronous so there's no easy way to know which
handles have finished writing and can therefore be closed. So to
save confusion we close all handles for you. If you have a resource
that must be closed after a batch is written, but is not closed by
your handle, override the close method of any Closeable handle and
close your resource there.
addaddDocumentWriteOperation,
to be batched and then written to the server when a batch is full
or flushAsync() or
flushAndWait() is
called.addflushAsync() or flushAndWait() is called.addflushAsync() or flushAndWait() is called.voidaddAlladdAsflushAsync() or flushAndWait() is called.flushAsync() or flushAndWait() is called.booleanbooleanawaitCompletionvoidvoidonBatchFailureonBatchSuccessvoidretryvoidretryWithFailureListenersvoidsetBatchFailureListenersvoidsetBatchSuccessListenerswithBatchSizedefault WriteBatcherwithForestConfigDataMovementManager.readForestConfig()
then set via withForestConfig.withJobNamewithTemporalCollectionwithThreadCountwithTransformgetBatchSize,
getForestConfig,
getJobEndTime, getJobId, getJobName, getJobStartTime, getPrimaryClient, getThreadCount, isStarted, isStoppedhandle - the passed in DocumentMetadataHandleoperations - is the DocumentWriteOperation stream
passed in.Add a document to be batched then written to the server when a
batch is full or flushAsync() or flushAndWait() is called.
uri - the document uricontentHandle - the document contentsAdd a document to be batched then written to the server when a
batch is full or flushAsync() or flushAndWait() is called.
uri - the document uricontent - the document contentsAdd a document to be batched then written to the server when a
batch is full or flushAsync() or flushAndWait() is called.
uri - the document urimetadataHandle - the metadata (collection,
permissions, metdata values, properties, quality)contentHandle - the document contentsAdd a document to be batched then written to the server when a
batch is full or flushAsync() or flushAndWait() is called.
uri - the document urimetadataHandle - the metadata (collection,
permissions, metdata values, properties, quality)content - the document contentsdocs - the batch of WriteEvents where each
WriteEvent represents one documentAdd a document, by passing in a DocumentWriteOperation,
to be batched and then written to the server when a batch is full
or flushAsync() or
flushAndWait() is
called.
writeOperation - the DocumentWriteOperation object
containing the document's details to be written to the serverlistener - the action which has to be done when
the batch gets written successfullyAdd a listener to run each time there is an exception writing a batch.
These listeners will not run when an exception is thrown by a listener registered with onBatchSuccess. To learn more, please see Handling Exceptions in Listeners
listener - the code to run when a failure
occursqueryEvent - the information about the batch that
failedlisteners - the WriteBatchListener instances this
batcher should uselisteners - the WriteFailureListener instances
this batcher should usecollection - The temporal collection to use for a
temporal document inserttransform - The ServerTransform to run on each
document from each batch.DataMovementManager.readForestConfig()
then set via withForestConfig.
withForestConfig in interface BatcherforestConfig - the updated
ForestConfigurationwithJobName in
interface BatcherjobName - the name you would like to assign to
this jobwithBatchSize in
interface BatcherbatchSize - the batch size -- must be 1 or
greaterwithThreadCount in
interface BatcherthreadCount - the number of threads to use in this
Batchertimeout - the maximum time to waitunit - the time unit of the timeout argumentInterruptedException - if interrupted
while waitinggetJobTicket in
interface BatcherIllegalStateException - if this job has
not yet been startedRetry in the same thread to send a batch that failed. If it fails again, all the failure listeners associated with the batcher using onBatchFailure method would be processed.
Note : Use this method with caution as there is a possibility of infinite loops. If a batch fails and one of the failure listeners calls this method to retry with failure listeners and if the batch again fails, this would go on as an infinite loop until the batch succeeds.
writeBatch - the information about the batch that
failedfilter - the function to apply before writingCopyright (c) 2010-2025 Progress Software Corporation and/or its subsidiaries or affiliates. All Rights Reserved.