WriteBatcher (MarkLogic Java Client API 6.6.1)

All Superinterfaces:

Batcher
```
public interface WriteBatcher
extends Batcher
```
To facilitate long-running write jobs, batches documents added by many external threads and coordinates internal threads to send the batches round-robin to all appropriate hosts in the cluster. Appropriate hosts are those containing a forest associated with the database for the DatabaseClient provided to DataMovementManager. Many external threads (threads not managed by WriteBatcher) can concurrently add documents by calling WriteBatcher add or addAs. Each time enough documents are added to make a batch, the batch is added to an internal queue where the first available internal thread will pick it up and write it to the server. Since batches are not written until they are full, you should always call flushAsync() or flushAndWait() when no more documents will be written to ensure that any partial batch is written.
Sample Usage:
```
     WriteBatcher whb = dataMovementManager.newWriteBatcher()
         .withBatchSize(100)
         .withThreadCount(20)
         .onBatchSuccess(batch -> {
             logger.debug("batch # {}, so far: {}", batch.getJobBatchNumber(), batch.getJobWritesSoFar());
         })
         .onBatchFailure((batch,throwable) -> throwable.printStackTrace() );
     JobTicket ticket = dataMovementManager.startJob(whb);
     whb.add  ("doc1.txt", new StringHandle("doc1 contents"));
     whb.addAs("doc2.txt", "doc2 contents");

     whb.flushAndWait(); // send the two docs even though they're not a full batch
     dataMovementManager.stopJob(ticket);
```
Note: All Closeable content or metadata handles passed to add methods will be closed as soon as possible (after the batch is written). This is to avoid IO resource leakage. This differs from the normal usage of the Java Client API because WriteBatcher is asynchronous so there's no easy way to know which handles have finished writing and can therefore be closed. So to save confusion we close all handles for you. If you have a resource that must be closed after a batch is written, but is not closed by your handle, override the close method of any Closeable handle and close your resource there.

Method Summary

All Methods Instance Methods Abstract Methods
Modifier and Type	Method and Description
`WriteBatcher`	`add(DocumentWriteOperation writeOperation)` Add a document, by passing in a `DocumentWriteOperation`, to be batched and then written to the server when a batch is full or `flushAsync()` or `flushAndWait()` is called.
`WriteBatcher`	`add(java.lang.String uri, AbstractWriteHandle contentHandle)` Add a document to be batched then written to the server when a batch is full or `flushAsync()` or `flushAndWait()` is called.
`WriteBatcher`	`add(java.lang.String uri, DocumentMetadataWriteHandle metadataHandle, AbstractWriteHandle contentHandle)` Add a document to be batched then written to the server when a batch is full or `flushAsync()` or `flushAndWait()` is called.
`WriteBatcher`	`add(WriteEvent... docs)` Add docs in the form of WriteEvents.
`void`	`addAll(java.util.stream.Stream<? extends DocumentWriteOperation> operations)` Writes a document stream to the database.
`WriteBatcher`	`addAs(java.lang.String uri, DocumentMetadataWriteHandle metadataHandle, java.lang.Object content)` Add a document to be batched then written to the server when a batch is full or `flushAsync()` or `flushAndWait()` is called.
`WriteBatcher`	`addAs(java.lang.String uri, java.lang.Object content)` Add a document to be batched then written to the server when a batch is full or `flushAsync()` or `flushAndWait()` is called.
`boolean`	`awaitCompletion()` Blocks until the job has finished or cancelled all queued tasks.
`boolean`	`awaitCompletion(long timeout, java.util.concurrent.TimeUnit unit)` Blocks until the job has finished or cancelled all queued tasks.
`void`	`flushAndWait()` Create a batch from any unbatched documents and write that batch, then wait for all batches to complete (the same as awaitCompletion().
`void`	`flushAsync()` Create a batch from any unbatched documents and write that batch asynchronously.
`WriteFailureListener[]`	`getBatchFailureListeners()` Get the array of WriteFailureListener instances registered via onBatchFailure including the HostAvailabilityListener registered by default.
`WriteBatchListener[]`	`getBatchSuccessListeners()` Get the array of WriteBatchListener instances registered via onBatchSuccess.
`DocumentMetadataHandle`	`getDocumentMetadata()`
`JobTicket`	`getJobTicket()` After the job has been started, returns the JobTicket generated when the job was started.
`java.lang.String`	`getTemporalCollection()` The temporal collection configured for temporal document inserts
`ServerTransform`	`getTransform()`
`WriteBatcher`	`onBatchFailure(WriteFailureListener listener)` Add a listener to run each time there is an exception writing a batch.
`WriteBatcher`	`onBatchSuccess(WriteBatchListener listener)` Add a listener to run each time a batch is successfully written.
`void`	`retry(WriteBatch queryEvent)` Retry in the same thread to send a batch that failed.
`void`	`retryWithFailureListeners(WriteBatch writeBatch)` Retry in the same thread to send a batch that failed.
`void`	`setBatchFailureListeners(WriteFailureListener... listeners)` Remove any existing WriteFailureListener instances registered via onBatchFailure including the HostAvailabilityListener registered by default and replace them with the provided listeners.
`void`	`setBatchSuccessListeners(WriteBatchListener... listeners)` Remove any existing WriteBatchListener instances registered via onBatchSuccess and replace them with the provided listeners.
`WriteBatcher`	`withBatchSize(int batchSize)` Sets the number of documents to send per batch.
`WriteBatcher`	`withDefaultMetadata(DocumentMetadataHandle handle)` Sets the DocumentMetadataHandle for write operations.
`WriteBatcher`	`withForestConfig(ForestConfiguration forestConfig)` If the server forest configuration changes mid-job, it can be re-fetched with `DataMovementManager.readForestConfig()` then set via withForestConfig.
`WriteBatcher`	`withJobId(java.lang.String jobId)` Sets the unique id of the job to help with managing multiple concurrent jobs and start the job with the specified job id.
`WriteBatcher`	`withJobName(java.lang.String jobName)` Sets the job name.
`WriteBatcher`	`withTemporalCollection(java.lang.String collection)` The temporal collection to use for a temporal document insert
`WriteBatcher`	`withThreadCount(int threadCount)` Sets the number of threads added to the internal thread pool for this instance to use for writing or reporting on batches of uris.
`WriteBatcher`	`withTransform(ServerTransform transform)` The ServerTransform to modify each document from each batch before it is written to the database.

Methods inherited from interface com.marklogic.client.datamovement.Batcher
getBatchSize, getForestConfig, getJobEndTime, getJobId, getJobName, getJobStartTime, getPrimaryClient, getThreadCount, isStarted, isStopped

- Method Detail
  - withDefaultMetadata
```
WriteBatcher withDefaultMetadata(DocumentMetadataHandle handle)
```
    Sets the DocumentMetadataHandle for write operations.
    
    Parameters:
    
    handle - the passed in DocumentMetadataHandle
    
    Returns:
    
    this write batcher for chaining configuration
  - addAll
```
void addAll(java.util.stream.Stream<? extends DocumentWriteOperation> operations)
```
    Writes a document stream to the database.
    
    Parameters:
    
    operations - is the DocumentWriteOperation stream passed in.
  - getDocumentMetadata
```
DocumentMetadataHandle getDocumentMetadata()
```
    Returns:
    
    the documentMetadatHandle associated with the WriteeBatcher.
  - add
```
WriteBatcher add(java.lang.String uri,
                 AbstractWriteHandle contentHandle)
```
    Add a document to be batched then written to the server when a batch is full or flushAsync() or flushAndWait() is called.
    See Also:
    the Java Guide for more on using handles
    
    Parameters:
    
    uri - the document uri
    
    contentHandle - the document contents
    
    Returns:
    
    WriteBatcher the batcher containing the documents added
  - addAs
```
WriteBatcher addAs(java.lang.String uri,
                   java.lang.Object content)
```
    Add a document to be batched then written to the server when a batch is full or flushAsync() or flushAndWait() is called.
    See Also:
    IO Shortcut in MarkLogic Java Client API for more on using the *As shortcut methods
    
    Parameters:
    
    uri - the document uri
    
    content - the document contents
    
    Returns:
    
    WriteBatcher the batcher containing the documents added
  - add
```
WriteBatcher add(java.lang.String uri,
                 DocumentMetadataWriteHandle metadataHandle,
                 AbstractWriteHandle contentHandle)
```
    Add a document to be batched then written to the server when a batch is full or flushAsync() or flushAndWait() is called.
    See Also:
    the Java Guide for more on using handles
    
    Parameters:
    
    uri - the document uri
    
    metadataHandle - the metadata (collection, permissions, metdata values, properties, quality)
    
    contentHandle - the document contents
    
    Returns:
    
    WriteBatcher the batcher containing the documents added
  - addAs
```
WriteBatcher addAs(java.lang.String uri,
                   DocumentMetadataWriteHandle metadataHandle,
                   java.lang.Object content)
```
    Add a document to be batched then written to the server when a batch is full or flushAsync() or flushAndWait() is called.
    See Also:
    IO Shortcut in MarkLogic Java Client API for more on using the *As shortcut methods
    
    Parameters:
    
    uri - the document uri
    
    metadataHandle - the metadata (collection, permissions, metdata values, properties, quality)
    
    content - the document contents
    
    Returns:
    
    WriteBatcher the batcher containing the documents added
  - add
```
WriteBatcher add(WriteEvent... docs)
```
    Add docs in the form of WriteEvents. This is a convenience method for re-adding documents from failed batches.
    
    Parameters:
    
    docs - the batch of WriteEvents where each WriteEvent represents one document
    
    Returns:
    
    WriteBatcher the batcher containing the documents added
  - add
```
WriteBatcher add(DocumentWriteOperation writeOperation)
```
    Add a document, by passing in a DocumentWriteOperation, to be batched and then written to the server when a batch is full or flushAsync() or flushAndWait() is called.
    
    Parameters:
    
    writeOperation - the DocumentWriteOperation object containing the document's details to be written to the server
    
    Returns:
    
    WriteBatcher the batcher containing the documents added
  - onBatchSuccess
```
WriteBatcher onBatchSuccess(WriteBatchListener listener)
```
    Add a listener to run each time a batch is successfully written.
    
    Parameters:
    
    listener - the action which has to be done when the batch gets written successfully
    
    Returns:
    
    this instance for method chaining
  - onBatchFailure
```
WriteBatcher onBatchFailure(WriteFailureListener listener)
```
    Add a listener to run each time there is an exception writing a batch.
    
    These listeners will not run when an exception is thrown by a listener registered with onBatchSuccess. To learn more, please see Handling Exceptions in Listeners
    
    Parameters:
    
    listener - the code to run when a failure occurs
    
    Returns:
    
    this instance for method chaining
  - retry
```
void retry(WriteBatch queryEvent)
```
    Retry in the same thread to send a batch that failed. This method will throw an Exception if it fails again, so it can be wrapped in a try-catch block.
    
    Parameters:
    
    queryEvent - the information about the batch that failed
  - getBatchSuccessListeners
```
WriteBatchListener[] getBatchSuccessListeners()
```
    Get the array of WriteBatchListener instances registered via onBatchSuccess.
    
    Returns:
    
    the WriteBatchListener instances this batcher is using
  - getBatchFailureListeners
```
WriteFailureListener[] getBatchFailureListeners()
```
    Get the array of WriteFailureListener instances registered via onBatchFailure including the HostAvailabilityListener registered by default.
    
    Returns:
    
    the WriteFailureListener instances this batcher is using
  - setBatchSuccessListeners
```
void setBatchSuccessListeners(WriteBatchListener... listeners)
```
    Remove any existing WriteBatchListener instances registered via onBatchSuccess and replace them with the provided listeners.
    
    Parameters:
    
    listeners - the WriteBatchListener instances this batcher should use
  - setBatchFailureListeners
```
void setBatchFailureListeners(WriteFailureListener... listeners)
```
    Remove any existing WriteFailureListener instances registered via onBatchFailure including the HostAvailabilityListener registered by default and replace them with the provided listeners.
    
    Parameters:
    
    listeners - the WriteFailureListener instances this batcher should use
  - withTemporalCollection
```
WriteBatcher withTemporalCollection(java.lang.String collection)
```
    The temporal collection to use for a temporal document insert
    
    Parameters:
    
    collection - The temporal collection to use for a temporal document insert
    
    Returns:
    
    this instance for method chaining
  - getTemporalCollection
```
java.lang.String getTemporalCollection()
```
    The temporal collection configured for temporal document inserts
    
    Returns:
    
    The temporal collection configured for temporal document inserts
  - withTransform
```
WriteBatcher withTransform(ServerTransform transform)
```
    The ServerTransform to modify each document from each batch before it is written to the database.
    
    Parameters:
    
    transform - The ServerTransform to run on each document from each batch.
    
    Returns:
    
    this instance for method chaining
  - getTransform
```
ServerTransform getTransform()
```
  - withForestConfig
```
WriteBatcher withForestConfig(ForestConfiguration forestConfig)
```
    If the server forest configuration changes mid-job, it can be re-fetched with DataMovementManager.readForestConfig() then set via withForestConfig.
    
    Specified by:
    
    withForestConfig in interface Batcher
    
    Parameters:
    
    forestConfig - the updated ForestConfiguration
    
    Returns:
    
    this instance for method chaining
  - withJobName
```
WriteBatcher withJobName(java.lang.String jobName)
```
    Sets the job name. Eventually, this may become useful for seeing named jobs in ops director.
    
    Specified by:
    
    withJobName in interface Batcher
    
    Parameters:
    
    jobName - the name you would like to assign to this job
    
    Returns:
    
    this instance for method chaining
  - withJobId
```
WriteBatcher withJobId(java.lang.String jobId)
```
    Sets the unique id of the job to help with managing multiple concurrent jobs and start the job with the specified job id.
    
    Specified by:
    
    withJobId in interface Batcher
    
    Parameters:
    
    jobId - the unique id you would like to assign to this job
    
    Returns:
    
    this instance (for method chaining)
  - withBatchSize
```
WriteBatcher withBatchSize(int batchSize)
```
    Sets the number of documents to send per batch. Since documents are large relative to uris, this number should be much lower than the batch size for QueryBatcher. The default batch size is 100.
    
    Specified by:
    
    withBatchSize in interface Batcher
    
    Parameters:
    
    batchSize - the batch size -- must be 1 or greater
    
    Returns:
    
    this instance for method chaining
  - withThreadCount
```
WriteBatcher withThreadCount(int threadCount)
```
    Sets the number of threads added to the internal thread pool for this instance to use for writing or reporting on batches of uris. Each time enough documents are added to fill a batch, a batch is created and a task is queued to write the batch. As a thread becomes available it grabs a task from the queue and performs the task (usually writing the batch to the server then reporting on the batch to listeners registered with onBatchSuccess and onBatchFailure). By default the number of threads is the number of hosts containing applicable forests. More threads should accommodate more throughput.
    
    Specified by:
    
    withThreadCount in interface Batcher
    
    Parameters:
    
    threadCount - the number of threads to use in this Batcher
    
    Returns:
    
    this instance for method chaining
  - flushAsync
```
void flushAsync()
```
    Create a batch from any unbatched documents and write that batch asynchronously.
  - flushAndWait
```
void flushAndWait()
```
    Create a batch from any unbatched documents and write that batch, then wait for all batches to complete (the same as awaitCompletion().
  - awaitCompletion
```
boolean awaitCompletion()
```
    Blocks until the job has finished or cancelled all queued tasks.
    
    Returns:
    
    true if the queue completed without InterruptedException, false if we hit the time limit or InterruptedException was thrown while waiting
  - awaitCompletion
```
boolean awaitCompletion(long timeout,
                        java.util.concurrent.TimeUnit unit)
                 throws java.lang.InterruptedException
```
    Blocks until the job has finished or cancelled all queued tasks.
    
    Parameters:
    
    timeout - the maximum time to wait
    
    unit - the time unit of the timeout argument
    
    Returns:
    
    true if the queue completed without timing out, false if we hit the time limit
    
    Throws:
    
    java.lang.InterruptedException - if interrupted while waiting
  - getJobTicket
```
JobTicket getJobTicket()
```
    After the job has been started, returns the JobTicket generated when the job was started.
    
    Specified by:
    
    getJobTicket in interface Batcher
    
    Returns:
    
    the JobTicket generated when this job was started
    
    Throws:
    
    java.lang.IllegalStateException - if this job has not yet been started
  - retryWithFailureListeners
```
void retryWithFailureListeners(WriteBatch writeBatch)
```
    Retry in the same thread to send a batch that failed. If it fails again, all the failure listeners associated with the batcher using onBatchFailure method would be processed.
    
    Note : Use this method with caution as there is a possibility of infinite loops. If a batch fails and one of the failure listeners calls this method to retry with failure listeners and if the batch again fails, this would go on as an infinite loop until the batch succeeds.
    
    Parameters:
    
    writeBatch - the information about the batch that failed

Interface WriteBatcher

Method Summary

Methods inherited from interface com.marklogic.client.datamovement.Batcher

Method Detail

withDefaultMetadata

addAll

getDocumentMetadata

add

addAs

add

addAs

add

add

onBatchSuccess

onBatchFailure

retry

getBatchSuccessListeners

getBatchFailureListeners

setBatchSuccessListeners

setBatchFailureListeners

withTemporalCollection

getTemporalCollection

withTransform

getTransform

withForestConfig

withJobName

withJobId

withBatchSize

withThreadCount

flushAsync

flushAndWait

awaitCompletion

awaitCompletion

getJobTicket

retryWithFailureListeners