Type Members

1.  class ErrorHandler extends KeyValueWriteErrorHandler

   Error handler that will receive any errors encountered by the Couchbase Spark Connector while trying to write Key-Value DataFrame operations.

   Error handler that will receive any errors encountered by the Couchbase Spark Connector while trying to write Key-Value DataFrame operations.

   Error handling is very application-dependent and users are encouraged to copy this code into their own application and adapt to their needs. It should be seen as an example.

   It is recommended that users prefer using KeyValueOptions.ErrorBucket instead, which will write errors to a specified collection on the Couchbase cluster. This is a much simpler and more versatile form of error handling, as Apache Spark's architecture means this ErrorHandler class does have some caveats that limit its usefulness:

   Where this executes ------------------- It's very important to note that this will execute on the Sparker executor (worker), and not inside the driver application. When doing initial testing with .master("local[*]") these are one and the same. But when deploying the application to a Spark cluster, the workers are usually separate nodes. Many things that will work fine in local testing - like maintaining a list of received errors and processing them in the application - will not work on a real Spark cluster. The ErrorHandler in the application will never execute.

   The critical point: There is no way for the application to programmatically get access to the values that are passed to this handler. To over-simplify what Spark does - it will run your application, and also copy your application to the executors and run it there. Each copy of that application will have their own independent copy of this ErrorHandler, running on separate JVM processes, generally on different nodes. The Spark Connector will be running key-value operations on the Spark executors, and sending failures to the copies of ErrorHandler that execute there.

   So ErrorHandler logs any failures and also appends them to a JSONL file, and it's crucial to understand that all of this logic will execute on the Spark executor (worker). So logging will appear in the Spark executor logs, and the file will be created on the Spark executor. It will be up to the application to find and copy these files from the Spark executors later. Per above, users may find KeyValueOptions.ErrorBucket a simpler solution - the application can simply fetch any errors from Couchbase after the job.

2.  class ErrorQueueManager extends Logging
3.  case class Get(id: String) extends KeyValueOperation with Product with Serializable
4.  class GetRDD extends RDD[GetResult] with Logging
5.  case class Insert[T](id: String, content: T) extends KeyValueOperation with Product with Serializable
6.  class InsertRDD[T] extends RDD[MutationResult] with Logging
7.  class KeyValueContinuousStream extends KeyValueDataStream with ContinuousStream
8.  class KeyValueDataStream extends SparkDataStream with Logging
9.  class KeyValueErrorDocumentHandler extends Logging with Serializable
10.  case class KeyValueInputPartition(schema: StructType, partitionOffset: KeyValuePartitionOffset, conf: CouchbaseConfig, config: KeyValueStreamConfig) extends InputPartition with Product with Serializable
11.  class KeyValueMicroBatchStream extends KeyValueDataStream with MicroBatchStream
12.  case class KeyValueOffset(offsets: List[PartitionOffset]) extends Offset with Product with Serializable
13.  sealed abstract class KeyValueOperation extends AnyRef
14.  class KeyValuePartition extends Partition
15.  case class KeyValuePartitionOffset(streamStartOffsets: Map[Int, KeyValueStreamOffset], streamEndOffsets: Option[Map[Int, KeyValueStreamOffset]]) extends PartitionOffset with Product with Serializable

    Holds the offset information for an individual vbucket.

16.  class KeyValuePartitionReader extends ContinuousPartitionReader[InternalRow] with Logging

    The KeyValuePartitionReader is responsible for actually streaming the mutations from a number of vbuckets.

17.  class KeyValueScan extends Scan
18.  class KeyValueScanBuilder extends ScanBuilder
19.  class KeyValueSourceInitialOffsetWriter extends HDFSMetadataLog[Map[Int, KeyValueStreamOffset]]
20.  case class KeyValueStreamConfig(streamFrom: StreamFrom, numInputPartitions: Int, bucket: String, scope: Option[String], collections: Seq[String], streamContent: Boolean, streamXattrs: Boolean, flowControlBufferSize: Option[Int], persistencePollingInterval: Option[String], connectionIdentifier: Option[String]) extends Product with Serializable
21.  case class KeyValueStreamOffset(vbuuid: Long, seqno: Long, snapshotStartSeqno: Long, snapshotEndSeqno: Long, collectionsManifestUid: Long) extends Product with Serializable
22.  class KeyValueTable extends SupportsRead
23.  class KeyValueTableProvider extends Logging with TableProvider with DataSourceRegister with CreatableRelationProvider
24.  case class KeyValueWriteConfig(bucket: String, scope: Option[String], collection: Option[String], idFieldName: String, casFieldName: Option[String], durability: Option[String], timeout: Option[String], connectionIdentifier: Option[String], errorHandler: Option[String], errorBucket: Option[String], errorScope: Option[String], errorCollection: Option[String], writeMode: Option[String]) extends Product with Serializable
25.  trait KeyValueWriteErrorHandler extends Serializable

    Trait for handling errors during Couchbase KeyValue write operations.

    Trait for handling errors during Couchbase KeyValue write operations.

    This allows applications to define custom error handling behavior for all write operations instead of having operations fail the entire job.

    Error handlers are processed asynchronously through a background queue system, so blocking operations are permitted. Each operation will block the queue until it completes.

    See ExampleErrorHandler.

26.  case class KeyValueWriteErrorInfo(bucket: String, scope: String, collection: String, documentId: String, throwable: Option[Throwable]) extends Product with Serializable

    Case class containing error information for KeyValue write operations.

    Case class containing error information for KeyValue write operations.

    bucket

    The bucket name where the error occurred

    scope

    The scope name where the error occurred

    collection

    The collection name where the error occurred

    documentId

    The document ID that failed to write

    throwable

    The original exception that occurred, or None if the exception could not be returned (which may be because it's not serializable)

27.  case class LookupIn(id: String, specs: Seq[LookupInSpec]) extends KeyValueOperation with Product with Serializable
28.  class LookupInRDD extends RDD[LookupInResult] with Logging
29.  case class MutateIn(id: String, specs: Seq[MutateInSpec], cas: Long = 0) extends KeyValueOperation with Product with Serializable
30.  class MutateInRDD extends RDD[MutateInResult] with Logging
31.  class RelationPartitionWriter extends ForeachPartitionFunction[String] with Logging
32.  case class Remove(id: String, cas: Long = 0) extends KeyValueOperation with Product with Serializable
33.  class RemoveRDD extends RDD[MutationResult] with Logging
34.  case class Replace[T](id: String, content: T, cas: Long = 0) extends KeyValueOperation with Product with Serializable
35.  class ReplaceRDD[T] extends RDD[MutationResult] with Logging
36.  case class Upsert[T](id: String, content: T) extends KeyValueOperation with Product with Serializable
37.  class UpsertRDD[T] extends RDD[MutationResult] with Logging