org.apache.mahout.sparkbindings.drm

CheckpointedDrmSpark

class CheckpointedDrmSpark[K] extends CheckpointedDrm[K]

Spark-specific optimizer-checkpointed DRM.

K

matrix key type (e.g. the keys of sequence files once persisted)

Linear Supertypes
CheckpointedDrm[K], DrmLike[K], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. CheckpointedDrmSpark
  2. CheckpointedDrm
  3. DrmLike
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CheckpointedDrmSpark(rddInput: DrmRddInput[K], _nrow: Long = -1L, _ncol: Int = -1, _cacheStorageLevel: StorageLevel = ..., partitioningTag: Long = scala.util.Random.nextLong(), _canHaveMissingRows: Boolean = false)(implicit arg0: ClassTag[K])

    rddInput

    underlying rdd to wrap over.

    _nrow

    number of rows; if unspecified, we will compute with an inexpensive traversal.

    _ncol

    number of columns; if unspecified, we will try to guess with an inexpensive traversal.

    _cacheStorageLevel

    storage level

    partitioningTag

    unique partitioning tag. Used to detect identically partitioned operands.

    _canHaveMissingRows

    true if the matrix is int-keyed, and if it also may have missing rows (will require a lazy fix for some physical operations.

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def cache(): CheckpointedDrmSpark[K]

  8. lazy val canHaveMissingRows: Boolean

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  9. def checkpoint(cacheHint: CacheHint): CheckpointedDrm[K]

    Action operator -- does not necessary means Spark action; but does mean running BLAS optimizer and writing down Spark graph lineage since last checkpointed DRM.

    Action operator -- does not necessary means Spark action; but does mean running BLAS optimizer and writing down Spark graph lineage since last checkpointed DRM.

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  11. def collect: Matrix

    Collecting DRM to fron-end in-core Matrix.

    Collecting DRM to fron-end in-core Matrix.

    If key in DRM is Int, then matrix is collected using key as row index. Otherwise, order of rows in result is undefined but key.toString is applied as rowLabelBindings of the in-core matrix .

    Note that this pre-allocates target matrix and then assigns collected RDD to it thus this likely would require about 2 times the RDD memory

    returns

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  12. def computeNCol: Int

    Attributes
    protected
  13. def computeNNonZero: Long

    Attributes
    protected
  14. def computeNRow: Long

    Attributes
    protected
  15. val context: DistributedContext

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  16. def dfsWrite(path: String): Unit

    Dump matrix as computed Mahout's DRM into specified (HD)FS path

    Dump matrix as computed Mahout's DRM into specified (HD)FS path

    path

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  17. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  19. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  20. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  21. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  22. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  23. def keyClassTag: ClassTag[K]

    Explicit extraction of key class Tag

    Explicit extraction of key class Tag

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  24. lazy val ncol: Int

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  25. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  26. def newRowCardinality(n: Int): CheckpointedDrm[K]

    Changes the number of rows in the DRM without actually touching the underlying data.

    Changes the number of rows in the DRM without actually touching the underlying data. Used to redimension a DRM after it has been created, which implies some blank, non-existent rows.

    n

    new row dimension

    returns

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  27. final def notify(): Unit

    Definition Classes
    AnyRef
  28. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  29. lazy val nrow: Long

    Definition Classes
    CheckpointedDrmSpark → DrmLike
  30. val partitioningTag: Long

    unique partitioning tag.

    unique partitioning tag. Used to detect identically partitioned operands.

    Attributes
    protected[org.apache.mahout]
    Definition Classes
    CheckpointedDrmSpark → DrmLike
  31. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  32. def toString(): String

    Definition Classes
    AnyRef → Any
  33. def uncache(): CheckpointedDrmSpark.this.type

    if matrix was previously persisted into cache, delete cached representation

    if matrix was previously persisted into cache, delete cached representation

    Definition Classes
    CheckpointedDrmSpark → CheckpointedDrm
  34. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  35. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  36. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from CheckpointedDrm[K]

Inherited from DrmLike[K]

Inherited from AnyRef

Inherited from Any

Ungrouped