org.apache.mahout.sparkbindings

SparkEngine

object SparkEngine extends DistributedEngine

Spark-specific non-drm-method operations

Linear Supertypes
DistributedEngine, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SparkEngine
  2. DistributedEngine
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def allreduceBlock[K](drm: CheckpointedDrm[K], bmf: ((Array[K], Matrix)) ⇒ Matrix, rf: (Matrix, Matrix) ⇒ Matrix)(implicit arg0: ClassTag[K]): Matrix

    Optional engine-specific all reduce tensor operation.

    Optional engine-specific all reduce tensor operation.

    Definition Classes
    SparkEngine → DistributedEngine
  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. def colMeans[K](drm: CheckpointedDrm[K])(implicit arg0: ClassTag[K]): Vector

    Engine-specific colMeans implementation based on a checkpoint.

    Engine-specific colMeans implementation based on a checkpoint.

    Definition Classes
    SparkEngine → DistributedEngine
  10. def colSums[K](drm: CheckpointedDrm[K])(implicit arg0: ClassTag[K]): Vector

    Definition Classes
    SparkEngine → DistributedEngine
  11. def drm2IntKeyed[K](drmX: DrmLike[K], computeMap: Boolean = false)(implicit arg0: ClassTag[K]): (DrmLike[Int], Option[DrmLike[K]])

    Convert non-int-keyed matrix to an int-keyed, computing optionally mapping from old keys to row indices in the new one.

    Convert non-int-keyed matrix to an int-keyed, computing optionally mapping from old keys to row indices in the new one. The mapping, if requested, is returned as a 1-column matrix.

    Definition Classes
    SparkEngine → DistributedEngine
  12. def drmBroadcast(m: Matrix)(implicit dc: DistributedContext): BCast[Matrix]

    Broadcast support

    Broadcast support

    Definition Classes
    SparkEngine → DistributedEngine
  13. def drmBroadcast(v: Vector)(implicit dc: DistributedContext): BCast[Vector]

    Broadcast support

    Broadcast support

    Definition Classes
    SparkEngine → DistributedEngine
  14. def drmDfsRead(path: String, parMin: Int = 0)(implicit sc: DistributedContext): CheckpointedDrm[_]

    Load DRM from hdfs (as in Mahout DRM format)

    Load DRM from hdfs (as in Mahout DRM format)

    path
    sc

    spark context (wanted to make that implicit, doesn't work in current version of scala with the type bounds, sorry)

    returns

    DRM[Any] where Any is automatically translated to value type

    Definition Classes
    SparkEngine → DistributedEngine
  15. def drmParallelizeEmpty(nrow: Int, ncol: Int, numPartitions: Int = 10)(implicit sc: DistributedContext): CheckpointedDrm[Int]

    This creates an empty DRM with specified number of partitions and cardinality.

    This creates an empty DRM with specified number of partitions and cardinality.

    Definition Classes
    SparkEngine → DistributedEngine
  16. def drmParallelizeEmptyLong(nrow: Long, ncol: Int, numPartitions: Int = 10)(implicit sc: DistributedContext): CheckpointedDrm[Long]

    Definition Classes
    SparkEngine → DistributedEngine
  17. def drmParallelizeWithRowIndices(m: Matrix, numPartitions: Int = 1)(implicit sc: DistributedContext): CheckpointedDrm[Int]

    Parallelize in-core matrix as spark distributed matrix, using row ordinal indices as data set keys.

    Parallelize in-core matrix as spark distributed matrix, using row ordinal indices as data set keys.

    Definition Classes
    SparkEngine → DistributedEngine
  18. def drmParallelizeWithRowLabels(m: Matrix, numPartitions: Int = 1)(implicit sc: DistributedContext): CheckpointedDrm[String]

    Parallelize in-core matrix as spark distributed matrix, using row labels as a data set keys.

    Parallelize in-core matrix as spark distributed matrix, using row labels as a data set keys.

    Definition Classes
    SparkEngine → DistributedEngine
  19. def drmSampleKRows[K](drmX: DrmLike[K], numSamples: Int, replacement: Boolean)(implicit arg0: ClassTag[K]): Matrix

    Definition Classes
    SparkEngine → DistributedEngine
  20. def drmSampleRows[K](drmX: DrmLike[K], fraction: Double, replacement: Boolean)(implicit arg0: ClassTag[K]): DrmLike[K]

    (Optional) Sampling operation.

    (Optional) Sampling operation. Consistent with Spark semantics of the same.

    K
    drmX
    fraction
    replacement
    returns

    Definition Classes
    SparkEngine → DistributedEngine
  21. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  22. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  23. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  24. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  25. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  26. var hdfsUtils: HDFSUtil

  27. def indexedDatasetDFSRead(src: String, schema: Schema = DefaultIndexedDatasetReadSchema, existingRowIDs: Option[BiDictionary] = None)(implicit sc: DistributedContext): IndexedDatasetSpark

    Returns an org.apache.mahout.sparkbindings.indexeddataset.IndexedDatasetSpark from default text delimited files.

    Returns an org.apache.mahout.sparkbindings.indexeddataset.IndexedDatasetSpark from default text delimited files. Reads a vector per row.

    src

    a comma separated list of URIs to read from

    schema

    how the text file is formatted

    Definition Classes
    SparkEngine → DistributedEngine
  28. def indexedDatasetDFSReadElements(src: String, schema: Schema = ..., existingRowIDs: Option[BiDictionary] = None)(implicit sc: DistributedContext): IndexedDatasetSpark

    Returns an org.apache.mahout.sparkbindings.indexeddataset.IndexedDatasetSpark from default text delimited files.

    Returns an org.apache.mahout.sparkbindings.indexeddataset.IndexedDatasetSpark from default text delimited files. Reads an element per row.

    src

    a comma separated list of URIs to read from

    schema

    how the text file is formatted

    Definition Classes
    SparkEngine → DistributedEngine
  29. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  30. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  31. def norm[K](drm: CheckpointedDrm[K])(implicit arg0: ClassTag[K]): Double

    Definition Classes
    SparkEngine → DistributedEngine
  32. final def notify(): Unit

    Definition Classes
    AnyRef
  33. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  34. def numNonZeroElementsPerColumn[K](drm: CheckpointedDrm[K])(implicit arg0: ClassTag[K]): Vector

    Definition Classes
    SparkEngine → DistributedEngine
  35. def optimizerRewrite[K](action: DrmLike[K])(implicit arg0: ClassTag[K]): DrmLike[K]

    Perform default expression rewrite.

    Perform default expression rewrite. Return physical plan that we can pass to exec(). <P>

    A particular physical engine implementation may choose to either use or not use these rewrites as a useful basic rewriting rule.<P>

    Definition Classes
    SparkEngine → DistributedEngine
  36. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  37. def toPhysical[K](plan: DrmLike[K], ch: CacheHint)(implicit arg0: ClassTag[K]): CheckpointedDrm[K]

    Second optimizer pass.

    Second optimizer pass. Translate previously rewritten logical pipeline into physical engine plan.

    Definition Classes
    SparkEngine → DistributedEngine
  38. def toString(): String

    Definition Classes
    AnyRef → Any
  39. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  40. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  41. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from DistributedEngine

Inherited from AnyRef

Inherited from Any

Ungrouped