com.yahoo.ml.caffe

DataSource

abstract class DataSource[T1, T2] extends Serializable

Base class for various data sources.

Each subclass must have a constructor with the following signature: (conf: Config, layerId: Int, isTrain: Boolean). This is required by CaffeOnSpark at startup.

T1

class of entries extracted from RDD

T2

class of data blob in batch

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Known Subclasses
ImageDataFrame, ImageDataSource, LMDB, SeqImageDataSource
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. DataSource
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DataSource(conf: Config, layerId: Int, isTrain: Boolean, STOP_MARK: T1)

    conf

    CaffeSpark configuration

    layerId

    the layer index in the network protocol file

    STOP_MARK

    stop mark to indicate source is exhausted

Abstract Value Members

  1. abstract def dummyDataBlobs(): Array[FloatBlob]

    make a dummy data blob to be used by Solver threads

    make a dummy data blob to be used by Solver threads

    returns

    a dummy data blob

  2. abstract def dummyDataHolder(): T2

    make a dummy data blob to be used by Solver threads

    make a dummy data blob to be used by Solver threads

    returns

    a dummy data blob

  3. abstract def makeRDD(sc: SparkContext): RDD[T1]

    construct a sample RDD

    construct a sample RDD

    sc

    spark context

    returns

    RDD created from this source

  4. abstract def nextBatch(sampleIds: Array[String], data: T2, labels: FloatBlob): Boolean

    create a batch of samples extracted from source queue

    create a batch of samples extracted from source queue

    This method is Invoked by Transformer thread. You should extract samples from source queue, parse it and produce a batch.

    sampleIds

    holder for sample Ids

    data

    holder for data blob

    labels

    holder for label blob

    returns

    true if successful

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. val STOP_MARK: T1

    stop mark to indicate source is exhausted

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  8. def batchSize(): Int

    batch size

  9. var batchSize_: Int

    Attributes
    protected

  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  11. val conf: Config

    CaffeSpark configuration

  12. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  13. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  14. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  15. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  16. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  17. def init(): Boolean

    initialization of a Source within a process

    initialization of a Source within a process

    returns

    true if successfully initialized

  18. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  19. val isTrain: Boolean

  20. val layerId: Int

    the layer index in the network protocol file

  21. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  22. final def notify(): Unit

    Definition Classes
    AnyRef

  23. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  24. def setBatchSize(size: Int): Unit

    adjust batch size

    adjust batch size

    size

    the new batch size

  25. var solverMode: Int

    Attributes
    protected

  26. var sourceFilePath: String

    Attributes
    protected

  27. var sourceQueue: ArrayBlockingQueue[T1]

  28. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  29. def toString(): String

    Definition Classes
    AnyRef → Any

  30. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  31. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  32. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped