com.yahoo.ml.caffe

ImageDataFrame

class ImageDataFrame extends ImageDataSource

ImageDataFrame is a built-in data source class using Spark dataframe format.

ImageDataFrame expects dataframe with 2 required columns (lable:String, data:byte[]), and 5 optional columns (id: String, channels :Int, height:Int, width:Int, encoded: Boolean).

ImageDataFrame could be configured via the following MemoryDataLayer parameter: (1) dataframe_column_select ... a collection of dataframe SQL selection statements (ex. "sampleId as id", "abs(height) as height") (2) image_encoded ... indicate whether image data are encoded or not. (default: false) (3) dataframe_format ... Dataframe Format. (default: parquet)

Linear Supertypes
ImageDataSource, DataSource[(String, String, Int, Int, Int, Boolean, Array[Byte]), MatVector], Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. ImageDataFrame
  2. ImageDataSource
  3. DataSource
  4. Serializable
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ImageDataFrame(conf: Config, layerId: Int, isTrain: Boolean)

    conf

    CaffeSpark configuration

    layerId

    the layer index in the network protocol file

    isTrain

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. val STOP_MARK: (String, String, Int, Int, Int, Boolean, Array[Byte])

    Definition Classes
    DataSource

  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  8. def batchSize(): Int

    batch size

    batch size

    Definition Classes
    DataSource

  9. var batchSize_: Int

    Attributes
    protected
    Definition Classes
    DataSource

  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  11. def dummyDataBlobs(): Array[FloatBlob]

    make a dummy data blob to be used by Solver threads

    make a dummy data blob to be used by Solver threads

    returns

    a dummy data blob

    Definition Classes
    ImageDataSource → DataSource

  12. def dummyDataHolder(): MatVector

    make a dummy data blob to be used by Solver threads

    make a dummy data blob to be used by Solver threads

    returns

    a dummy data blob

    Definition Classes
    ImageDataSource → DataSource

  13. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  14. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  15. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  16. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  17. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  18. def init(): Boolean

    initialization of a Source within a process

    initialization of a Source within a process

    returns

    true if successfully initialized

    Definition Classes
    ImageDataSource → DataSource

  19. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  20. var log: Logger

    Attributes
    protected
    Definition Classes
    ImageDataSource

  21. def makeRDD(sc: SparkContext): RDD[(String, String, Int, Int, Int, Boolean, Array[Byte])]

    construct a sample RDD

    construct a sample RDD

    sc

    spark context

    returns

    RDD created from this source

    Definition Classes
    ImageDataFrame → DataSource

  22. var memdatalayer_param: MemoryDataParameter

    Attributes
    protected
    Definition Classes
    ImageDataSource

  23. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  24. def nextBatch(sampleIds: Array[String], mats: MatVector, labels: FloatBlob): Boolean

    create a batch of samples extracted from source queue

    create a batch of samples extracted from source queue

    This method is Invoked by Transformer thread. You should extract samples from source queue, parse it and produce a batch.

    sampleIds

    holder for sample Ids

    labels

    holder for label blob

    returns

    true if successful

    Definition Classes
    ImageDataSource → DataSource

  25. final def notify(): Unit

    Definition Classes
    AnyRef

  26. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  27. def setBatchSize(size: Int): Unit

    adjust batch size

    adjust batch size

    size

    the new batch size

    Definition Classes
    DataSource

  28. var solverMode: Int

    Attributes
    protected
    Definition Classes
    DataSource

  29. var sourceFilePath: String

    Attributes
    protected
    Definition Classes
    DataSource

  30. var sourceQueue: ArrayBlockingQueue[(String, String, Int, Int, Int, Boolean, Array[Byte])]

    Definition Classes
    DataSource

  31. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  32. def toString(): String

    Definition Classes
    AnyRef → Any

  33. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  34. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  35. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from ImageDataSource

Inherited from DataSource[(String, String, Int, Int, Int, Boolean, Array[Byte]), MatVector]

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped