com.yahoo.ml.caffe

ImageDataFrame

class ImageDataFrame extends ImageDataSource

ImageDataFrame is a built-in data source class using Spark dataframe format.

ImageDataFrame expects dataframe with 2 required columns (lable:String, data:byte[]), and 5 optional columns (id: String, channels :Int, height:Int, width:Int, encoded: Boolean).

ImageDataFrame could be configured via the following MemoryDataLayer parameter: (1) dataframe_column_select ... a collection of dataframe SQL selection statements (ex. "sampleId as id", "abs(height) as height") (2) image_encoded ... indicate whether image data are encoded or not. (default: false) (3) dataframe_format ... Dataframe Format. (default: parquet)

Linear Supertypes
ImageDataSource, DataSource[(String, String, Int, Int, Int, Boolean, Array[Byte]), MatVector], Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. ImageDataFrame
  2. ImageDataSource
  3. DataSource
  4. Serializable
  5. Serializable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ImageDataFrame(conf: Config, layerId: Int, isTrain: Boolean)

    conf

    CaffeSpark configuration

    layerId

    the layer index in the network protocol file

    isTrain

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. val STOP_MARK: (String, String, Int, Int, Int, Boolean, Array[Byte])

    Definition Classes
    DataSource
  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def batchSize(): Int

    batch size

    batch size

    Definition Classes
    DataSource
  9. var batchSize_: Int

    Attributes
    protected
    Definition Classes
    DataSource
  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  11. def dummyDataBlobs(): Array[FloatBlob]

    make a dummy data blob to be used by Solver threads

    make a dummy data blob to be used by Solver threads

    returns

    a dummy data blob

    Definition Classes
    ImageDataSourceDataSource
  12. def dummyDataHolder(): MatVector

    make a dummy data blob to be used by Solver threads

    make a dummy data blob to be used by Solver threads

    returns

    a dummy data blob

    Definition Classes
    ImageDataSourceDataSource
  13. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  14. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  15. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  16. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  17. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  18. def init(): Boolean

    initialization of a Source within a process

    initialization of a Source within a process

    returns

    true if successfully initialized

    Definition Classes
    ImageDataSourceDataSource
  19. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  20. var log: Logger

    Attributes
    protected
    Definition Classes
    ImageDataSource
  21. def makeRDD(sc: SparkContext): RDD[(String, String, Int, Int, Int, Boolean, Array[Byte])]

    construct a sample RDD

    construct a sample RDD

    sc

    spark context

    returns

    RDD created from this source

    Definition Classes
    ImageDataFrameDataSource
  22. var memdatalayer_param: MemoryDataParameter

    Attributes
    protected
    Definition Classes
    ImageDataSource
  23. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  24. def nextBatch(sampleIds: Array[String], mats: MatVector, labels: FloatBlob): Boolean

    create a batch of samples extracted from source queue

    create a batch of samples extracted from source queue

    This method is Invoked by Transformer thread. You should extract samples from source queue, parse it and produce a batch.

    sampleIds

    holder for sample Ids

    labels

    holder for label blob

    returns

    true if successful

    Definition Classes
    ImageDataSourceDataSource
  25. final def notify(): Unit

    Definition Classes
    AnyRef
  26. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  27. def setBatchSize(size: Int): Unit

    adjust batch size

    adjust batch size

    size

    the new batch size

    Definition Classes
    DataSource
  28. var solverMode: Int

    Attributes
    protected
    Definition Classes
    DataSource
  29. var sourceFilePath: String

    Attributes
    protected
    Definition Classes
    DataSource
  30. var sourceQueue: ArrayBlockingQueue[(String, String, Int, Int, Int, Boolean, Array[Byte])]

    Definition Classes
    DataSource
  31. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  32. def toString(): String

    Definition Classes
    AnyRef → Any
  33. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  35. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from ImageDataSource

Inherited from DataSource[(String, String, Int, Int, Int, Boolean, Array[Byte]), MatVector]

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped