tensorflowonspark.TFParallel module

run(sc, map_fn, tf_args, num_executors, use_barrier=True)

Runs the user map_fn as parallel, independent instances of TF on the Spark executors.

Args:

sc

SparkContext

map_fun

user-supplied TensorFlow “main” function

tf_args

argparse args, or command-line ARGV. These will be passed to the map_fun.

num_executors

number of Spark executors. This should match your Spark job’s --num_executors.

use_barrier

Boolean indicating if TFParallel should use Spark barrier execution mode to wait for all executors.

Returns:

None