Skip to main content

Task - CustomTaskBuilder

Task.CustomTaskBuilder

A helper class to build a custom task to run on the ELARA platform.

The custom task executes a command inside a dockerized sandbox. It is given some input files as blob data, and produces some output files as blob data. The custom task can run any linux-compatible binary program (which can be provided as one of the input files) inside our default Ubuntu container, or any docker image you upload to ELARA via a blob

A corresponding

Template can be created using .toTemplate().

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input_data.json output_data.json")

Type parameters

Name	Type
`Outputs`	extends `Record` =
`Results`	extends `Record` =

Hierarchy

Builder

↳ CustomTaskBuilder

CustomScenario

dataDir

▸ dataDir(dir):

CustomTaskBuilder

Provide the directory where the input and output data is located.

This directory is a docker "volume mount" - if this directory already exists in the image, files inside will not be visible inside the container. If necessary, copy files into the data directory from elsewhere in the image via a

shell command.

Note: the directory is also set as the current working directory before the task command is executed.

Parameters

Name	Type	Description
`dir`	`string`	the directory to mount the data inside the docker container

Returns

CustomTaskBuilder

Example

// Optimize a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_scenario = new CustomScenarioBuilder("MyScearnio")
  .dataDir("/my/custom/dir")
  .input("my_program", my_program_stream, file => file, true)
  .input("input.json", input_data_stream, data => Utf8Encode(ToJson(data)))
  .output("output.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input.json output.json")
  .objective(outputs => outputs["output.json"])
  .optimizeEvery("input.json", "amount", { min: () => Const(0), max: () => Const(100) })

Task

constructor

• new CustomTaskBuilder(name, module?):

CustomTaskBuilder

Construct a CustomTaskBuilder to build a custom task, which executes a command given some input files as blob

Streams, and produces some output files as blob streams. The custom task can run any linux-compatible binary program, which can be provided as one of the input files.

Type parameters

Name	Type
`Outputs`	extends `Record` =
`Results`	extends `Record` =

Parameters

Name	Type	Description
`name`	`string`	the name of the custom task
`module?`	`ModulePath` \| `ModuleBuilder`	-

Returns

CustomTaskBuilder

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input_data.json output_data.json")

Overrides

Builder.constructor

dockerImage

▸ dockerImage(stream):

CustomTaskBuilder

Provide a user-defined docker image to execute the custom task. The docker image is provided via a blob

Stream, containing an OCI container in a .tar, .tar.gz, .tar.bz2 file format as generated by docker via docker image save ... or similar methods.

If no docker image is provided, then an Ubuntu LTS image is used by default. If no command is provided the image's default command is used.

Parameters

Name	Type	Description
`stream`	`Stream`	the stream containing the docker image

Returns

CustomTaskBuilder

Example

// Create a custom task which uses a user-provided docker image to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .dockerImage(my_docker_image_stream)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))

env

▸ env(name, value):

CustomTaskBuilder

Set an environment variable for the custom task

Parameters

Name	Type	Description
`name`	`string`	the name of the environment variable
`value`	`string`	the value of the environment variable

Returns

CustomTaskBuilder

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .env("MY_PROGRAM_SETTING", "TRUE")
  .shell("./my_program input_data.json output_data.json")

exec

▸ exec(command, ...args):

CustomTaskBuilder

Provide the command used to execute the custom task with raw arguemnts. This command will be executed in a sandbox. As the command is not be interpretted by any shell, it is not necessary to quote the arguments (e.g. filenames containing spaces, etc).

Parameters

Name	Type	Description
`command`	`string`	the filename of the program to be executed
`...args`	`string`[]	the command-line arguments given to the program

Returns

CustomTaskBuilder

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .exec("./my_program", "input data.json", "output data.json")

input

▸ input(filename, stream, executable?, toBlob?):

CustomTaskBuilder

Add an input to the custom task. A file with the provided filename is created from the data stored in a blob

Stream before the custom task command is launched.

Note that the input could be a linux-compatible program to execute.

Type parameters

Name	Type
`T`	extends `EastType`

Parameters

Name	Type	Default value	Description
`filename`	`string`	`undefined`	the filename of the input
`stream`	`Stream`	`undefined`	a blob stream contain the data to place in the file
`executable`	`boolean`	`false`	set to `true` if the file is an executable program (default `false`)
`toBlob?`	(`input`: `Variable`) => `EastFunction`	`undefined`	an optional function to create an Expression to serialize the input data into a raw file (blob)

Returns

CustomTaskBuilder

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input_data.json output_data.json")

logStdErr

▸ logStdErr(enabled):

CustomTaskBuilder

Enable or disable logging of stderr from the custom task.

Parameters

Name	Type	Description
`enabled`	`boolean`	whether logging is enabled (default `true`)

Returns

CustomTaskBuilder

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input_data.json output_data.json")
  .logStdErr(false)

logStdOut

▸ logStdOut(enabled):

CustomTaskBuilder

Enable or disable logging of stdout from the custom task.

Parameters

Name	Type	Description
`enabled`	`boolean`	whether logging is enabled (default `true`)

Returns

CustomTaskBuilder

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input_data.json output_data.json")
  .logStdOut(false)

output

▸ output(filename, fromBlob?, result?):

CustomTaskBuilder<Outputs & { [K in string]: Stream }, Results & { [K in string]: BlobType }>

Add an output to the custom task. A file with the provided filename is expected to exist after the custom task command is completed. The output data is then copied to a

Type parameters

Name	Type
`Name`	extends `string`

Parameters

Name	Type	Description
`filename`	`Name`	the filename of the output
`fromBlob?`	`undefined`	an optional function to create an Expression to deserialize (or parse) the output data from a raw file (blob)
`result?`	`true`	whether this file is written to a datastream (default `true`)

Returns

CustomTaskBuilder<Outputs & { [K in string]: Stream }, Results & { [K in string]: BlobType }>

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input_data.json output_data.json")

outputStreams

▸ outputStreams(): { [K in string | number | symbol]: Stream }

Fetch the output streams of the custom task. This returns a record of data streams, where the keys are the expected filenames.

Returns

{ [K in string | number | symbol]: Stream }

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input_data.json output_data.json")

// Get the decoded output data stream
const output_data = my_task.outputStreams()["output_data.json"]

result

▸ result(name, result):

CustomTaskBuilder<Outputs, Results & { [K in string]: ReturnType["type"] }>

Add a result to the custom task which is computed from the output files.

Type parameters

Name	Type
`Name`	extends `string`
`F`	extends (`output`: { [K in string \| number \| symbol]: Variable }) => `EastFunction`

Parameters

Name	Type	Description
`name`	`Name`	the name of the computed result
`result`	`F`	an function to create an EastFunction computing the desired result

Returns

CustomTaskBuilder<Outputs, Results & { [K in string]: ReturnType["type"] }>

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file, and get a field from the result
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .result("field_a", outputs => GetField(outputs["output_data.json"], "a"))
  .shell("./my_program input_data.json output_data.json")

shell

▸ shell(command):

CustomTaskBuilder

Provide the command used to execute the custom task in shell. This command will be executed by the linux bash shell in a sandboxed environment.

Parameters

Name	Type	Description
`command`	`string`	the command to be executed

Returns

CustomTaskBuilder

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
const my_task = new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input_data.json output_data.json")

toTemplate

▸ toTemplate(): Object

Return a

Template fragment containing the Streams and TaskDescriptions built by this builder.

Returns

Object

Name	Type
`streams`	`Record`
`tasks`

Example

// Create a custom task which uses a user-provided program to transform a JSON file into another JSON file
export default new CustomTaskBuilder("MyTask")
  .input("my_program", my_program_stream, true)
  .input("input_data.json", input_data_stream, false, data => Utf8Encode(ToJson(data)))
  .output("output_data.json", blob => FromJson(MyOutputType, Utf8Decode(blob)))
  .shell("./my_program input_data.json output_data.json")
  .toTemplate()

Overrides

Builder.toTemplate

On this page

Type parameters
Hierarchy
CustomScenario
- dataDir
Task