Pipeline - PipelineBuilder
Pipeline.PipelineBuilderA PipelineBuilder
to build a data
Methods on the PipelineBuilder
enable you to directly transform data
from
any
transform
operation. If the Stream is tabular,
specialised operations can be applied to select
, filter
, join
, aggregate
,
disaggregate
and offset
each entry in the Stream.
Quality of data can be observed using assert
and warn
by providing expression based conditions
and messages to the whole
A corresponding
Template can be created using.toTemplate()
.
Remarks
See
Transform Data for a related learning module.Example
const username = Stream("Username", StringType);
const password = Stream("Password", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.error({
if: username => Equal(username, ""),
message: "Username is empty"
})
.input({ name: "password", stream: password })
.warn({
if: (_, { password }) => Equal(password, ""),
message: () => Const("Password is empty")
})
.transform((username, { password }) => StringJoin`${username}:${password}`)
.transform(str => AsciiToBase64(str))
.toTemplate();
Pipeline
constructor
• new PipelineBuilder(name
, module?
):
PipelineBuilder
Create a PipelineBuilder
to build a data
Methods on the PipelineBuilder
enable you to directly transform data
from
any
transform
operation. If the Stream is tabular,
specialised operations can be applied to select
, filter
, join
, aggregate
,
disaggregate
and offset
each entry in the Stream.
Quality of data can be observed using assert
and warn
by providing expression based conditions
and messages to the whole
A corresponding
Template can be created using.toTemplate()
.
Parameters
Name | Type | Description |
---|---|---|
name | string | the name of the Pipeline to create |
module? | ModulePath | ModuleBuilder | - |
Returns
PipelineBuilder
Remarks
See
Transform Data for a related learning module.Example
const username = Stream("Username", StringType);
const password = Stream("Password", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.error({
if: username => Equal(username, ""),
message: "Username is empty"
})
.input({ name: "password", stream: password })
.warn({
if: (_, { password }) => Equal(password, ""),
message: () => Const("Password is empty")
})
.transform((username, { password }) => StringJoin`${username}:${password}`)
.transform(str => AsciiToBase64(str))
.toTemplate();
from
▸ from(stream
): T
extends
DictType
? TabularPipelineBuilder
: T
extends BlobType
? BlobPipelineBuilder
: GenericPipelineBuilder
Define the
Stream to construct the Pipeline from.Type parameters
Name | Type |
---|---|
T | extends EastType |
Parameters
Name | Type | Description |
---|---|---|
stream | Stream | the input Stream for the Pipeline |
Returns
T
extends
DictType
? TabularPipelineBuilder
: T
extends BlobType
? BlobPipelineBuilder
: GenericPipelineBuilder
a new
PipelineBuilderExample
const username = Stream("Username", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.toTemplate();