Pipeline - GenericPipelineBuilder
Pipeline.GenericPipelineBuilderA GenericPipelineBuilder
to build a data
Methods on the GenericPipelineBuilder
enable you to directly transform data
from
any
transform
operation.
Quality of data can be observed using assert
and warn
by providing expression based conditions
and messages to the whole
A corresponding
Template can be created using.toTemplate()
.
Remarks
See
Transform Data for a related learning module.Example
// get a predefined pipeline table
const username = Stream("Username", StringType);
const password = Stream("Password", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.error({
if: username => Equal(username, ""),
message: "Username is empty"
})
.input({ name: "password", stream: password })
.warn({
if: (_, { password }) => Equal(password, ""),
message: () => Const("Password is empty")
})
.transform((username, { password }) => StringJoin`${username}:${password}`)
.transform(str => AsciiToBase64(str))
.toTemplate();
Type parameters
Name | Type |
---|---|
Output | extends EastType |
Inputs | extends Record |
Hierarchy
-
Builder
↳
GenericPipelineBuilder
Pipeline
error
▸ error(config
):
GenericPipelineBuilder
Add an assertion on the pipeline inputs and output to identify errors. When the if
predicate returns true
the pipeline will be terminated with an error message and output data will not be produced.
Parameters
Name | Type | Description |
---|---|---|
config | Object | the error message and predicate |
config.if | (value : Variable , inputs : Inputs ) => EastFunction | If true an error will be created |
config.message | string | (value : Variable , inputs : Inputs ) => EastFunction | The message in the case that an error is created |
Returns
GenericPipelineBuilder
Example
const username = Stream("Username", StringType);
const password = Stream("Password", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.error({
if: username => Equal(username, ""),
message: "Username is empty"
})
.input({ name: "password", stream: password })
.transform((username, { password }) => StringJoin`${username}:${password}`)
.error({
if: str => Equal(str, ":"),
message: () => Const("Unexpected string")
})
.toTemplate();
input
▸ input(config
):
GenericPipelineBuilder
<Output
, Inputs
& { [K in string]: Variable }>
Add an additional named input
Stream to the Pipeline.Type parameters
Name | Type |
---|---|
Name | extends string |
T | extends EastType |
Parameters
Name | Type | Description |
---|---|---|
config | Object | the input stream and the resulting variable name |
config.name | Name extends "input" | keyof Inputs ? never : Name | - |
config.stream | Stream | - |
Returns
GenericPipelineBuilder
<Output
, Inputs
& { [K in string]: Variable }>
a new
PipelineBuilderExample
const username = Stream("Username", StringType);
const password = Stream("Password", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.input({ name: "password", stream: password })
.toTemplate();
let
▸ let(name
):
GenericPipelineBuilder
<Output
, Inputs
& { [K in string]: Variable }>
Give a name to the current
Pipeline output to be used as an input later in the Pipeline (i.e. after the next operation)Type parameters
Name | Type |
---|---|
Name | extends string |
Parameters
Name | Type |
---|---|
name | Name extends "input" | keyof Inputs ? never : Name |
Returns
GenericPipelineBuilder
<Output
, Inputs
& { [K in string]: Variable }>
a new
PipelineBuilderExample
const username = Stream("Username", StringType);
const password = Stream("Password", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.let("username")
.input({ name: "password", stream: password })
.transform((username, { password }) => StringJoin`${username}:${password}`)
.transform((str, inputs) => Struct({
Username: inputs.username,
Hash: str
}))
.toTemplate();
log
▸ log(config
):
GenericPipelineBuilder
Produce a log message depending on the pipeline inputs and ouputs. When the if
the predicate returns true
the
pipeline will produce a log message and proceed.
Parameters
Name | Type | Description |
---|---|---|
config | Object | the log message and optional predicate |
config.if? | (value : Variable , inputs : Inputs ) => EastFunction | If true a log message will be produced (optional) |
config.message | string | (value : Variable , inputs : Inputs ) => EastFunction | The log message |
Returns
GenericPipelineBuilder
Example
const username = Stream("Username", StringType);
const password = Stream("Password", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.log({
if: username => Equal(username, ""),
message: "Username is empty"
})
.input({ name: "password", stream: password })
.transform((username, { password }) => StringJoin`${username}:${password}`)
.log({
if: str => Equal(str, ":"),
message: () => Const("Unexpected string")
})
.toTemplate();
outputStream
▸ outputStream():
Stream
Return the
Stream containing the output of the pipeline.Returns
Stream
Example
// Create a datastream where a password can be written by end-user at runtime.
const hourly = new SourceBuilder("DatabasePassword")
.writeable(StringType)
.outputStream()
toJson
▸ toJson(config
):
BlobPipelineBuilder
Unparse a
Stream into a BlobType Stream containing JSON data.Type parameters
Name | Type |
---|---|
S | extends (value : Variable , inputs : Inputs ) => EastFunction |
Parameters
Name | Type | Description |
---|---|---|
config | Object | the configuration of the JSON parsing |
config.value | S | the function to parse the value |
Returns
BlobPipelineBuilder
a new
PipelineBuilderExample
const username = Stream("Username", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.toJson({
value: (value) => value
})
.toTemplate();
toTemplate
▸ toTemplate():
Template
Convert the built pipeline into an
Template, for usage in an EDK project.Returns
Template
a
TemplateExample
const username = Stream("Username", StringType);
const template = new PipelineBuilder("BasicAuth")
.from(username)
.toTemplate();
Overrides
Builder.toTemplate
transform
▸ transform(f
): ReturnType
["type"
] extends
DictType
? TabularPipelineBuilder
: GenericPipelineBuilder
<ReturnType
["type"
], Inputs
>
Transform the entire input
Stream based on an EastFunction.Type parameters
Name | Type |
---|---|
F | extends (value : Variable , inputs : Inputs ) => EastFunction |
Parameters
Name | Type | Description |
---|---|---|
f | F | an EastFunction function that generates the output Expression |
Returns
ReturnType
["type"
] extends
DictType
? TabularPipelineBuilder
: GenericPipelineBuilder
<ReturnType
["type"
], Inputs
>
a new
PipelineBuilderExample
const username = Stream("Username", StringType);
const password = Stream("Password", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.input({ name: "password", stream: password })
.transform((username, { password }) => StringJoin`${username}:${password}`)
.toTemplate();
warn
▸ warn(config
):
GenericPipelineBuilder
Add a warning on the pipeline inputs and ouputs to identify problems. When the if
predicate returns true
the
pipeline will register a warning with a message, but will proceed to proceed to produce output data.
Parameters
Name | Type | Description |
---|---|---|
config | Object | the warning message and predicate |
config.if | (value : Variable , inputs : Inputs ) => EastFunction | If true a warning will be produced |
config.message | string | (value : Variable , inputs : Inputs ) => EastFunction | The message in the case that a warning is produced |
Returns
GenericPipelineBuilder
Example
const username = Stream("Username", StringType);
const password = Stream("Password", StringType);
const pipeline = new PipelineBuilder("BasicAuth")
.from(username)
.warn({
if: username => Equal(username, ""),
message: "Username is empty"
})
.input({ name: "password", stream: password })
.transform((username, { password }) => StringJoin`${username}:${password}`)
.warn({
if: str => Equal(str, ":"),
message: () => Const("Unexpected string")
})
.toTemplate();