Sink - S3SinkBuilder

Sink.S3SinkBuilder

Create a SinkBuilder to build a data

Sink.

Methods on the SinkBuilder enable you to emit stream data to external systems. If the

Stream is a BlobType, specialised operations can be applied to perform an ftp put of the Stream.

Quality of data can be observed using assert and warn by providing expression based conditions and messages to the whole

Stream.

A corresponding

Template can be created using .toTemplate().

Remarks

See

Transform Data for a related learning module.

Type parameters

Name	Type
B	extends BlobType
Inputs	extends Record

Hierarchy

• Builder

  ↳ S3SinkBuilder

Sink

toTemplate

▸ toTemplate():

Template

Return a

Template fragment containing the Streams and TaskDescriptions built by this builder.

Returns

Template

Example

 // get a predefined blob stream
 const file = Stream("File", BlobType);
 // get a predefined path stream
 const key = Stream("key", StringType)
 // put the contents of the file at the uri
 const sink = new SinkBuilder("Ftp")
     .from(file)
     .input({ name: "key", stream: key })
     .warn({
         predicate: key => NotEqual(key, ''),
         message: "Key is empty"
     })
     .s3({
        region: (inputs) => Const("__REGION__"),
        bucket: (inputs) => Const("__BUCKET__"),
        key: (inputs) => inputs.key,
        access_key_id: (inputs) => Const("__ACCESS_KEY_ID__"),
        secret_access_key: (inputs) => Const("__SECRET_KEY__")
     })
     .toTemplate();

Overrides

Builder.toTemplate