Sink - BlobSinkBuilder

Sink.BlobSinkBuilder

Create a SinkBuilder to build a data

Methods on the SinkBuilder enable you to emit stream data to external systems. If the

Stream is a BlobType, specialised operations can be applied to perform an ftp put of the Stream.

Quality of data can be observed using assert and warn by providing expression based conditions and messages to the whole

Stream.

A corresponding

Template can be created using .toTemplate().

Remarks

See

Transform Data for a related learning module.

Type parameters

Name	Type
`Output`	extends `BlobType`
`Inputs`	extends `Record`

Sink

error

▸ error(config):

BlobSinkBuilder

Check the sink inputs for validity and identify any errors. When the if predicate returns true the sink will be terminated with an error message and sink output will not be produced.

Parameters

Name	Type	Description
`config`	`Object`	the error message and predicate
`config.if`	(`inputs`: `Inputs`) => `EastFunction`	If true an error will be created
`config.message`	`string` \| (`inputs`: `Inputs`) => `EastFunction`	The message in the case that an error is created

Returns

BlobSinkBuilder

a new

BlobSinkBuilder

Example

 // get a predefined blob stream
 const file = Stream("File", BlobType);
 // get a predefined uri stream
 const uri = Stream("uri", StringType)
 // put the contents of the file at the uri
 const sink = new SinkBuilder("Ftp")
     .from(file)
     .input({ name: "uri", stream: uri })
     .error({
         if: uri => Equal(uri, ''),
         message: "Uri is empty"
     })
     .ftp({
        uri: (inputs) => inputs.uri,
     })
     .toTemplate();

ftp

▸ ftp(config):

FtpSinkBuilder

Build a sink to put a blob into an ftp file uri.

Type parameters

Name	Type
`U`	extends (`inputs`: `Inputs`) => `EastFunction`
`P`	extends (`inputs`: `Inputs`) => `EastFunction`

Parameters

Name	Type	Description
`config`	`Object`	the configuration of the ftp put
`config.key?`	`U`	An EastFunction for the ftp private key
`config.password?`	`U`	An EastFunction for the ftp password
`config.port?`	`P`	An EastFunction for the ftp port
`config.uri`	`U`	An EastFunction for the ftp uri
`config.username?`	`U`	An EastFunction for the ftp username

Returns

FtpSinkBuilder

a new

BlobSinkBuilder

Example

 // get a predefined blob stream
 const file = Stream("File", BlobType);
 // get a predefined uri stream
 const uri = Stream("uri", StringType)
 // put the contents of the file at the uri
 const sink = new SinkBuilder("Ftp")
     .from(file)
     .input({ name: "uri", stream: uri })
     .warn({
         predicate: uri => NotEqual(uri, ''),
         message: "Uri is empty"
     })
     .ftp({
        uri: (inputs) => inputs.uri,
     })
     .toTemplate();

input

▸ input(config):

BlobSinkBuilder<Output, Inputs & { [K in string]: Variable }>

Add a stream as input to the BlobSinkBuilder, which can be used in expressions to configure how the

Sink will access the data (such as a URL or password), the output data, or any data assertions (preconditions and postconditions).

Type parameters

Name	Type
`Name`	extends `string`
`T`	extends `EastType`

Parameters

Name	Type	Description
`config`	`Object`	the configuration for the `input`
`config.name`	`Name` extends `"input"` \| keyof `Inputs` ? `never` : `Name`	the name to give the input Variable
`config.stream`	`Stream`	the input stream (the stream and associated preconditions)

Returns

BlobSinkBuilder<Output, Inputs & { [K in string]: Variable }>

a new

BlobSinkBuilder

Example

 // get a predefined blob stream
 const file = Stream("File", BlobType);
 // get a predefined uri stream
 const uri = Stream("uri", StringType)
 // put the contents of the file at the uri
 const sink = new SinkBuilder("Ftp")
     .from(file)
     .input({ name: "uri", stream: uri })
     .assert({
         predicate: uri => NotEqual(uri, ''),
         message: "Uri is empty"
     })
     .ftp({
        uri: (inputs) => inputs.uri,
     })
     .toTemplate();

log

▸ log(config):

BlobSinkBuilder

Produce log messages based on the sink inputs. When the if predicate returns true the sink will register a log message and proceed to proceed to output data.

Parameters

Name	Type	Description
`config`	`Object`	the log message and optional predicate
`config.if?`	(`inputs`: `Inputs`) => `EastFunction`	If true a log message will be produced (optional)
`config.message`	`string` \| (`inputs`: `Inputs`) => `EastFunction`	The log message

Returns

BlobSinkBuilder

a new

BlobSinkBuilder

Example

 // get a predefined blob stream
 const file = Stream("File", BlobType);
 // get a predefined uri stream
 const uri = Stream("uri", StringType)
 // put the contents of the file at the uri
 const sink = new SinkBuilder("Ftp")
     .from(file)
     .input({ name: "uri", stream: uri })
     .warn({
         if: uri => Equal(uri, ''),
         message: "Uri is empty"
     })
     .ftp({
        uri: (inputs) => inputs.uri,
     })
     .toTemplate();

s3

▸ s3(config):

S3SinkBuilder

Build a sink to put a blob into an s3 path.

Type parameters

Name	Type
`U`	extends (`inputs`: `Inputs`) => `EastFunction`

Parameters

Name	Type	Description
`config`	`Object`	the configuration of the s3 blob
`config.access_key_id`	`U`	An EastFunction with the AWS access key ID
`config.bucket`	`U`	An EastFunction with a bucket name containing the object
`config.key`	`U`	An EastFunction with the key of the object to get.
`config.region`	`U`	An EastFunction with the aws region
`config.secret_access_key`	`U`	An EastFunction with the AWS secret access key

Returns

S3SinkBuilder

a new

BlobSinkBuilder

Example

 // get a predefined blob stream
 const file = Stream("File", BlobType);
 // get a predefined path stream
 const key = Stream("key", StringType)
 // put the contents of the file at the uri
 const sink = new SinkBuilder("Ftp")
     .from(file)
     .input({ name: "key", stream: key })
     .warn({
         predicate: key => NotEqual(key, ''),
         message: "Key is empty"
     })
     .s3({
        region: (inputs) => Const("__REGION__"),
        bucket: (inputs) => Const("__BUCKET__"),
        key: (inputs) => inputs.key,
        access_key_id: (inputs) => Const("__ACCESS_KEY_ID__"),
        secret_access_key: (inputs) => Const("__SECRET_KEY__")
     })
     .toTemplate();

warn

▸ warn(config):

BlobSinkBuilder

Check the sink inputs for validity and raise warnings as necessary. When the if predicate returns true the sink will register a warning with a message, but will proceed to proceed to output data.

Parameters

Name	Type	Description
`config`	`Object`	the warning message and predicate
`config.if`	(`inputs`: `Inputs`) => `EastFunction`	If true a warning will be produced
`config.message`	`string` \| (`inputs`: `Inputs`) => `EastFunction`	The message in the case that a warning is created

Returns

BlobSinkBuilder

a new

BlobSinkBuilder

Example

 // get a predefined blob stream
 const file = Stream("File", BlobType);
 // get a predefined uri stream
 const uri = Stream("uri", StringType)
 // put the contents of the file at the uri
 const sink = new SinkBuilder("Ftp")
     .from(file)
     .input({ name: "uri", stream: uri })
     .warn({
         if: uri => Equal(uri, ''),
         message: "Uri is empty"
     })
     .ftp({
        uri: (inputs) => inputs.uri,
     })
     .toTemplate();