Source - S3SourceBuilder

Source.S3SourceBuilder

A helper class to build a data

Source, which populates a data Stream with data. Data sources can range from simple clocks to complex database queries or API requests.

A corresponding

Template can be created using .toTemplate().

Remarks

See

Ingest Structured Data for a related learning module.

Type parameters

Name	Type
B	extends BlobType
Inputs	extends Record

Hierarchy

• AbstractSourceBuilder

  ↳ S3SourceBuilder

Source

error

▸ error(config):

S3SourceBuilder

Check the source outputs for validity and identify any errors. When the if predicate returns true the source will be terminated with an error message and output data will not be produced.

Parameters

Name	Type	Description
config	Object	the error message and predicate
config.if	(value: Variable, inputs: Inputs) => EastFunction	If true an error will be created
config.message	string | (value: Variable, inputs: Inputs) => EastFunction	The message in the case that an error is created

Returns

S3SourceBuilder

a new

SourceBuilder

Example

 const source = new SourceBuilder("FloatSource")
     .input({ name: "max", stream: max_stream })
     .writeable(FloatType)
     .error({
         if: (value, { max }) => Greater(value, max),
         message: (value) => StringJoin`The value ${value} exceeds maximum`
     })

Inherited from

AbstractSourceBuilder.error

---

log

▸ log(config):

S3SourceBuilder

Produce log messages based on the source outputs and inputs. When the if predicate returns true source will register a warning with a message, but will proceed to proceed to produce output data.

Parameters

Name	Type	Description
config	Object	the log message and optional predicate
config.if?	(value: Variable, inputs: Inputs) => EastFunction	If true a log message will be produced (optional)
config.message	string | (value: Variable, inputs: Inputs) => EastFunction	The message in the case that a warning is created

Returns

S3SourceBuilder

a new

SourceBuilder

Example

 const source = new SourceBuilder("FloatSource")
     .input({ name: "max", stream: max_stream })
     .writeable(FloatType)
     .log({
         if: (value, { max }) => Greater(value, max),
         message: (value) => StringJoin`The value ${value} exceeds maximum`
     })

Inherited from

AbstractSourceBuilder.log

---

outputStream

▸ outputStream():

Stream

Return the

Stream containing the output of the datasource.

Returns

Stream

Example

// Create an s3 datastream
const source = new SourceBuilder("My S3 Blob")
     .s3({
        region: (inputs) => Const("__REGION__"),
        bucket: (inputs) => Const("__BUCKET__"),
        key: (inputs) => Const("__KEY__"),
        access_key_id: (inputs) => Const("__ACCESS_KEY_ID__"),
        secret_access_key: (inputs) => Const("__SECRET_KEY__")
     })
    .outputStream()

Overrides

AbstractSourceBuilder.outputStream

---

toTemplate

▸ toTemplate():

Template

Return a

Template fragment containing the Streams and TaskDescriptions built by this builder.

Returns

Template

Example

// Create an s3 datastream
export default new SourceBuilder("My S3 Blob")
     .s3({
        region: (inputs) => Const("__REGION__"),
        bucket: (inputs) => Const("__BUCKET__"),
        key: (inputs) => Const("__KEY__"),
        access_key_id: (inputs) => Const("__ACCESS_KEY_ID__"),
        secret_access_key: (inputs) => Const("__SECRET_KEY__")
     })
    .toTemplate()

Overrides

AbstractSourceBuilder.toTemplate

---

warn

▸ warn(config):

S3SourceBuilder

Check the source outputs for validity and produce warnings as necessary. When the if predicate returns true the source will register a warning with a message, but will proceed to proceed to produce output data.

Parameters

Name	Type	Description
config	Object	the warning message and predicate
config.if	(value: Variable, inputs: Inputs) => EastFunction	If true a warning will be produced
config.message	string | (value: Variable, inputs: Inputs) => EastFunction	The message in the case that a warning is created

Returns

S3SourceBuilder

a new

SourceBuilder

Example

 const source = new SourceBuilder("FloatSource")
     .input({ name: "max", stream: max_stream })
     .writeable(FloatType)
     .warn({
         if: (value, { max }) => Greater(value, max),
         message: (value) => StringJoin`The value ${value} exceeds maximum`
     })

Inherited from

AbstractSourceBuilder.warn