Source - FtpSourceBuilder
Source.FtpSourceBuilderA helper class to build a data
Source, which populates a data Stream with data. Data sources can range from simple clocks to complex database queries or API requests.A corresponding
Template can be created using.toTemplate()
.
Remarks
See
Ingest Structured Data for a related learning module.Type parameters
Name | Type |
---|---|
B | extends BlobType |
Inputs | extends Record |
Hierarchy
-
AbstractSourceBuilder
↳
FtpSourceBuilder
Source
error
▸ error(config
):
FtpSourceBuilder
Check the source outputs for validity and identify any errors. When the if
predicate returns true
the source will be terminated with an error message and output data will not be produced.
Parameters
Name | Type | Description |
---|---|---|
config | Object | the error message and predicate |
config.if | (value : Variable , inputs : Inputs ) => EastFunction | If true an error will be created |
config.message | string | (value : Variable , inputs : Inputs ) => EastFunction | The message in the case that an error is created |
Returns
FtpSourceBuilder
a new
SourceBuilderExample
const source = new SourceBuilder("FloatSource")
.input({ name: "max", stream: max_stream })
.writeable(FloatType)
.error({
if: (value, { max }) => Greater(value, max),
message: (value) => StringJoin`The value ${value} exceeds maximum`
})
Inherited from
AbstractSourceBuilder.error
log
▸ log(config
):
FtpSourceBuilder
Produce log messages based on the source outputs and inputs. When the if
predicate returns true
source will register a warning with a message, but will proceed to proceed to produce output data.
Parameters
Name | Type | Description |
---|---|---|
config | Object | the log message and optional predicate |
config.if? | (value : Variable , inputs : Inputs ) => EastFunction | If true a log message will be produced (optional) |
config.message | string | (value : Variable , inputs : Inputs ) => EastFunction | The message in the case that a warning is created |
Returns
FtpSourceBuilder
a new
SourceBuilderExample
const source = new SourceBuilder("FloatSource")
.input({ name: "max", stream: max_stream })
.writeable(FloatType)
.log({
if: (value, { max }) => Greater(value, max),
message: (value) => StringJoin`The value ${value} exceeds maximum`
})
Inherited from
AbstractSourceBuilder.log
outputStream
▸ outputStream():
Stream
Return the
Stream containing the output of the datasource.Returns
Stream
Example
// Create an ftp datastream which is updated at the beginning of every hour
const hourly = new SourceBuilder("Hourly")
.ftp({
cron: "0 8 * * *",
uri: () => Const("ftp://ftp.bom.gov.au/anon/gen/clim_data/IDCKWCDEA0/tables/qld/applethorpe/applethorpe-200902.csv"),
})
.outputStream()
Overrides
AbstractSourceBuilder.outputStream
toTemplate
▸ toTemplate():
Template
Return a
Template fragment containing the Streams and TaskDescriptions built by this builder.Returns
Template
Example
// Create a datastream which is updated to the current time at the beginning of every hour
export default new SourceBuilder("Hourly")
.ftp({
cron: "0 8 * * *",
uri: () => Const("ftp://ftp.bom.gov.au/anon/gen/clim_data/IDCKWCDEA0/tables/qld/applethorpe/applethorpe-200902.csv"),
})
.toTemplate()
Overrides
AbstractSourceBuilder.toTemplate
warn
▸ warn(config
):
FtpSourceBuilder
Check the source outputs for validity and produce warnings as necessary. When the if
predicate returns true
the source will register a warning with a message, but will proceed to proceed to produce output data.
Parameters
Name | Type | Description |
---|---|---|
config | Object | the warning message and predicate |
config.if | (value : Variable , inputs : Inputs ) => EastFunction | If true a warning will be produced |
config.message | string | (value : Variable , inputs : Inputs ) => EastFunction | The message in the case that a warning is created |
Returns
FtpSourceBuilder
a new
SourceBuilderExample
const source = new SourceBuilder("FloatSource")
.input({ name: "max", stream: max_stream })
.writeable(FloatType)
.warn({
if: (value, { max }) => Greater(value, max),
message: (value) => StringJoin`The value ${value} exceeds maximum`
})
Inherited from
AbstractSourceBuilder.warn