Source
The
Source module facilitates creation of streaming interfaces with external systems (such as an api) or files to provide continuous streams of input data.Example
// Create a blob datastream containing the data in an ftp file, updated daily
export default new SourceBuilder("Hourly")
.ftp({
cron: "0 8 * * *",
uri: () => Const("ftp://ftp.bom.gov.au/anon/gen/clim_data/IDCKWCDEA0/tables/qld/applethorpe/applethorpe-200902.csv"),
})
.toTemplate()
Example
const source = new SourceBuilder("FileSource")
.file({
path: "./my_big_csv_file.csv",
})
Example
// Create a blob datastream containing the response from a http get request, updated daily
export default new SourceBuilder("Hourly")
.api({
cron: "0 8 * * *",
url: () => Const("https://covid.ourworldindata.org/data/owid-covid-data.json"),
retry: (_, attempts) => Less(attempts, 5n)
})
.toTemplate()