Creating a Data Source

We have now created the pipeline, but still, that is just an S3 source that is confusing for any non technical person. They want to look at graphs or things that are actually visualized, and not random numbers.

That is why the Platform also covers the part to visualize and make random numbers become useful. That part is called Dashboards.

To create a Dashboard, all that is needed is some data of any format. Luckily, we have just the right one created from the previous section.

In order to get some data into a Dashboard, we need to use the Data Lake exporter in our pipeline. It is worth noting that a Data Lake exporter is just a Parquet with a predefined path set by the Platform.

First of all, let us create a version in our pipeline My First Pipeline called to_dashboard, as in the image below:

Creating a version of a pipeline

Then, set the content of the version called to_dashboard as following:

Exporting parquet data to Data Lake

Fill in the Data Lake exporter as below:

Data Lake exporter options

It is important to note here that you can’t use versioning with the yyyy-mm-dd format, as you can’t have the “-” symbol in the topic of a Data Lake exporter, instead you may use the default yyyy_MM_dd format for this example.

Finally, run the pipeline and wait for it to finish. Once the pipeline is finished, refresh the page (so the Data Sources page fetches the latest data).

Listing available data sources

The data source is now visible under Data Sources page. Now, with a ready data source, we can finally create a Dashboard.