Pipeline Creation

What is a pipeline?

What can be seen so far is that we are definitely going to be dealing with pipelines, but what actually are they?

Having in mind that VDH is an ETL Tool, and its keyword is pipeline, we can assume that a pipeline is an ETL process. This assumption would be correct, as we use them to extract data from sources (importer), transform them, and write them as output (exporter). This will become clearer in the next sections. For now, let's create an empty pipeline.

In the Create Your Project section, you have created your project. We will create your first pipeline there. If you are not sure whether you are currently on your project, in the image below you can see how to navigate to it.

Description of home options

Now that we are at the right place (at the right time 😉), check the image below to navigate to VDH.

Navigating to Visual Data Hub

Creating a pipeline

As there are no pipelines created so far in your project, let's go ahead and create the first one. To do it, click on this button , located at the bottom-right corner of the screen. A window like this will open:

Creating a pipeline form

The parameters are as follows:

• Name: A project can have multiple pipelines, and we use this parameter to identify the pipeline within the project. For the sake of our example, name it: My First Pipeline

• Description: Here you may write a short description to help others understand what this pipeline does, or where it is needed. On our example, please write: This pipeline is created as part of 'A Data Primer Onboarding Program'

• Version Name: We use versioning to make our life easier, and we strongly recommend you do so too. Doing so, you will be in control of the pipeline throughout its life span, handling changes in a way that does not affect functioning pipelines. We mostly use dates as version names, in order to have an instinctive understanding of when the pipeline had changes. A pipeline without any versions can not exist, therefore, we need to create one as well. You can write the date in the following format: YYYY_MM_DD

• Project: Every pipeline must belong to one project, and yours should reside in the chosen project (yours).

• Permissions: This section is used to manage permissions within the project. The rules of User Management apply on pipelines as well. For this pipeline, set your permissions as follows:

  1. Viewers: Public

  2. Editors: Your User

Click on Save Changes!

CONGRATULATIONS!!! You have created your first pipeline and it should look something like this:

Pipeline card explanation

As each pipeline consists of its own rectangle box, we can manage the pipeline within that structure.

• Modifying versions: To modify a version of the pipeline, click on the dots on the right. There you will have the choices to:

  1. Edit: opens the window that was opened when creating the pipeline will be opened, and you can make the changes on it.

  2. Clone: provides the means of creating a new version, starting from the previous one.

  3. Remove: removes a version of the pipeline.

• Adding new versions: this is a straightforward process, by clicking on the button, and using a new version name. This new version will be blank, unlike the Clone option that we previously talked about.

There is more to VDH and it's features, but it will all be uncovered within the next sections. Stay tuned! 😉