Gee
Dec 31, 2020

Nice write-up! One thing I want to add to this list is data validation. As data engineers, we should also validate/test data we generated after a round of transformation. You mentioned dbt but it was referred to as a tool for test automation. I think we should articulate that 'data validation' is a step we cannot dismiss to keep good data quality inside an organization. Some of the tools include Great Expectations, dbt.

Gee
Gee

Written by Gee

Data/Machine Learning Engineer

No responses yet