Analytics without managing data pipelines seems like something impossible. Dashboards and reports start with data. Data must by definition be structured in order to gain insight. Setting up business intelligence solutions therefore begins with taking stock and unlocking data sources. Because the infrastructure on which data warehouses operate has become much more powerful in recent years, we have come to a point where the traditional ETL procedures, which write the data from an operational database to the data warehouse, are slowly but surely changing to an ELT (Extract , Load, Transform instead of Extract, Transform, Load). No matter how you turn this around, a degree of standardization is needed to create a clear view of the data.
Business intelligence projects often consist of aligning physical data sources to something that is presentable to the business. This means that the data from the data sources must be modeled in such a way that the way the business looks at the data must correspond to how the data is physically located in the data warehouse. This is to a large extent the T of ETL. However, this ensures that the transformation that takes place in the ETL procedure is opaque and unnecessarily complex. End users do not have access to the transformation logic and therefore often do not know how certain values are created.
A few years ago, a few smart entrepreneurs started making standard models for specific data sources. This has brought us a step closer to a clear view of the data. Specific knowledge of the source system, translated into a loading procedure and a tightly modeled business model that creates clarity for the end user. Any changes to the data model of the source system will automatically pick up changes. This way you are “upgrade-proof”.
With the arrival of Fivetran , analytics has become possible without managing data pipelines. Fivetran has developed standard connectors for more than a hundred of the most used data sources. Fivetran chooses to offer managed connectors so that any changed fields in the source system can always be made immediately available for charging. The advantage of these managed data pipelines is that you do not have to worry about your loading procedure and that you have standardized schemes at your disposal. There is also extensive documentation of all connectors available so that you have an overview end-to-end when it comes to data management.
Do you want to know more about Fivetran or how DataBright can help you with a modern analytics setup? Please contact us.