Talend-dbt Migration: a passport to modernize data

Talend-dbt migration: a passport to modernization its data Talend ETL is very well known for its OS version, "OS" for Open Studio 😊.  Talend, the publisher of the eponymous ETL, was acquired by Qlik in May 2023. The free version of Talend ETL (“TOS”) has not been distributed since early 2024. This means that “TOS” is no longer maintained or updated. Talend Data Fabric, the paid version of "TOS", takes over, with better support, new features, etc. Many companies that had opted for "TOS" are considering leaving the platform to escape the paid model, among other reasons.  Furthermore, a tool called "dbt" (Data Build Tool) is experiencing explosive growth. In this article, we explain how to approach a transition between TOS and dbt.  ETLs ​ Talend, among others, are generally less popular.  The end of ETL?    ETLs, which allow the data transformation process to be "outsourced" to third-party machines, and thus to "relieve" the burden on databases, have less added value with the advent of powerful Cloud databases, such as BigQuery, Snowflake, Redshift and others. With an ELT, the transformations are performed directly within the DWH. The unparalleled added value of SQL    Workflows designed with ETL interfaces are easy to learn, but rigid, whereas the latest generation ELT frameworks, in "quasi" SQL, have great flexibility and use a universal language: SQL (or dialects of SQL) 😊!   dbt, it's what ? Among the fastest-growing frameworks is dbt (from dbt Labs), which was founded in 2016 and already has 3500 user companies!   Some of its characteristics: dbt allows you to modularize SQL using models  and macros (via Jinja). dbt uses simple YAML files  to manage configurations  / define dependencies and specify relationships between models (in a DAG: Directed Acyclic Graph). The quality tests are defined in the YAML files. dbt is versioned with Git , the most widely used version control tool in the world. … And dbt offers an open source version  (dbt Core), and therefore a "freemium" model which, incidentally, had allowed Talend to emerge rapidly a few years ago... Beyond this migration from Talend to dbt which can achieve consensus, the deep history of the source platform, with intricate flows that may have flourished for years, implies that the  migration will necessarily be a complex challenge. In any case, in "paper/pencil" mode 😊.    We believe that automating this migration is essential.