Accelerate the exit [ DataStage ]

 

Accelerate the

exit

DataStage ]

 

IBM's DataStage is a bit like an old diesel engine : powerful, robust, but expensive to maintain, complex to configure , and not really designed for modern cloud architectures.

To complete the picture, young computer scientists don't really want to get involved in it.


The future of this ETL seems uncertain at present.

 

Faced with this, the most commonly implemented option for DataStage migrations remains "copy-paste" (Lift & Shift) to one target or another.

 

And here, three major problems arise:

  1. After 10, 15 or 20 years of development, DataStage flows look (perhaps 😊) like a huge "plate of spaghetti": technical debt, accumulated complexity... good luck.
  2. Migrating to "T&M" mode is long, it's expensive... and on top of that, it's uncertain.
  3. The chosen target technology, however modern it may be today, will sooner or later end up burdened with debt, inflationary licensing costs — in short, almost a problem in the making.

1. "Job Graph" view in DataStage

Understanding the overall job flow - 

This view allows you to visualize the complete pipeline of a DataStage job in graphical form.

Each node represents a step in the processing (source, transformation, join, aggregation, target table) and the links materialize the data flows.

This is the overall understanding view of the DataStage job and its execution logic.

2. "Job Lineage" view in DataStage

Trace each field, from the target to the sources - 

This view presents the detailed downward traceability of the data, field by field.

It allows you to trace the origin of a field from the target table back to the sources, transformation after transformation, by precisely identifying the rules applied.

This is the audit and fine-grained explainability view in DataStage

3. “Data Lineage multi techno” view

Connecting technologies and understanding information usage - 

A macroscopic and cross-sectional view of data flows, beyond DataStage. 

It connects the different drivers of transformation and business uses within a single interface, and highlights the actual consumption of information.

This is the holistic view of data flows and uses .

3 minutes to understand everything
 

Commentaires

Posts les plus consultés de ce blog

E-book / Maîtriser & Simplifier une plateforme SAP BO avec {openAudit}

Power BI libère les utilisateurs… Mais comment garder la maîtrise de sa plateforme dans le temps ?

Migration Talend-dbt : un passeport pour moderniser ses données