From source to dashboard cell: Mapping the IS to rebuild it intelligently

From the source to the dashboard cell:     Mapping the information system to rebuild it intelligently In IT transformation projects, there is one injunction that keeps coming up: "We must migrate!" ... to the Cloud, to modern architectures, to more agile solutions, etc.    In recent years we have seen the emergence of new data solutions that are sweeping everything in their path: for example, Power BI for data visualization, dbt for transformation management or Snowflake for storage: simple, efficient, scalable, cloud native, often cheap, etc. When we decide to take action, a few questions come to mind: "  What exactly should we migrate  ?" "  And therefore, what can we throw away   ?" "  And how do we rebuild what has value in the target without breaking everything?  " Below is an approach based primarily on the real-time analysis of all internal processes of a company.     To modernize, to migrate, it will necessarily be necessary to understand how information flows, from operational sources to its final exposure, column by column, to the dashboard cell accessed by the business.   And knowing what the information produced is actually used for 😊! The findings are often quite surprising... Action #1:  Inventory all continuous flows, from sources to the dashboard cell, with a complete reverse engineering of the information system. In the feeder layers, it will be a matter of analyzing stored procedures, ETL/ELT jobs, nested views, encapsulated SQL transformations, FTP transfers, subqueries, cursors, etc. ➡️ Output;  At each stage, the technical data lineage will highlight the transformation tools used (such as ETL, scripts, SQL queries…), the nature of the processing applied (filters, aggregations, joins, etc.) as well as the sequence of operations or flows (via schedulers, execution scripts, etc.):  In the data visualization layer, the aim will be to push the analysis down to the cell level of a dashboard, even in complex multi-technology environments: analysis of the business rules embedded in the dataviz objects, the intermediate semantic layer, expressions, filters, etc. ➡️ Output: A data lineage in the data visualization layer allows zooming from a source field in the DWH to the cell accessed by the business. This data lineage is connected to the data lineage in the sources to provide a complete view of a data flow.   Action #2:  Define the levels of use: know what each feed flow is used for, from operational sources to the dashboard cell.  The main technical stack will need to be analyzed  to understand all the data consumed within and outside of the batch chains. Data consumed by satellites (applications not "parsed") will also be analyzed  to identify the completeness of useful information. This dual analysis will be configured to take into account the target business  : regulatory information can be consumed very periodically, while still having significant added value. Combine action #1 and action #2:  Combining data lineage and usage analysis: the winning combination for transforming a system!  Trace back through the streams to isolate unnecessary chains: Data lineage allows you to trace the pipeline back from unused data to the first table that originated information consumed in another branch. From this branching point, it will be possible to remove the unnecessary portion of the chain  without any impact: our various projects allow us to suggest that 50% of tables and processes can be eliminated on average. The same is true (often much more) for dashboards… Let's go! In the "remaining tasks to migrate" section, each flow and each dashboard can be scored based on internal considerations.  This will allow for precise prioritization of the migration. Automated migration to the Cloud:  The (continuous) reverse engineering technique will allow us to expose the processing chains (ETL/ELT jobs, procedures in the feeder layers, business rules in the data visualization layer). This ultra-granular knowledge of the source, and its upstream rationalization, will allow for the intelligent reconstruction of data flows within the target technologies,  data feeds, and data visualizations (in addition to the template and any semantic layer). The pivot will be largely performed using SQL. This will truly open the door to the transformation of the information system.  Migrating food supplies:  Technological scope addressed by {openAudit}:  Migrate the dataviz layer:  Technological scope addressed by {openAudit}:  Understanding in order to transform better: that is the whole role of data lineage and usage analysis. With  {openAudit} , you have a robust and automated solution for mapping, streamlining, and securing IT modernization projects—from the first column to the last cell of the dashboard. Our diagnostic and migration projects are offered at fixed prices.