Operations Dashboard

Clarifier SemanticInputRef vs ForeignKeyRef (doc + éventuelle unification)

Record traceability refactor — event log via identity-resolver

Refondre la gestion du target_id entre actions et entre sinks

E2E cross-source entity resolution — eng_instances résolues par clés métier uniquement

Revoir l'architecture sinkId vs datasetId

Refactor ACK mechanism toward direct mapping model

Retroactive identity resolution when entity catalog is declared after ingestion

Rename noesisId → recordId and noesisURN/globalId → globalId across codebase

[Frontend] Identity Hub — fixer le dashboard, renommer terminologie, page Configuration specs

E2E Bug - incremental_ingestion scenario failing

[Frontend] Visualisation du schéma des sinks dans l'éditeur de pipeline

MVP Phase 1 — Stabiliser + rendre démontrable (7-18 avril)

MVP Phase 2 — Compléter le MVP fonctionnel (21 avril - 9 mai)

MVP Phase 3 — Déployer en SaaS + mise en vitrine (12-30 mai)

Infra — Déploiement cloud (Docker Compose → hébergement cloud)

Infra — Auth & multi-tenant basique (Keycloak cloud)

Infra — HTTPS, persistence, backups

Frontend — Landing page produit noesis.io

UX — Onboarding simplifié (source → pipeline → résultats en < 30 min)

Démo — Scénarios de démo documentés (PLM/XPDM + multi-sources)

Core - Sink schema management from operations

E2E Bug — consolidated_join and chained_consolidated_joins scenarios failing

E2E Bug — identity_global_resolution scenario failing

E2E - Fixer admin_user_roles: ajouter API creation de user ou provisioning Keycloak

Core/API — Capabilities & routing des tâches (extract/import/transform)

Core/API — Gouvernance de concurrence (global/capability/tool/source/sink)

Workers — Registration/heartbeat avec metadata de capacité et resource class

Observabilité — Métriques & alerting de saturation/queue/fairness

Frontend — Vue dynamique de l’infrastructure workers/pools/queues

E2E + docs — Validation ADR-010 (scheduling/caps/backpressure)

Core/API — Uniformiser le statut QUEUED sur ingestion/import/transform

Import & Dependency Resolution

[Core] Simplification des schémas des actions vers opérations vers sink

Connector - Add connector on demand

Core - Dynamic schema - Customer extensions in datasets

Streams - Per-file status tracking and retry

UI - Data source - Discovery running

When updating noesis ID, we also need to update lineage targetting it

UI - Administration Panel for Users

UI - Global refresh toolbar

UI - Data Source - Capabilities - We should see which datasets can be imported/ingested with which tool

Target schema auto-creation

Flink - Review incremental_join problems

Core - Sink schema management

Core - Python Transform Infrastructure (user-defined transformations with lake access)

Delete propagation

Reporting - Run event data model and Postgres storage

Reporting - API endpoints GET /runs/{id}/report and /errors

Flink - Study impact of upgrading to latest Flink version (1.20.3 → 2.x)

E2E - Tests avances de dependances cross-datasource (multi-source vers une cible)

E2E Bug — streaming_join scenario failing

UI - Data Lineage - Data Lineage should have a better display, currently it's only statistics

Architecture Pilot/Workers (ADR-010) — capacités, scheduling et gouvernance de concurrence

[Archi] Supprimer stepContext du workflow multi-step — remplacer par lookup en base

Infra - Étude multi-tenant: architecture, isolation, impact sur le code existant

Infra - Étude comparative des cloud providers pour Noesis SaaS (focus Europe/France)

Core + UI - Pipeline schema validation: conformité des data models entrée/sortie par opération

Core + UI - Pipeline pause/resume/restart transparent (masquer la complexité Flink)

UI - Pipeline live monitoring: real-time DAG visualization with per-operation metrics

Core - Pipeline bootstrap from lake when Kafka retention is expired (historical data access)

Core - Kafka Source Connector (streaming data source with JSON support)

UI - Demo review session: real-time feedback (NOE-23 + NOE-25)

Manage dataset dependencies to allow correct import orchestration

[Frontend] Construire IHM de dashboard/pilotage/suivi des dépendances et des déblocages

Batch Import Tool — Stratégies de regroupement configurables (ADR-011)

Plan d'ingestion — Sub-ingestions parallèles (batchId/subBatchId, SUB_BATCH_END)

Core — Stratégie fixed-size + time-window

Core — Stratégie tree (regroupement par sous-arbre hiérarchique — cas XPDM)

Core — Stratégie custom (script Python via infra transform)

Core — API CRUD BatchStrategy + état accumulation

3DX — Finaliser le connecteur XPDM batch (PackageBuilder + XML format NOE-232)

3DX Simulator — Support XPDM batch import (PR #3 en review)

E2E — Scénarios batch import (partial + complémentaire + timeout + erreurs)

Frontend — Configuration batch strategy sur pipeline + vue progression packages

UI - Pipeline - It should be possible to have a global view of the pipeline with also its impact (like sink -> data source tools and source with data sync info)

UI - Pipeline - It should be possible to see schema info on operation (in and out)

Core — Renommer import-resolver en import-orchestrator

UI - Data sync activity when running

[E2E] Error quarantine + resolution test scenarios

Core — Modèle domain BatchStrategy + ToolModePackage + migrations

Core — Câbler le mode PACKAGE dans ImportMessageHandler + import resolver

Core — BatchAccumulator dans l'import resolver (Redis state + triggers BATCH_END/timeout)