ETL monitoring and management solution increases visibility by 87% and productivity by 43%

THE CHALLENGE

At the time, our banking customer had no central ETL monitoring and management solution for their data warehouse. They did not have visibility into the jobs that were scheduled or underway or which jobs had failed. They had no method for automatically reinitiating jobs that had failed. They were also unable to pinpoint the reason a job failed.

 They needed a centralized dashboard to handle their business side as well as the ability to monitor and manage hundreds of attributes around the various jobs, including job status, job flows, errors and failures, successful completions, failure reports, ticket handling, and more. Our customer also needed the ability to retrigger failed jobs on an ad-hoc basis along with process lineage visibility.

THE SOLUTION

Beyondsoft created a centralized ETL monitoring and management solution, including multiple MVPs within just seven months. The solution enables our customer to monitor incoming jobs from several data pipelines from multiple source systems in different formats. The solution delivers insights on job status including run time, success or failure status, reason of failure, and other key measures.

Beyondsoft also created a tool to automatically handle job failures. When a job fails, the tool creates a failure report with in-depth details including who initiated the job, how long the job had been running, the reason for failure, and other details. The solution then automatically generates a ticket that is escalated to the appropriate party.

ETL monitoring and management solution architecture overview
ETL monitoring and management solution architecture
THE RESULTS

The ETL monitoring and management solution provides our customer with a central dashboard for monitoring as well as in-depth visibility and control through error detection, error enrichment, event triggering services, unified monitoring service, unified metadata, SLA impact, and recovery. Our customer also benefits from enhancements in their existing components like Airflow, Job Server, Ingestion, Computes, and DaaS. The solution from Beyondsoft delivered the following benefits:

  • Increased productivity by 43%.
  • Reduced losses from job failures by 76%.
  • Cut required headcount in half, from 16 to 8.
  • Increased observability of the entire process by 87%.
TECHNOLOGIES USED

Java 1.8, Spring Boot, Spring Cloud, Spring MVP, Spring Data, Spring Security, Kafka, ELK, Filebeat, Healthbeat, Kubernetes, Docker, Jenkins, React js, and Kibana.

なぜ私たちを選べますか

私たちのオンショア、ニアショア、オフショアのデリバリーサービスは、24時間365日、お客様のビジネスをサポートします。日本の大手SI企業に対し数十年にわたりサービスを提供しています。1999年以来、日本で25年の経験を持っています。日本には500-600人のスタッフがいます。長年にわたる成功事例は、私たちがお客様の投資対効果においてどれほど重要であるかを証明しています。シンガポールは私たちのグローバル本部であり、世界各地に14の地域オフィスを持っています。

30年以上にわたる強力なITコンサルティングサービスの経験

COBOL, C, Java, Pythonなど幅広い開発言語や開発環境に対応

SAPにおけるABAP, BTP, Fioriなど幅広く対応

40以上のグローバルデリバリーネットワークを持つ4大陸をカバー

CMMI 5、ISO 9001、ISO 14001、ISO 20000、ISO 27001、ISO 22301、ISO 45001、TMMi5の認証

マイクロソフトの専門家であるAzure MSP

Beyondsoftと共にビジネスの潜在力を最大限に引き出しましょう。私たちがどのようにイノベーションを推進し、効率を向上させ、ビジネスの成長を実現するのか、ぜひご相談ください。