Process
To tackle this challenge, the team embarked on designing, building and deploying a cloud-based system capable of monitoring the cloud service at one-minute intervals.
The first few weeks focused on design and architecture, with the goal of achieving a cost-effective and reliable solution.
The design involved 3 milestones:
1. Onboarding system
This system facilitated the integration of new clients into the monitoring infrastructure by storing their configuration in a DynamoDB database.
2. Test system
The test system consisted of test executors responsible for retrieving client information from the database, and test clients who actively tested client endpoints and reported their findings.
3. Dashboards
In collaboration with another internal client team responsible for analytics visualization, the team developed dashboards using Grafana. These dashboards served as the backbone for interpreting and monitoring all incoming data, quickly alerting of any anomaly detected.
Highlights
- Developing the infrastructure as code using CDK.
- Setting up onboarding for new cloud service clusters.
- Designing test clients for various interfaces. (OTT, proprietary DRM, Widevine, Fairplay, Playready).
- Automating deployment with Codefresh.
- Building Grafana dashboards.
- Ensuring multi-region support and encryption for test results.
- Onboarding the client's customers into the monitoring solution.
- Encountering the first alert for a production customer.
