Case Study
Helping a leader in Skill Assessment and Proctoring to scale and handle more load
Challenge
The client, a leader in skill assessment and proctoring, initially managed 2,500-3,000 concurrent proctoring sessions. A new government project demanded scaling to 15,000-20,000 sessions, which surged to 60,000 due to COVID-19, peaking at 200,000 concurrent sessions.
Key challenges:
-
Media Streaming Optimization: Media streaming clusters were not effectively optimized, resulting in sub-optimally routed requests
-
Database Inefficiencies: Suboptimal database queries led to high call volumes, increasing latency and performance bottlenecks.
-
RDS Instance Overload: Using the same Relational Database Service (RDS) instance for every microservice created difficulties in scaling and a single point of failure.
-
Database Connection Costs: High costs and performance issues were incurred due to inefficient database connections and lack of optimization.
-
Classroom Management Bottlenecks: Vertical scaling limitations in classroom management created a potential single point of failure.
-
Manual AI Proctoring Processes: Manual launching and shutting of machines for AI proctoring led to inefficiencies in both performance and costs
Solution
-
Media Streaming Cluster Optimization: Conducted virtual load balancing and cluster optimization for Kurento media streaming servers, ensuring requests were efficiently directed.
-
Database Query Optimization: Cleaned up database functions and converted triggers to code, reducing unnecessary database calls.
-
Transition to MongoDB: Replaced Neo4j with MongoDB, as Neo4j’s architecture struggled with the high write loads, leading to limited scalability and increased system strain.
-
Microservice-specific RDS Instances: Implemented separate RDS instances with database engines tailored to each microservice’s needs, enhancing performance and reliability.
-
Database Caching with Apache Ignite: Implemented Apache Ignite to reduce database load and increase response times.
-
Optimized Database Connections: Built an architecture with connection pools, allowing for rapid and repeated reuse of ready database connections.
-
Bloom Filters Implementation: Applied bloom filters to minimize unnecessary database load.
-
Horizontal Scaling of Classroom Management: Adapted the system architecture to enable horizontal scaling, removing the single point of failure risk.
-
Dynamic Autoscaling: Automated the process of launching and shutting down AI proctoring machines using Lambda functions, optimizing both performance and cost.
Xponentium Impact
-
Achieved Desired Scalability: Successfully handled peak concurrency of 200,000 sessions, meeting the expanded demand.
-
Improved System Performance: Optimized database and media streaming processes significantly reduced latency and improved overall system efficiency.
-
Enhanced Customer Satisfaction: Meeting the SLAs consistently boosted customer satisfaction and trust.
-
Cost Efficiency: The automated scaling solution and optimized database architecture reduced operational costs while maintaining high performance.