Confidential client

The challenge

Build a platform that monitors news, web and social media across the Middle East and beyond, in Arabic, English and more, and turns millions of posts and articles into analytics clients can act on.

What we built

• Collection layer: 900+ news/web crawlers (838 active) plus scheduled pipelines for Facebook, Instagram, TikTok, YouTube, X and Kwai, each with its own schedule, limits and runtime controls.
• Kafka pipelines into MongoDB and Elasticsearch; 118M+ indexed articles and approximately 6 GB downloaded per day.
• Multilingual named-entity recognition (130K+ mentions processed through a monitored NER queue), sentiment and emotion, topic and trend detection, influencer ranking and audience profiling.
• Client dashboard with multilingual Boolean queries, team workspaces, mentions, share of voice, reach, engagement, advertising value, country and language breakdowns, word clouds and top influencers.
• Admin tooling for crawler health, pipeline scheduling and NER process monitoring.

Results

Stack