Skip to content
All projects
AI / LLMFeaturedAirtel X Labs

AI-Powered Natural-Language Data Agent

A conversational agent that turns plain-English questions into Hive SQL, runs them on the cluster and replies with results, charts or files.

automated tests passing
202/202
pass rate when the harness started
35/44

Context

The spam-detection team needed fast answers from Hive tables and HDFS parquet data without writing SQL.

Problem

Routine questions such as lookups, counts and comparisons across loads all needed someone to write a query against the cluster. That made the people who were comfortable with SQL and the cluster a bottleneck, and slowed down the rest of the team.

My role

I designed and built the agent: the chat interface, LLM integration, SQL-first execution on Spark, the knowledge-base pipeline and the automated test harness.

Approach & architecture

  • Conversational interface on Telegram, group-chat aware, so the team can ask questions where they already talk.
  • The agent classifies intent, generates Hive SQL with a hosted Gemma 31B LLM, runs it on the cluster and formats the reply as text, a chart or a file.
  • Runs on YARN via spark-submit, so queries execute next to the data.
  • Migrated the agent from direct parquet reads to a SQL-first Hive architecture.
  • Knowledge-base pipeline: the team maintains table metadata in Confluence, and a weekly sync job converts it to structured JSON that the agent loads at startup. This replaced a hard-coded schema catalogue.
  • Features include LLM-generated matplotlib charts, CSV and multi-sheet Excel export, result caching with TTL, and proactive anomaly-detection alerts on batch loads.
Userplain-English questionChat interfacegroup-chat awareKnowledge baseConfluence → JSONAgentintent + SQL generationHosted LLMGemma 31BSpark / Hiveon YARNFormattertext · chart · fileReplyresults in chatScheduled monitorsbatch-load checksAlertsanomaly detection
A user asks a question in a chat interface. The agent uses a schema knowledge base and a hosted LLM to classify intent and generate SQL, runs it on Spark and Hive on YARN, and a formatter replies to the user with text, a chart or a file. Separately, scheduled monitors on the data send anomaly alerts to the chat.

Results

  • Built an automated test harness that calls the agent directly against a real Spark session, and drove the pass rate from 35/44 to 202/202 across lookups, aggregations, edge cases, routing and reply quality.
  • The spam-detection team can query Hive and HDFS data in plain English, without writing SQL.
  • Schema knowledge is now owned by the team in Confluence instead of being hard-coded in the agent.

Tech stack

  • Python
  • PySpark
  • Hive
  • HDFS
  • YARN
  • LLMs
  • Gemma
  • Prompt Engineering
  • Telegram API
  • Confluence

What I'd do next

  • Add query-cost guardrails, such as partition-filter checks and row limits, before SQL reaches the cluster.
  • Track answer accuracy from real usage and feed failures back into the test suite.
  • Move from loading the whole knowledge base at startup to retrieving only the relevant table metadata as the catalogue grows.