AI Data Engineer · 4+ Years

PhaniKumar S

Building pipelines at scale, with AI on top

Senior Data Engineer specializing in scalable data pipelines, real-time ingestion, and cloud-native architectures on Azure — extended with agentic RAG, Databricks Mosaic AI, and AI/BI Genie for natural-language, self-service data access. Currently at Amegy Bank, Houston.

Get In Touch GitHub ↗ LinkedIn ↗

Years Experience

99.9%

SLA Adherence

60%

Processing Reduced

$80M

Revenue Impact

Houston, Texas, US

AVAILABLE FOR OPPORTUNITIES

Career

Experience

Amegy Bank

Sep 2025 – Present

Senior Data Engineer · Houston, TX

Orchestrated 25+ Azure Data Factory pipelines with dependency management, retry logic, and alerting via Azure Monitor, achieving 99.9% SLA adherence across all production pipelines.
Built a data validation and reconciliation framework using Azure Databricks with PySpark, improving transaction data accuracy to 99% and reducing financial reporting errors carrying an estimated $2M+ compliance risk exposure.
Developed automated failure detection via Azure Monitor and Logic Apps with real-time Microsoft Teams alerts, reducing incident response time.
Implemented Azure RBAC and Azure Active Directory (Entra ID) role-based access control policies, alongside Unity Catalog for centralized data governance, table/column-level access controls, and audit logging, to enforce least-privilege access on sensitive banking datasets in Azure Data Lake Storage Gen2.
Migrated full-load batch pipelines to incremental load architecture using watermarking and Delta Lake MERGE upserts on Azure Databricks, reducing data processing time and cutting annual cloud compute costs by $15K.
Automated CI/CD pipelines using Azure DevOps for data pipeline deployments, version control, and automated testing, improving code quality and reducing deployment time.
Built an agentic RAG POC using Databricks Mosaic AI Agent Framework, dynamically routing natural language queries between Genie (structured/SQL) and Mosaic AI Vector Search (semantic similarity) based on query intent, enabling accurate responses across banking data.

Wipro

oct 2021 – Dec 2023

Data Engineer · Hyderabad

Designed automated ADF pipelines with real-time triggers, enhancing operational reliability by 30%.
Optimized Spark jobs via partition pruning, caching, and Z-ordering on Nestlé sales data — improving query performance by 20%.
Developed automated pipeline failure detection using Azure Monitor + Teams alerts, achieving 50% faster incident response.
Implemented Delta Lake upserts with watermarking, reducing ETL processing time by 40%.
Supported Nestlé's $80M YoY revenue growth in 2022 through optimized reporting pipelines.

Infosys

Mar 2020 – Sep 2021

Data Engineer · Hyderabad

Designed and deployed DBT models to ingest and transform insurance claims, policy, and provider data into the warehouse, processing 50M+ records daily and reducing reporting query time by 45%.
Developed complex transformation logic using Spark SQL, multi-level joins and window functions to validate claims adjudication, membership eligibility, and provider network data — improving data accuracy to 98% across 200+ end users.
Orchestrated insurance data pipelines using Apache Airflow DAGs — automating end-to-end claims ingestion, transformation, and loading into the warehouse for downstream analytics reporting.
Implemented DBT-based data lineage and access control policies to enforce data security compliance across sensitive policyholder datasets.

25+

Airflow DAGs Built

10+

Upstream Systems

40%

ETL Time Saved

Top-Tier Companies

Selected Work

Projects

★ Featured — Agentic RAG

Azure AI Chatbot — Agentic RAG on Databricks

Built a RAG chatbot on Databricks using LangChain, Azure AI Foundry, and Azure AI Search with hybrid retrieval and grounded LLM responses.
Implemented hybrid (vector + keyword) search retrieval to improve answer relevance over pure semantic search.
Designed idempotent document ingestion using content-hash IDs to prevent duplicate entries across a 5,600+ page document set.
Extended into an agentic RAG proof of concept using Databricks Mosaic AI Agent Framework, dynamically routing natural language queries between Genie (structured/SQL) and Mosaic AI Vector Search (semantic similarity) based on query intent.

Databricks Mosaic AI LangChain Azure AI Search AI/BI Genie RAG

Data Engineering

Flight Data Analysis Pipeline

End-to-end Azure Databricks pipeline using Databricks Autoloader for incremental ingestion (raw to bronze).
Delta Live Tables for silver-layer transformations and data quality enforcement.
Curated analytical views for top airlines and bookings-per-year trends.

Databricks Autoloader Delta Live Tables Medallion Architecture

PhaniKumar S

Technical Stack

Experience

Projects

Education

Get In Touch