Prateek Rai
I build data platforms that scale — and I use LLMs to do it faster.
Senior Software Engineer at Atlan, working on the metadata catalog that helps Fortune 500 enterprises understand and govern their data. My work sits at the intersection of distributed systems, data infrastructure and AI-augmented engineering.
Experience
Atlan
Remote
Senior Software Engineer I
Mar 2026 – Present
Architecting and building the next generation of data connectivity and platform services for Atlan's metadata catalog.
- Re-architected a high-traffic enterprise connector from Argo to Temporal on a next-generation SDK, cutting peak memory by 13x and eliminating recurring out-of-memory failures through streaming extraction and chunk-wise enrichment.
- Led the design of reusable intermediary services and SDK components for partner integrations, creating a generic ingestion pipeline and standardised asset model adopted by 10+ engineering teams.
- Drove end-to-end resolution for 300+ high-impact customer issues across 25+ Fortune 500 enterprises.
- Shipped 5+ joint catalog and lineage integrations from preview to general availability through cross-functional collaboration.
Data Engineer
Apr 2024 – Mar 2026
Built and delivered core data-source connectors for Atlan's metadata catalog.
- Designed and shipped multiple production connectors and platform services, owning architecture, implementation and release end-to-end.
- Developed ingestion pipelines and reusable integration components adopted by downstream engineering teams.
- Drove resolution for customer issues across enterprise accounts, working cross-functionally with engineering and partner teams.
BYJU'S
Bangalore, Karnataka
Member of Technical Staff I
Oct 2023 – Apr 2024
- Engineered OLAP pipelines, introducing event triggers for real-time JSON delivery and orchestrating scheduled jobs on Apache Airflow alongside OLTP pipelines on Kafka Connect.
- Owned and optimised ETL pipelines with Talend and AWS Glue, holding a 95% success rate on bulk-load jobs, and managed the Snowflake warehouse with Python/SQL tasks, complex role structures and data masking.
- Spearheaded an Apache Superset implementation for dynamic dashboarding, cutting Snowflake credit usage by 10% through a query and cost monitoring dashboard on a Dockerised EC2 deployment.
Engineering Intern
Jan 2023 – Sep 2023
- Engineered Spark-based data transfer utilities handling 10+ TB and 100M+ records across Parquet, CSV and JSON.
- Streamlined Spark job creation across 30+ data pipelines using AWS Glue, and improved data accessibility for 50+ datasets through AWS Athena.
- Worked on the Snowflake warehouse with Python for data processing and automation, reducing processing errors by 40% across 20+ sources.
Skills
Core — Systems Design, Software Architecture, Distributed Systems, Microservices
Languages — Python, Go, SQL, TypeScript
Platform — Temporal, Argo Workflows, Apache Airflow, Kafka, Kubernetes, Docker, Terraform, FastAPI
Data — Snowflake, BigQuery, PostgreSQL, Apache Iceberg, Apache Atlas, Elasticsearch, Spark
Cloud — AWS, GCP, Azure
AI engineering — LLM agents, Model Context Protocol, context engineering, Claude Agent SDK
Projects
DocAid
Jan 2022 – Jan 2023
GitHub · Capstone project, Thapar University
- Engineered an IoT wearable on Arduino Uno with pulse, oxygen, temperature and ECG sensors, streaming 100 readings/min to a Firebase realtime database.
- Built a real-time Django/Channels dashboard monitoring up to 500 patients.
GitRep Scorer
Jun 2022 – Jul 2022
- Built a GitHub repository rating tool in Python, scoring a 10,000-repo Kaggle dataset with BeautifulSoup and TOPSIS analysis.
- Deployed as a Streamlit app.
Education
BE, Computer Science and Engineering
June 2023
Thapar Institute of Engineering and Technology · Patiala, Punjab
- CGPA: 8.84
- Minor: Data Science
Class 12, CBSE
June 2018
Army Public School · Ambala Cantt, Haryana
- Percentage: 94.2%
- Subjects: Physics, Chemistry and Mathematics, with English and Information Technology
Class 10, CBSE
June 2016
Army Public School · Ambala Cantt, Haryana
- CGPA: 10