Hi, I'm Devansh
👋
AI Engineer · AI Agents · LLM Evals · RAG · Fine-tuning · Noida, India
DV

About

AI Engineer with 2.7+ years shipping LLM-backed features to production in TypeScript and Python. At Forest AI I build and own tool-calling AI agents run in production by AXA Group and Alteos, plus the evaluation harnesses and release gates that decide what ships — my evals raised agent accuracy from 78% to 95%. I also fine-tune small LLMs with LoRA and QLoRA.

Skills

drag me

AI Agents & Pipelines:

AI Agents
Tool Calling
Deterministic Workflows
LangGraph
Vercel AI SDK
Mastra
LangChain
Gemini / OpenAI / Anthropic APIs
Multimodal (OCR, Vision)
RAG
Hybrid Search
Pinecone
FAISS
pgvector

Evals, Safety & Reliability:

Evaluation Harnesses
Eval Datasets & Metrics
Release Gates
LLM-as-a-Judge
Backtesting & Replay
Regression Suites
Red-teaming
Prompt-injection Testing
Idempotent / Resumable Pipelines
Prompt & Dataset Versioning
Langfuse

Fine-tuning:

LoRA
QLoRA
PEFT
4-bit Quantization
PyTorch
Hugging Face Transformers
MLX
Precision / Recall / F1

Engineering:

TypeScript
Python
SQL
Node.js
FastAPI
PostgreSQL
MongoDB
Docker
AWS (SNS, Lambda)
GCP Cloud Run
CI/CD
React
Next.js
Playwright

Work Experience

F
Forest AI
Aug 2025 - Present
AI Engineer
Remote
  • Built and shipped tool-calling AI agents to production (Vercel AI SDK, Gemini) for AXA Group and Alteos insurance claims across WhatsApp, Telegram and web, with provider fallbacks for model failures.
  • Designed the email agent as a deterministic LangGraph workflow: a multi-step pipeline that is idempotent, retryable and resumable after failure via checkpointing.
  • Designed the evaluation harness and backtesting framework with versioned eval datasets (regression suites, benchmarks, adversarial sets); raised agent accuracy 78% → 95% and cut manual testing ~70%.
  • Used eval results as release gates, blocking releases on system-prompt leakage under prompt-injection turns and on latency regressions traced via Langfuse.
  • Made model behaviour reproducible by versioning prompts (Langfuse) and eval datasets, and replaying past conversations to reconstruct decisions.
  • Built an agent-simulation engine (Mastra, ElevenLabs) monitoring 8 production agents with 50 persona conversations/day, scored by LLM-as-a-judge.
  • Built a multimodal document-understanding pipeline (OCR, vision, rule checks) for per-claim risk scores that fails closed, and a RAG retrieval system on Pinecone at ~60 ms p50.
  • Fine-tuned Gemma 3 1B with LoRA for 13-intent JSON routing: macro-F1 0.455 → 0.962, valid JSON 12.5% → 100%, 2.2x lower latency.
  • Owned agents after launch: observability, AWS SNS alerting, GCP Cloud Run deploys, CI gates and Playwright end-to-end tests.
Userology
Aug 2024 - Jul 2025
Software Engineer
Remote
  • Debugged production issues in an LLM-backed user-research analysis pipeline, raising accuracy to 95% by tracing data-quality failures from raw transcripts to model outputs.
  • Rebuilt the campaign-creation flow (Next.js), improving recruitment efficiency ~30%, and cut re-renders ~40% by migrating state to Zustand.
I
Infitout Financial Technology
Mar 2024 - Aug 2024
Frontend Developer
India
  • Built the frontend of 1lyoptions, a real-time options-trading terminal on REST APIs.
  • Built a strategy-builder module reused across products.

Education

B.Tech, Computer Science
Noida, India
Featured

Talk to Website

Multi-tenant SaaS that turns any website or PDF into an embeddable RAG chatbot.

Talk to Website
  • •

    Shipped a multi-tenant SaaS turning any website or PDF into an embeddable RAG pipeline, with hybrid vector + keyword retrieval (reciprocal rank fusion).

  • •

    Idempotent content-hash sync so only changed sources are re-embedded.

  • •

    Protected ingestion against adversarial input through system design: SSRF-safe URL fetching and HMAC-signed visitor tokens.

Next.js
TypeScript
LangChain
FAISS
pgvector
Gemini
RAG
Hybrid Search

More work

Selected work

Fine-tuning experiments measured on held-out data, and contributions to open-source LLM tooling.

LLM fine-tuning

ClauseFinder

A 1.5B model fine-tuned to pull exact legal clauses out of contracts — without making quotes up.

Held-out F1
0.6640.924
Hallucinated quotes
21.7%0%
  • Fine-tuned Qwen2.5-1.5B with QLoRA (4-bit) on 1,626 CUAD contracts, trained locally with MLX.
  • Beats 3-shot prompting of the same model (F1 0.740) on the held-out set.
Qwen2.5-1.5B
QLoRA
PEFT
MLX
Python
Contact

Get in Touch

Want to talk about a project or an opportunity? Email me at vdevwish12312@gmail.com.