Tech Lead, Data Engineering

Design and maintain scalable data pipelines tailored to LLM requirements, including preprocessing unstructured text data from various sources, implementing chunking strategies, and optimizing embedding generation for vector databases. Build and manage data infrastructure, including data warehouses, data lakes, and streaming solutions, specifically optimized for LLM workflows. Deploy LLMs into production environments using containerization (Docker) and orchestration tools (Kubernetes). Automate CI/CD pipelines for model versioning, A/B testing, and rollback procedures, ensuring seamless updates to fine-tuned models. Optimize data systems for performance, reliability, and scalability, particularly for real-time inference for applications like chatbots or document analysis. Implement MLOps-driven model deployment and monitoring, tracking key metrics such as inference latency, token usage costs, and output quality drift. Manage vector databases (e.g., Qdrant, Pinecone, FAISS) and design indexing strategies for Retrieval-Augmented Generation (RAG) architectures. Collaborate with data scientists/analysts, and other stakeholders to understand data and LLM requirements and deliver solutions. Create and maintain documentation for all data-related processes, procedures, and workflows, including LLM-specific pipelines and deployments. Research and stay up-to-date with the latest trends, technologies, and best practices in data engineering, MLOps, and LLM technologies. Mentor junior engineers, conduct technical reviews and provide active guidance. Contribute to technical roadmap planning, architectural decision-making and lead technical initiatives. Implement data governance best practices, establish and enforce data quality standards across teams and projects. Identify and mitigate technical risks in data infrastructure and LLM deployments.