Senior Data Platform Engineer

India, Remote·Posted today

mldata-platformspythonawsgcpterraformkafkasparkragllm

<div class="content-intro">Who we are DigiCert is a global leader in intelligent trust. We protect the digital world by ensuring the security, privacy, and authenticity of every interaction. Our AI-powered DigiCert ONE platform unifies PKI, DNS, and certificate lifecycle management, to secure infrastructure, software, devices, messages, AI content and agents. Learn why more than 100,000 organizations, including 90% of the Fortune 500, choose DigiCert to stop today’s threats and prepare for a quantum-safe future at <a href="http://www.digicert.com/">www.digicert.com</a></div>  Job summary We are looking for a Senior Data Platform Engineer to design, build, and operate the foundational data and ML infrastructure that powers analytics, reporting, and machine learning across DigiCert. This role sits at the intersection of data engineering and ML platform work — you will own the systems, pipelines, and tooling that data scientists, analysts, and engineers rely on every day. You bring deep expertise in Databricks, a strong engineering mindset, and hands-on experience building and maintaining ML pipelines in production. You will partner closely with data science, analytics, product, and business teams to deliver a platform that is reliable, governed, and built to scale.   What you will do <ul> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Design, build, and maintain scalable data and ML pipelines using Python and SQL, processing large-scale datasets across batch and streaming workloads</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Own and evolve core platform infrastructure on Databricks — including Delta Lake table architecture, Unity Catalog governance, Databricks Workflows orchestration, and compute optimization</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Build and maintain end-to-end ML pipelines: feature engineering, model training pipelines, experiment tracking (MLflow), and model deployment/serving infrastructure</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Collaborate with data scientists to operationalize models — bridging the gap between experimentation and production-grade ML systems</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Define and enforce data platform standards: ingestion patterns, data modeling conventions, medallion architecture (Bronze/Silver/Gold), and pipeline reliability practices</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Implement data quality, observability, and monitoring frameworks to ensure platform health and data trustworthiness</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Optimize pipelines for performance, cost, and reliability at scale using Spark and PySpark</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Evaluate, integrate, and govern new platform tooling and data sources within the Databricks ecosystem</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Contribute to architectural decisions and help drive the long-term data platform roadmap</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Participate in code reviews, technical design discussions, and engineering standards</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Mentor junior engineers and elevate overall platform and data engineering practices</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Document platform architecture, pipeline design, and operational runbooks</li> </ul>   What you will have <ul> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">6+ years of experience in data engineering, data platform, or ML engineering roles</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Strong proficiency in Python and SQL, with a track record of building production-grade data pipelines using both</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Hands-on Databricks expertise: Delta Lake, Unity Catalog, Databricks Workflows, PySpark, and the Databricks ecosystem broadly</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Experience building and maintaining ML pipelines in production — feature engineering, training pipelines, experiment tracking, and model deployment</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Familiarity with MLflow or comparable experiment tracking and model registry tools</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Experience working on cloud data platforms (AWS, Azure, or GCP)</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Strong understanding of data modeling, dimensional design, and analytics-friendly data architecture</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Experience with batch and incremental/CDC pipeline patterns</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Proficiency with Git, version control, and CI/CD practices for data and ML workflows</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Strong engineering judgment — you think about reliability, maintainability, and cost, not just correctness</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Clear communication and comfort working with both technical and non-technical stakeholders</li> </ul>   Nice to have <ul> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Experience with streaming or near real-time pipelines (Kafka, Kinesis, Spark Structured Streaming)</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Familiarity with feature store platforms (Databricks Feature Store, Feast, or Tecton)</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Experience with LLM pipelines, RAG architectures, or AI/BI tooling (Genie, AI Functions)</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Knowledge of data quality and observability tooling (Great Expectations, Monte Carlo, etc.)</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Exposure to dbt or similar SQL-based transformation frameworks</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Infrastructure-as-code experience (Terraform, Databricks Asset Bundles)</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Experience working in Agile or Scrum environments</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Prior experience mentoring engineers or shaping platform standards</li> </ul>   Benefits <ul> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Generous time off policies</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Top shelf benefits</li> <li style="font-family: arial, helvetica, sans-serif; font-size: 12pt;">Education, wellness and lifestyle support</li> </ul>   To protect candidate information and maintain a secure hiring process, all applications must be submitted through our careers portal. Resumes or CVs sent directly via email will not be reviewed or considered.   #LI-SD1

Apply for this role