Senior Data Engineer — Azure, Databricks & ML Pipelines | Remote

<img src="https://we-work-remotely.imgix.net/logos/0171/6035/logo.gif?ixlib=rails-4.0.0&w=50&h=50&dpr=2&fit=fill&auto=compress" /> <p> <strong>Headquarters:</strong> Remote <br /><strong>URL:</strong> <a href="https://www.toptal.com/">https://www.toptal.com/</a> </p> <h2>About the Role</h2> <p>We're looking for a Senior Data Engineer to design, build, and maintain scalable data pipelines and ML-ready infrastructure on Azure and Databricks. This is a hands-on engineering role: you'll own the full data pipeline lifecycle — ingestion, transformation, orchestration, and deployment — while supporting machine learning workflows with clean, reliable data. If you're comfortable owning infrastructure decisions and writing production-quality Python at scale, this role is built for that.</p> <h2>What You'll Do</h2> <ul> <li> <p>Design, build, and maintain data pipelines using Databricks and Azure-native data services</p> </li> <li> <p>Develop and optimize ETL/ELT processes to support analytics and machine learning workloads</p> </li> <li> <p>Build and maintain CI/CD pipelines for data engineering and ML deployment workflows</p> </li> <li> <p>Write clean, efficient, production-quality Python for data processing and pipeline automation</p> </li> <li> <p>Support machine learning teams with well-structured, high-quality datasets and feature pipelines</p> </li> <li> <p>Design and manage data architecture across Azure services (e.g., Azure Data Factory, Azure Data Lake, Azure Synapse)</p> </li> <li> <p>Monitor pipeline performance, troubleshoot data quality issues, and implement reliability improvements</p> </li> <li> <p>Implement data governance, security, and access control best practices</p> </li> <li> <p>Collaborate with data scientists, analysts, and software engineers to align data infrastructure with business needs</p> </li> <li> <p>Participate in code reviews, architecture discussions, and technical planning</p> </li> </ul> <h2>What You Bring</h2> <ul> <li> <p>Strong hands-on experience with Azure cloud data services</p> </li> <li> <p>Proven experience building and maintaining pipelines on Databricks</p> </li> <li> <p>Solid experience designing and managing CI/CD pipelines for data or ML workflows</p> </li> <li> <p>Strong Python skills for data engineering and pipeline development</p> </li> <li> <p>Working knowledge of machine learning workflows and how data engineering supports them</p> </li> <li> <p>Experience with SQL and relational/distributed data systems</p> </li> <li> <p>Understanding of data pipeline orchestration, monitoring, and reliability practices</p> </li> <li> <p>Strong problem-solving skills and ability to work independently on complex data infrastructure challenges</p> </li> <li> <p>Solid communication skills for collaborating with data science and engineering teams</p> </li> </ul> <h2>Nice to Have</h2> <ul> <li> <p>Experience with MLOps practices and tools (MLflow, Azure ML)</p> </li> <li> <p>Familiarity with Spark internals and performance tuning within Databricks</p> </li> <li> <p>Experience with infrastructure-as-code (Terraform, Bicep, ARM templates)</p> </li> <li> <p>Exposure to real-time/streaming data pipelines (Kafka, Event Hubs, Structured Streaming)</p> </li> <li> <p>Relevant Azure or Databricks certifications</p> </li> </ul> <h2>Why This Role</h2> <ul> <li> <p>Full pipeline ownership: Own data infrastructure end to end, from ingestion through ML-ready delivery</p> </li> <li> <p>Modern data stack: Work with Azure and Databricks, leading platforms in enterprise data engineering</p> </li> <li> <p>Cross-functional impact: Directly enable machine learning and analytics outcomes, not just move data</p> </li> <li> <p>Flexibility: Remote-friendly engagement structure</p> </li> </ul> <h2>How to Apply</h2> <p>Ready to bring your data engineering expertise to Azure and Databricks-powered ML infrastructure? Apply through Toptal here: https://www.toptal.com/talent/apply</p> <p>&nbsp;</p> <p><strong>To apply:</strong> <a href="https://weworkremotely.com/remote-jobs/toptal-senior-data-engineer-azure-databricks-ml-pipelines-remote">https://weworkremotely.com/remote-jobs/toptal-senior-data-engineer-azure-databricks-ml-pipelines-remote</a></p>