Senior Data/ML Engineer — Databricks Forecasting Platform | Remote

Anywhere in the World·Posted 6d ago

python

<img src="https://we-work-remotely.imgix.net/logos/0171/6036/logo.gif?ixlib=rails-4.0.0&w=50&h=50&dpr=2&fit=fill&auto=compress" /> Headquarters: Remote URL: <a href="https://www.toptal.com/">https://www.toptal.com/</a> <h2>About the Role</h2> We're looking for a Senior Data/ML Engineer to refactor, operationalize, and improve an existing time series forecasting platform that's already live and driving real business value. This is not a greenfield build — the focus is modernizing a Databricks-based forecasting system that's been maintained primarily by a single developer for years. You'll reduce technical debt, strengthen testing and observability, and raise the engineering bar on a system the business already depends on. If you'd rather bring discipline and maturity to an existing production system than start from a blank slate, this is built for that. <h2>What You'll Do</h2> <ul> <li> Review the current forecasting platform architecture and identify areas for improvement </li> <li> Refactor existing Databricks, Python, and PySpark implementations </li> <li> Move business logic out of Databricks notebooks and into reusable Python modules or packages </li> <li> Improve separation of concerns between orchestration and core business logic </li> <li> Establish stronger engineering standards and help define what "good" looks like for the platform </li> <li> Implement or improve automated testing practices and validation mechanisms for forecasting workflows </li> <li> Build or improve monitoring and observability, increasing visibility into how predictions are generated </li> <li> Help monitor model behavior and operational health over time </li> <li> Improve reliability of scheduled training workflows, reducing manual intervention on failure </li> <li> Improve failure handling, retries, and overall workflow resilience </li> <li> Maintain and extend existing forecasting capabilities as needed </li> </ul> <h2>What You Bring</h2> <ul> <li> Strong professional experience with Databricks, including workspaces, notebooks, scheduled workflows, and CI/CD processes </li> <li> Strong Python engineering experience, including designing reusable modules or packages </li> <li> Strong PySpark experience with production data pipelines or distributed data processing </li> <li> Experience refactoring production code and improving maintainability </li> <li> Familiarity with time series forecasting concepts and workflows </li> <li> Ability to understand and work effectively within an existing, unfamiliar codebase </li> <li> Experience improving software quality, testing strategy, and engineering standards </li> <li> Experience implementing automated testing practices </li> <li> Experience improving monitoring, observability, or operational visibility for production systems </li> <li> Strong judgment around technical debt, refactoring priorities, and maintainable architecture </li> <li> Ability to work with existing systems rather than only building from scratch </li> </ul> <h2>Why This Role</h2> <ul> <li> Real production impact: Improve a system the business already relies on, not a proof-of-concept </li> <li> Engineering maturity focus: Bring testing, observability, and maintainability to a platform that's outgrown its current state </li> <li> Meaningful ownership: Help define engineering standards for the forecasting platform going forward </li> <li> Flexible location: Preference for Toronto or St. Louis, but open to remote consultants globally with North American working-hours overlap </li> </ul> <h2>How to Apply</h2> Ready to bring engineering rigor to a production forecasting platform? Apply through Toptal here: https://www.toptal.com/talent/apply To apply: <a href="https://weworkremotely.com/remote-jobs/toptal-senior-data-ml-engineer-databricks-forecasting-platform-remote">https://weworkremotely.com/remote-jobs/toptal-senior-data-ml-engineer-databricks-forecasting-platform-remote</a>

Apply for this role