Fast, accurate and flexible cluster scheduling with Firmament
Event details
| Date | 17.06.2015 |
| Hour | 15:15 |
| Speaker | Malte Schwarzkopf, University of Cambridge (UK) |
| Location | |
| Category | Conferences - Seminars |
Scheduling work on "warehouse-scale" clusters in modern data centres
is a challenging undertaking: thousands of tasks must be placed
rapidly, and may interact in complex ways on the cluster substrate
consisting of thousands of many-core servers. Moreover, existing
schedulers typically tightly couple policy and implementation. An
ideal cluster scheduler would flexibly support used-defined scheduling
policies, ensure high machine utilisation, avoid pathological
interference on shared resources, and find the optimal assignments quickly.
In this talk, I explain how we square this circle in our Firmament
scheduler. Firmament models the scheduling problem as an incremental
minimum-cost, maximum-flow optimisation. This approach considers all
possible assignments concurrently and makes placement decisions that
are optimal for the given scheduling policy. By modelling tasks'
preferences and interactions as costs on a flow network representing
the cluster topology, Firmament simultaneously optimises for multiple objectives.
Via two case-studies, I illustrate how this enables practical, useful
scheduling policies to be expressed concisely. To scale to large
clusters, Firmament solves the optimisation problem incrementally. On
a Google cluster of 12,500 machines, this takes 200ms on average, a
10x improvement over solving the problem from scratch.
Using test-bed experiments on a physical cluster and trace-driven
simulation on a Google cluster, I show that Firmament is a flexible
scheduling platform (supporting many different policies) which makes
accurate, high-quality decisions (optimal with regards to the policy)
at interactive time-scales (sub-second even at Google-scale).
is a challenging undertaking: thousands of tasks must be placed
rapidly, and may interact in complex ways on the cluster substrate
consisting of thousands of many-core servers. Moreover, existing
schedulers typically tightly couple policy and implementation. An
ideal cluster scheduler would flexibly support used-defined scheduling
policies, ensure high machine utilisation, avoid pathological
interference on shared resources, and find the optimal assignments quickly.
In this talk, I explain how we square this circle in our Firmament
scheduler. Firmament models the scheduling problem as an incremental
minimum-cost, maximum-flow optimisation. This approach considers all
possible assignments concurrently and makes placement decisions that
are optimal for the given scheduling policy. By modelling tasks'
preferences and interactions as costs on a flow network representing
the cluster topology, Firmament simultaneously optimises for multiple objectives.
Via two case-studies, I illustrate how this enables practical, useful
scheduling policies to be expressed concisely. To scale to large
clusters, Firmament solves the optimisation problem incrementally. On
a Google cluster of 12,500 machines, this takes 200ms on average, a
10x improvement over solving the problem from scratch.
Using test-bed experiments on a physical cluster and trace-driven
simulation on a Google cluster, I show that Firmament is a flexible
scheduling platform (supporting many different policies) which makes
accurate, high-quality decisions (optimal with regards to the policy)
at interactive time-scales (sub-second even at Google-scale).
Links
Practical information
- General public
- Free
Organizer
- Willy Zwaenepoel
Contact
- Sylvie Thomet