Thanks! Yes, I am aware of Horovod's elastic feature, and Ray. Just so happens I did review them - but didn't include here for "real estate" reasons.

Take a look here, see what you think:

https://3dfy.ai/post/building-a-cloud-based-infrastructure