Hi! Just wanted to say great article.

In the same spirit I have also took a swing at this type of problem by adding batching: https://medium.com/better-programming/improving-your-prediction-api-with-dynamic-batching-50b98e5054f7

Also made a OSS package to help with this: https://github.com/DACUS1995/async-batcher.

Thanks again.