Summary
The user is seeking advice on designing an application to run on Kubernetes for scalable parallel processing tasks. They want to ensure that the application can handle multiple queries running in parallel efficiently.
Question
Hi Community.
I have been tasked to build an application that can run on Kubernetes and be as scalable as possible.
Basically the challenge is that the application should receive requests to make certain processing tasks and these tasks are basically a group of queries that run against a database.
Now there are two things, these queries should run in parallel as much as possible and different users might trigger many processes at same time.
Considering the bottleneck is not the database…
What is the best design to ensure that the application scales in Kubernetes by managing new pods etc?
If I am running a parallel loop lets say in python, can Kubernetes break it down into multiple pods?
Thank you for the help while I try to learn these new concepts to app design.
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.