I have some nodes with different amount of gpu cards. I want to be able to assign a specific pod to a certain gpu card, something like a reservation when I am assigning a task to it.
I have already seen, that you can label a node like that:
kubectl label node nodaA project-type=ProjectA
But this is just labeling a node, which a pod will be assigned to it. But I want to be able to be more specific.
Let's say I have a node called NodeA, which has 3 GPU cards from Nvidia. The first card is reserved for a specific task called "CertainOne", the other two cards are just there available for other pods.
When a user submits pods which is not the "CertainOne", the pods should be assigned to these 3 cards, no matter what label. But when another user submits a pod with the label "CertainOne", then the reserved card should always be used (the other running pod which could be using at that moment should be dropped and the pod with the label CertainOne should use it as a priority)
But I wasn't able to find a solution for that. The whole idea is, that I can change the reservation card at anytime, for example to another card, but also to change the range, let's say not only the first one, but also the second one, so that pods with the label "CertainOne" should be assigned to card 1 or card 2 in this second scenario.
Is this even possible? Or is there another similar solution to that?