NodePool is the Schema for the NodePools API
Last changed in Karpenter v0.33.8
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
NodePoolSpec is the top level nodepool specification. Nodepools launch nodes in response to pods that are unschedulable. A single nodepool is capable of managing a diverse set of nodes. Node properties are determined from a combination of nodepool and pod scheduling constraints.
Disruption contains the parameters that relate to Karpenter's disruption logic
Budgets is a list of Budgets. If there are multiple active budgets, Karpenter uses the most restrictive value. If left undefined, this will default to one budget with a value to 10%.
Duration determines how long a Budget is active since each Schedule hit. Only minutes and hours are accepted, as cron does not work in seconds. If omitted, the budget is always active. This is required if Schedule is set. This regex has an optional 0s at the end since the duration.String() always adds a 0s at the end.
Nodes dictates the maximum number of NodeClaims owned by this NodePool that can be terminating at once. This is calculated by counting nodes that have a deletion timestamp set, or are actively being deleted by Karpenter. This field is required when specifying a budget. This cannot be of type intstr.IntOrString since kubebuilder doesn't support pattern checking for int nodes for IntOrString nodes. Ref: https://github.com/kubernetes-sigs/controller-tools/blob/55efe4be40394a288216dab63156b0a64fb82929/pkg/crd/markers/validation.go#L379-L388
Reasons is a list of disruption methods that this budget applies to. If Reasons is not set, this budget applies to all methods. Otherwise, this will apply to each reason defined. allowed reasons are Underutilized, Empty, and Drifted.
Schedule specifies when a budget begins being active, following the upstream cronjob syntax. If omitted, the budget is always active. Timezones are not supported. This field is required if Duration is set.
ConsolidateAfter is the duration the controller will wait before attempting to terminate nodes that are underutilized. Refer to ConsolidationPolicy for how underutilization is considered.
ConsolidationPolicy describes which nodes Karpenter can disrupt through its consolidation algorithm. This policy defaults to "WhenEmptyOrUnderutilized" if not specified
Limits define a set of bounds for provisioning capacity.
Template contains the template of possibilities for the provisioning logic to launch a NodeClaim with. NodeClaims launched from this NodePool will often be further constrained than the template specifies.
Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations
Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels
NodeClaimTemplateSpec describes the desired state of the NodeClaim in the Nodepool NodeClaimTemplateSpec is used in the NodePool's NodeClaimTemplate, with the resource requests omitted since users are not able to set resource requests in the NodePool.
ExpireAfter is the duration the controller will wait before terminating a node, measured from when the node is created. This is useful to implement features like eventually consistent node upgrade, memory leak protection, and disruption testing.
NodeClassRef is a reference to an object that defines provider specific configuration
API version of the referent
Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds"
Name of the referent; More info: http://kubernetes.io/docs/user-guide/identifiers#names
Requirements are layered with GetLabels and applied to every node.
The label key that the selector applies to.
This field is ALPHA and can be dropped or replaced at any time MinValues is the minimum number of unique values required to define the flexibility of the specific requirement.
Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch.
StartupTaints are taints that are applied to nodes upon startup which are expected to be removed automatically within a short period of time, typically by a DaemonSet that tolerates the taint. These are commonly used by daemonsets to allow initialization and enforce startup ordering. StartupTaints are ignored for provisioning purposes in that pods are not required to tolerate a StartupTaint in order to have nodes provisioned for them.
Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.
Required. The taint key to be applied to a node.
TimeAdded represents the time at which the taint was added. It is only written for NoExecute taints.
The taint value corresponding to the taint key.
Taints will be applied to the NodeClaim's node.
Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.
Required. The taint key to be applied to a node.
TimeAdded represents the time at which the taint was added. It is only written for NoExecute taints.
The taint value corresponding to the taint key.
TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated. Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation. This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period. When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached. Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod. If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout, that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds. The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks. If left undefined, the controller will wait indefinitely for pods to be drained.
Weight is the priority given to the nodepool during scheduling. A higher numerical weight indicates that this nodepool will be ordered ahead of other nodepools with lower weights. A nodepool with no weight will be treated as if it is a nodepool with a weight of 0.
NodePoolStatus defines the observed state of NodePool
Conditions contains signals for health and readiness
lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
message is a human readable message indicating details about the transition. This may be an empty string.
observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
status of the condition, one of True, False, Unknown.
type of condition in CamelCase or in foo.example.com/CamelCase.
Resources is the list of resources that have been provisioned.
Source: karpenter.sh_nodepools.yaml
Changes to NodePool (karpenter.sh/v1) in each Karpenter version, compared to the version before it.
2 properties have changed the description
Released · View NodePool in v0.33.8
NodePool was first seen on this version of Karpenter
Released · View NodePool in v0.33.6
2 ready-to-use manifests of NodePool to start from.
apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: # Constrain the maximum size of the node pool limits: cpu: 256 memory: 1024Gi disruption: # Allow Karpenter to consolidate nodes when they are empty or underutilized consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 5m budgets: # Allow voluntary disruptions to 5% of the nodes in the pool for any reason - nodes: "5%" template: spec: # Use the EC2NodeClass named "private" as the node template in this pool nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: private # Force node replacement after about a month expireAfter: 720h requirements: # Select from standard instance families (compute, general, memory) - key: karpenter.k8s.aws/instance-category operator: In values: ["c", "m", "r"] # Only choose instances greater than this generation - key: karpenter.k8s.aws/instance-generation operator: Gt values: ["2"] # Only use on-demand instances in this pool - key: karpenter.sh/capacity-type operator: In values: ["on-demand"] - key: kubernetes.io/arch operator: In # Allow AMD64 and ARM64 (e.g., Graviton) proceseors values: ["amd64", "arm64"] # Only use Linux as the OS in this pool - key: kubernetes.io/os operator: In values: ["linux"]
apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: gpu spec: # Constrain the maximum size of the node pool limits: cpu: 256 memory: 1024Gi disruption: # Allow Karpenter to consolidate nodes when they are empty or underutilized consolidationPolicy: WhenEmptyOrUnderutilized consolidateAfter: 5m budgets: # Allow voluntary disruptions to 5% of the nodes in the pool for any reason - nodes: "5%" template: spec: # Use the EC2NodeClass named "private" as the node template in this pool nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: private # Never force replacement of nodes expireAfter: Never # Taint nodes so arbitrary workloads don't wind up provisioning expensive # GPU instances. taints: - effect: NoSchedule key: karpenter.sh/nodepool value: "gpu" requirements: # Only use instances with NVIDIA GPUs - key: karpenter.k8s.aws/instance-gpu-manufacturer operator: In values: ["nvidia"] # Only use on-demand instances in this pool - key: karpenter.sh/capacity-type operator: In values: ["on-demand"] # Only use Linux as the OS in this pool - key: kubernetes.io/os operator: In values: ["linux"]
kubespec.dev