Recently, a support ticket was raised where a large microservices project was having issues with Kubernetes Camel Cluster Service. We had experienced this issue before and provided the following details.
If you have a rather large topology (e.g. deploying multiple replicas of a full granular deployment) and you are seeing substantial CPU usage on the Kubernetes API Master Nodes/Control Plane due to repeated API calls, additional configuration is likely warranted. When using the Apache Camel Kubernetes Cluster Service, the default settings can cause high CPU utilization on the Kubernetes/OpenShift master nodes. This is due to the underlying Fabric8 client library making frequent LIST API calls.
To mitigate this, you can tune the Apache Camel Kubernetes configuration options. We recommend adjusting the following properties:
camel.component.kubernetes-cluster-service.renew-deadline-millis: This is the duration that non-leader candidates will wait to force acquire leadership. This is measured in milliseconds.camel.component.kubernetes-cluster-service.jitter-factor: This is a jitter factor to apply to the renew deadline.camel.component.kubernetes-cluster-service.lease-duration-millis: This is the duration that leaders will hold the lease. This is measured in milliseconds.camel.component.kubernetes-cluster-service.retry-period-millis: This is the time between two subsequent attempts to check and acquire the leadership. It is randomized using the jitter factor. This is measured in milliseconds.
Increasing the values for these properties will reduce the frequency of API calls to the Kubernetes master.
For an example, these are settings that have helped when a BLC microservices cluster was under load:
CAMEL_CLUSTER_KUBERNETES_LEASE_DURATION_MILLIS = 60000CAMEL_CLUSTER_KUBERNETES_RENEW_DEADLINE_MILLIS = 40000CAMEL_CLUSTER_KUBERNETES_RETRY_PERIOD_MILLIS = 10000CAMEL_CLUSTER_KUBERNETES_JITTER_FACTOR = 2
Our documentation has been updated with the above information here: Camel Cluster Service - Broadleaf Dev Central