# Apache Camel Cluster Service Environment Debugging Tips

**URL:** <https://forums.broadleafcommerce.com/t/apache-camel-cluster-service-environment-debugging-tips/68>\
**Category:** Dev Ops\
**Created:** [April 1, 2025, 5:08pm UTC](https://forums.broadleafcommerce.com/t/apache-camel-cluster-service-environment-debugging-tips/68 "2025-04-01T17:08:47Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![broadleaf-devops](https://yyz1.discourse-cdn.com/flex011/user_avatar/forums.broadleafcommerce.com/broadleaf-devops/32/20_2.png) [@broadleaf-devops](https://forums.broadleafcommerce.com/u/broadleaf-devops)\
**Post date:** [April 1, 2025, 5:08pm UTC](https://forums.broadleafcommerce.com/t/apache-camel-cluster-service-environment-debugging-tips/68/1 "2025-04-01T17:08:47Z")

</div>

In certain scenarios, it may be necessary to debug issues related to the Apache Camel Cluster Service. Problems that can manifest from a non-working setup include seeing some “unacked” records in `blc_notification_state` tables, indicating that the [Retry Cluster Service](https://javadocs.ci.broadleafcommerce.com/broadleaf-common-messaging/1.5.9-GA/com/broadleafcommerce/common/messaging/notification/RetryClusterService.html) is not working for some reason.

If you are unaware of how Broadleaf leverage’s Apache Camel’s Cluster Service, you can read more about it here: [Camel Cluster Service - Broadleaf Dev Central](https://developer.broadleafcommerce.com/production/configuration/camel-cluster-service)

The retry logic mainly kicks in for two scenarios:

1. to send notifications for the notification states have been inserted as seed data (or directly in the DB with an “unacked” status), OR
2. if the initial message send we do during a CRUD operation fails for whatever reason (e.g. the broker is down temporarily). Most of the time, with (2), it will succeed and never even hit the retry flow. If there’s something wrong with the retry flow, you’d only know if that initial send failed, which it typically doesn’t.

## Debug Tip 1: Check ENV Properties

When running on Kubernetes, there are specific ENV properties that need to be set to tell the system which Camel Cluster implementation to use, e.g. `file` based for local dev and `kubernetes` or `zookeeper` when deploying to a higher environment (e.g. on Kubernetes in the cloud)

Double check the appropriate ENV properties are set and injected properly on your pods: [Camel Cluster Service - Broadleaf Dev Central](https://developer.broadleafcommerce.com/production/configuration/camel-cluster-service#kubernetesclusterservice_non_local)

**Additional Tip:** Turn on the Broadleaf Environment Report and verify that your ENV properties are being properly picked up as expected. For example, in your config server, add the following property for all your flex packages:

```auto
broadleaf:
  environment:
    report:
      disabled: false

```

## Debug Tip 2: When using the Kubernetes Camel Cluster Service - check `leases`

If you have configured the Kubernetes Camel Cluster Service - then the number one check is to verify that each of your deployed applications is able to create a kubernetes `lease`. You can verify this by running: `kubectl get leases` and should see an output similar to what is presented here: [Camel Cluster Service - Broadleaf Dev Central](https://developer.broadleafcommerce.com/production/configuration/camel-cluster-service#validation)

Important: each type of FLEX PACKAGE should have ownership to various leases. For example, if you deploy the `Balanced` composition, then you should see lease holders for the `auth`, `browse`, `cart`, `processing`, and `supporting` flex packages. If you notice that one or more of the following are missing, then that likely points to some issue with that flex package type unable to interact with the Camel Cluster Service.

## Debug Tip 3: Turn on more granular logging

To see what the Camel Cluster Service implementation is doing, you may find it useful to turn on additional logging. Setting the following logs may be helpful :

```auto
logging:
  level:
    io:
      fabric8: DEBUG
    org:
      apache:
        camel: DEBUG (you may also want to set to TRACE for more visibility, but does get noisy)
    com:
      broadleafcommerce:
        common:
          messaging:
            notification: TRACE

```

**Additional Tip:** From an Flex Package startup log perspective, one other indication that there may be a problem is looking at the logs during startup - the following shows screenshots of normal `INFO` level logging turned on (assuming you have Kubernetes as your backing cluster implementation):

A successful startup should have logs emitted that look like this:

 ![427043696-76678b1f-60fe-4ad7-8032-49231d7a1099](https://canada1.discourse-cdn.com/flex011/uploads/blc/original/1X/b6ce848698b96ebe6f7501166455225deb9d4ea3.jpeg)

A startup for a Flex Package that has a problem communicating with the Kubernetes Camel Cluster Service might look like this:

 ![427044198-37cf6bb3-ff70-436b-817f-2bb4e06242df](https://canada1.discourse-cdn.com/flex011/uploads/blc/original/1X/f43d34da054c07e93be3539e326a8f79877fb5af.jpeg)

## Debug Tip 4: Hook Up a Remote JVM Debugger

If possible, it may be useful to hook up a remove JVM debugger to the pod that is having Camel Cluster Service issues.

To hook up a remote debugger to a pod, the application must have the remote debug port enabled. The default HELM charts for Broadleaf’s flex packages have a property that can be set to enable this. For example, on the `blc-browse` flex package chart, there is a property on the `values.yml` file that you can set like:

```auto
debug:
  enabled: false
  port: 9004

```

Next - you will need to port forward the debug port to your local machine (You can do this via `kubectl` or via a tool like Lens)

 ![427050024-70cc5230-7da8-439c-b5ac-1fb4f873065d-1](https://canada1.discourse-cdn.com/flex011/uploads/blc/original/1X/6c4426c4ddffd5a9b2a72cc8d7e19edd968d642e.jpeg)

Once you have that setup, now you can start a remove JVM debug session in intelliJ. You can set this up using a Debug configuration similar to the one shown here:

 ![427050901-26bb4ef0-c2ce-4517-b263-f172a84b145a](https://canada1.discourse-cdn.com/flex011/uploads/blc/original/1X/c8621515f83c3264e68e7a668dd66bb544325843.jpeg)

## Debug Tip 5: Understanding the Kubernetes Camel Cluster Service Flow

It’s helpful to understand the flow of how the Kubernetes Camel Cluster Service works to set appropriate breakpoints.

The general call stack looks like this:

`DeferServiceStartupListener` → `BaseService` → `AbstractCamelClusterService` → `KubernetesClusterView`

On initialization of `DeferServiceStartupListener`, it has 2 services list: `earlyServices` and `services`.

During execution of this class, the `doStart` method gets called twice with both lists… first the `earlyServices` list (in the method `onCamelContextStarting` ) and then with the services list (in the method `onCamelContextStarted`)

The `KubernetesClusterService` is supposed to be loaded in the `services` list (not the earlyServices). If you notice that the `KubernetesClusterService` class is NOT present in the `services` list, then that indicates a Spring bean loading problem/conflict. You’ll want to check all instantiations of the `CamelClusterService` in your codebase for issues.

A successful breakpoint and step through of this flow would look something like this:

1. `DeferServiceStartupListener` starts up and iterates through the list of `earlyServices` first and starts all them up.  

2. `DeferServiceStartupListener` next iterates through the list of `services` and starts all them up (it should include the `KubernetesClusterService`  

3. The `BaseService#start()` method which is a parent class of `KubernetesClusterView` will be called  

4. The `BaseService#start()` method which is a parent class of `KubernetesClusterService` will be called  

5. The `AbstractCamelClusterService#doStart()` method will be called  

6. The `KubernetesClusterView#doStart()` method will be called and try to get a connection to the running Kubernetes Control Plain API  

## Additional Resources and Context:

- [camel-spring-boot/components-starter/camel-kubernetes-cluster-service-starter at camel-spring-boot-4.7.0 · apache/camel-spring-boot · GitHub](https://github.com/apache/camel-spring-boot/tree/camel-spring-boot-4.7.0/components-starter/camel-kubernetes-cluster-service-starter)
- [camel/components/camel-kubernetes/src/main/java/org/apache/camel/component/kubernetes/cluster/KubernetesClusterView.java at main · apache/camel · GitHub](https://github.com/apache/camel/blob/main/components/camel-kubernetes/src/main/java/org/apache/camel/component/kubernetes/cluster/KubernetesClusterView.java)