How do I pass my logs to ELK?

Dev Ops
1

There aren’t any special steps that are required outside the documentation that Elastic provides. For our hosted clients we use Elastic Cloud which is a hosted service provided by Elastic - Elastic Cloud: Hosted Elasticsearch, Hosted Search | Elastic.

Since ELK is open source, there is also the option to self host. This is obviously a more involved approach but certainly doable. We use the cloud hosting mostly to reduce the overhead of managing our own cluster. Here is a tutorial on the self-hosting approach - Tutorial 1: Installing a self-managed Elastic Stack | Elastic Installation and Upgrade Guide [8.16] | Elastic

Here are the basic steps for setting up Fleet Server to get your logs aggregated. Fleet Server remotely manages the Elastic Agents that are deployed to each node. The basic steps for setting up Fleet Server:

  1. Log into Kibana and go to Fleet Server
  2. From Fleet Server, create an Agent Policy - name it “Kubernetes” or something similar. The policy defines what all you want monitored. As a starting point you can disable everything except “Collect Kubernetes container logs”. You can read through the other policy items and decide if there is anything else you want to include.
  3. Once the Agent Policy is created, go back to the Fleet Server and create an Enrollment Token. This is a security token that the agent will use. Again, name it “Kubernetes” or something similar.
  4. Finally, from Fleet Server, go to Agent and create a new Agent. The agent will need
    1. The Agent Policy you created. Expand the Authentication Settings and make sure your Enrollment Token is selected.
    2. Select “Enroll in Fleet” option.
    3. Install the Elastic Agent on your Host - it will produce a kubernetes manifest for you to use to create the agent and the necessary permissions. This can also be abstracted out to Helm Charts if desired. Note that there are specific Fleet settings in the DaemonSet definition
    4. Once the agent is install on Kubernetes, it should report back to your cluster. At this point the logs from the nodes should be collected in Elastic.

Those should be the basic steps. A little troubleshooting may be required but the primary item is to get the Elastic Agent installed and registered with the Fleet Server.