By Ian - Tue Oct 27 2020 - 3 min read
I'm a huge fan of Kubernetes. I'm also a huge fan of ElasticSearch. So I was a bit let down when the process of deploying wasn't very seamless. In this post I'll show you an easy and scalable Kubernetes config that works perfectly.
The first and easiest step is adding
ResourceDefinitions to your cluster. This can be done with an awesome oneliner:
kubectl --kubeconfig KUBECONFIG_LOCATION apply -f https://download.elastic.co/downloads/eck/1.2.1/all-in-one.yaml
This is the config for running the ElasticSearch cluster on your Kube Cluster.
# This sample sets up an Elasticsearch cluster with 2 nodes. apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: production-elasticsearch spec: version: 7.7.1 nodeSets: - name: default config: xpack.security.authc: anonymous: username: anonymous roles: superuser authz_exception: false # most Elasticsearch configuration parameters are possible to set, e.g: node.attr.attr_name: attr_value node.master: true node.data: true node.ingest: true node.store.allow_mmap: false podTemplate: metadata: labels: # additional labels for pods app: APP_NAME tier: elasticsearch spec: containers: - name: elasticsearch # specify resource limits and requests resources: limits: memory: 0.7G cpu: 0.2 requests: memory: 0.7G cpu: 0.2 env: - name: ES_JAVA_OPTS value: "-Xms325m -Xmx325m" count: 2 http: tls: selfSignedCertificate: disabled: true
Two important things to note:
env: - name: ES_JAVA_OPTS value: "-Xms325m -Xmx325m"
These values reference two things:
Xms (Minimum Heap Size) and
Xmx (Maximum Heap Size). The heap size refers to the
JVM Heap Size, and by default ElasticSearch sets it to 1Gb. They should be the same size and when moving to production you need to set it
to no more than 50% or your instance's RAM.
http: tls: selfSignedCertificate: disabled: true
This is set to
false, because my ElasticSearch cluster is running on a closed system (VPN) on kubernetes. Meaning than
it's not reachable from the internet. This makes https useless, so I prefer the simplicity of http.
Save this config as
deploy-elastic.yaml and run it using:
kubectl --kubeconfig KUBECONFIG_LOCATION apply -f deploy-elastic.yaml
After the ElasticSearch cluster finished deploying you can easily connect to it with the following URL:
Do note that this URL is only reachable when you are inside the kubernetes cluster. If you want to publicly expose your ElasticSearch cluster I highly recommend a different setup! One that uses HTTPS and has users setup.
You can reach it using an application that's already running in your cluster, or using a VPN.
This config is not connected to an autoscaling group. If you want to scale your cluster to run more than two instances, you can simple change
to a different integer.
Kubernetes is mostly designed to run stateless containers. So running something like a database isn't really recommended. Some user do use ElasticSearch as a database, which is fine most of the time, especially when you run it on a dedicated VPS. You should truly not do this when running ElasticSearch on Kubernetes. Treat your ElasticSearch cluster as an ephemeral index.
You should always be able to reconstruct the search index when necessary.
For the people in the back: DO NOT USE ELASTICSEARCH AS A DATABASE.
So with that out of the way, I hope you found this useful. I'm aware that there's also a very well documented helm chart available for ElasticSearch. Do you want me to make a post on that? Let me know!