I am trying to setup a 3 node cluster on RHEL 8.10 VMs with elasticsearch-8.18.1-linux-x86_64.tar.gz. The ES_JAVA_HOME is configured with jdk-21.0.5
I have followed the steps from Set up HTTPS | Elastic Docs and generated http-ca.p12
My primary master elasticsearch.yml looks like the below:
cluster.name: DevAdvSearch
node.name: DevES1
discovery.seed_hosts: ["xx.xxx.xxx.xx","xx.xxx.xxx.xx","xx.xxx.xxx.xx"]
cluster.initial_master_nodes: ["DevES1"]
node.roles: [master]
path.data: /appln_storage/data
path.logs: /appln_storage/logs
http.port: 9200
network.host: 0.0.0.0
transport.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.type: PKCS12
keystore.path: http-ca.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.type: PKCS12
keystore.path: http-ca.p12
The other 2 data node's elasticsearch.yml looks like below:
cluster.name: DevAdvSearch
node.name: DevES2 (DevES3 for the 3rd node)
discovery.seed_hosts: ["xx.xxx.xxx.xx","xx.xxx.xxx.xx","xx.xxx.xxx.xx"]
cluster.initial_master_nodes: ["DevES1"]
node.roles: [data]
path.data: /appln_storage/data
path.logs: /appln_storage/logs
http.port: 9200
network.host: 0.0.0.0
transport.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.type: PKCS12
keystore.path: http-ca.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.type: PKCS12
keystore.path: http-ca.p12
The password for the below were stored at keystore
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password
Problem Statement:
When I run ./bin/elasticsearch from the master node, was getting couple of warnings and the below from log:
[2025-06-01T14:22:19,662][WARN ][o.e.c.c.Coordinator ] [DevES1] This node is a fully-formed single-node cluster with cluster UUID [jHdfrWBySPqGSmlv58-0Ew], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[xx.xxx.xxx.xx,xx.xxx.xxx.xx,xx.xxx.xxx.xx]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message. See [Bootstrapping a cluster | Elasticsearch Guide [8.18] | Elastic] for more information.
[2025-06-01T14:22:24,876][ERROR][o.e.x.d.l.DeprecationIndexingComponent] [DevES1] Bulk write of deprecation logs encountered some failures: [[Z_TeK5cBg1URBLnUZm8E org.elasticsearch.action.UnavailableShardsException: [.ds-.logs-deprecation.elasticsearch-default-2025.06.01-000001][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.ds-.logs-deprecation.elasticsearch-default-2025.06.01-000001][0]] containing [index {[.logs-deprecation.elasticsearch-default][Z_TeK5cBg1URBLnUZm8E], source[{"@timestamp":"2025-06-01T14:20:49.772Z", "log.level": "WARN", "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.event.category":"indices","event.code":"dot-prefix","message":"Index [.monitoring-es-8-*] name begins with a dot (.), which is deprecated, and will not be allowed in a future Elasticsearch version." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[DevES1][generic][T#8]","log.logger":"org.elasticsearch.deprecation.validation.DotPrefixValidator","elasticsearch.cluster.uuid":"jHdfrWBySPqGSmlv58-0Ew","elasticsearch.node.id":"7NkAJJsCT12KZCGud0CXFg","elasticsearch.node.name":"DevES1","elasticsearch.cluster.name":"DevAdvSearch"}
]}]]]]
[2025-06-01T14:22:28,948][ERROR][o.e.x.s.InitialNodeSecurityAutoConfiguration] [DevES1] Failed to generate credentials for the elastic built-in superuserorg.elasticsearch.action.UnavailableShardsException: [.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][reserved-user-elastic], source[{"password":"xxxxxxxxxxxxxxxxxxx","enabled":true,"type":"reserved-user"}]}] and a refresh]
See logs for more details.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Elasticsearch security features have been automatically configured!
Authentication is enabled and cluster connections are encrypted.
Unable to auto-generate the password for the elastic built-in superuser.
HTTP CA certificate SHA-256 fingerprint:
Unable to generate an enrollment token for Kibana instances, try invoking bin/elasticsearch-create-enrollment-token -s kibana.
An enrollment token to enroll new nodes wasn't generated. To add nodes and enroll them into this cluster:
• On this node:
⁃ Create an enrollment token with bin/elasticsearch-create-enrollment-token -s node.
⁃ Restart Elasticsearch.
• On other nodes:
⁃ Start Elasticsearch with bin/elasticsearch --enrollment-token <token>, using the enrollment token that you generated.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[2025-06-01T14:22:29,936][ERROR][o.e.x.i.h.ILMHistoryStore] [DevES1] failures: [{afTeK5cBg1URBLnUeW_L=org.elasticsearch.action.UnavailableShardsException: [.ds-ilm-history-7-2025.06.01-000001][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.ds-ilm-history-7-2025.06.01-000001][0]] containing [3] requests], avTeK5cBg1URBLnUeW_L=org.elasticsearch.action.UnavailableShardsException: [.ds-ilm-history-7-2025.06.01-000001][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.ds-ilm-history-7-2025.06.01-000001][0]] containing [3] requests], aPTeK5cBg1URBLnUeW_K=org.elasticsearch.action.UnavailableShardsException: [.ds-ilm-history-7-2025.06.01-000001][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.ds-ilm-history-7-2025.06.01-000001][0]] containing [3] requests]}]
[2025-06-01T14:22:34,429][WARN ][o.e.h.AbstractHttpServerTransport] [DevES1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/xx.xxx.xxx.xx:9200, remoteAddress=/xx.xx.xxx.xx:55135}io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: illegal_parameter
at io.netty.codec@4.1.118.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500)
at io.netty.codec@4.1.118.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
See logs for more details.
[2025-06-01T14:22:34,484][WARN ][o.e.h.AbstractHttpServerTransport] [DevES1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/xx.xxx.xxx.xx:9200, remoteAddress=/xx.xx.xxx.xx:55136}io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: illegal_parameter
at io.netty.codec@4.1.118.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500)
at io.netty.codec@4.1.118.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
See logs for more details.
[2025-06-01T14:40:19,710][WARN ][o.e.c.c.Coordinator ] [DevES1] This node is a fully-formed single-node cluster with cluster UUID [jHdfrWBySPqGSmlv58-0Ew], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[xx.xxx.xxx.xx,xx.xxx.xxx.xx,xx.xxx.xxx.xx]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message. See [Bootstrapping a cluster | Elasticsearch Guide [8.18] | Elastic] for more information.
When I tried generating the enrollment token from the other window of same VM which I am running, getting the below error
$ ./bin/elasticsearch-create-enrollment-token -s node
warning: ignoring JAVA_HOME=/usr/java/jdk17; using ES_JAVA_HOME
14:00:51.025 [main] WARN org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [xx.xxx.xxx.xx]; the server provided a certificate with subject name [CN=Elastic Certificate Tool Autogenerated CA], fingerprint [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx], keyUsage [keyCertSign, cRLSign] and no extendedKeyUsage; the certificate is valid between [2025-06-01T12:37:58Z] and [2028-05-31T12:37:58Z] (current time is [2025-06-01T14:00:51.022816617Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate does not have any subject alternative names; the certificate is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: Composite-Trust{JDK-trusted-certs,StoreTrustConfig{path=http-ca.p12, password=, type=PKCS12, algorithm=PKIX}})])
java.security.cert.CertificateException: No subject alternative names present
at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:138) ~[?:?]
at sun.security.util.HostnameChecker.match(HostnameChecker.java:101) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:466) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:432) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:81) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1302) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:447) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:206) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:586) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141) ~[?:?]
at org.elasticsearch.xpack.core.common.socket.SocketAccess.lambda$doPrivileged$0(SocketAccess.java:42) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:41) ~[?:?]
at org.elasticsearch.xpack.core.security.CommandLineHttpClient.execute(CommandLineHttpClient.java:178) ~[?:?]
at org.elasticsearch.xpack.core.security.CommandLineHttpClient.execute(CommandLineHttpClient.java:112) ~[?:?]
at org.elasticsearch.xpack.security.tool.BaseRunAsSuperuserCommand.checkClusterHealthWithRetries(BaseRunAsSuperuserCommand.java:214) ~[?:?]
at org.elasticsearch.xpack.security.tool.BaseRunAsSuperuserCommand.execute(BaseRunAsSuperuserCommand.java:127) ~[?:?]
at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55) ~[elasticsearch-8.18.1.jar:8.18.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:101) ~[elasticsearch-cli-8.18.1.jar:8.18.1]
at org.elasticsearch.cli.Command.main(Command.java:54) ~[elasticsearch-cli-8.18.1.jar:8.18.1]
at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:65) ~[cli-launcher-8.18.1.jar:8.18.1]
ERROR: Failed to determine the health of the cluster. , with exit code 69
Would like to understand what is missing and how to get this fixed.
Appreciate your help!!