Elasticsearch7在容器中运行的时候是怎么感知到内存limit然后传递给java的xms和xmx参数
来自三线的随记
首先，看一下elasticsearch容器的启动命令，避免被exec 迷惑
[root@gzu1 ~]# docker inspect 1.1.1.1/library/elasticsearch:7.17.5| jq '.[].ContainerConfig | {Cmd,Entrypoint} '
{
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"eswrapper\"]"
],
"Entrypoint": [
"/bin/tini",
"--",
"/usr/local/bin/docker-entrypoint.sh"
]
}
进入到 es 容器中确认，与1号进程是预期一致的
[root@gzu1 ~]# kubectl exec -it -n es-system es-data-0 -- bash root@es-data-0:/usr/share/elasticsearch# ps auxwwww USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 2500 540 ? Ss Mar27 3:07 /bin/tini -- /usr/local/bin/docker-entrypoint.sh eswrapper elastic+ 7 103 4.7 103849888 18890788 ? Sl Mar27 124591:05 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-2689440660253749261 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Des.cgroups.hierarchy.override=/ -Xms16384m -Xmx16384m -XX:MaxDirectMemorySize=8589934592 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch elastic+ 224 0.0 0.0 108392 4636 ? Sl Mar27 0:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller root 50256 0.0 0.0 4244 2324 pts/0 Ss+ 04:05 0:00 bash root 64422 0.3 0.0 4376 2328 pts/1 Ss 05:22 0:00 bash root 64430 0.0 0.0 5900 1492 pts/1 R+ 05:22 0:00 ps auxwwww
可以看到这里实际的运行 elasticsearch 的 java 命令中，居然有Xms 和Xmx，而且是 pod 内存限制的一半
而在当前pod yaml中，并没有指定这些参数:
[root@gzu1 ~]# kubectl get pods -n es-system es-data-0 -o json | jq '.spec.containers[].resources'
{
"limits": {
"cpu": "8",
"memory": "32Gi"
},
"requests": {
"cpu": "4",
"memory": "32Gi"
}
}
[root@gzu1 ~]# kubectl get pods -n es-system es-data-0 -o json | jq '.spec.containers[].env'
[
{
"name": "POD_IP",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "status.podIP"
}
}
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "NODE_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "spec.nodeName"
}
}
},
{
"name": "NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "PROBE_PASSWORD_PATH",
"value": "/mnt/elastic-internal/probe-user/elastic-internal-probe"
},
{
"name": "PROBE_USERNAME",
"value": "elastic-internal-probe"
},
{
"name": "READINESS_PROBE_PROTOCOL",
"value": "http"
},
{
"name": "HEADLESS_SERVICE_NAME",
"value": "dmp-es-data"
},
{
"name": "NSS_SDB_USE_CACHE",
"value": "no"
}
]
[root@gzu1 ~]# kubectl get pods -n es-system es-data-0 -o json | jq '.spec.containers[]'|grep -i env # 确认没有通过configmap等其他形式挂入env
"env": [
[root@gzu1 ~]#
这时候打开 /usr/local/bin/docker-entrypoint.sh 脚本开始检查 该 elasticsearch 的初始化过程，能发现
/usr/local/bin/docker-entrypoint.sh 经过一些简单的初始化以后，还会按需通过读取一些文件，转化为本文暂时不关注的环境变量，最后会调用 /usr/share/elasticsearch/bin/elasticsearch
而 /usr/share/elasticsearch/bin/elasticsearch ,众所周知，他还是一个脚本
在该脚本中，他会执行 source "`dirname "$0"`"/elasticsearch-env ，也就是通过 /usr/share/elasticsearch/bin/elasticsearch-env 做一些环境初始化， 例如定义ES_JAVA_HOME 环境变量，以及决定用内置jdk还是通过env指定的jdk。还会按需通过读取一些文件，转化为本文暂时不关注的环境变量
然后在指定完需要用哪个 java 以后，回到/usr/share/elasticsearch/bin/elasticsearch
可以看到存在一些不起眼的代码
# CONTROLLING STARTUP: # # This script relies on a few environment variables to determine startup # behavior, those variables are: # # ES_PATH_CONF -- Path to config directory # ES_JAVA_OPTS -- External Java Opts on top of the defaults set # # Optionally, exact memory values can be set using the `ES_JAVA_OPTS`. Example # values are "512m", and "10g". # # ES_JAVA_OPTS="-Xms8g -Xmx8g" ./bin/elasticsearch ........ if [ -z "$ES_TMPDIR" ]; then ES_TMPDIR=`"$JAVA" "$XSHARE" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.TempDirectory` fi ....... # The JVM options parser produces the final JVM options to start Elasticsearch. # It does this by incorporating JVM options in the following way: # - first, system JVM options are applied (these are hardcoded options in the # parser) # - second, JVM options are read from jvm.options and jvm.options.d/*.options # - third, JVM options from ES_JAVA_OPTS are applied # - fourth, ergonomic JVM options are applied ES_JAVA_OPTS=`export ES_TMPDIR; "$JAVA" "$XSHARE" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JvmOptionsParser "$ES_PATH_CONF" "$ES_HOME/plugins"`
我们在 elasticsearch 容器中同样执行这些代码看看效果:
root@es-data-0:/usr/share/elasticsearch# /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.tools.launchers.TempDirectory /tmp/elasticsearch-9132815636720135492 root@es-data-0:/usr/share/elasticsearch# export ES_TMPDIR=/tmp/elasticsearch-9132815636720135492 root@es-data-0:/usr/share/elasticsearch# /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.tools.launchers.JvmOptionsParser /usr/share/elasticsearch/config /usr/share/elasticsearch/plugins -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-9132815636720135492 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Xms16384m -Xmx16384m -XX:MaxDirectMemorySize=8589934592 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25
在这里找到了Xms 和 Xmx的定义，也就是说 /usr/share/elasticsearch/bin/elasticsearch 脚本会通过执行 org.elasticsearch.tools.launchers.TempDirectory 产生一个临时文件夹，然后继续执行 org.elasticsearch.tools.launchers.JvmOptionsParser 产生合适的 Xms 及 Xmx 参数
而如果我们在环境变量中预先定义了 ES_JAVA_OPTS ，就会有如下效果(在该脚本注释中也有提到)
root@es-data-0:/usr/share/elasticsearch# export ES_JAVA_OPTS="-Xms8g -Xmx8g" root@es-data-0:/usr/share/elasticsearch# /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.tools.launchers.JvmOptionsParser /usr/share/elasticsearch/config /usr/share/elasticsearch/plugins -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-9132815636720135492 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Xms8g -Xmx8g -XX:MaxDirectMemorySize=4294967296 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25
