三线的随记 - 用户贡献 [zh-cn]

容器化freessl-acme.sh更新nginx实现https证书自动续期

2026-04-09T09:19:51Z

Admin：

'''⚠️ 注意：freessl目前acme地址已经更改为litessl.com。下述内容并没有跟进此更新，仅供参考'''

=== 赘述 ===
由于有一定的OS环境洁癖，所以不打算直接在Linux OS环境中直接安装acme.sh，采用容器化的方式安装acme.sh

acme.sh官方其实有做好的docker images,但是由于太闲了所以又重新写了一份(班门弄斧)

=== 正题1: 制作能自签域名证书的镜像 ===
先构建一个带有 acme.sh 基础镜像
cat <<\EOF > acme.sh.Dockerfile
FROM alpine:3.15.3
RUN apk --no-cache add curl wget openssl && \
rm -rf /var/cache/apk && \
curl -s <nowiki>https://get.acme.sh</nowiki> | sh -s email=my@example.com
WORKDIR /root/.acme.sh
ENTRYPOINT ["/usr/sbin/crond", "-f", "-S","/dev/stdout"]
EOF

docker build -f acme.sh.Dockerfile -t san3xian/freessl_acme.sh:v2 .
然后到<nowiki/>https://freessl.cn/acme-deploy 注册登陆

并在该站点的 [https://freessl.cn/acme-deploy acme页面中]添加域名，完成域名的DCV预检测(即配置CNAME，域名解析状态可以在这里查询: https://myssl.com/dns_check.html)

获取token (即ACME 地址中 ****/directory/ 后的字符串)

接着根据实际需求构建域名专属acme镜像用于签发证书，安装证书以及触发容器化或者节点直接部署运行的nginx reload(后面按需运行的时候修改该容器pid mode和reloadcmd即可)

这里 <code>--issue</code> 命令中由于没有带--days参数，所以默认签的是60天的证书

将下述内容按需修改存于Dockerfile中
FROM san3xian/freessl_acme.sh:v2

ARG acme_token="your_token" \
acme_domain="your_domain"
RUN /root/.acme.sh/acme.sh --issue -d ${acme_domain} --dns dns_dp \
--server <nowiki>https://acme.freessl.cn/v2/DV90/directory/${acme_token}</nowiki> && \
mkdir -pv /certs && \
/root/.acme.sh/acme.sh --install-cert -d ${acme_domain} \
--key-file /certs/${acme_domain}_key.pem \
--fullchain-file /certs/${acme_domain}_cert.pem \
--reloadcmd "/bin/kill -s HUP 1"
使用docker build构建镜像
docker build -t san3xian/freessl_acme.sh:yourdomain .
如果上述步骤都没错的话，在构建这个带有域名证书信息的镜像的时候，就能看到申请下来的证书会被打印输出到控制台

并且根据回显内容可知，申请下来的证书保存在了 <code>/root/.acme.sh/${acme_domain}/</code> 路径下，而且证书会被拷贝一份到 <code>/certs</code> 路径下(此路径按实际需求在Dockerfile中修改即可)

在freessl.cn的acme页面中也能看到相应的客户端ID信息

证书的相关配置信息(如install-path和ReloadCmd)在<code>/root/.acme.sh/${acme_domain}/${acme_domain}.conf</code> 文件中

到这里acme.sh自动签证书的步骤就已经基本做完了

acme.sh配置参考: https://blog.freessl.cn/acme-quick-start/

=== 正题2: 适配实际的tls证书使用环境 ===
如前所述，本文是为了用容器化的freessl acme.sh向容器化的nginx进行证书自签，所以下面还会有docker-compose的内容，当然实际操作中用docker run 直接运行容器也可以

前提展开: 由于我这里nginx容器是使用常规镜像启动的，所以我的nginx master的pid在容器的PID namespace就会是1，遂在上文的reloadcmd 中写死了pid

然后我的docker-compose.yml是在server文件夹下的，所以docekr-compose拉起的容器默认会带 <code>server_</code> 前缀，即如docker-compose help中所述:
docker-compose --help
-p, --project-name NAME Specify an alternate project name
(default: directory name)
docker-compose.yml内容，部分内容按实际需求修改
version: "2"

services:
nginx:
restart: always
image: nginx:1.21-alpine
volumes:
- "/var/lib/nginx/conf.d:/etc/nginx/conf.d"
- "/var/lib/nginx/nginx.conf:/etc/nginx/nginx.conf"
- "/var/lib/nginx/certs:/etc/nginx/certs"
- "/var/lib/nginx/sites-enabled:/etc/nginx/sites-enabled"
ports:
- "80:80" # http port for private network
- "443:443" # https port for private network
freessl:
restart: always
image: san3xian/freessl_acme.sh:yourdomain #修改为你在上文实际打出来的容器镜像
volumes:
- "/var/lib/nginx/certs:/certs"
tty: true # alpine下不分配伪tty的话BusyBox crond会有的日志不输出到stdout
pid: container:server_nginx_1 # 与nginx共享同一个PID namespace, 这里暂时没有找到在docker-compose文件中关联其它container的写法，所以只能完全写死容器名了
可见freessl 和 nginx 容器都挂载了同一个/var/lib/nginx/certs目录，这样nginx就能读到由freessl acme.sh自签下发的tls证书

接着docker-compose up -d 启动了相应的容器后

由于在前文签证书的时候，freessl acme.sh容器中/certs路径还没挂载到节点上，所以此时还需要重新拷贝/签一份证书到certs路径下
docker exec server_freessl_1 "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" --force
如果上述步骤都没错的话，此时nginx也会被触发reload，可以docker logs server_nginx_1 看相关信息，按照实际修改nginx的配置以让tls证书生效即可

由于freessl acme.sh容器中还有crond的守护进程运行着，所以acme.sh也会被定时触发，可以通过docker logs server_freessl_1看相关日志

crontab -l 也可以看到相关任务定时触发配置信息，建议按需将acme.sh触发命令的重定向( <code>> /dev/null</code> )去除以方便查看acme.sh的cron日志
[root@localhost server]# docker exec server_freessl_1 crontab -l
# do daily/weekly/monthly maintenance
# min hour day month weekday command
*/15 * * * * run-parts /etc/periodic/15min
0 * * * * run-parts /etc/periodic/hourly
0 2 * * * run-parts /etc/periodic/daily
0 3 * * 6 run-parts /etc/periodic/weekly
0 5 1 * * run-parts /etc/periodic/monthly

1 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null

如果需要添加多个域名的话，可以直接在freessl容器中运行命令
/root/.acme.sh/acme.sh --issue -d ${acme_domain} --dns dns_dp --server <nowiki>https://acme.freessl.cn/v2/DV90/directory/${acme_token}</nowiki>
和
/root/.acme.sh/acme.sh --install-cert -d ${acme_domain} --key-file /certs/${acme_domain}_key.pem --fullchain-file /certs/${acme_domain}_cert.pem --reloadcmd "/bin/kill -s HUP 1"
即可，这样下一次acme.sh的cron任务触发的时候，多个域名都会被执行证书更新检查
[[分类:Linux]]
[[分类:Docker]]
[[分类:Docker-compose]]
{{DEFAULTSORT:docker容器化freessl_acme.sh配合nginx实现https证书自动续期}}

容器化freessl-acme.sh更新nginx实现https证书自动续期

2026-04-09T08:32:22Z

Admin：

注意：freessl目前acme地址已经更改为litessl.com。下述内容并没有跟进此更新，仅供参考

=== 赘述 ===
由于有一定的OS环境洁癖，所以不打算直接在Linux OS环境中直接安装acme.sh，采用容器化的方式安装acme.sh

acme.sh官方其实有做好的docker images,但是由于太闲了所以又重新写了一份(班门弄斧)

=== 正题1: 制作能自签域名证书的镜像 ===
先构建一个带有 acme.sh 基础镜像
cat <<\EOF > acme.sh.Dockerfile
FROM alpine:3.15.3
RUN apk --no-cache add curl wget openssl && \
rm -rf /var/cache/apk && \
curl -s <nowiki>https://get.acme.sh</nowiki> | sh -s email=my@example.com
WORKDIR /root/.acme.sh
ENTRYPOINT ["/usr/sbin/crond", "-f", "-S","/dev/stdout"]
EOF

docker build -f acme.sh.Dockerfile -t san3xian/freessl_acme.sh:v2 .
然后到<nowiki/>https://freessl.cn/acme-deploy 注册登陆

并在该站点的 [https://freessl.cn/acme-deploy acme页面中]添加域名，完成域名的DCV预检测(即配置CNAME，域名解析状态可以在这里查询: https://myssl.com/dns_check.html)

获取token (即ACME 地址中 ****/directory/ 后的字符串)

接着根据实际需求构建域名专属acme镜像用于签发证书，安装证书以及触发容器化或者节点直接部署运行的nginx reload(后面按需运行的时候修改该容器pid mode和reloadcmd即可)

这里 <code>--issue</code> 命令中由于没有带--days参数，所以默认签的是60天的证书

将下述内容按需修改存于Dockerfile中
FROM san3xian/freessl_acme.sh:v2

ARG acme_token="your_token" \
acme_domain="your_domain"
RUN /root/.acme.sh/acme.sh --issue -d ${acme_domain} --dns dns_dp \
--server <nowiki>https://acme.freessl.cn/v2/DV90/directory/${acme_token}</nowiki> && \
mkdir -pv /certs && \
/root/.acme.sh/acme.sh --install-cert -d ${acme_domain} \
--key-file /certs/${acme_domain}_key.pem \
--fullchain-file /certs/${acme_domain}_cert.pem \
--reloadcmd "/bin/kill -s HUP 1"
使用docker build构建镜像
docker build -t san3xian/freessl_acme.sh:yourdomain .
如果上述步骤都没错的话，在构建这个带有域名证书信息的镜像的时候，就能看到申请下来的证书会被打印输出到控制台

并且根据回显内容可知，申请下来的证书保存在了 <code>/root/.acme.sh/${acme_domain}/</code> 路径下，而且证书会被拷贝一份到 <code>/certs</code> 路径下(此路径按实际需求在Dockerfile中修改即可)

在freessl.cn的acme页面中也能看到相应的客户端ID信息

证书的相关配置信息(如install-path和ReloadCmd)在<code>/root/.acme.sh/${acme_domain}/${acme_domain}.conf</code> 文件中

到这里acme.sh自动签证书的步骤就已经基本做完了

acme.sh配置参考: https://blog.freessl.cn/acme-quick-start/

=== 正题2: 适配实际的tls证书使用环境 ===
如前所述，本文是为了用容器化的freessl acme.sh向容器化的nginx进行证书自签，所以下面还会有docker-compose的内容，当然实际操作中用docker run 直接运行容器也可以

前提展开: 由于我这里nginx容器是使用常规镜像启动的，所以我的nginx master的pid在容器的PID namespace就会是1，遂在上文的reloadcmd 中写死了pid

然后我的docker-compose.yml是在server文件夹下的，所以docekr-compose拉起的容器默认会带 <code>server_</code> 前缀，即如docker-compose help中所述:
docker-compose --help
-p, --project-name NAME Specify an alternate project name
(default: directory name)
docker-compose.yml内容，部分内容按实际需求修改
version: "2"

services:
nginx:
restart: always
image: nginx:1.21-alpine
volumes:
- "/var/lib/nginx/conf.d:/etc/nginx/conf.d"
- "/var/lib/nginx/nginx.conf:/etc/nginx/nginx.conf"
- "/var/lib/nginx/certs:/etc/nginx/certs"
- "/var/lib/nginx/sites-enabled:/etc/nginx/sites-enabled"
ports:
- "80:80" # http port for private network
- "443:443" # https port for private network
freessl:
restart: always
image: san3xian/freessl_acme.sh:yourdomain #修改为你在上文实际打出来的容器镜像
volumes:
- "/var/lib/nginx/certs:/certs"
tty: true # alpine下不分配伪tty的话BusyBox crond会有的日志不输出到stdout
pid: container:server_nginx_1 # 与nginx共享同一个PID namespace, 这里暂时没有找到在docker-compose文件中关联其它container的写法，所以只能完全写死容器名了
可见freessl 和 nginx 容器都挂载了同一个/var/lib/nginx/certs目录，这样nginx就能读到由freessl acme.sh自签下发的tls证书

接着docker-compose up -d 启动了相应的容器后

由于在前文签证书的时候，freessl acme.sh容器中/certs路径还没挂载到节点上，所以此时还需要重新拷贝/签一份证书到certs路径下
docker exec server_freessl_1 "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" --force
如果上述步骤都没错的话，此时nginx也会被触发reload，可以docker logs server_nginx_1 看相关信息，按照实际修改nginx的配置以让tls证书生效即可

由于freessl acme.sh容器中还有crond的守护进程运行着，所以acme.sh也会被定时触发，可以通过docker logs server_freessl_1看相关日志

crontab -l 也可以看到相关任务定时触发配置信息，建议按需将acme.sh触发命令的重定向( <code>> /dev/null</code> )去除以方便查看acme.sh的cron日志
[root@localhost server]# docker exec server_freessl_1 crontab -l
# do daily/weekly/monthly maintenance
# min hour day month weekday command
*/15 * * * * run-parts /etc/periodic/15min
0 * * * * run-parts /etc/periodic/hourly
0 2 * * * run-parts /etc/periodic/daily
0 3 * * 6 run-parts /etc/periodic/weekly
0 5 1 * * run-parts /etc/periodic/monthly

1 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null

如果需要添加多个域名的话，可以直接在freessl容器中运行命令
/root/.acme.sh/acme.sh --issue -d ${acme_domain} --dns dns_dp --server <nowiki>https://acme.freessl.cn/v2/DV90/directory/${acme_token}</nowiki>
和
/root/.acme.sh/acme.sh --install-cert -d ${acme_domain} --key-file /certs/${acme_domain}_key.pem --fullchain-file /certs/${acme_domain}_cert.pem --reloadcmd "/bin/kill -s HUP 1"
即可，这样下一次acme.sh的cron任务触发的时候，多个域名都会被执行证书更新检查
[[分类:Linux]]
[[分类:Docker]]
[[分类:Docker-compose]]
{{DEFAULTSORT:docker容器化freessl_acme.sh配合nginx实现https证书自动续期}}

VMware不关机扩容linux磁盘

2026-04-09T07:36:32Z

Admin：

__FORCETOC__

=== 在虚拟化平台对磁盘扩容后 fdisk -l 识别不到分区 (partition丢失,GPT分区表不建议使用fdisk) ===
注意在部分OS下，可能在扩容前 Disk 的相关 partition 信息在fdisk中是可以正常识别的，但是在 vsphere / esxi 对磁盘进行扩容以后

就发现 partition 丢了【fdisk -l /dev/sda 看不到某些分区】(特别是centos7.9 + fdisk -l + GPT)

这时候可以直接 <code>parted /dev/sda</code>

进入以后应该就会提示
Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that
another operating system believes the disk is smaller. Fix, by moving the backup to the end (and
removing the old backup)?
直接进行Fix并扩容即可

也可以找到原本的扇区记录，然后进入fdisk ，修改 partition table 为 GPT，根据相关扇区记录，把分区重新建立回来即可

=== lvm类型的分区扩容 ===

==== 利用fdisk - 传统步骤扩容分区(旧版本fdisk对GPT兼容不好，GPT分区表建议使用parted) ====
磁盘名sda, vg名 cl, lv名 root 等自行根据实际情况修改!
# vsphere(虚拟化平台)层面完成磁盘扩容
# 进入系统 shell 中，执行命令 lsblk 查看对应的磁盘是否有感知到新的容量。有时候会碰到 os 无法感知到 disk 容量变化的情况出现，这时候需要手动rescan一下<code>echo 1 > /sys/class/block/sda/device/rescan</code>
# <code>fdisk -l /dev/sda</code> 确认相关分区及扇区信息
# 使用<code>fdisk /dev/sda</code> 进入交互式操作删除重建需要扩容的分区(增大分区容量,保留原本的LVM2_member signature, 起始扇区与原本一致)
# <code>partprobe /dev/sda </code>
# <code>pvresize /dev/sda3 </code>
# <code>lvextend /dev/cl/root /dev/sda3 </code>
# <code>xfs_growfs /dev/mapper/cl-root </code>
# 完成

==== 利用parted工具一行命令直接扩容第三分区到100% ====
磁盘名sda, vg名 cl, lv名 root 等自行根据实际情况修改!
parted /dev/sda --script -- resizepart 3 100%

pvresize /dev/sda3 && lvextend -l +100%FREE /dev/ol/root && xfs_growfs /

=== 非lvm类型虚拟磁盘扩容(磁盘直接作为了挂载点) ===
磁盘名sda, vg名 cl, lv名 root 等自行根据实际情况修改!
# vm层面完成磁盘扩容
# 同样手动rescan一下让os感知磁盘容量变化<code>echo 1 > /sys/class/block/sda/device/rescan</code>
#根据文件系统类型执行fs扩容，如<code>xfs_growfs</code> / <code>resize2fs</code>
#例如⬇️
[root@192-168-157-39 ~]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 150G 0 disk /var/lib/containers

[root@192-168-157-39 ~]# echo 1 > /sys/class/block/sdb/device/rescan

[root@192-168-157-39 ~]# lsblk /dev/sdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 155G 0 disk /var/lib/containers

[root@192-168-157-39 ~]# cat /etc/fstab |grep cont
/dev/sdb /var/lib/containers xfs defaults,pquota 0 0

[root@192-168-157-39 ~]# xfs_growfs /dev/sdb
meta-data=/dev/sdb isize=512 agcount=4, agsize=9830400 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=39321600, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=19200, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 39321600 to 40632320

[root@192-168-157-39 ~]# df -h /var/lib/containers
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 155G 19G 137G 12% /var/lib/containers

[root@192-168-157-39 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
├─sda1 8:1 0 1019M 0 part /boot
└─sda2 8:2 0 49G 0 part
└─centos-root 253:0 0 49G 0 lvm /
sdb 8:16 0 155G 0 disk /var/lib/containers
sr0 11:0 1 9.5G 0 rom
 
额外延伸：[https://www.2daygeek.com/scan-detect-luns-scsi-disks-on-redhat-centos-oracle-linux/ How to scan\detect new LUN’s & SCSI disks in Linux?]

通过命令 <code>lsscsi -g</code> 能看 vmware 磁盘和 OS disk 的对照关系
[[category: Linux]]

Fish shell或bash shell下的ssh-agent现有会话探测脚本

2026-02-27T17:23:37Z

Admin：

一个随便糊的用于发现设备上现有的ssh-agent会话脚本，方便在不同的terminal tab中复用一个ssh-agent

适配:

=== macOS + fish shell (macos v26.3+) ===
#!/usr/bin/env fish

#######################################################
# Author: SanXian
# To find the exists ssh-agent connection or create
# a new ssh-agent connection for macOS with fish shell
# version: 1.5
#######################################################

#set base_dir "/var/folders/4w/hx1kgdwd2ll9chvbzv6y70r80000gn/T"
# Linux
# /tmp/ssh-*/agent.*

# macOS-new version:
# set base_dir "$TMPDIR/ssh-*/agent.*"
set base_dir "~/.ssh/agent/*agent.*"
for i in (sh -c "ls -1 --color=never $base_dir 2>/dev/null")
set agent_file (echo -n $i | grep --color=never -Eo "agent.*")
echo "found: $agent_file"
#set PID (echo -n $agent_file | awk -F . '{print $2 + 1}')
set PID (lsof -n $i | awk '/ssh-agent/{print $2}')
echo PID: $PID
export SSH_AGENT_PID=$PID
export SSH_AUTH_SOCK=$i
env | grep -i ssh_
exit
end

# ssh agent sock not found
echo "ssh agent sock file not found, creating"
eval (ssh-agent -c -t 4h)
#ssh-add -t 0
ssh-add -t 360d

=== Linux bash: ===
#!/usr/bin/env bash

#######################################################
# Author: SanXian
# To find the exists ssh-agent connection or create
# a new ssh-agent connection for macOS with fish shell
# version: 1.5
#######################################################

#set base_dir "/var/folders/4w/hx1kgdwd2ll9chvbzv6y70r80000gn/T"
# Linux
base_dir=/tmp/ssh-*/agent.*

# macOS
#set base_dir $TMPDIR/ssh-*/agent.*
agent_paths=`sh -c "ls -1 --color=never $base_dir 2>/dev/null"`
for i in $agent_paths
do
agent_file=`echo -n $i | grep --color=never -Eo "agent.*"`
echo "found: $agent_file"
PID=`echo -n $agent_file | awk -F . '{print $2 + 1}'`
echo PID: $PID
export SSH_AGENT_PID=$PID
export SSH_AUTH_SOCK=$i
env | grep -i ssh_
done

if <nowiki>[[ -z $agent_paths ]]</nowiki>
then
# ssh agent sock not found
echo "ssh agent sock file not found, creating"
eval `ssh-agent -t 4h`
#ssh-add -t 0
ssh-add -t 360d
fi

=== 食用方法 ===
执行:
source ssh-search
或
. ssh-search

扩展阅读:

[[SSH-add脚本化载入带密码(passphrase)的私钥]]
[[分类:Linux]]
[[分类:MacOS]]
[[分类:SSH]]

Etcd命令随记

2026-02-02T09:49:39Z

Admin：

=== 切换 ETCDCTL API version ===

==== with docker exec ====
docker exec -e "ETCDCTL_API=2" etcd_container_name_or_id etcdctl --help

==== env ====
export ETCDCTL_API=2

=== 排障相关命令 ===

* 如果etcd是基于k8s manifest启动的,且该manifest初始化了ETCDCTL_ENDPOINTS变量，在指定endpoints只能通过环境变量指定，或者在bash环境下unset "ETCDCTL_ENDPOINTS" 取消环境变量定义然后再通过参数传入,不然会提示配置冲突:

2021-10-19 03:25:28.520086 C | pkg/flags: conflicting environment variable "ETCDCTL_ENDPOINTS" is shadowed by corresponding command-line flag (either unset environment variable or disable flag)

==== etcdctl v3 related ====
etcdctl v3可用的环境变量可见文档 https://github.com/etcd-io/etcd/blob/main/etcdctl/README.md

其中常见的包括
export ETCDCTL_ENDPOINTS=<nowiki>https://127.0.0.1:2379,https://10.255.251.102:2379,https://10.255.251.103:2379</nowiki>
export ETCDCTL_CACERT=/etc/kubernetes/ssl/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/ssl/etcd/peer.crt
export ETCDCTL_KEY=/etc/kubernetes/ssl/etcd/peer.key

===== k8s 的 etcd 可以通过下述命令直接在k8s controller shell 上初始化 etcdctl 环境变量 =====
export ETCD_PEER_PORT=2379
eval "$(kubectl get pods -n kube-system -o custom-columns='name:.metadata.name,ip:.status.podIP' --no-headers | awk '/etcd/{ eps = eps (eps?",":"") "https://" $2 ":<nowiki>''</nowiki>$ETCD_PEER_PORT" } END { print "export ETCDCTL_ENDPOINTS=\"" eps "\"" }')"
export ETCDCTL_CACERT=/etc/kubernetes/ssl/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/ssl/etcd/peer.crt
export ETCDCTL_KEY=/etc/kubernetes/ssl/etcd/peer.key

etcdctl member list -w table
etcdctl endpoint status -w table

===== etcd member list (v3 API) =====
* etcdctl member list

[root@test kubelet]# docker exec -e "ETCDCTL_ENDPOINTS=<nowiki>https://192.168.150.12:12379,https://192.168.150.13:12379,https://192.168.150.14:12379</nowiki>" `docker ps | awk '/etcd /{print $1}'` etcdctl member list -w table
+------------------+---------+-------------------------+-----------------------------+------------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+-------------------------+-----------------------------+------------------------------+------------+
| 8a8a69237e3e00ef | started | dce-etcd-192.168.150.12 | <nowiki>http://192.168.150.12:12380</nowiki> | <nowiki>https://192.168.150.12:12379</nowiki> | false |
| d58cc05313738455 | started | dce-etcd-192.168.150.13 | <nowiki>http://192.168.150.13:12380</nowiki> | <nowiki>https://192.168.150.13:12379</nowiki> | false |
| ed2566e796a749a6 | started | dce-etcd-192.168.150.14 | <nowiki>http://192.168.150.14:12380</nowiki> | <nowiki>https://192.168.150.14:12379</nowiki> | false |
+------------------+---------+-------------------------+-----------------------------+------------------------------+------------+

===== etcd endpoint health (v3 API) =====
*etcdctl endpoint health

docker exec -e "ETCDCTL_API=3" -e "ETCDCTL_ENDPOINTS=<nowiki>https://192.168.155.22:12379,https://192.168.155.23:12379,https://192.168.155.24:12379</nowiki>" etcd_container_name_or_id etcdctl endpoint health -w table

+------------------------------+--------+--------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+------------------------------+--------+--------------+-------+
| <nowiki>https://192.168.155.23:12379</nowiki> | true | 14.2308ms | |
| <nowiki>https://192.168.155.22:12379</nowiki> | true | 14.572283ms | |
| <nowiki>https://192.168.155.24:12379</nowiki> | true | 351.572429ms | |
+------------------------------+--------+--------------+-------+

===== etcd endpoint status (v3 API) =====

* etcdctl endpoint status

docker exec -e "ETCDCTL_API=3" -e "ETCDCTL_ENDPOINTS=<nowiki>https://192.168.155.22:12379,https://192.168.155.23:12379,https://192.168.155.24:12379</nowiki>" etcd_container_name_or_id etcdctl endpoint status -w table

+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| <nowiki>https://192.168.155.22:12379</nowiki> | 5710b6824446f271 | 3.4.1 | 35 MB | false | false | 52 | 7925759 | 7925759 | |
| <nowiki>https://192.168.155.23:12379</nowiki> | 2a8509b66bfae6b6 | 3.4.1 | 35 MB | true | false | 52 | 7925759 | 7925759 | |
| <nowiki>https://192.168.155.24:12379</nowiki> | 72f4884011f8a2b | 3.4.1 | 35 MB | false | false | 52 | 7925760 | 7925760 | |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

==== etcd cluster-health (v2 API) ====
etcdctl v2可用的环境变量可见文档 https://github.com/etcd-io/etcd/blob/main/etcdctl/READMEv2.md

etcdctl v2常用的环境变量
ETCDCTL_ENDPOINT
ETCDCTL_CA_FILE
ETCDCTL_KEY_FILE
ETCDCTL_CERT_FILE
* etcdctl cluster-health

docker exec -e "ETCDCTL_API=2" -e "ETCDCTL_ENDPOINTS=<nowiki>https://192.168.155.22:12379,https://192.168.155.23:12379,https://192.168.155.24:12379</nowiki>" etcd_container_name_or_id etcdctl cluster-health
member 72f4884011f8a2b is healthy: got healthy result from <nowiki>https://192.168.155.24:12379</nowiki>
member 2a8509b66bfae6b6 is healthy: got healthy result from <nowiki>https://192.168.155.23:12379</nowiki>
member 5710b6824446f271 is healthy: got healthy result from <nowiki>https://192.168.155.22:12379</nowiki>
cluster is healthy

docker exec -it -e ETCDCTL_API=2 `docker ps | awk '/etcd /{print $1}'` etcdctl cluster-health

=== 磁盘/网络性能要求相关 ===
[https://etcd.io/docs/v3.5/op-guide/hardware/ official documentation]<blockquote>etcd is very sensitive to disk write latency. Typically 50 sequential IOPS (e.g., a 7200 RPM disk) is required. For heavily loaded clusters, 500 sequential IOPS (e.g., a typical local SSD or a high performance virtualized block device) is recommended. Note that most cloud providers publish concurrent IOPS rather than sequential IOPS; the published concurrent IOPS can be 10x greater than the sequential IOPS. To measure actual sequential IOPS, we suggest using a disk benchmarking tool such as diskbench or fio.

</blockquote>[https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.10/html-single/scalability_and_performance/index#recommended-etcd-practices_recommended-host-practices openshift_container_platform 推荐的 etcd 实践]<blockquote>就延迟而言，应该在一个可最少以 50 IOPS 按顺序写入 8000 字节的块设备上运行。也就是说，当有一个 20ms 的延迟时，使用 fdatasync 来同步 WAL 中的写入操作。对于高负载的集群，建议使用 8000 字节的连续 500 IOPS (2 毫秒)。要测量这些数字，您可以使用基准测试工具，如 fio。

</blockquote>[https://www.jianshu.com/p/f31ef5e7bdd0 简书 - etcd 性能测试与调优]

[https://bbs.pceva.com.cn/thread-24244-1-1.html 对硬盘性能的深度解析之二]<blockquote>IO延迟与Queue Depth（队列深度）/Queue Length （队列长度）

IO延迟是指控制器将IO指令发出之后，直到IO完成的过程中总共花费的时间。早前业界不成文的规定为，只要IO延迟在20ms内，IO性能对于应用程序来说都是可以接受的，但是如果大于20ms，应用程序的性能将会受到较大影响。（JMF602的小文件随机写入IOPS是个位数，所以你们觉得卡）

这样算下来，存储设备应当满足最低的IOPS要求应该为1S/20ms=50IOPS，所以只要区区50IOPS就可以满足这个要求了。单块机械硬盘的IOPS一般在80附近（7200转），固态硬盘的话就比较夸张了，对于大型的存储设备，通过并行N个IO通道工作，达到几十万甚至几百万IOPS都不是问题。

然而不能总以最低标准来要求存储设备。当接收到的IO很少的时候，IO延迟也会很小。比如一块Intel X25-M Gen2 34nm 80G固态硬盘，即使延迟平均在0.1ms的话，每个IO通道的IOPS=1000/0.1=10000，但是这块固态硬盘被厂家标称35000的读取IOPS，这里就引出另一个概念：Queue Depth（队列深度，也可以叫队列长度)

控制器向存储设备发起的指令，不是一条条发送的，而是一批批的发送，存储目标设备批量执行IO，然后把数据和结果返回控制器。只要存储设备肚量和消化能力足够强，在IO比较少的时候，处理一条指令和同时处理多条指令将会消耗几乎相同的时间。控制器发出的批量指令的最大条数，由控制器上的Queue Depth（队列深度）决定。（一般好的固态硬盘主控，队列深度都支持到32了）

如果给出队列深度，IOPS，IO延迟三者中的任意两者，则可以推算出第三者，公式：IOPS=(队列深度）/ (IO延迟）。实际上，随着队列深度的增加，IO延迟也在增加，二者是互相促进的关系，所以，随着IO数目的增多，将很快达到存储设备提供的最大IOPS处理能力，此时IO延迟将会陡峭升高，而IOPS则增加缓慢。（消化不良）</blockquote>[https://etcd.io/docs/v3.5/faq/#what-does-the-etcd-warning-failed-to-send-out-heartbeat-on-time-mean wal_fsync_duration_seconds 官方建议] <blockquote>Usually this issue is caused by a slow disk. Before the leader sends heartbeats attached with metadata, it may need to persist the metadata to disk. The disk could be experiencing contention among etcd and other applications, or the disk is too simply slow (e.g., a shared virtualized disk). To rule out a slow disk from causing this warning, monitor wal_fsync_duration_seconds (p99 duration should be less than 10ms) to confirm the disk is reasonably fast. If the disk is too slow, assigning a dedicated disk to etcd or using faster disk will typically solve the problem. To tell whether a disk is fast enough for etcd, a benchmarking tool such as fio can be used. Read here for an example.</blockquote>
[[分类:Linux]]
[[分类:K8s]]

Etcd命令随记

2026-02-02T09:32:33Z

Admin：

=== 切换 ETCDCTL API version ===

==== with docker exec ====
docker exec -e "ETCDCTL_API=2" etcd_container_name_or_id etcdctl --help

==== env ====
export ETCDCTL_API=2

=== 排障相关命令 ===

* 如果etcd是基于k8s manifest启动的,且该manifest初始化了ETCDCTL_ENDPOINTS变量，在指定endpoints只能通过环境变量指定，或者在bash环境下unset "ETCDCTL_ENDPOINTS" 取消环境变量定义然后再通过参数传入,不然会提示配置冲突:

2021-10-19 03:25:28.520086 C | pkg/flags: conflicting environment variable "ETCDCTL_ENDPOINTS" is shadowed by corresponding command-line flag (either unset environment variable or disable flag)

==== etcdctl v3 related ====
etcdctl v3可用的环境变量可见文档 https://github.com/etcd-io/etcd/blob/main/etcdctl/README.md

其中常见的包括
export ETCDCTL_ENDPOINTS=<nowiki>https://127.0.0.1:2379,https://10.255.251.102:2379,https://10.255.251.103:2379</nowiki>
export ETCDCTL_CACERT=/etc/kubernetes/ssl/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/ssl/etcd/peer.crt
export ETCDCTL_KEY=/etc/kubernetes/ssl/etcd/peer.key

===== k8s 的 etcd 可以通过下述命令直接在k8s controller shell 上初始化 etcdctl 环境变量 =====
export ETCD_PEER_PORT=2379
eval $(kubectl get pods -n kube-system -o custom-columns="name:.metadata.name,ip:.status.podIP" --no-headers|awk '/etcd/{printf "https://"$2":'$ETCD_PEER_PORT',"}'|awk '{gsub(",$","");print "export ETCDCTL_ENDPOINTS=\""$1"\""}')
export ETCDCTL_CACERT=/etc/kubernetes/ssl/etcd/ca.crt
export ETCDCTL_CERT=/etc/kubernetes/ssl/etcd/peer.crt
export ETCDCTL_KEY=/etc/kubernetes/ssl/etcd/peer.key

etcdctl member list -w table
etcdctl endpoint status -w table

===== etcd member list (v3 API) =====
* etcdctl member list

[root@test kubelet]# docker exec -e "ETCDCTL_ENDPOINTS=<nowiki>https://192.168.150.12:12379,https://192.168.150.13:12379,https://192.168.150.14:12379</nowiki>" `docker ps | awk '/etcd /{print $1}'` etcdctl member list -w table
+------------------+---------+-------------------------+-----------------------------+------------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+-------------------------+-----------------------------+------------------------------+------------+
| 8a8a69237e3e00ef | started | dce-etcd-192.168.150.12 | <nowiki>http://192.168.150.12:12380</nowiki> | <nowiki>https://192.168.150.12:12379</nowiki> | false |
| d58cc05313738455 | started | dce-etcd-192.168.150.13 | <nowiki>http://192.168.150.13:12380</nowiki> | <nowiki>https://192.168.150.13:12379</nowiki> | false |
| ed2566e796a749a6 | started | dce-etcd-192.168.150.14 | <nowiki>http://192.168.150.14:12380</nowiki> | <nowiki>https://192.168.150.14:12379</nowiki> | false |
+------------------+---------+-------------------------+-----------------------------+------------------------------+------------+

===== etcd endpoint health (v3 API) =====
*etcdctl endpoint health

docker exec -e "ETCDCTL_API=3" -e "ETCDCTL_ENDPOINTS=<nowiki>https://192.168.155.22:12379,https://192.168.155.23:12379,https://192.168.155.24:12379</nowiki>" etcd_container_name_or_id etcdctl endpoint health -w table

+------------------------------+--------+--------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+------------------------------+--------+--------------+-------+
| <nowiki>https://192.168.155.23:12379</nowiki> | true | 14.2308ms | |
| <nowiki>https://192.168.155.22:12379</nowiki> | true | 14.572283ms | |
| <nowiki>https://192.168.155.24:12379</nowiki> | true | 351.572429ms | |
+------------------------------+--------+--------------+-------+

===== etcd endpoint status (v3 API) =====

* etcdctl endpoint status

docker exec -e "ETCDCTL_API=3" -e "ETCDCTL_ENDPOINTS=<nowiki>https://192.168.155.22:12379,https://192.168.155.23:12379,https://192.168.155.24:12379</nowiki>" etcd_container_name_or_id etcdctl endpoint status -w table

+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| <nowiki>https://192.168.155.22:12379</nowiki> | 5710b6824446f271 | 3.4.1 | 35 MB | false | false | 52 | 7925759 | 7925759 | |
| <nowiki>https://192.168.155.23:12379</nowiki> | 2a8509b66bfae6b6 | 3.4.1 | 35 MB | true | false | 52 | 7925759 | 7925759 | |
| <nowiki>https://192.168.155.24:12379</nowiki> | 72f4884011f8a2b | 3.4.1 | 35 MB | false | false | 52 | 7925760 | 7925760 | |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

==== etcd cluster-health (v2 API) ====
etcdctl v2可用的环境变量可见文档 https://github.com/etcd-io/etcd/blob/main/etcdctl/READMEv2.md

etcdctl v2常用的环境变量
ETCDCTL_ENDPOINT
ETCDCTL_CA_FILE
ETCDCTL_KEY_FILE
ETCDCTL_CERT_FILE
* etcdctl cluster-health

docker exec -e "ETCDCTL_API=2" -e "ETCDCTL_ENDPOINTS=<nowiki>https://192.168.155.22:12379,https://192.168.155.23:12379,https://192.168.155.24:12379</nowiki>" etcd_container_name_or_id etcdctl cluster-health
member 72f4884011f8a2b is healthy: got healthy result from <nowiki>https://192.168.155.24:12379</nowiki>
member 2a8509b66bfae6b6 is healthy: got healthy result from <nowiki>https://192.168.155.23:12379</nowiki>
member 5710b6824446f271 is healthy: got healthy result from <nowiki>https://192.168.155.22:12379</nowiki>
cluster is healthy

docker exec -it -e ETCDCTL_API=2 `docker ps | awk '/etcd /{print $1}'` etcdctl cluster-health

=== 磁盘/网络性能要求相关 ===
[https://etcd.io/docs/v3.5/op-guide/hardware/ official documentation]<blockquote>etcd is very sensitive to disk write latency. Typically 50 sequential IOPS (e.g., a 7200 RPM disk) is required. For heavily loaded clusters, 500 sequential IOPS (e.g., a typical local SSD or a high performance virtualized block device) is recommended. Note that most cloud providers publish concurrent IOPS rather than sequential IOPS; the published concurrent IOPS can be 10x greater than the sequential IOPS. To measure actual sequential IOPS, we suggest using a disk benchmarking tool such as diskbench or fio.

</blockquote>[https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.10/html-single/scalability_and_performance/index#recommended-etcd-practices_recommended-host-practices openshift_container_platform 推荐的 etcd 实践]<blockquote>就延迟而言，应该在一个可最少以 50 IOPS 按顺序写入 8000 字节的块设备上运行。也就是说，当有一个 20ms 的延迟时，使用 fdatasync 来同步 WAL 中的写入操作。对于高负载的集群，建议使用 8000 字节的连续 500 IOPS (2 毫秒)。要测量这些数字，您可以使用基准测试工具，如 fio。

</blockquote>[https://www.jianshu.com/p/f31ef5e7bdd0 简书 - etcd 性能测试与调优]

[https://bbs.pceva.com.cn/thread-24244-1-1.html 对硬盘性能的深度解析之二]<blockquote>IO延迟与Queue Depth（队列深度）/Queue Length （队列长度）

IO延迟是指控制器将IO指令发出之后，直到IO完成的过程中总共花费的时间。早前业界不成文的规定为，只要IO延迟在20ms内，IO性能对于应用程序来说都是可以接受的，但是如果大于20ms，应用程序的性能将会受到较大影响。（JMF602的小文件随机写入IOPS是个位数，所以你们觉得卡）

这样算下来，存储设备应当满足最低的IOPS要求应该为1S/20ms=50IOPS，所以只要区区50IOPS就可以满足这个要求了。单块机械硬盘的IOPS一般在80附近（7200转），固态硬盘的话就比较夸张了，对于大型的存储设备，通过并行N个IO通道工作，达到几十万甚至几百万IOPS都不是问题。

然而不能总以最低标准来要求存储设备。当接收到的IO很少的时候，IO延迟也会很小。比如一块Intel X25-M Gen2 34nm 80G固态硬盘，即使延迟平均在0.1ms的话，每个IO通道的IOPS=1000/0.1=10000，但是这块固态硬盘被厂家标称35000的读取IOPS，这里就引出另一个概念：Queue Depth（队列深度，也可以叫队列长度)

控制器向存储设备发起的指令，不是一条条发送的，而是一批批的发送，存储目标设备批量执行IO，然后把数据和结果返回控制器。只要存储设备肚量和消化能力足够强，在IO比较少的时候，处理一条指令和同时处理多条指令将会消耗几乎相同的时间。控制器发出的批量指令的最大条数，由控制器上的Queue Depth（队列深度）决定。（一般好的固态硬盘主控，队列深度都支持到32了）

如果给出队列深度，IOPS，IO延迟三者中的任意两者，则可以推算出第三者，公式：IOPS=(队列深度）/ (IO延迟）。实际上，随着队列深度的增加，IO延迟也在增加，二者是互相促进的关系，所以，随着IO数目的增多，将很快达到存储设备提供的最大IOPS处理能力，此时IO延迟将会陡峭升高，而IOPS则增加缓慢。（消化不良）</blockquote>[https://etcd.io/docs/v3.5/faq/#what-does-the-etcd-warning-failed-to-send-out-heartbeat-on-time-mean wal_fsync_duration_seconds 官方建议] <blockquote>Usually this issue is caused by a slow disk. Before the leader sends heartbeats attached with metadata, it may need to persist the metadata to disk. The disk could be experiencing contention among etcd and other applications, or the disk is too simply slow (e.g., a shared virtualized disk). To rule out a slow disk from causing this warning, monitor wal_fsync_duration_seconds (p99 duration should be less than 10ms) to confirm the disk is reasonably fast. If the disk is too slow, assigning a dedicated disk to etcd or using faster disk will typically solve the problem. To tell whether a disk is fast enough for etcd, a benchmarking tool such as fio can be used. Read here for an example.</blockquote>
[[分类:Linux]]
[[分类:K8s]]

Kubectl随记

2026-01-28T06:33:30Z

Admin：

=== kubectl set default editor ===
export KUBE_EDITOR=/usr/bin/vim

===kubectl completion bash not working 排障随记===
需要已经安装 bash-completion

如果提示 <code>_get_comp_words_by_ref: command not found</code> 错误的话，需要执行
source /usr/share/bash-completion/bash_completion
执行命令 <code>set</code> 看一下当前环境的相关配置有没有kube相关项
set | grep -i kube
没有的话需要执行
source <(kubectl completion bash)

正常的话,打开 kubectl debug
__kubectl_debug() {
if [[ -n ${BASH_COMP_DEBUG_FILE} ]]; then
echo "$*" >> "${BASH_COMP_DEBUG_FILE}"
fi
}

export BASH_COMP_DEBUG_FILE=****
根据debug file里面的记录去排查出错点

=== kubectl to get all api-resource and filter by verbs ===
kubectl api-resources -o name --cached --request-timeout=5s --verbs=get

kubectl api-resources --api-group=networking.k8s.io

==== get groupVersion when a resource exists in multiple api groups ====

==== explain api-resource with sprcific api version ====
kubectl api-resources --api-group=networking.k8s.io -v8 2>&1| grep ingress

kubectl explain ingress --api-version="networking.k8s.io/v1beta1"

=== get service account token faster / 一条命令结合jsonpath揭秘base64处理后的secret ===
kubectl get secret sa-secret -o jsonpath={.data.token} | base64 -d

=== Check k8s account privilege / k8s 对象权限相关 ===
kubectl options:
--as=: Username to impersonate for the operation
--as-group=[]: Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
假设有名为registry-test的serviceAccount位于kube-system，这时候我们要测试他的RBAC权限，可以使用如下命令
kubectl --as "system:serviceaccount:kube-system:registry-test" get serviceaccount

Usage: kubectl auth can-i VERB [TYPE | TYPE/NAME | NONRESOURCEURL] [options]

# Check to see if I can do everything in my current namespace ("*" means all)
kubectl auth can-i '*' '*'

[root@node-a ~]# kubectl --as "system:serviceaccount:kube-system:registry-test" auth can-i list pods
yes
PS: 在RBAC定义中(例如role / clusterrole)对 <code>resources</code> 的 <code>pods/*</code> 声明不会对诸如 <code>pods/exec pods/log pods/status</code> 等资源授权起效果

原因: [[K8s-rbac关于子资源授权的一些记录]]

===kubectl get resources with custom column / 利用custom-columns自定义输出字段 (语法相对go-template简单)===

=====通过custom-columns只输出pod所在租户，pod名字和pod uid=====
kubectl get pods -o custom-columns='namespace:metadata.namespace,pod:metadata.name,uid:metadata.uid'

=====通过custom-columns只输出namespace名字和ns annotation中定义的node selector=====
kubectl get namespaces -o custom-columns="NAMESPACE:.metadata.name, NODE_SELECTOR:.metadata.annotations.scheduler\.alpha\.kubernetes\.io/node-selector"
按照节点输出基础统计信息及CPU&内存信息
kubectl get nodes -o custom-columns='NAME:.metadata.name,STATUS:.status.conditions[?(@.type=="Ready")].status,AGE:.metadata.creationTimestamp,VERSION:.status.nodeInfo.kubeletVersion,INTERNAL-IP:.status.addresses[?(@.type=="InternalIP")].address,OS-IMAGE:.status.nodeInfo.osImage,KERNEL-VERSION:.status.nodeInfo.kernelVersion,CONTAINER-RUNTIME:.status.nodeInfo.containerRuntimeVersion,CPU-CAPACITY:.status.capacity.cpu,MEMORY-CAPACITY:.status.capacity.memory'

===Get node info order by node ip address / 根据ip地址排序获取节点信息===
kubectl get nodes -owide --sort-by status.addresses[0].address

===Get pods order by running node name / 根据pod运行节点排序获取pod信息===
kubectl get pods -owide --sort-by .spec.nodeName

kubectl get pods -owide --sort-by spec.nodeName
===Get pod order by pod create time / 根据pod创建时间排序获取pod===
kubectl get pods -n efk-system --sort-by status.startTime

kubectl get pods -n efk-system --sort-by status.startTime| grep filebeat

kubectl get pods -n efk-system --sort-by status.startTime| grep filebeat|tac

=== GET Pod Volumes / 获取某个租户下所有pod所绑定的volumes (依赖jq) ===
过滤了 POD 所挂载的 k8s service account token
kubectl get pods -n ${your_name_space} -o json | jq -r '.items[] | [
.metadata.name,
([.spec.volumes[] | select(.name | startswith("kube-api-access") | not)] | tostring)
] | @tsv' | column -t

* @tsv是为了将输出格式化成两列，column命令在这里起到对齐的作用
如果需要过滤configmap和secret，也可以用命令
kubectl get pods -n ${your_name_space} -o json | jq -r '.items[] | [
.metadata.name,
([.spec.volumes[] |
select( (.name | startswith("kube-api-access") | not)
and (.configMap == null)
and (.secret == null) )] | tostring)
] | @tsv' | column -t

===kubectl get resources with go-template / 利用go-template自定义输出(功能扩展性高度自由,但是有一定的编码调试成本)===

====Deployments - image / 只获取deployment名字和image====
<nowiki>kubectl get deployments -o go-template --template '{{range .items}}{{.metadata.name}}{{" -- "}}{{range .spec.template.spec.containers}}{{.image}}{{" "}}{{end}}{{"\n"}}{{end}}'</nowiki>
稍微格式化输出，输出deployment名字 + container名字 + 对应的image
kubectl get deployments.apps -o go-template --template '<nowiki>{{range .items}}</nowiki><nowiki>{{printf "%-30s " .metadata.name}}</nowiki><nowiki>{{range .spec.template.spec.containers}}</nowiki><nowiki>{{.name}}</nowiki><nowiki>{{":"}}</nowiki><nowiki>{{.image}}</nowiki><nowiki>{{" "}}</nowiki><nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

====Pods - image / 只获取pod名字和image====
<nowiki>kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{" -- "}}{{range .spec.containers}}{{.image}}{{end}}{{"\n"}}{{end}}'</nowiki>
====Get the ip address of the specified node / 只获取节点名字和对应的Internal IP地址====
<nowiki>kubectl get nodes -l kubernetes.io/hostname=nodename -o go-template --template '{{range .items}}{{range .status.addresses}}{{ if eq .type "InternalIP" }}{{.address}}{{end}}{{end}}{{end}}{{"\n"}}'</nowiki>
====Get node taints / 只获取节点名字和taints====
<nowiki>kubectl get nodes -o go-template --template '{{range .items}}{{.metadata.name}}{{":\n"}}{{range .spec.taints}}{{.key}}{{"="}}{{.value}}{{":"}}{{.effect}}{{" "}}{{end}}{{"\n\n"}}{{end}}'</nowiki>

<nowiki>kubectl get nodes -l kubernetes.io/hostname=nodename -o go-template --template '{{range .items}}{{.metadata.name}}{{":\n"}}{{range .spec.taints}}{{.key}}{{"="}}{{.value}}{{":"}}{{.effect}}{{" "}}{{end}}{{"\n\n"}}{{end}}'</nowiki>

==== Get node labels and format output / 只获取节点名字和labels并格式化输出 ====
kubectl get node -o go-template --template '<nowiki>{{range .items}}</nowiki><nowiki>{{.metadata.name}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{range $key, $value := .metadata.labels}}</nowiki><nowiki>{{"\t\t"}}</nowiki><nowiki>{{$key}}</nowiki><nowiki>{{":"}}</nowiki><nowiki>{{$value}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

==== 获取namespace名字和ns annotation中定义的node selector ====
kubectl get namespaces -o go-template='<nowiki>{{range .items}}</nowiki><nowiki>{{.metadata.name}}</nowiki><nowiki>{{" "}}</nowiki><nowiki>{{index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector" -}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

===== 稍微处理一下，格式化对齐输出 =====
kubectl get namespaces -o go-template='<nowiki>{{range .items}}</nowiki><nowiki>{{printf "%-30s " .metadata.name}}</nowiki><nowiki>{{if index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector"}}</nowiki><nowiki>{{index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector"}}</nowiki><nowiki>{{else}}</nowiki> <nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

===kubectl get event filter by pod name / 根据对象进行过滤获取k8s事件===
kubectl get event -n kube-system --field-selector involvedObject.name=${pod_name}

===kubectl get event sort by lastTime / 根据事件最后触发时间进行排序===
kubectl get event -n kube-system --sort-by=.lastTimestamp

=== kubectl patch ===
[[K8s的patch命令随记]]

kubectl patch 调整 annotation
<nowiki>kubectl patch ns demo -p '{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/node-selector": "kubernetes.io/hostname=node1"}}}'</nowiki>
[[分类:K8s]]

Kubectl随记

2025-11-12T11:03:01Z

Admin：

=== kubectl set default editor ===
export KUBE_EDITOR=/usr/bin/vim

===kubectl completion bash not working 排障随记===
需要已经安装 bash-completion

如果提示 <code>_get_comp_words_by_ref: command not found</code> 错误的话，需要执行
source /usr/share/bash-completion/bash_completion
执行命令 <code>set</code> 看一下当前环境的相关配置有没有kube相关项
set | grep -i kube
没有的话需要执行
source <(kubectl completion bash)

正常的话,打开 kubectl debug
__kubectl_debug() {
if [[ -n ${BASH_COMP_DEBUG_FILE} ]]; then
echo "$*" >> "${BASH_COMP_DEBUG_FILE}"
fi
}

export BASH_COMP_DEBUG_FILE=****
根据debug file里面的记录去排查出错点

=== kubectl to get all api-resource and filter by verbs ===
kubectl api-resources -o name --cached --request-timeout=5s --verbs=get

kubectl api-resources --api-group=networking.k8s.io

==== get groupVersion when a resource exists in multiple api groups ====

==== explain api-resource with sprcific api version ====
kubectl api-resources --api-group=networking.k8s.io -v8 2>&1| grep ingress

kubectl explain ingress --api-version="networking.k8s.io/v1beta1"

=== get service account token faster / 一条命令结合jsonpath揭秘base64处理后的secret ===
kubectl get secret sa-secret -o jsonpath={.data.token} | base64 -d

=== Check k8s account privilege / k8s 对象权限相关 ===
kubectl options:
--as=: Username to impersonate for the operation
--as-group=[]: Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
假设有名为registry-test的serviceAccount位于kube-system，这时候我们要测试他的RBAC权限，可以使用如下命令
kubectl --as "system:serviceaccount:kube-system:registry-test" get serviceaccount

Usage: kubectl auth can-i VERB [TYPE | TYPE/NAME | NONRESOURCEURL] [options]

# Check to see if I can do everything in my current namespace ("*" means all)
kubectl auth can-i '*' '*'

[root@node-a ~]# kubectl --as "system:serviceaccount:kube-system:registry-test" auth can-i list pods
yes
PS: 在RBAC定义中(例如role / clusterrole)对 <code>resources</code> 的 <code>pods/*</code> 声明不会对诸如 <code>pods/exec pods/log pods/status</code> 等资源授权起效果

原因: [[K8s-rbac关于子资源授权的一些记录]]

===kubectl get resources with custom column / 利用custom-columns自定义输出字段 (语法相对go-template简单)===

=====通过custom-columns只输出pod所在租户，pod名字和pod uid=====
kubectl get pods -o custom-columns='namespace:metadata.namespace,pod:metadata.name,uid:metadata.uid'

=====通过custom-columns只输出namespace名字和ns annotation中定义的node selector=====
kubectl get namespaces -o custom-columns="NAMESPACE:.metadata.name, NODE_SELECTOR:.metadata.annotations.scheduler\.alpha\.kubernetes\.io/node-selector"

===Get node info order by node ip address / 根据ip地址排序获取节点信息===
kubectl get nodes -owide --sort-by status.addresses[0].address

===Get pods order by running node name / 根据pod运行节点排序获取pod信息===
kubectl get pods -owide --sort-by .spec.nodeName

kubectl get pods -owide --sort-by spec.nodeName
===Get pod order by pod create time / 根据pod创建时间排序获取pod===
kubectl get pods -n efk-system --sort-by status.startTime

kubectl get pods -n efk-system --sort-by status.startTime| grep filebeat

kubectl get pods -n efk-system --sort-by status.startTime| grep filebeat|tac

=== GET Pod Volumes / 获取某个租户下所有pod所绑定的volumes (依赖jq) ===
过滤了 POD 所挂载的 k8s service account token
kubectl get pods -n ${your_name_space} -o json | jq -r '.items[] | [
.metadata.name,
([.spec.volumes[] | select(.name | startswith("kube-api-access") | not)] | tostring)
] | @tsv' | column -t

* @tsv是为了将输出格式化成两列，column命令在这里起到对齐的作用
如果需要过滤configmap和secret，也可以用命令
kubectl get pods -n ${your_name_space} -o json | jq -r '.items[] | [
.metadata.name,
([.spec.volumes[] |
select( (.name | startswith("kube-api-access") | not)
and (.configMap == null)
and (.secret == null) )] | tostring)
] | @tsv' | column -t

===kubectl get resources with go-template / 利用go-template自定义输出(功能扩展性高度自由,但是有一定的编码调试成本)===

====Deployments - image / 只获取deployment名字和image====
<nowiki>kubectl get deployments -o go-template --template '{{range .items}}{{.metadata.name}}{{" -- "}}{{range .spec.template.spec.containers}}{{.image}}{{" "}}{{end}}{{"\n"}}{{end}}'</nowiki>
稍微格式化输出，输出deployment名字 + container名字 + 对应的image
kubectl get deployments.apps -o go-template --template '<nowiki>{{range .items}}</nowiki><nowiki>{{printf "%-30s " .metadata.name}}</nowiki><nowiki>{{range .spec.template.spec.containers}}</nowiki><nowiki>{{.name}}</nowiki><nowiki>{{":"}}</nowiki><nowiki>{{.image}}</nowiki><nowiki>{{" "}}</nowiki><nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

====Pods - image / 只获取pod名字和image====
<nowiki>kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{" -- "}}{{range .spec.containers}}{{.image}}{{end}}{{"\n"}}{{end}}'</nowiki>
====Get the ip address of the specified node / 只获取节点名字和对应的Internal IP地址====
<nowiki>kubectl get nodes -l kubernetes.io/hostname=nodename -o go-template --template '{{range .items}}{{range .status.addresses}}{{ if eq .type "InternalIP" }}{{.address}}{{end}}{{end}}{{end}}{{"\n"}}'</nowiki>
====Get node taints / 只获取节点名字和taints====
<nowiki>kubectl get nodes -o go-template --template '{{range .items}}{{.metadata.name}}{{":\n"}}{{range .spec.taints}}{{.key}}{{"="}}{{.value}}{{":"}}{{.effect}}{{" "}}{{end}}{{"\n\n"}}{{end}}'</nowiki>

<nowiki>kubectl get nodes -l kubernetes.io/hostname=nodename -o go-template --template '{{range .items}}{{.metadata.name}}{{":\n"}}{{range .spec.taints}}{{.key}}{{"="}}{{.value}}{{":"}}{{.effect}}{{" "}}{{end}}{{"\n\n"}}{{end}}'</nowiki>

==== Get node labels and format output / 只获取节点名字和labels并格式化输出 ====
kubectl get node -o go-template --template '<nowiki>{{range .items}}</nowiki><nowiki>{{.metadata.name}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{range $key, $value := .metadata.labels}}</nowiki><nowiki>{{"\t\t"}}</nowiki><nowiki>{{$key}}</nowiki><nowiki>{{":"}}</nowiki><nowiki>{{$value}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

==== 获取namespace名字和ns annotation中定义的node selector ====
kubectl get namespaces -o go-template='<nowiki>{{range .items}}</nowiki><nowiki>{{.metadata.name}}</nowiki><nowiki>{{" "}}</nowiki><nowiki>{{index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector" -}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

===== 稍微处理一下，格式化对齐输出 =====
kubectl get namespaces -o go-template='<nowiki>{{range .items}}</nowiki><nowiki>{{printf "%-30s " .metadata.name}}</nowiki><nowiki>{{if index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector"}}</nowiki><nowiki>{{index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector"}}</nowiki><nowiki>{{else}}</nowiki> <nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

===kubectl get event filter by pod name / 根据对象进行过滤获取k8s事件===
kubectl get event -n kube-system --field-selector involvedObject.name=${pod_name}

===kubectl get event sort by lastTime / 根据事件最后触发时间进行排序===
kubectl get event -n kube-system --sort-by=.lastTimestamp

=== kubectl patch ===
[[K8s的patch命令随记]]

kubectl patch 调整 annotation
<nowiki>kubectl patch ns demo -p '{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/node-selector": "kubernetes.io/hostname=node1"}}}'</nowiki>
[[分类:K8s]]

Kubectl随记

2025-11-12T10:40:36Z

Admin：

=== kubectl set default editor ===
export KUBE_EDITOR=/usr/bin/vim

===kubectl completion bash not working 排障随记===
需要已经安装 bash-completion

如果提示 <code>_get_comp_words_by_ref: command not found</code> 错误的话，需要执行
source /usr/share/bash-completion/bash_completion
执行命令 <code>set</code> 看一下当前环境的相关配置有没有kube相关项
set | grep -i kube
没有的话需要执行
source <(kubectl completion bash)

正常的话,打开 kubectl debug
__kubectl_debug() {
if [[ -n ${BASH_COMP_DEBUG_FILE} ]]; then
echo "$*" >> "${BASH_COMP_DEBUG_FILE}"
fi
}

export BASH_COMP_DEBUG_FILE=****
根据debug file里面的记录去排查出错点

=== kubectl to get all api-resource and filter by verbs ===
kubectl api-resources -o name --cached --request-timeout=5s --verbs=get

kubectl api-resources --api-group=networking.k8s.io

==== get groupVersion when a resource exists in multiple api groups ====

==== explain api-resource with sprcific api version ====
kubectl api-resources --api-group=networking.k8s.io -v8 2>&1| grep ingress

kubectl explain ingress --api-version="networking.k8s.io/v1beta1"

=== get service account token faster / 一条命令结合jsonpath揭秘base64处理后的secret ===
kubectl get secret sa-secret -o jsonpath={.data.token} | base64 -d

=== Check k8s account privilege / k8s 对象权限相关 ===
kubectl options:
--as=: Username to impersonate for the operation
--as-group=[]: Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
假设有名为registry-test的serviceAccount位于kube-system，这时候我们要测试他的RBAC权限，可以使用如下命令
kubectl --as "system:serviceaccount:kube-system:registry-test" get serviceaccount

Usage: kubectl auth can-i VERB [TYPE | TYPE/NAME | NONRESOURCEURL] [options]

# Check to see if I can do everything in my current namespace ("*" means all)
kubectl auth can-i '*' '*'

[root@node-a ~]# kubectl --as "system:serviceaccount:kube-system:registry-test" auth can-i list pods
yes
PS: 在RBAC定义中(例如role / clusterrole)对 <code>resources</code> 的 <code>pods/*</code> 声明不会对诸如 <code>pods/exec pods/log pods/status</code> 等资源授权起效果

原因: [[K8s-rbac关于子资源授权的一些记录]]

===kubectl get resources with custom column / 利用custom-columns自定义输出字段 (语法相对go-template简单)===

=====通过custom-columns只输出pod所在租户，pod名字和pod uid=====
kubectl get pods -o custom-columns='namespace:metadata.namespace,pod:metadata.name,uid:metadata.uid'

=====通过custom-columns只输出namespace名字和ns annotation中定义的node selector=====
kubectl get namespaces -o custom-columns="NAMESPACE:.metadata.name, NODE_SELECTOR:.metadata.annotations.scheduler\.alpha\.kubernetes\.io/node-selector"

===Get node info order by node ip address / 根据ip地址排序获取节点信息===
kubectl get nodes -owide --sort-by status.addresses[0].address

===Get pods order by running node name / 根据pod运行节点排序获取pod信息===
kubectl get pods -owide --sort-by .spec.nodeName

kubectl get pods -owide --sort-by spec.nodeName
===Get pod order by pod create time / 根据pod创建时间排序获取pod===
kubectl get pods -n efk-system --sort-by status.startTime

kubectl get pods -n efk-system --sort-by status.startTime| grep filebeat

kubectl get pods -n efk-system --sort-by status.startTime| grep filebeat|tac

=== GET Pod Volumes / 获取某个租户下所有pod所绑定的volumes (依赖jq) ===
过滤了 POD 所挂载的 k8s service account token
kubectl get pods -n ${your-name-space} -o json | jq -r '.items[] | [
.metadata.name,
([.spec.volumes[] | select(.name | startswith("kube-api-access") | not)] | tostring)
] | @tsv' | column -t

* @tsv是为了将输出格式化成两列，column命令在这里起到对齐的作用

===kubectl get resources with go-template / 利用go-template自定义输出(功能扩展性高度自由,但是有一定的编码调试成本)===

====Deployments - image / 只获取deployment名字和image====
<nowiki>kubectl get deployments -o go-template --template '{{range .items}}{{.metadata.name}}{{" -- "}}{{range .spec.template.spec.containers}}{{.image}}{{" "}}{{end}}{{"\n"}}{{end}}'</nowiki>
稍微格式化输出，输出deployment名字 + container名字 + 对应的image
kubectl get deployments.apps -o go-template --template '<nowiki>{{range .items}}</nowiki><nowiki>{{printf "%-30s " .metadata.name}}</nowiki><nowiki>{{range .spec.template.spec.containers}}</nowiki><nowiki>{{.name}}</nowiki><nowiki>{{":"}}</nowiki><nowiki>{{.image}}</nowiki><nowiki>{{" "}}</nowiki><nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

====Pods - image / 只获取pod名字和image====
<nowiki>kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{" -- "}}{{range .spec.containers}}{{.image}}{{end}}{{"\n"}}{{end}}'</nowiki>
====Get the ip address of the specified node / 只获取节点名字和对应的Internal IP地址====
<nowiki>kubectl get nodes -l kubernetes.io/hostname=nodename -o go-template --template '{{range .items}}{{range .status.addresses}}{{ if eq .type "InternalIP" }}{{.address}}{{end}}{{end}}{{end}}{{"\n"}}'</nowiki>
====Get node taints / 只获取节点名字和taints====
<nowiki>kubectl get nodes -o go-template --template '{{range .items}}{{.metadata.name}}{{":\n"}}{{range .spec.taints}}{{.key}}{{"="}}{{.value}}{{":"}}{{.effect}}{{" "}}{{end}}{{"\n\n"}}{{end}}'</nowiki>

<nowiki>kubectl get nodes -l kubernetes.io/hostname=nodename -o go-template --template '{{range .items}}{{.metadata.name}}{{":\n"}}{{range .spec.taints}}{{.key}}{{"="}}{{.value}}{{":"}}{{.effect}}{{" "}}{{end}}{{"\n\n"}}{{end}}'</nowiki>

==== Get node labels and format output / 只获取节点名字和labels并格式化输出 ====
kubectl get node -o go-template --template '<nowiki>{{range .items}}</nowiki><nowiki>{{.metadata.name}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{range $key, $value := .metadata.labels}}</nowiki><nowiki>{{"\t\t"}}</nowiki><nowiki>{{$key}}</nowiki><nowiki>{{":"}}</nowiki><nowiki>{{$value}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

==== 获取namespace名字和ns annotation中定义的node selector ====
kubectl get namespaces -o go-template='<nowiki>{{range .items}}</nowiki><nowiki>{{.metadata.name}}</nowiki><nowiki>{{" "}}</nowiki><nowiki>{{index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector" -}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

===== 稍微处理一下，格式化对齐输出 =====
kubectl get namespaces -o go-template='<nowiki>{{range .items}}</nowiki><nowiki>{{printf "%-30s " .metadata.name}}</nowiki><nowiki>{{if index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector"}}</nowiki><nowiki>{{index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector"}}</nowiki><nowiki>{{else}}</nowiki> <nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

===kubectl get event filter by pod name / 根据对象进行过滤获取k8s事件===
kubectl get event -n kube-system --field-selector involvedObject.name=${pod_name}

===kubectl get event sort by lastTime / 根据事件最后触发时间进行排序===
kubectl get event -n kube-system --sort-by=.lastTimestamp

=== kubectl patch ===
[[K8s的patch命令随记]]

kubectl patch 调整 annotation
<nowiki>kubectl patch ns demo -p '{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/node-selector": "kubernetes.io/hostname=node1"}}}'</nowiki>
[[分类:K8s]]

Docker拉取镜像认证报文简析

2025-10-28T08:52:40Z

Admin：

有时候会有一些奇怪的镜像操作需要用到curl等工具去实现，遂有了本篇记录

这里主要记录集中与registry API交互过程中，auth -> get manifest info部分

=== 针对 hub.docker.com 例子 ===
" 一个 docker pull 指令会拉两部分，一部分是 manifest，一部分是 layer，前者指定了一个 image 相关的信息和 layer 的信息（一个 JSON 文件），后者就是一些大文件（layer）"

会用到的关联域名:

Location of Docker index: <nowiki>https://index.docker.io/</nowiki>

Location of the remote repository: <nowiki>https://registry-1.docker.io</nowiki>

实现:

假设需要拉取的镜像为alpine:3.17.0_rc1 即:
➤ docker pull alpine:3.17.0_rc1
那么在curl中的实现即为(是的，他有个library!):
➤ curl <nowiki>https://registry-1.docker.io/v2/library/alpine/manifests/3.17.0_rc1</nowiki>
这时候直接请求是会401的，要根据返回的www-authenticate header去进行认证
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< content-type: application/json
< docker-distribution-api-version: registry/2.0
< www-authenticate: Bearer realm="<nowiki>https://auth.docker.io/token</nowiki>",service="registry.docker.io",scope="repository:library/alpine:pull"
< date: Sun, 20 Nov 2022 09:08:33 GMT
< content-length: 157
< strict-transport-security: max-age=31536000
< docker-ratelimit-source: 8.210.192.22
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"library/alpine","Action":"pull"}]}]}
* Connection #1 to host registry-1.docker.io left intact
根据回显进行认证(匿名认证):

注意这里的service也是需要一一对应的，scope value可以不进行URL编码
➤ curl "<nowiki>https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io</nowiki>"
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsIng1YyI6WyJNSUlDK1RDQ0FwK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakJHTVVRd1FnWURWUVFERXp0U1RVbEdPbEZNUmpRNlEwZFFNenBSTWtWYU9sRklSRUk2VkVkRlZUcFZTRlZNT2taTVZqUTZSMGRXV2pwQk5WUkhPbFJMTkZNNlVVeElTVEFlRncweU1qQXhNVEF5TWpJeE5EbGFGdzB5TXpBeE1qVXlNakl4TkRsYU1FWXhSREJDQmdOVkJBTVRPMUJaUTFJNlNWQmFRanBJUWxGWE9qZE1SVms2UWtGV1FqcEhTRGRhT2xWSVZ6TTZVa3RMVWpwRE4wNHpPbGxOTkZJNlNWaE5TRHBLVkZCQ01JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBbE5JOGFtMlBERnZzNndOeVl2d0dkZWZXVXJXQWdMZ3N2MzA2MnZycG5VNkN0WUpEZDZ2K2NEVG1FN1FlTllEaFMyaU1wU3djZkRCL2RFclEwdnhIZE4ycDIvODZmZy9TeWlIMnhmMGFVTjlDV1dud0JPaTIvS3hLditpbFNlQ01HYXRwRlg3SmYxcWI4N0Q5NUxOVDBvOU9OTmYxT3RidjY5ck9tL1RIVFh3clUvV3dTZlUyWktUbEw4SVRXRkRXN09ZK3hXdUJ0WUpteVhqcVpsaWRBbUNTdTdHY0Y0MVB5em9KTFFTMnJCdXJwOXc0cWgxMFk1bUNIcWdsaEI1Rk9aOUs0T2pUaVhUUHJFUk5WcnArUFVIR3JBYVRPRTBwQzgyUHBuWVZhNzNDUkdsMEdDdC9RckJwVjRpdmswdzF0eEtkV1NiSDNnRmtqZ2g1N0tOcDhRSURBUUFCbzRHeU1JR3ZNQTRHQTFVZER3RUIvd1FFQXdJSGdEQVBCZ05WSFNVRUNEQUdCZ1JWSFNVQU1FUUdBMVVkRGdROUJEdFFXVU5TT2tsUVdrSTZTRUpSVnpvM1RFVlpPa0pCVmtJNlIwZzNXanBWU0Zjek9sSkxTMUk2UXpkT016cFpUVFJTT2tsWVRVZzZTbFJRUWpCR0JnTlZIU01FUHpBOWdEdFNUVWxHT2xGTVJqUTZRMGRRTXpwUk1rVmFPbEZJUkVJNlZFZEZWVHBWU0ZWTU9rWk1WalE2UjBkV1dqcEJOVlJIT2xSTE5GTTZVVXhJU1RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlFQTdIY1VyVm1namo1cE01MXhZVHd2eGE1VnRqd2hub0dRZjFxTU52UGVHeVlDSUFwYm4vWFkvS1F5WWFWRnRjMWtsb0lmZzd4L3hlbkZhbkp4L0F2cURGdFgiXX0.eyJhY2Nlc3MiOlt7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6ImxpYnJhcnkvYWxwaW5lIiwiYWN0aW9ucyI6WyJwdWxsIl0sInBhcmFtZXRlcnMiOnsicHVsbF9saW1pdCI6IjEwMCIsInB1bGxfbGltaXRfaW50ZXJ2YWwiOiIyMTYwMCJ9fV0sImF1ZCI6InJlZ2lzdHJ5LmRvY2tlci5pbyIsImV4cCI6MTY2ODkzNTY5MiwiaWF0IjoxNjY4OTM1MzkyLCJpc3MiOiJhdXRoLmRvY2tlci5pbyIsImp0aSI6ImRja3JfanRpX19FanZUOWtRS041YVJSZXUybE51WWROQU84VT0iLCJuYmYiOjE2Njg5MzUwOTIsInN1YiI6IiJ9.WcwQ-5wGEUUXsET0cqfWGwk-zAxx5HCSdRPNKi47oM04qeybNolZQejeFYsH-yiEEnQU8X0qoYSrJoy77xMaOAfTw_k2V4bYdKNct16g1WKpkQadd5hubIWShL59JblIhC21gfO30KcVcbsoZiepRKfCwCJ5vVm_Zt8BLsqIySTLnV-Vfyd_jCvSuNNpnOpC2r7ChTGTeUslRfMl2ZK4UGzCIy6rYwgcp4at5d_GLVyjZtS50WAmPlb9xBcawCZE3ulzGc7H9-OcmAtP04GV5exkQ9G34cfGdPvjKmwA1PkD6dQf2nsDE31LxeGqJcWdw7cYyGo31yPFHGJmQ905xQ","access_token":"xxxxxxxxxxxxxxx","expires_in":300,"issued_at":"2022-11-20T09:09:52.106027636Z"}
拿到需要的token，在http header中带上token，重新请求manifests信息

注意这里的实例是一种application/vnd.docker.distribution.manifest.v1+prettyjws类型的manifests信息(具体返回的是什么类型的镜像格式同样可以在http response header: content-type 看出来)
➤ curl <nowiki>https://registry-1.docker.io/v2/library/alpine/manifests/3.17.0_rc1</nowiki> -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsIng1YyI6WyJNSUlDK1RDQ0FwK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakJHTVVRd1FnWURWUVFERXp0U1RVbEdPbEZNUmpRNlEwZFFNenBSTWtWYU9sRklSRUk2VkVkRlZUcFZTRlZNT2taTVZqUTZSMGRXV2pwQk5WUkhPbFJMTkZNNlVVeElTVEFlRncweU1qQXhNVEF5TWpJeE5EbGFGdzB5TXpBeE1qVXlNakl4TkRsYU1FWXhSREJDQmdOVkJBTVRPMUJaUTFJNlNWQmFRanBJUWxGWE9qZE1SVms2UWtGV1FqcEhTRGRhT2xWSVZ6TTZVa3RMVWpwRE4wNHpPbGxOTkZJNlNWaE5TRHBLVkZCQ01JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBbE5JOGFtMlBERnZzNndOeVl2d0dkZWZXVXJXQWdMZ3N2MzA2MnZycG5VNkN0WUpEZDZ2K2NEVG1FN1FlTllEaFMyaU1wU3djZkRCL2RFclEwdnhIZE4ycDIvODZmZy9TeWlIMnhmMGFVTjlDV1dud0JPaTIvS3hLditpbFNlQ01HYXRwRlg3SmYxcWI4N0Q5NUxOVDBvOU9OTmYxT3RidjY5ck9tL1RIVFh3clUvV3dTZlUyWktUbEw4SVRXRkRXN09ZK3hXdUJ0WUpteVhqcVpsaWRBbUNTdTdHY0Y0MVB5em9KTFFTMnJCdXJwOXc0cWgxMFk1bUNIcWdsaEI1Rk9aOUs0T2pUaVhUUHJFUk5WcnArUFVIR3JBYVRPRTBwQzgyUHBuWVZhNzNDUkdsMEdDdC9RckJwVjRpdmswdzF0eEtkV1NiSDNnRmtqZ2g1N0tOcDhRSURBUUFCbzRHeU1JR3ZNQTRHQTFVZER3RUIvd1FFQXdJSGdEQVBCZ05WSFNVRUNEQUdCZ1JWSFNVQU1FUUdBMVVkRGdROUJEdFFXVU5TT2tsUVdrSTZTRUpSVnpvM1RFVlpPa0pCVmtJNlIwZzNXanBWU0Zjek9sSkxTMUk2UXpkT016cFpUVFJTT2tsWVRVZzZTbFJRUWpCR0JnTlZIU01FUHpBOWdEdFNUVWxHT2xGTVJqUTZRMGRRTXpwUk1rVmFPbEZJUkVJNlZFZEZWVHBWU0ZWTU9rWk1WalE2UjBkV1dqcEJOVlJIT2xSTE5GTTZVVXhJU1RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlFQTdIY1VyVm1namo1cE01MXhZVHd2eGE1VnRqd2hub0dRZjFxTU52UGVHeVlDSUFwYm4vWFkvS1F5WWFWRnRjMWtsb0lmZzd4L3hlbkZhbkp4L0F2cURGdFgiXX0.eyJhY2Nlc3MiOlt7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6ImxpYnJhcnkvYWxwaW5lIiwiYWN0aW9ucyI6WyJwdWxsIl0sInBhcmFtZXRlcnMiOnsicHVsbF9saW1pdCI6IjEwMCIsInB1bGxfbGltaXRfaW50ZXJ2YWwiOiIyMTYwMCJ9fV0sImF1ZCI6InJlZ2lzdHJ5LmRvY2tlci5pbyIsImV4cCI6MTY2ODkzNTY5MiwiaWF0IjoxNjY4OTM1MzkyLCJpc3MiOiJhdXRoLmRvY2tlci5pbyIsImp0aSI6ImRja3JfanRpX19FanZUOWtRS041YVJSZXUybE51WWROQU84VT0iLCJuYmYiOjE2Njg5MzUwOTIsInN1YiI6IiJ9.WcwQ-5wGEUUXsET0cqfWGwk-zAxx5HCSdRPNKi47oM04qeybNolZQejeFYsH-yiEEnQU8X0qoYSrJoy77xMaOAfTw_k2V4bYdKNct16g1WKpkQadd5hubIWShL59JblIhC21gfO30KcVcbsoZiepRKfCwCJ5vVm_Zt8BLsqIySTLnV-Vfyd_jCvSuNNpnOpC2r7ChTGTeUslRfMl2ZK4UGzCIy6rYwgcp4at5d_GLVyjZtS50WAmPlb9xBcawCZE3ulzGc7H9-OcmAtP04GV5exkQ9G34cfGdPvjKmwA1PkD6dQf2nsDE31LxeGqJcWdw7cYyGo31yPFHGJmQ905xQ"
{
<nowiki> </nowiki> "schemaVersion": 1,
<nowiki> </nowiki> "name": "library/alpine",
<nowiki> </nowiki> "tag": "3.17.0_rc1",
<nowiki> </nowiki> "architecture": "amd64",
<nowiki> </nowiki> "fsLayers": [
<nowiki> </nowiki> {
<nowiki> </nowiki> "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
<nowiki> </nowiki> },
<nowiki> </nowiki> {
<nowiki> </nowiki> "blobSum": "sha256:7b26c2f269ea232d8dd57ee7696652248b61fda9653d7c7f06a9fea2bd2f009e"
<nowiki> </nowiki> }
<nowiki> </nowiki> ],
<nowiki> </nowiki> "history": [
<nowiki> </nowiki> {
<nowiki> </nowiki> "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\"],\"Image\":\"sha256:118ada46def154efd2774e3211fef3356ad1f268e143dba1136af919d7670b0b\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"container\":\"bcb0af1ba8750c9284eb4ae69a2d444d288ccc718e2f9d329f34bbbd06deaad5\",\"container_config\":{\"Hostname\":\"bcb0af1ba875\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD [\\\"/bin/sh\\\"]\"],\"Image\":\"sha256:118ada46def154efd2774e3211fef3356ad1f268e143dba1136af919d7670b0b\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{}},\"created\":\"2022-11-16T22:19:19.655513526Z\",\"docker_version\":\"20.10.12\",\"id\":\"fecb21f6263ecee0ce8a3d6c50c432b455c23f8afb7ebe929d54f6ca520b032b\",\"os\":\"linux\",\"parent\":\"f163dc23345785e0ac6ebc8d6c6262f19f2565ca420f0cd194c09e1029317881\",\"throwaway\":true}"
<nowiki> </nowiki> },
<nowiki> </nowiki> {
<nowiki> </nowiki> "v1Compatibility": "{\"id\":\"f163dc23345785e0ac6ebc8d6c6262f19f2565ca420f0cd194c09e1029317881\",\"created\":\"2022-11-16T22:19:19.533331062Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:a7bed337c392105c21fe1c7067e51f5d0dedbc488b2be5daf17a18783fa29d04 in / \"]}}"
<nowiki> </nowiki> }
<nowiki> </nowiki> ],
<nowiki> </nowiki> "signatures": [
<nowiki> </nowiki> {
<nowiki> </nowiki> "header": {
<nowiki> </nowiki> "jwk": {
<nowiki> </nowiki> "crv": "P-256",
<nowiki> </nowiki> "kid": "GLVQ:OCEF:4TWE:RS63:BQJ5:VXYZ:EK7Y:DZQG:QNIT:4ZN3:PLT5:CBSK",
<nowiki> </nowiki> "kty": "EC",
<nowiki> </nowiki> "x": "Z4i5ZWH8GizJjOrkPpUAPT1lZH8frbh6XVlZZ6rwXtA",
<nowiki> </nowiki> "y": "EuvtBs7t1sKcHzX2t5WOsXAyT8jnuetPK_IsojxGb0E"
<nowiki> </nowiki> },
<nowiki> </nowiki> "alg": "ES256"
<nowiki> </nowiki> },
<nowiki> </nowiki> "signature": "gJk_3VSshExIjeBeaLhprRpriQMANg5C-5inX-mBexjGHiAKTRjeg2c-JOcHnFpku_rdQFAEJdyLfCBkTFDKeA",
<nowiki> </nowiki> "protected": "eyJmb3JtYXRMZW5ndGgiOjIxMDEsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAyMi0xMS0yMFQwOToxMzoxNVoifQ"
<nowiki> </nowiki> }
<nowiki> </nowiki> ]
}
ps: 对于不存在的tag, hub.docker.com返回的是(http code: 404)
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":"unknown tag=3.17.0_rc2"}]}

=== 题外话: 手动认证请求registry的 /_catalog接口的例子 ===
PS: 注意，有的镜像仓库<code>_catalog</code> 接口认证实现不一样，例如harbor，访问<code>_catalog</code>接口的时候，返回的401请求header里面包含了一个信息<code>www-authenticate: Basic realm="harbor"</code> 这意味着，在harbor中，这个接口只能通过harbor用户密码进行basic认证，而不是token认证。

先匿名请求接口，遇到401
# curl 10.10.217.241/v2/_catalog -v

* About to connect() to 10.10.217.241 port 80 (#0)
* Trying 10.10.217.241...
* Connected to 10.10.217.241 (10.10.217.241) port 80 (#0)
> GET /v2/_catalog HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.10.217.241
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Server: nginx
< Date: Wed, 24 May 2023 10:32:57 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 145
< Connection: keep-alive
< Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="<nowiki>http://10.10.217.241/registry/token</nowiki>",service="BUILDIN-REGISTRY",scope="registry:catalog:*"
< X-Content-Type-Options: nosniff
< Docker-Distribution-Api-Version: registry/2.0
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"registry","Class":"","Name":"catalog","Action":"*"}]}]}
* Connection #0 to host 10.10.217.241 left intact
获取token:

注意这里的service也是需要一一对应的，scope value可以不进行URL编码
# curl "<nowiki>http://10.10.217.241/registry/token?service=BUILDIN-REGISTRY&scope=registry:catalog:*</nowiki>" -u admin:password-password

{
"token":"xxxx-token-x"
}
带上token，成功请求:
# curl 10.10.217.241/v2/_catalog -H "Authorization: Bearer xxxx-token-x"

{"repositories":[xxxxxxxxxxxxxx]}

=== Related articles ===

* [https://mp.weixin.qq.com/s?__biz=MzI3MTI2NzkxMA==&mid=2247493326&idx=1&sn=3e7415f94e8f4f99e0165ab2c089dee9 奇妙的Linux世界 - Docker Hub 镜像拉取有限制？手把手教你破解它！]

* [https://docs.docker.com/registry/spec/auth/token/ docker official doc: Token Authentication Specification]

__强显目录__
[[分类:Linux]]
[[分类:Docker]]

容器进程因io导致cacheMemory较高处理随记

2025-10-22T10:20:17Z

Admin：

就好像harbor的registry容器，因为经常有io，所以比较容易触发这种现象。

或者频繁追加写日志到文件的进程，也会触发这种现象

如果是cgroup2，就可以用cgroup2 的memory.high处理。找到相应的cgroup2路径，配置memory.high即可

以harbor为例

nerdctl ps | awk '/registry-photon/{print "nerdctl inspect --mode native -f \"<nowiki>{{json .Spec.linux.cgroupsPath}}</nowiki>\" "$1}' | sh |cut -c 2-|cut -c -144|awk 'BEGIN{FS=":"};{print "cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/"$1"/"$2"-"$3".scope/memory.high"}'

nerdctl ps | awk '/registry-photon/{print "nerdctl inspect --mode native -f \"<nowiki>{{json .Spec.linux.cgroupsPath}}</nowiki>\" "$1}' | sh |cut -c 2-|cut -c -144|awk 'BEGIN{FS=":"};{print "echo 2G > /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/"$1"/"$2"-"$3".scope/memory.high"}'

[[分类:K8s]]
{{DEFAULTSORT:container容器进程因io导致cacheMemory较高处理随记}}

容器进程因io导致cacheMemory较高处理随记

2025-10-22T10:20:02Z

Admin：

就好像harbor的registry容器，因为经常有io，所以比较容易触发这种现象。

或者频繁追加写日志到文件的进程，也会触发这种现象

如果是cgroup2，就可以用cgroup2 的mem.high处理。找到相应的cgroup2路径，配置mem.high即可

以harbor为例

nerdctl ps | awk '/registry-photon/{print "nerdctl inspect --mode native -f \"<nowiki>{{json .Spec.linux.cgroupsPath}}</nowiki>\" "$1}' | sh |cut -c 2-|cut -c -144|awk 'BEGIN{FS=":"};{print "cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/"$1"/"$2"-"$3".scope/memory.high"}'

nerdctl ps | awk '/registry-photon/{print "nerdctl inspect --mode native -f \"<nowiki>{{json .Spec.linux.cgroupsPath}}</nowiki>\" "$1}' | sh |cut -c 2-|cut -c -144|awk 'BEGIN{FS=":"};{print "echo 2G > /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/"$1"/"$2"-"$3".scope/memory.high"}'

[[分类:K8s]]
{{DEFAULTSORT:container容器进程因io导致cacheMemory较高处理随记}}

容器进程因io导致cacheMemory较高处理随记

2025-10-22T10:19:17Z

Admin：

就好像harbor的registry容器，因为经常有io，所以比较容易触发这种现象。

或者频繁追加写日志到文件的进程，也会触发这种现象

如果是cgroup2，就可以用cgroup2 的mem.high处理。找到相应的cgroup2路径，配置mem.high即可

以harbor为例

<nowiki>
nerdctl ps|grep registry-photon|awk '{print "nerdctl inspect --mode native -f \"{{json .Spec.linux.cgroupsPath}}\" "$1}'|sh|cut -c 2-|cut -c -144|awk 'BEGIN{FS=":"};{print "cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/"$1"/"$2"-"$3".scope/memory.high"}'

nerdctl ps|grep registry-photon|awk '{print "nerdctl inspect --mode native -f \"{{json .Spec.linux.cgroupsPath}}\" "$1}'|sh|cut -c 2-|cut -c -144|awk 'BEGIN{FS=":"};{print "echo 2G > /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/"$1"/"$2"-"$3".scope/memory.high"}'
</nowiki>

[[分类:K8s]]
{{DEFAULTSORT:container容器进程因io导致cacheMemory较高处理随记}}

容器进程因io导致cacheMemory较高处理随记

2025-10-22T10:18:45Z

Admin：创建页面，内容为“就好像harbor的registry容器，因为经常有io，所以比较容易触发这种现象。或者频繁追加写日志到文件的进程，也会触发这种现象如果是cgroup2，就可以用cgroup2 的mem.high处理。找到相应的cgroup2路径，配置mem.high即可以harbor为例 ffffff 分类:K8s {{DEFAULTSORT:container容器进程因io导致cacheMemory较高处理随记}}”

就好像harbor的registry容器，因为经常有io，所以比较容易触发这种现象。

或者频繁追加写日志到文件的进程，也会触发这种现象

如果是cgroup2，就可以用cgroup2 的mem.high处理。找到相应的cgroup2路径，配置mem.high即可

以harbor为例

ffffff
[[分类:K8s]]
{{DEFAULTSORT:container容器进程因io导致cacheMemory较高处理随记}}

Elasticsearch的一些api随记

2025-10-21T03:27:52Z

Admin：

===Health===
/_cat/health

/_cluster/health

===Indices health===
按条件查看索引状态
/_cat/indices?help
/_cat/indices?health=red&v&s=store.size:desc,index

/_cat/indices?health=yellow&v&s=store.size:desc,index

/_cat/indices?health=green&v&s=store.size:desc,index
注意在配合 * 通配符搜索/操作索引的时候，如果涉及隐藏的datastream / 隐藏的index，默认不会匹配命中，必须指定 <code>expand_wildcards=all</code> 参数

-> https://www.elastic.co/guide/en/elasticsearch/reference/7.16/multi-index.html#hidden

===Nodes===
/_cat/nodes?v
查看es各节点磁盘空间占用、分片数目等
/_cat/allocation?v

/_cat/nodeattrs

===Get master node===
/_cat/master?v

===Cluster allocation explain related===
可以用于定位分片状态以及分片为何故障
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

/_cluster/allocation/explain

===Shards===
粗略查看分片情况，特别是查看分片分布节点或大小/状态
GET /_cat/shards

GET /_cat/shards?index=index_name

GET /_cat/shards?index=index_na*
查看分片分配失败原因
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

==== Recovery API ====
Returns information about ongoing and completed shard recoveries, similar to the index recovery API.

For data streams, the API returns information about the stream’s backing indices

可以查看当前正在 relocating 的分片，也能查到各分片处理进度百分比
GET /_cat/recovery?active_only=true&s=index&v

=== Adds a data stream or index to an alias, and sets the write index or data stream for the alias ===
为别名设置可写索引或数据流

If the alias doesn’t exist, the <code>add</code> action creates it.
POST /_aliases
{
"actions": [
{
"add": {
"index": "es-k8s-logs-000020",
"alias": "es-k8s-logs-alias",
"is_write_index": true
}
}
]
}

===Thread pool related===
https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-thread-pool.html
/_cluster/settings?pretty&include_defaults=true | grep processors

====Get maximum number of threads info====
curl "127.1:9200/_cat/thread_pool?v&h=ip,node_name,id,name,max,size,queue_size,queue,active,rejected&pretty"

=== Tasks ===
概览
GET /_cat/tasks?v&s=action,node,start_time
tasks 详情
GET /_tasks

GET /_tasks?group_by=parents&actions=*forcemerge*&detailed=true

=== Templates 模板 ===
/_cat/templates?v
⚠️ /_template/${template_name} is legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.

新版本中使用 <code>/_index_template</code> 取代
GET/PUT /_template/${template_name}
但是 index_template 对比 legacy template有个很明显的差异就是，legacy template可以直接根据priority进行叠加覆盖，而index_template哪个template priority高，就只有哪个生效 https://www.elastic.co/guide/en/elasticsearch/reference/7.17/index-templates.html<blockquote>If a new data stream or index matches more than one index template, the index template with the highest priority is used.</blockquote>

==== Use template to change the replicas settings of all indexes (Legacy index template) ====
Multiple index templates can potentially match an index, in this case, both the settings and mappings are merged into the final configuration of the index.

The order of the merging can be controlled using the <code>order</code> parameter, with lower order being applied first, '''and higher orders overriding them.'''

legacy es template 中, 取值范围为 0 - 2^31-1 (0~2147483647)
PUT /_template/${template_name}
{
"order": 2147483647,
"index_patterns": [
"*"
],
"settings": {
"index": {
"number_of_replicas": "0"
}
}
}
[[使用jq批量修改es index template的lifecycle配置]]

=== ILM (index lifecycle policy) 索引生命周期 ===
顾名思义，ilm另外也可用于做ES集群的冷热温架构。

不同的阶段(phase)能做哪些事可以在这个 [https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html Document] 查看

比较难受的是，ilm目前没有类似 <code>_cat/templates</code> 的接口一次性只查看这个集群已配置的 ILM 策略名字，只能一次性获取全部策略具体定义 (不过可以利用浏览器的F12 json preview折叠来曲线救国)
GET /_ilm/policy

==== Get specific ilm policy detail 获取特定ILM策略定义 ====
GET /_ilm/policy/${ilm_name}

PUT /_ilm/policy/ilm-30d-delete
{
"policy": {
"phases": {
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}

==== Get index's ilm status 获取索引当前 ILM 状态 ====
GET /${index_name}/_ilm/explain

GET /${index_name}/_ilm/explain?human

==== Move index's ilm to step 修改索引的ILM阶段状态(人为触发ILM action执行) ====
https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-move-to-step.html
POST _ilm/move/my-index-000001
{
"current_step": {
"phase": "new",
"action": "complete",
"name": "complete"
},
"next_step": {
"phase": "warm",
"action": "forcemerge",
"name": "forcemerge"
}
}

==== Manually create an index that is managed by template and ILM (including rollover operations by day) ====

===== 手动创建原本应由template和ilm管控的索引，且索引名内包含日期(动态索引名) =====
⚠️ 这种索引不能直接粗暴地 <code>PUT /index-name-2022.10.23-000022</code> 以创建索引，否则手动创建出来的索引，在rollover滚动 (例如rollover-max_age:1d)的时候，创建出来的新索引名字仍然是创建索引时定义的日期，而不是当天轮滚发生时的日期(如 <code>index-name-2022.10.23-000023</code>)

这个现象可以通过判断 <code>GET /index-name/_settings</code> 中 <code>index.provided_name</code> 属性看出来

解法:
PUT %3Cindex-name-%7Bnow%2Fd%7D-000099%3E
注意在kibana-Dev Tools中不要做URL Decode，他就是这样的需要编码一下(解码后就是: <code><index-name-{now/d}-000099></code>)

ps: 如果怕创建错名字的话，可以使用 <code>GET %3Cindex-name-%7Bnow%2Fd%7D-000099%3E/_settings</code> 预览一下生成的索引名效果

对于索引最后的这个序号，[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html#increment-index-names-for-alias 无论前一个索引的名称是什么，该编号始终为 6 个字符，且为零填充]。即使手动创建的索引结尾是<code>-00001</code>，在rollover发生以后，索引后缀序号依然会变成<code>-000002</code>

==== 故障处理: 有的时候对现有的索引修改了其引用的 ilm policy 为别的 policy,或者修改了其引用的 ilm policy中的 phase 定义。会导致索引ilm故障 ====
有可能导致他的ilm处理会出问题（不记得怎么告警），没记错的话通过 <code>GET {index_name}/_ilm/explain</code> 能看到 error 信息，能看到卡在某个 phase 失败

这时候需要人为修改 index 的 ilm phase 修复，如
POST _ilm/move/insight-es-k8s-logs-dce5-aliyun-default-prd-2024.01.16
{
"current_step": {
"phase": "hot",
"action": "rollover",
"name": "ERROR"
},
"next_step": {
"phase": "cold"
}
}
或者尝试通过 <code>POST {index_name}/_ilm/retry</code> 接口重试

===Cluster setting related ES集群参数设置===
/_cluster/settings?include_defaults=true&pretty

/_cluster/settings?include_defaults=true

==== Wildcard expressions or all indices are not allowed ====

===== 允许泛匹配删除索引 =====
PUT /_cluster/settings
{
"persistent": {
"action": {
"destructive_requires_name": "false"
}
}
}

====primaries recovery settings ====
=====控制索引恢复或者relocating的并发数=====
{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 10,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_outgoing_recoveries": null,
"node_concurrent_recoveries": 20
}
}
}
}
}

{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": null,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": null
}
}
}
}
}

{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 30,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": 10
}
}
}
}
}
cluster.routing.allocation.node_concurrent_recoveries: A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>.
# PUT /_cluster/settings
{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_concurrent_recoveries": 8
}
}
}
}
}

=== recovery.max_bytes_per_sec [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/recovery.html 修改relocating时并发传输数据量] ===
加大此数值可以有效缩短es relocating index的耗时

indices.recovery.max_bytes_per_sec: Limits total inbound and outbound recovery traffic for each node. Applies to both peer recoveries as well as snapshot recoveries (i.e., restores from a snapshot). Defaults to <code>40mb</code> unless the node is a dedicated cold or frozen node, in which case the default relates to the total memory available to the node. 

===Index settings===

====modify the number of replicas in bulk====

===== 批量/单个设置索引副本数 =====
PUT /index_name*/_settings
{
"index": {
"number_of_replicas": 1
}
}

=== Search Documents ===
通过指定 <code>_source</code> 可以控制结果只保留某些字段

==== match_all 搜索 ====
GET /sw_segment-20230914/_search
{
"query": {
"match_all": {}
},
"size": 1
}

==== 单字段排序匹配搜索(match) ====
GET /sw_segment-20230914/_search
{
"query": {
"match": {
"segment_id": "b7bb26fae59e4f45b101346cb83ff796.69.16946808855979526"
}
},
"sort": [
{
"start_time": {
"order": "desc"
}
}
],
"size": 1
}

==== 对某个字段的值进行聚合(查询该字段有多少种类型的值) ====
GET /.monitoring-es-7-2024.12.13/_search
{
"size": 0,
"aggs": {
"unique_types": {
"terms": {
"field": "type",
"size": 10
}
}
}
}

==== 对某个字段的值进行计数统计(查询该字段排名前10的值) ====
(top_10_endpoints 可以自定义, 自定义的聚合名)
GET sw_segment-20251021/_search
{
"size": 0,
"aggs": {
"top_10_endpoints": {
"terms": {
"field": "endpoint_id",
"size": 10,
"order": {
"_count": "desc"
}
}
}
}
}

=== Elastic Cloud on Kubernetes (ECK / Elastic operator) ===
ECK operator下管理的Elasticsearch如果要修改<code>cluster.routing.allocation.exclude</code> 的参数配置，需要先为 elasticsearch 实例配置annotation: 'eck.k8s.elastic.co/managed=false'，不然会配置一会就会被刷回原状

===Other===

====对于有大量索引的刚重启的es集群====
(主分片在1w-2w)

=====加快es集群恢复速度=====
结合es节点资源监控图，观测节点cpu压力，以及cpu IO wait

适当通过update cluster settings接口动态增加node_initial_primaries_recoveries (Defaults to <code>4</code>)

和 node_concurrent_recoveries

(A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>

Defaults to <code>2</code>)数值

通过使用 cluster settings + include_defaults=true 筛选查到当前配置值

减少集群从red状态到yellow状态的耗时：增加索引副本数量，增加node_initial_primaries_recoveries值

减少集群从yellow状态到green状态的耗时：增加 node_concurrent_recoveries 值

通过访问 <code>/_cluster/allocation/explain</code> 接口查到阻碍集群 to green(yellow)的原因

=====在es集群恢复期间因节点内存压力大(node was low on resources: memory.)而被k8s Evicted=====
调整缩小 jvm 配置值，尽量不超配(requests 和 limit尽量一致或提高requests值)

=== Error ===

==== 集群分片数达到maximum错误 ====
集群分片数达到maximum错误会有如下log信息，但是集群的健康状态不会改变
2022-11-10T10:26:03.643184618Z org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [3] shards, but this cluster currently has [1999]/[2000] maximum normal shards open;
解决:

调整index ilm 策略或者调整集群的max_shards_per_node配置

临时生效配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"transient": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
永久更改性配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"persistent": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
[[分类:Elasticsearch]]
{{DEFAULTSORT:api随记}}

Elasticsearch的一些api随记

2025-10-21T03:20:29Z

Admin：

===Health===
/_cat/health

/_cluster/health

===Indices health===
按条件查看索引状态
/_cat/indices?help
/_cat/indices?health=red&v&s=store.size:desc,index

/_cat/indices?health=yellow&v&s=store.size:desc,index

/_cat/indices?health=green&v&s=store.size:desc,index
注意在配合 * 通配符搜索/操作索引的时候，如果涉及隐藏的datastream / 隐藏的index，默认不会匹配命中，必须指定 <code>expand_wildcards=all</code> 参数

-> https://www.elastic.co/guide/en/elasticsearch/reference/7.16/multi-index.html#hidden

===Nodes===
/_cat/nodes?v
查看es各节点磁盘空间占用、分片数目等
/_cat/allocation?v

/_cat/nodeattrs

===Get master node===
/_cat/master?v

===Cluster allocation explain related===
可以用于定位分片状态以及分片为何故障
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

/_cluster/allocation/explain

===Shards===
粗略查看分片情况，特别是查看分片分布节点或大小/状态
GET /_cat/shards

GET /_cat/shards?index=index_name

GET /_cat/shards?index=index_na*
查看分片分配失败原因
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

==== Recovery API ====
Returns information about ongoing and completed shard recoveries, similar to the index recovery API.

For data streams, the API returns information about the stream’s backing indices

可以查看当前正在 relocating 的分片，也能查到各分片处理进度百分比
GET /_cat/recovery?active_only=true&s=index&v

=== Adds a data stream or index to an alias, and sets the write index or data stream for the alias ===
为别名设置可写索引或数据流

If the alias doesn’t exist, the <code>add</code> action creates it.
POST /_aliases
{
"actions": [
{
"add": {
"index": "es-k8s-logs-000020",
"alias": "es-k8s-logs-alias",
"is_write_index": true
}
}
]
}

===Thread pool related===
https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-thread-pool.html
/_cluster/settings?pretty&include_defaults=true | grep processors

====Get maximum number of threads info====
curl "127.1:9200/_cat/thread_pool?v&h=ip,node_name,id,name,max,size,queue_size,queue,active,rejected&pretty"

=== Tasks ===
概览
GET /_cat/tasks?v&s=action,node,start_time
tasks 详情
GET /_tasks

GET /_tasks?group_by=parents&actions=*forcemerge*&detailed=true

=== Templates 模板 ===
/_cat/templates?v
⚠️ /_template/${template_name} is legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.

新版本中使用 <code>/_index_template</code> 取代
GET/PUT /_template/${template_name}
但是 index_template 对比 legacy template有个很明显的差异就是，legacy template可以直接根据priority进行叠加覆盖，而index_template哪个template priority高，就只有哪个生效 https://www.elastic.co/guide/en/elasticsearch/reference/7.17/index-templates.html<blockquote>If a new data stream or index matches more than one index template, the index template with the highest priority is used.</blockquote>

==== Use template to change the replicas settings of all indexes (Legacy index template) ====
Multiple index templates can potentially match an index, in this case, both the settings and mappings are merged into the final configuration of the index.

The order of the merging can be controlled using the <code>order</code> parameter, with lower order being applied first, '''and higher orders overriding them.'''

legacy es template 中, 取值范围为 0 - 2^31-1 (0~2147483647)
PUT /_template/${template_name}
{
"order": 2147483647,
"index_patterns": [
"*"
],
"settings": {
"index": {
"number_of_replicas": "0"
}
}
}
[[使用jq批量修改es index template的lifecycle配置]]

=== ILM (index lifecycle policy) 索引生命周期 ===
顾名思义，ilm另外也可用于做ES集群的冷热温架构。

不同的阶段(phase)能做哪些事可以在这个 [https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html Document] 查看

比较难受的是，ilm目前没有类似 <code>_cat/templates</code> 的接口一次性只查看这个集群已配置的 ILM 策略名字，只能一次性获取全部策略具体定义 (不过可以利用浏览器的F12 json preview折叠来曲线救国)
GET /_ilm/policy

==== Get specific ilm policy detail 获取特定ILM策略定义 ====
GET /_ilm/policy/${ilm_name}

PUT /_ilm/policy/ilm-30d-delete
{
"policy": {
"phases": {
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}

==== Get index's ilm status 获取索引当前 ILM 状态 ====
GET /${index_name}/_ilm/explain

GET /${index_name}/_ilm/explain?human

==== Move index's ilm to step 修改索引的ILM阶段状态(人为触发ILM action执行) ====
https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-move-to-step.html
POST _ilm/move/my-index-000001
{
"current_step": {
"phase": "new",
"action": "complete",
"name": "complete"
},
"next_step": {
"phase": "warm",
"action": "forcemerge",
"name": "forcemerge"
}
}

==== Manually create an index that is managed by template and ILM (including rollover operations by day) ====

===== 手动创建原本应由template和ilm管控的索引，且索引名内包含日期(动态索引名) =====
⚠️ 这种索引不能直接粗暴地 <code>PUT /index-name-2022.10.23-000022</code> 以创建索引，否则手动创建出来的索引，在rollover滚动 (例如rollover-max_age:1d)的时候，创建出来的新索引名字仍然是创建索引时定义的日期，而不是当天轮滚发生时的日期(如 <code>index-name-2022.10.23-000023</code>)

这个现象可以通过判断 <code>GET /index-name/_settings</code> 中 <code>index.provided_name</code> 属性看出来

解法:
PUT %3Cindex-name-%7Bnow%2Fd%7D-000099%3E
注意在kibana-Dev Tools中不要做URL Decode，他就是这样的需要编码一下(解码后就是: <code><index-name-{now/d}-000099></code>)

ps: 如果怕创建错名字的话，可以使用 <code>GET %3Cindex-name-%7Bnow%2Fd%7D-000099%3E/_settings</code> 预览一下生成的索引名效果

对于索引最后的这个序号，[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html#increment-index-names-for-alias 无论前一个索引的名称是什么，该编号始终为 6 个字符，且为零填充]。即使手动创建的索引结尾是<code>-00001</code>，在rollover发生以后，索引后缀序号依然会变成<code>-000002</code>

==== 故障处理: 有的时候对现有的索引修改了其引用的 ilm policy 为别的 policy,或者修改了其引用的 ilm policy中的 phase 定义。会导致索引ilm故障 ====
有可能导致他的ilm处理会出问题（不记得怎么告警），没记错的话通过 <code>GET {index_name}/_ilm/explain</code> 能看到 error 信息，能看到卡在某个 phase 失败

这时候需要人为修改 index 的 ilm phase 修复，如
POST _ilm/move/insight-es-k8s-logs-dce5-aliyun-default-prd-2024.01.16
{
"current_step": {
"phase": "hot",
"action": "rollover",
"name": "ERROR"
},
"next_step": {
"phase": "cold"
}
}
或者尝试通过 <code>POST {index_name}/_ilm/retry</code> 接口重试

===Cluster setting related ES集群参数设置===
/_cluster/settings?include_defaults=true&pretty

/_cluster/settings?include_defaults=true

==== Wildcard expressions or all indices are not allowed ====

===== 允许泛匹配删除索引 =====
PUT /_cluster/settings
{
"persistent": {
"action": {
"destructive_requires_name": "false"
}
}
}

====primaries recovery settings ====
=====控制索引恢复或者relocating的并发数=====
{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 10,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_outgoing_recoveries": null,
"node_concurrent_recoveries": 20
}
}
}
}
}

{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": null,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": null
}
}
}
}
}

{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 30,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": 10
}
}
}
}
}
cluster.routing.allocation.node_concurrent_recoveries: A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>.
# PUT /_cluster/settings
{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_concurrent_recoveries": 8
}
}
}
}
}

=== recovery.max_bytes_per_sec [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/recovery.html 修改relocating时并发传输数据量] ===
加大此数值可以有效缩短es relocating index的耗时

indices.recovery.max_bytes_per_sec: Limits total inbound and outbound recovery traffic for each node. Applies to both peer recoveries as well as snapshot recoveries (i.e., restores from a snapshot). Defaults to <code>40mb</code> unless the node is a dedicated cold or frozen node, in which case the default relates to the total memory available to the node. 

===Index settings===

====modify the number of replicas in bulk====

===== 批量/单个设置索引副本数 =====
PUT /index_name*/_settings
{
"index": {
"number_of_replicas": 1
}
}

=== Search Documents ===
通过指定 <code>_source</code> 可以控制结果只保留某些字段

==== match_all 搜索 ====
GET /sw_segment-20230914/_search
{
"query": {
"match_all": {}
},
"size": 1
}

==== 单字段排序匹配搜索(match) ====
GET /sw_segment-20230914/_search
{
"query": {
"match": {
"segment_id": "b7bb26fae59e4f45b101346cb83ff796.69.16946808855979526"
}
},
"sort": [
{
"start_time": {
"order": "desc"
}
}
],
"size": 1
}

==== 对某个字段的值进行聚合(查询该字段有多少种类型的值) ====
GET /.monitoring-es-7-2024.12.13/_search
{
"size": 0,
"aggs": {
"unique_types": {
"terms": {
"field": "type",
"size": 10
}
}
}
}

=== Elastic Cloud on Kubernetes (ECK / Elastic operator) ===
ECK operator下管理的Elasticsearch如果要修改<code>cluster.routing.allocation.exclude</code> 的参数配置，需要先为 elasticsearch 实例配置annotation: 'eck.k8s.elastic.co/managed=false'，不然会配置一会就会被刷回原状

===Other===

====对于有大量索引的刚重启的es集群====
(主分片在1w-2w)

=====加快es集群恢复速度=====
结合es节点资源监控图，观测节点cpu压力，以及cpu IO wait

适当通过update cluster settings接口动态增加node_initial_primaries_recoveries (Defaults to <code>4</code>)

和 node_concurrent_recoveries

(A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>

Defaults to <code>2</code>)数值

通过使用 cluster settings + include_defaults=true 筛选查到当前配置值

减少集群从red状态到yellow状态的耗时：增加索引副本数量，增加node_initial_primaries_recoveries值

减少集群从yellow状态到green状态的耗时：增加 node_concurrent_recoveries 值

通过访问 <code>/_cluster/allocation/explain</code> 接口查到阻碍集群 to green(yellow)的原因

=====在es集群恢复期间因节点内存压力大(node was low on resources: memory.)而被k8s Evicted=====
调整缩小 jvm 配置值，尽量不超配(requests 和 limit尽量一致或提高requests值)

=== Error ===

==== 集群分片数达到maximum错误 ====
集群分片数达到maximum错误会有如下log信息，但是集群的健康状态不会改变
2022-11-10T10:26:03.643184618Z org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [3] shards, but this cluster currently has [1999]/[2000] maximum normal shards open;
解决:

调整index ilm 策略或者调整集群的max_shards_per_node配置

临时生效配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"transient": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
永久更改性配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"persistent": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
[[分类:Elasticsearch]]
{{DEFAULTSORT:api随记}}

Nexus3-运维随记

2025-09-29T09:15:10Z

Admin：

sonatype nexus3

一些orientdb SQL操作记录

=== 基础登陆 ===
java -jar ./lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/nexus-data/db/component admin admin

=== 按照repository 统计空间占用 ===
select bucket.repository_name as repository,sum(size) as bytes from asset group by bucket.repository_name order by bytes desc limit 10;

=== Maven 仓库：按 <code>groupId</code> 汇总 (AI生成) ===
在仓库级别统计
SELECT gid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT attributes.maven2.groupId AS gid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket WHERE repository_name = 'upload-snapshot') AND attributes.maven2.groupId IS NOT NULL ) GROUP BY gid ORDER BY bytes DESC LIMIT 200;

=== Maven 仓库：按 <code>Name</code>(artifactId) 汇总 (AI生成) ===
在全局下按名字统计并过滤仓库:
SELECT repo, gid, aid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT bucket.repository_name AS repo, attributes.maven2.groupId AS gid, attributes.maven2.artifactId AS aid, size FROM asset WHERE bucket.repository_name = ''''upload-snapshot'''<nowiki/>' AND attributes.maven2.groupId IS NOT NULL AND attributes.maven2.artifactId IS NOT NULL ) GROUP BY repo, gid, aid ORDER BY bytes DESC LIMIT 200;

在全局下按名字统计并且不过滤仓库:
SELECT repo, gid, aid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT bucket.repository_name AS repo, attributes.maven2.groupId AS gid, attributes.maven2.artifactId AS aid, size FROM asset WHERE attributes.maven2.groupId IS NOT NULL AND attributes.maven2.artifactId IS NOT NULL ) GROUP BY repo, gid, aid ORDER BY bytes DESC LIMIT 200;

Nexus3-运维随记

2025-09-29T09:13:46Z

Admin：

sonatype nexus3

一些orientdb SQL操作记录

=== 基础登陆 ===
java -jar ./lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/nexus-data/db/component admin admin

=== 按照repository 统计空间占用 ===
select bucket.repository_name as repository,sum(size) as bytes from asset group by bucket.repository_name order by bytes desc limit 10;

=== Maven 仓库：按 <code>groupId</code> 汇总 (AI生成) ===
SELECT gid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT attributes.maven2.groupId AS gid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket WHERE repository_name = 'upload-snapshot') AND attributes.maven2.groupId IS NOT NULL ) GROUP BY gid ORDER BY bytes DESC LIMIT 200;

=== Maven 仓库：按 <code>Name</code>(artifactId) 汇总 (AI生成) ===
按名字统计并过滤仓库:
SELECT repo, gid, aid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT bucket.repository_name AS repo, attributes.maven2.groupId AS gid, attributes.maven2.artifactId AS aid, size FROM asset WHERE bucket.repository_name = ''''upload-snapshot'''<nowiki/>' AND attributes.maven2.groupId IS NOT NULL AND attributes.maven2.artifactId IS NOT NULL ) GROUP BY repo, gid, aid ORDER BY bytes DESC LIMIT 200;

按名字统计并且不过滤仓库:
SELECT repo, gid, aid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT bucket.repository_name AS repo, attributes.maven2.groupId AS gid, attributes.maven2.artifactId AS aid, size FROM asset WHERE attributes.maven2.groupId IS NOT NULL AND attributes.maven2.artifactId IS NOT NULL ) GROUP BY repo, gid, aid ORDER BY bytes DESC LIMIT 200;

Nexus3-运维随记

2025-09-29T09:01:08Z

Admin：

sonatype nexus3

一些orientdb SQL操作记录

=== 基础登陆 ===
java -jar ./lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/nexus-data/db/component admin admin

=== 按照repository 统计空间占用 ===
select bucket.repository_name as repository,sum(size) as bytes from asset group by bucket.repository_name order by bytes desc limit 10;

=== Maven 仓库：按 <code>groupId</code> 汇总 (AI生成) ===
SELECT gid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT attributes.maven2.groupId AS gid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket WHERE repository_name = 'upload-snapshot') AND attributes.maven2.groupId IS NOT NULL ) GROUP BY gid ORDER BY bytes DESC LIMIT 200;

=== Maven 仓库：按 <code>Name</code>(artifactId) 汇总 (AI生成) ===
SELECT gid, aid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT attributes.maven2.groupId AS gid, attributes.maven2.artifactId AS aid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket WHERE repository_name = 'upload-snapshot') AND attributes.maven2.groupId IS NOT NULL AND attributes.maven2.artifactId IS NOT NULL ) GROUP BY gid, aid ORDER BY bytes DESC LIMIT 200;

SELECT bucket, gid, aid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT bucket, attributes.maven2.groupId AS gid, attributes.maven2.artifactId AS aid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket) AND attributes.maven2.groupId IS NOT NULL AND attributes.maven2.artifactId IS NOT NULL ) GROUP BY gid, aid ORDER BY bytes DESC LIMIT 200;

Nexus3-运维随记

2025-09-29T08:58:27Z

Admin：

sonatype nexus3

一些orientdb SQL操作记录

=== 基础登陆 ===
java -jar ./lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/nexus-data/db/component admin admin

=== 按照repository 统计空间占用 ===
select bucket.repository_name as repository,sum(size) as bytes from asset group by bucket.repository_name order by bytes desc limit 10;

=== Maven 仓库：按 <code>groupId</code> 汇总 (AI生成) ===
SELECT gid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT attributes.maven2.groupId AS gid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket WHERE repository_name = 'upload-snapshot') AND attributes.maven2.groupId IS NOT NULL ) GROUP BY gid ORDER BY bytes DESC LIMIT 200;

=== Maven 仓库：按 <code>Name</code>(artifactId) 汇总 (AI生成) ===
SELECT gid, aid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT attributes.maven2.groupId AS gid, attributes.maven2.artifactId AS aid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket WHERE repository_name = 'upload-snapshot') AND attributes.maven2.groupId IS NOT NULL AND attributes.maven2.artifactId IS NOT NULL ) GROUP BY gid, aid ORDER BY bytes DESC LIMIT 200;

Nexus3-运维随记

2025-09-29T04:02:24Z

Admin：

sonatype nexus3

一些orientdb SQL操作记录

=== 基础登陆 ===
java -jar ./lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/nexus-data/db/component admin admin

=== 按照repository 统计空间占用 ===
select bucket.repository_name as repository,sum(size) as bytes from asset group by bucket.repository_name order by bytes desc limit 10;

=== Maven 仓库：按 <code>groupId</code> 汇总 (AI生成) ===
SELECT gid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT attributes.maven2.groupId AS gid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket WHERE repository_name = 'upload-snapshot') AND attributes.maven2.groupId IS NOT NULL ) GROUP BY gid ORDER BY bytes DESC LIMIT 200;

=== Maven 仓库：按 <code>Name</code>(artifactId) 汇总 (AI生成) ===
SELECT gid, aid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT attributes.maven2.groupId AS gid, attributes.maven2.artifactId AS aid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket WHERE repository_name = 'upload-snapshot') AND attributes.maven2.groupId IS NOT NULL AND attributes.maven2.artifactId IS NOT NULL ) GROUP BY gid, aid ORDER BY bytes DESC LIMIT 200;

Nexus3-运维随记

2025-09-29T03:49:15Z

Admin：

Nexus3-运维随记

2025-09-26T11:27:06Z

Admin：

sonatype nexus3

=== 按照repository 统计空间占用 ===
java -jar ./lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/nexus-data/db/component admin admin

select bucket.repository_name as repository,sum(size) as bytes from asset group by bucket.repository_name order by bytes desc limit 10;

=== Maven 仓库：按 <code>groupId</code> 汇总 ===
SELECT gid, SUM(size) AS bytes, COUNT(*) AS files FROM ( SELECT attributes.maven2.groupId AS gid, size FROM asset WHERE bucket IN (SELECT @rid FROM bucket WHERE repository_name = 'upload-snapshot') AND attributes.maven2.groupId IS NOT NULL ) GROUP BY gid ORDER BY bytes DESC LIMIT 200;

Nexus3-运维随记

2025-09-26T10:41:51Z

Admin：创建页面，内容为“sonatype nexus3 === 按照repository 统计空间占用 === java -jar ./lib/support/nexus-orient-console.jar CONNECT PLOCAL:/nexus-data/db/component admin admin select bucket.repository_name as repository,sum(size) as bytes from asset group by bucket.repository_name order by bytes desc limit 10;”

容器主动回收cache-memory随记

2025-09-22T07:38:45Z

Admin：

暂记，待详细测试

通过修改cgroup2 的memory.high 可以触发系统到达特定值时回收内存

影响面待评估。

对那种因为操作文件读写带来的cache memory占用，特别使用，如harbor 的 registry 进程

有意思的文章，关于memory.* : https://support.huaweicloud.com/usermanual-hce/hce_02_0072.html

需要更进一步的测试：

在不人为干预memory.high 的情况下，如果一个容器cache memory 非常高，rss非常低，到达99系统是否会回收？为什么回收？什么参数控制?

内核参数 <code>vm.min_free_kbytes</code> 和 <code>vm.watermark_scale_factor</code> ?

PS: <code>echo 3 > /proc/sys/vm/drop_caches</code> 也可以触发回收，不过是节点级别，不是容器级别，会影响该主机全部进程

[[分类:K8s]]
{{DEFAULTSORT:Memory容器主动回收cache-memory随记}}

容器主动回收cache-memory随记

2025-09-22T07:35:33Z

Admin：创建页面，内容为“暂记，待详细测试通过修改cgroup2 的memory.high 可以触发系统到达特定值时回收内存影响面待评估。对那种因为操作文件读写带来的cache memory占用，特别使用，如harbor 的 registry 进程需要更进一步的测试：在不人为干预memory.high 的情况下，如果一个容器cache memory 非常高，rss非常低，到达99系统是否会回收？为什么回收？什么参数控制? 内核参数 <c…”

暂记，待详细测试

通过修改cgroup2 的memory.high 可以触发系统到达特定值时回收内存

影响面待评估。

对那种因为操作文件读写带来的cache memory占用，特别使用，如harbor 的 registry 进程

需要更进一步的测试：

在不人为干预memory.high 的情况下，如果一个容器cache memory 非常高，rss非常低，到达99系统是否会回收？为什么回收？什么参数控制?

内核参数 <code>vm.min_free_kbytes</code> 和 <code>vm.watermark_scale_factor</code> ?

PS: <code>echo 3 > /proc/sys/vm/drop_caches</code> 也可以触发回收，不过是节点级别，不是容器级别，会影响该主机全部进程

[[分类:K8s]]
{{DEFAULTSORT:Memory容器主动回收cache-memory随记}}

K8s的一些小坑或者bug简要记录

2025-09-18T08:47:52Z

Admin：

__TOC__
=== kubectl===

==== kubectl rollout history====
kubectl rollout history 在 v1.26之前，如果带上-o yaml或者-o json之类的-o 参数，输出的内容会是错误的版本内容

相关Issue https://github.com/kubernetes/kubectl/issues/598#issuecomment-1230824762

==== kubectl apply 在特定情况下可能有bug或者非预期行为====
前提提要: kubectl apply 的工作涉及到了计算行为 [https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#how-apply-calculates-differences-and-merges-changes How apply calculates differences and merges changes]

例如如果在kubectl 1.18，kubectl apply操作hostAliases的时候可能是追加而不是替换 [[在使用kubectl apply操作workload产生的非预期行为|在使用kubectl_apply操作hostalias产生的非预期行为]]

还有一个修改probe配置，apply会有异常的，这个基本也是跟apply计算实现有关（只出现在1.18,不是很记得怎么复现，有缘再补）

kubernetes中apply命令执行的全过程源码解析：https://juejin.cn/post/6968106028642598949

=== kubelet===

==== kubelet 1.27前串行拉取容器镜像====
https://kubernetes.io/docs/concepts/containers/images/#serial-and-parallel-image-pulls

By default, kubelet pulls images serially. In other words, kubelet sends only one image pull request to the image service at a time. Other image pull requests have to wait until the one being processed is complete.

kubernetes 节点上kubelet在1.27版本之前对于容器镜像是串行拉取的，串行值为1，这在拉公网镜像的时候会有可能导致其它容器镜像一直处在拉取状态，在1.27中改成了并行镜像拉取

==== kubelet 不断刷大量的 'Path "/var/lib/kubelet/pods/${pod_ID}/volumes" does not exist' 日志报错 ====
关联原因Issue里面介绍是runc cgroup GC异常

issue:

https://github.com/kubernetes/kubernetes/issues/112124

底部有cgroup清理脚本，但是KUBE_POD_IDS的取值逻辑要根据实际环境调整，而且就算改完了，rmdir cgroup directory会提示Device or resource busy错误

继续关联issue:

https://github.com/kubernetes/kubernetes/issues/112151#issuecomment-1285261341

issue解释诱因: 磁盘IO

==== kubelet 刷 vol_data.json: no such file or directory 日志 ====
报错日志样式:

operationExecutor.UnmountVolume failed

failed to open volume data file [/var/lib/kubelet/pods/${pod_id}/volumes/kubernetes.io~csi/${pvc_id}/vol_data.json]: open /var/lib/kubelet/pods/${pod_id}/volumes/kubernetes.io~csi/${pvc_id}/vol_data.json: no such file or directory

Issue:

https://github.com/kubernetes/kubernetes/issues/85280

里面issue creator有提及: <blockquote>When there is something wrong to execute os.Remove(volPath), volume path is left on node. However, mount path and vol_data.json is deleted.</blockquote>这时候实践下来，可以手动umount，重启kubelet错误即可解除

其他issue中有提及<nowiki/>https://github.com/kubernetes/kubernetes/issues/116847#issuecomment-1721540974<blockquote>Alright one last update. If anyone is running into problems like these, make sure your CSI driver implements <code>NodeUnpublish</code> correctly at very minimum and idempotent. This issue imo is almost entirely caused by problematic CSI driver implementations.</blockquote>

==== k8s-1.18 subPath & configmap 在 configmap 修改以后，对应的容器会无法重启 ====
https://github.com/kubernetes/kubernetes/issues/68211

=== kube-proxy ===
自k8s '''1.24'''开始，kube-proxy不再会默认对service占用的 nodeport进行tcp socket监听，而会全权交给iptables/ipvs负责相应的流量处理。'''所以不要再单纯依靠ss -nplt / netstat -antl 来判定一个svc是否监听正常。可以依靠tcpdump等手段判定流量已经进入到了本节点'''

https://github.com/kubernetes/kubernetes/pull/108496
[[分类:K8s]]

Kubectl随记

2025-08-22T10:49:27Z

Admin：

=== kubectl set default editor ===
export KUBE_EDITOR=/usr/bin/vim

===kubectl completion bash not working 排障随记===
需要已经安装 bash-completion

如果提示 <code>_get_comp_words_by_ref: command not found</code> 错误的话，需要执行
source /usr/share/bash-completion/bash_completion
执行命令 <code>set</code> 看一下当前环境的相关配置有没有kube相关项
set | grep -i kube
没有的话需要执行
source <(kubectl completion bash)

正常的话,打开 kubectl debug
__kubectl_debug() {
if [[ -n ${BASH_COMP_DEBUG_FILE} ]]; then
echo "$*" >> "${BASH_COMP_DEBUG_FILE}"
fi
}

export BASH_COMP_DEBUG_FILE=****
根据debug file里面的记录去排查出错点

=== kubectl to get all api-resource and filter by verbs ===
kubectl api-resources -o name --cached --request-timeout=5s --verbs=get

kubectl api-resources --api-group=networking.k8s.io

==== get groupVersion when a resource exists in multiple api groups ====

==== explain api-resource with sprcific api version ====
kubectl api-resources --api-group=networking.k8s.io -v8 2>&1| grep ingress

kubectl explain ingress --api-version="networking.k8s.io/v1beta1"

=== get service account token faster / 一条命令结合jsonpath揭秘base64处理后的secret ===
kubectl get secret sa-secret -o jsonpath={.data.token} | base64 -d

=== Check k8s account privilege / k8s 对象权限相关 ===
kubectl options:
--as=: Username to impersonate for the operation
--as-group=[]: Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
假设有名为registry-test的serviceAccount位于kube-system，这时候我们要测试他的RBAC权限，可以使用如下命令
kubectl --as "system:serviceaccount:kube-system:registry-test" get serviceaccount

Usage: kubectl auth can-i VERB [TYPE | TYPE/NAME | NONRESOURCEURL] [options]

# Check to see if I can do everything in my current namespace ("*" means all)
kubectl auth can-i '*' '*'

[root@node-a ~]# kubectl --as "system:serviceaccount:kube-system:registry-test" auth can-i list pods
yes
PS: 在RBAC定义中(例如role / clusterrole)对 <code>resources</code> 的 <code>pods/*</code> 声明不会对诸如 <code>pods/exec pods/log pods/status</code> 等资源授权起效果

原因: [[K8s-rbac关于子资源授权的一些记录]]

===kubectl get resources with custom column / 利用custom-columns自定义输出字段 (语法相对go-template简单)===

=====通过custom-columns只输出pod所在租户，pod名字和pod uid=====
kubectl get pods -o custom-columns='namespace:metadata.namespace,pod:metadata.name,uid:metadata.uid'

=====通过custom-columns只输出namespace名字和ns annotation中定义的node selector=====
kubectl get namespaces -o custom-columns="NAMESPACE:.metadata.name, NODE_SELECTOR:.metadata.annotations.scheduler\.alpha\.kubernetes\.io/node-selector"

===Get node info order by node ip address / 根据ip地址排序获取节点信息===
kubectl get nodes -owide --sort-by status.addresses[0].address

===Get pods order by running node name / 根据pod运行节点排序获取pod信息===
kubectl get pods -owide --sort-by .spec.nodeName

kubectl get pods -owide --sort-by spec.nodeName
===Get pod order by pod create time / 根据pod创建时间排序获取pod===
kubectl get pods -n efk-system --sort-by status.startTime

kubectl get pods -n efk-system --sort-by status.startTime| grep filebeat

kubectl get pods -n efk-system --sort-by status.startTime| grep filebeat|tac

===kubectl get resources with go-template / 利用go-template自定义输出(功能扩展性高度自由,但是有一定的编码调试成本)===

====Deployments - image / 只获取deployment名字和image====
<nowiki>kubectl get deployments -o go-template --template '{{range .items}}{{.metadata.name}}{{" -- "}}{{range .spec.template.spec.containers}}{{.image}}{{" "}}{{end}}{{"\n"}}{{end}}'</nowiki>
稍微格式化输出，输出deployment名字 + container名字 + 对应的image
kubectl get deployments.apps -o go-template --template '<nowiki>{{range .items}}</nowiki><nowiki>{{printf "%-30s " .metadata.name}}</nowiki><nowiki>{{range .spec.template.spec.containers}}</nowiki><nowiki>{{.name}}</nowiki><nowiki>{{":"}}</nowiki><nowiki>{{.image}}</nowiki><nowiki>{{" "}}</nowiki><nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

====Pods - image / 只获取pod名字和image====
<nowiki>kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{" -- "}}{{range .spec.containers}}{{.image}}{{end}}{{"\n"}}{{end}}'</nowiki>
====Get the ip address of the specified node / 只获取节点名字和对应的Internal IP地址====
<nowiki>kubectl get nodes -l kubernetes.io/hostname=nodename -o go-template --template '{{range .items}}{{range .status.addresses}}{{ if eq .type "InternalIP" }}{{.address}}{{end}}{{end}}{{end}}{{"\n"}}'</nowiki>
====Get node taints / 只获取节点名字和taints====
<nowiki>kubectl get nodes -o go-template --template '{{range .items}}{{.metadata.name}}{{":\n"}}{{range .spec.taints}}{{.key}}{{"="}}{{.value}}{{":"}}{{.effect}}{{" "}}{{end}}{{"\n\n"}}{{end}}'</nowiki>

<nowiki>kubectl get nodes -l kubernetes.io/hostname=nodename -o go-template --template '{{range .items}}{{.metadata.name}}{{":\n"}}{{range .spec.taints}}{{.key}}{{"="}}{{.value}}{{":"}}{{.effect}}{{" "}}{{end}}{{"\n\n"}}{{end}}'</nowiki>

==== Get node labels and format output / 只获取节点名字和labels并格式化输出 ====
kubectl get node -o go-template --template '<nowiki>{{range .items}}</nowiki><nowiki>{{.metadata.name}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{range $key, $value := .metadata.labels}}</nowiki><nowiki>{{"\t\t"}}</nowiki><nowiki>{{$key}}</nowiki><nowiki>{{":"}}</nowiki><nowiki>{{$value}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

==== 获取namespace名字和ns annotation中定义的node selector ====
kubectl get namespaces -o go-template='<nowiki>{{range .items}}</nowiki><nowiki>{{.metadata.name}}</nowiki><nowiki>{{" "}}</nowiki><nowiki>{{index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector" -}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

===== 稍微处理一下，格式化对齐输出 =====
kubectl get namespaces -o go-template='<nowiki>{{range .items}}</nowiki><nowiki>{{printf "%-30s " .metadata.name}}</nowiki><nowiki>{{if index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector"}}</nowiki><nowiki>{{index .metadata.annotations "scheduler.alpha.kubernetes.io/node-selector"}}</nowiki><nowiki>{{else}}</nowiki> <nowiki>{{end}}</nowiki><nowiki>{{"\n"}}</nowiki><nowiki>{{end}}</nowiki>'

===kubectl get event filter by pod name / 根据对象进行过滤获取k8s事件===
kubectl get event -n kube-system --field-selector involvedObject.name=${pod_name}

===kubectl get event sort by lastTime / 根据事件最后触发时间进行排序===
kubectl get event -n kube-system --sort-by=.lastTimestamp

=== kubectl patch ===
[[K8s的patch命令随记]]

kubectl patch 调整 annotation
<nowiki>kubectl patch ns demo -p '{"metadata":{"annotations":{"scheduler.alpha.kubernetes.io/node-selector": "kubernetes.io/hostname=node1"}}}'</nowiki>
[[分类:K8s]]

使用jq批量修改es index template的lifecycle配置

2025-08-22T08:56:07Z

Admin：

aka: use jq tool to modify es index template lifecycle setting in bulk

新版本的es index template不支持通过结合索引匹配+优先级配置的方式来叠加批量修改已存在的索引模版(legacy template 支持)，只支持通过compose实现类似用法，部分场景会带来大量的搬砖成本(例如修改skywalking的索引模板中的生命周期配置或者索引副本数配置)

引入脚本:<pre>
<nowiki>
#!/bin/bash
#set -exu
set -ue
host="127.0.0.1:9200"
username="elastic"
password="elastic"
ilm_name="sw-default-ilm"

curl_options=(
"-u" "$username:$password"
"-s"
)

es_sw_templates=`curl ${curl_options[@]} "$host/_cat/templates/sw*?s=name" | awk '{print $1}'`
# bak
curl ${curl_options[@]} "${host}/_index_template?pretty" > es-${host}-index-templates_`date +%F%T`.bak.json

for template in ${es_sw_templates[@]}
do
if [[ $template == "sw_segment" ]]
then
continue
fi
echo PUT /_index_template/${template}
new_template=""
new_template=`curl ${curl_options[@]} "${host}/_index_template/${template}" | jq '.index_templates[0].index_template | .template.settings.index.lifecycle.name |= "'${ilm_name}'"'`
curl -X PUT -H "Content-Type: application/json" ${curl_options[@]} "${host}/_index_template/${template}" -d "${new_template}"
echo
done
</nowiki></pre>
[[分类:Elasticsearch]]
[[分类:Linux]]
也可以将修改 lifecycle.name 改为修改 compose 以及修改 refresh_interval
new_template=`curl ${curl_options[@]} "${host}/_index_template/${template}" | jq '.index_templates[0].index_template | .composed_of = ["sw_default_ilm"] | .template.settings.index.refresh_interval = "180s"'`
{{DEFAULTSORT:jq}}

Ansible随记

2025-08-06T06:42:04Z

Admin：

For linux and kubernetes

 

===Hosts config===
[cluster1_controller]
10.219.235.210
10.219.235.213
10.219.235.214

[cluster1_compute]
10.219.235.[215:219]

[cluster1_nodes:children]
cluster1_controller
cluster1_compute

[cluster1_nodes:vars]
ansible_ssh_user=guest2admin
ansible_ssh_pass=@users
ansible_python_interpreter=/usr/bin/python
ansible_ssh_common_args=-c aes256-cbc -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
ansible_connection=ssh
ansible_become_password=root@root@su
ansible_become_method=su
ansible_become_exe=sudo su -

#############################################

[k8s]
k8s-node-1 ansible_ssh_host=172.16.139.102
k8s-node-2 ansible_ssh_host=172.16.139.103
k8s-node-3 ansible_ssh_host=172.16.139.104

[test]
192.168.1.250 ansible_ssh_port=1234
192.168.1.251 ansible_ssh_user=xxx ansible_ssh_pass=yyy

[other-test]
192.168.1.250:1234

 

===Ansible command parameters===
-f 'FORKS', --forks 'FORKS'
specify number of parallel processes to use (default=5)
 
===Module===

====authorized_key 模块实现节点批量免密====

===== module 简要参数说明: =====
user=root : 将密钥推送到远程主机的哪个用户下

key=’<nowiki>{{ lookup('file', '/root/.ssh/authorized_keys')}}</nowiki>’ : 指定要推送的公钥文件所在的路径(常用应该是 id_rsa.pub )

path=’/root/.ssh/authorized_keys’ : 将密钥推送到远程主机的哪个目录下并重命名 [Default: (user home dir)+/.ssh/authorized_keys]

manage_dir=no : 指定模块是否应该管理 authorized key 文件所在的目录。如果设置为 yes，模块会创建目录，以及设置一个已存在目录的拥有者和权限。如果通过 path 选项，重新指定了一个 authorized key 文件所在目录，那么应该将该选项设置为 no

exclusive : 是否移除 authorized_keys 文件中其它非指定 key [default: no]

state (Choices: present, absent) : present 添加指定 key 到 authorized_keys 文件中；absent 从 authorized_keys 文件中移除指定 key [Default: present]

-k : 本次操作通过密码认证节点

ps: 如果是第一次ssh连接对端节点，可以考虑 ssh-keyscan -H ${node_ip} >> ~/.ssh/known_hosts 批量添加节点ssh指或者直接将ansible host_key_checking 设置为False (ssh-keyscan 中 -H参数代表将相关指纹和主机名结果都进行哈希化加密)

===== command line example: =====
ansible all -m authorized_key -a "user=root key='<nowiki>{{ lookup('file', '/root/.ssh/authorized_keys')}}</nowiki>' path='/root/.ssh/authorized_keys' manage_dir=no" -k
 
====selinux====
ansible k8s -m selinux -m selinux -a state=disabled
https://my.oschina.net/ozakilsc/blog/693023
 
====shell====
ansible k8s -m shell -a getenforce

ansible k8s -m shell -a hostname

ansible k8s -m shell -a "iptables -F && iptables -X && iptables -F -t nat && iptables -t nat -X && iptables -t raw -F && iptables -t raw -X && iptables -t mangle -F && iptables -t mangle -X"

ansible k8s -m shell -a "modprobe bridge && modprobe br_netfilter && sysctl -p /etc/sysctl.d/kubernetes.conf"

ansible k8s -m shell -a "timedatectl set-timezone Asia/Shanghai && timedatectl status"

ansible all -m shell -a "rpm -Uvh <nowiki>http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm</nowiki>"

ansible all -m shell -a "yum --enablerepo=elrepo-kernel install -y kernel-lt"

ansible all -m shell -a "grub2-set-default 0"
自动切到root用户(-b)
ansible all -b -m shell -a "cat <<< \$(jq '. + {\"bip\": \"192.168.0.1/28\"}' /etc/docker/daemon.json) > /etc/docker/daemon.json"
如果需要强制指定 shell 为 bash shell，可以如下
ansible all -m shell -e 'ansible_shell_executable=/usr/bin/bash' -a "ls"

====ping====
(用于判断远程客户端是否在线)
ansible k8s -m ping

====command====
(ansible default module) 

 

====yum====
(default state:installed)
ansible k8s -m yum -a 'name=vim state=installed'

ansible k8s -m yum -a 'name=vim'

ansible k8s -m yum -a 'name=vim, httpd'

ansible k8s -km yum -a "name=yum-utils,chrony,conntrack,ipvsadm,ipset,jq,iptables,curl,sysstat,libseccomp,wget,socat,git"

ansible k8s -m yum -a 'name=vsftpd state=removed'

ansible k8s -m yum -a "name=bridge-utils"

ansible all -m yum -a "name=epel-release,chrony,conntrack,ipvsadm,ipset,jq,iptables,curl,sysstat,libseccomp,wget,socat,git,bind-utils state=installed"
 

====service====
ansible k8s -m service -a " name='nginx' enabled=yes"

ansible k8s -m service -a "name=httpd state=started"

ansible k8s -m service -a "name=firewalld state=stopped enabled=no"

ansible k8s -km service -a "name=postfix state=stopped enabled=no"

ansible k8s -m service -a "name=chronyd enabled=yes state=started"

====copy====
ansible k8s -m copy -a "src=./kubernetes.conf dest=/etc/sysctl.d/"

ansible all -m copy -a "src=./authorized_keys dest=~/.ssh/authorized_keys mode=600" -k

====file====
ansible k8s -m file -a "path=/opt/k8s/bin state=directory"

ansible k8s -m file -a "path=/opt/k8s/work state=directory"

ansible k8s -m file -a "path=/opt/k8s/work state=absent"

====others====
ansible k8s -m shell -a "rpm --import file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7"

===Playbook===

====copy====
---
- hosts: all
<nowiki> </nowiki> tasks:
<nowiki> </nowiki> - name: copy kubernetes server executeable file to master node
<nowiki> </nowiki> copy:
<nowiki> </nowiki> src: '<nowiki>{{ item.src }}</nowiki>'
<nowiki> </nowiki> dest: '<nowiki>{{item.dest}}</nowiki>'
<nowiki> </nowiki> mode: '0744'
<nowiki> </nowiki> with_items:
<nowiki> </nowiki> - {src: './apiextensions-apiserver', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kubeadm', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kube-apiserver', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kube-controller-manager', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kubectl', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kubelet', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kube-proxy', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kube-scheduler', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './mounter', dest: '/opt/k8s/bin/'}

[[分类:Linux]]
 

===Other operations===

====use the passphrase protected ssh private key without input the password====
使用带密码保护的私钥并且避免每次都要输入私钥密码(本次会话生效)
$ eval `ssh-agent` # you might have agent already running so this might not be needed
$ ssh-add ~/.ssh/id_rsa
 
__无编辑段落__
__无新段落链接__

Ansible随记

2025-08-06T06:41:40Z

Admin：

For linux and kubernetes

 

===Hosts config===
[cluster1_controller]
10.219.235.210
10.219.235.213
10.219.235.214

[cluster1_compute]
10.219.235.[215:219]

[cluster1_nodes:children]
cluster1_controller
cluster1_compute

[cluster1_nodes:vars]
ansible_ssh_user=guest2admin
ansible_ssh_pass=@users
ansible_python_interpreter=/usr/bin/python
ansible_ssh_common_args=-c aes256-cbc -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
ansible_connection=ssh
ansible_become_password=root@root@su
ansible_become_method=su
ansible_become_exe=sudo su -

#############################################

[k8s]
k8s-node-1 ansible_ssh_host=172.16.139.102
k8s-node-2 ansible_ssh_host=172.16.139.103
k8s-node-3 ansible_ssh_host=172.16.139.104

[test]
192.168.1.250 ansible_ssh_port=1234
192.168.1.251 ansible_ssh_user=xxx ansible_ssh_pass=yyy

[other-test]
192.168.1.250:1234

 

===Ansible command parameters===
-f 'FORKS', --forks 'FORKS'
specify number of parallel processes to use (default=5)
 
===Module===

====authorized_key 模块实现节点批量免密====

===== module 简要参数说明: =====
user=root : 将密钥推送到远程主机的哪个用户下

key=’<nowiki>{{ lookup('file', '/root/.ssh/authorized_keys')}}</nowiki>’ : 指定要推送的公钥文件所在的路径(常用应该是 id_rsa.pub )

path=’/root/.ssh/authorized_keys’ : 将密钥推送到远程主机的哪个目录下并重命名 [Default: (user home dir)+/.ssh/authorized_keys]

manage_dir=no : 指定模块是否应该管理 authorized key 文件所在的目录。如果设置为 yes，模块会创建目录，以及设置一个已存在目录的拥有者和权限。如果通过 path 选项，重新指定了一个 authorized key 文件所在目录，那么应该将该选项设置为 no

exclusive : 是否移除 authorized_keys 文件中其它非指定 key [default: no]

state (Choices: present, absent) : present 添加指定 key 到 authorized_keys 文件中；absent 从 authorized_keys 文件中移除指定 key [Default: present]

-k : 本次操作通过密码认证节点

ps: 如果是第一次ssh连接对端节点，可以考虑 ssh-keyscan -H ${node_ip} >> ~/.ssh/known_hosts 批量添加节点ssh指或者直接将ansible host_key_checking 设置为False (ssh-keyscan 中 -H参数代表将相关指纹和主机名结果都进行哈希化加密)

===== command line example: =====
ansible all -m authorized_key -a "user=root key='<nowiki>{{ lookup('file', '/root/.ssh/authorized_keys')}}</nowiki>' path='/root/.ssh/authorized_keys' manage_dir=no" -k
 
====selinux====
ansible k8s -m selinux -m selinux -a state=disabled
https://my.oschina.net/ozakilsc/blog/693023
 
====shell====
ansible k8s -m shell -a getenforce

ansible k8s -m shell -a hostname

ansible k8s -m shell -a "iptables -F && iptables -X && iptables -F -t nat && iptables -t nat -X && iptables -t raw -F && iptables -t raw -X && iptables -t mangle -F && iptables -t mangle -X"

ansible k8s -m shell -a "modprobe bridge && modprobe br_netfilter && sysctl -p /etc/sysctl.d/kubernetes.conf"

ansible k8s -m shell -a "timedatectl set-timezone Asia/Shanghai && timedatectl status"

ansible all -m shell -a "rpm -Uvh <nowiki>http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm</nowiki>"

ansible all -m shell -a "yum --enablerepo=elrepo-kernel install -y kernel-lt"

ansible all -m shell -a "grub2-set-default 0"
自动切到root用户(-b)
ansible all -b -m shell -a "cat <<< \$(jq '. + {\"bip\": \"192.168.0.1/28\"}' /etc/docker/daemon.json) > /etc/docker/daemon.json"
如果需要强制指定 shell 为 bash shell，可以如下
ansible all -e 'ansible_shell_executable=/usr/bin/bash' -a "ls"

====ping====
(用于判断远程客户端是否在线)
ansible k8s -m ping

====command====
(ansible default module) 

 

====yum====
(default state:installed)
ansible k8s -m yum -a 'name=vim state=installed'

ansible k8s -m yum -a 'name=vim'

ansible k8s -m yum -a 'name=vim, httpd'

ansible k8s -km yum -a "name=yum-utils,chrony,conntrack,ipvsadm,ipset,jq,iptables,curl,sysstat,libseccomp,wget,socat,git"

ansible k8s -m yum -a 'name=vsftpd state=removed'

ansible k8s -m yum -a "name=bridge-utils"

ansible all -m yum -a "name=epel-release,chrony,conntrack,ipvsadm,ipset,jq,iptables,curl,sysstat,libseccomp,wget,socat,git,bind-utils state=installed"
 

====service====
ansible k8s -m service -a " name='nginx' enabled=yes"

ansible k8s -m service -a "name=httpd state=started"

ansible k8s -m service -a "name=firewalld state=stopped enabled=no"

ansible k8s -km service -a "name=postfix state=stopped enabled=no"

ansible k8s -m service -a "name=chronyd enabled=yes state=started"

====copy====
ansible k8s -m copy -a "src=./kubernetes.conf dest=/etc/sysctl.d/"

ansible all -m copy -a "src=./authorized_keys dest=~/.ssh/authorized_keys mode=600" -k

====file====
ansible k8s -m file -a "path=/opt/k8s/bin state=directory"

ansible k8s -m file -a "path=/opt/k8s/work state=directory"

ansible k8s -m file -a "path=/opt/k8s/work state=absent"

====others====
ansible k8s -m shell -a "rpm --import file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7"

===Playbook===

====copy====
---
- hosts: all
<nowiki> </nowiki> tasks:
<nowiki> </nowiki> - name: copy kubernetes server executeable file to master node
<nowiki> </nowiki> copy:
<nowiki> </nowiki> src: '<nowiki>{{ item.src }}</nowiki>'
<nowiki> </nowiki> dest: '<nowiki>{{item.dest}}</nowiki>'
<nowiki> </nowiki> mode: '0744'
<nowiki> </nowiki> with_items:
<nowiki> </nowiki> - {src: './apiextensions-apiserver', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kubeadm', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kube-apiserver', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kube-controller-manager', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kubectl', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kubelet', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kube-proxy', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './kube-scheduler', dest: '/opt/k8s/bin/'}
<nowiki> </nowiki> - {src: './mounter', dest: '/opt/k8s/bin/'}

[[分类:Linux]]
 

===Other operations===

====use the passphrase protected ssh private key without input the password====
使用带密码保护的私钥并且避免每次都要输入私钥密码(本次会话生效)
$ eval `ssh-agent` # you might have agent already running so this might not be needed
$ ssh-add ~/.ssh/id_rsa
 
__无编辑段落__
__无新段落链接__

一些routeros-script折腾随记

2025-07-28T07:18:08Z

Admin：Admin移动页面一些mikrotik-routeros-script折腾随记至一些routeros-script折腾随记，不留重定向

{{DISPLAYTITLE:routeros-script折腾随记}}

第三方cn docs: https://mikrotik-doc-cn.readthedocs.io/zh/latest/source/Scripts/content.html

=== 常用操作: ===
变量定义: 可以通过 <code>:local</code> 或者 <code>:global</code> 定义，区别就是作用域不一样
:local pppoeInterface "pppoe-sfp-telecom"
:local pppoeUptime
<code>:put</code> 可以将定义好的变量输出到console，但是不能输出到log

一般情况下，Scripts 在运行的时候，如果出现异常，不会有异常信息，只有简单的一句话:<blockquote>script,error executing script script-test from xxxxxxx failed, please check it manually</blockquote>可以通过类似以下方式把 exception (e) 打到log中方便排查
:onerror e {
:local pppoeInterface "pppoe-sfp-telecom"
:local pppoeUptime
[/interface/pppoe-client/monitor $pppoeInterface once do={:set pppoeUptime $uptime} ]
:log info "PPPoE interface uptime: $pppoeUptime"

} do {:log info "script running error: $e"}

some example:

用于判断 pppoe 连接时间，如果不在 3-5 点，就运行指定的脚本
:onerror e {
:local pppoeInterface "pppoe-sfp-telecom"
:local pppoeUptime
:local pppoeUptimeHour
:local dayPos
:local hourSemicolonPos
:local timeCutPosStart 0

[/interface/pppoe-client/monitor $pppoeInterface once do={:set pppoeUptime $uptime} ]
:log info "PPPoE interface uptime: $pppoeUptime"

:set dayPos [:find $pppoeUptime "d"]
:if ($dayPos >0) do={
:set timeCutPosStart ($dayPos + 1)
#:log debug "day pos found."
}
:log info "timeCutPosStart: $timeCutPosStart"

:set hourSemicolonPos [:find $pppoeUptime ":"]
:set pppoeUptimeHour [:pick $pppoeUptime $timeCutPosStart $hourSemicolonPos]
:log info "pppoeUptimeHour: $pppoeUptimeHour"

:if ( $pppoeUptimeHour > 5 or $pppoeUptimeHour < 3 ) do={
:log info "PPPOE connection uptime not as expected! will be restarted..."
/interface/pppoe-client/disable $pppoeInterface
:log info "$pppoeInterface disabled"
delay 3
/interface/pppoe-client/enable $pppoeInterface
:log info "$pppoeInterface enabled"
}

} do {:log info "script running error: $e"}

关联关键词: mikrotik
[[分类:Router]]

一些routeros-script折腾随记

2025-07-28T07:17:10Z

Admin：

一些routeros-script折腾随记

2025-07-28T07:14:59Z

Admin：Admin移动页面一些routeros-script折腾随记至一些mikrotik-routeros-script折腾随记，不留重定向

Kubectl edit或get cm格式混乱

2025-07-24T11:24:48Z

Admin：

configmap中的内容需要同时满足以下条件以后，在kubectl edit/get cm yaml下才不会被二次转义编码

# 换行前不要有空白字符 <pre>:%s/\\s\\n/\\n/g</pre>
# 没有tab键(\t) <pre>:%s/\\t/ /g</pre>
# 文尾没有换行符或空白字符(整个data object的最后一个字符不能是\n)

PS: 通过 <code>kubectl get cm xxxx -o jsonpath={.data}</code> 也可以避免输出的内容被二次转义编码
[[分类:K8s]]

Elasticsearch的一些api随记

2025-07-01T03:52:05Z

Admin：

===Health===
/_cat/health

/_cluster/health

===Indices health===
按条件查看索引状态
/_cat/indices?help
/_cat/indices?health=red&v&s=store.size:desc,index

/_cat/indices?health=yellow&v&s=store.size:desc,index

/_cat/indices?health=green&v&s=store.size:desc,index
注意在配合 * 通配符搜索/操作索引的时候，如果涉及隐藏的datastream / 隐藏的index，默认不会匹配命中，必须指定 <code>expand_wildcards=all</code> 参数

-> https://www.elastic.co/guide/en/elasticsearch/reference/7.16/multi-index.html#hidden

===Nodes===
/_cat/nodes?v
查看es各节点磁盘空间占用、分片数目等
/_cat/allocation?v

/_cat/nodeattrs

===Get master node===
/_cat/master?v

===Cluster allocation explain related===
可以用于定位分片状态以及分片为何故障
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

/_cluster/allocation/explain

===Shards===
粗略查看分片情况，特别是查看分片分布节点或大小/状态
GET /_cat/shards

GET /_cat/shards?index=index_name

GET /_cat/shards?index=index_na*
查看分片分配失败原因
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

==== Recovery API ====
Returns information about ongoing and completed shard recoveries, similar to the index recovery API.

For data streams, the API returns information about the stream’s backing indices

可以查看当前正在 relocating 的分片，也能查到各分片处理进度百分比
GET /_cat/recovery?active_only=true&s=index&v

=== Adds a data stream or index to an alias, and sets the write index or data stream for the alias ===
为别名设置可写索引或数据流

If the alias doesn’t exist, the <code>add</code> action creates it.
POST /_aliases
{
"actions": [
{
"add": {
"index": "es-k8s-logs-000020",
"alias": "es-k8s-logs-alias",
"is_write_index": true
}
}
]
}

===Thread pool related===
https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-thread-pool.html
/_cluster/settings?pretty&include_defaults=true | grep processors

====Get maximum number of threads info====
curl "127.1:9200/_cat/thread_pool?v&h=ip,node_name,id,name,max,size,queue_size,queue,active,rejected&pretty"

=== Tasks ===
概览
GET /_cat/tasks?v&s=action,node,start_time
tasks 详情
GET /_tasks

GET /_tasks?group_by=parents&actions=*forcemerge*&detailed=true

=== Templates 模板 ===
/_cat/templates?v
⚠️ /_template/${template_name} is legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.

新版本中使用 <code>/_index_template</code> 取代
GET/PUT /_template/${template_name}
但是 index_template 对比 legacy template有个很明显的差异就是，legacy template可以直接根据priority进行叠加覆盖，而index_template哪个template priority高，就只有哪个生效 https://www.elastic.co/guide/en/elasticsearch/reference/7.17/index-templates.html<blockquote>If a new data stream or index matches more than one index template, the index template with the highest priority is used.</blockquote>

==== Use template to change the replicas settings of all indexes (Legacy index template) ====
Multiple index templates can potentially match an index, in this case, both the settings and mappings are merged into the final configuration of the index.

The order of the merging can be controlled using the <code>order</code> parameter, with lower order being applied first, '''and higher orders overriding them.'''

legacy es template 中, 取值范围为 0 - 2^31-1 (0~2147483647)
PUT /_template/${template_name}
{
"order": 2147483647,
"index_patterns": [
"*"
],
"settings": {
"index": {
"number_of_replicas": "0"
}
}
}
[[使用jq批量修改es index template的lifecycle配置]]

=== ILM (index lifecycle policy) 索引生命周期 ===
顾名思义，ilm另外也可用于做ES集群的冷热温架构。

不同的阶段(phase)能做哪些事可以在这个 [https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html Document] 查看

比较难受的是，ilm目前没有类似 <code>_cat/templates</code> 的接口一次性只查看这个集群已配置的 ILM 策略名字，只能一次性获取全部策略具体定义 (不过可以利用浏览器的F12 json preview折叠来曲线救国)
GET /_ilm/policy

==== Get specific ilm policy detail 获取特定ILM策略定义 ====
GET /_ilm/policy/${ilm_name}

PUT /_ilm/policy/ilm-30d-delete
{
"policy": {
"phases": {
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}

==== Get index's ilm status 获取索引当前 ILM 状态 ====
GET /${index_name}/_ilm/explain

GET /${index_name}/_ilm/explain?human

==== Move index's ilm to step 修改索引的ILM阶段状态(人为触发ILM action执行) ====
https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-move-to-step.html
POST _ilm/move/my-index-000001
{
"current_step": {
"phase": "new",
"action": "complete",
"name": "complete"
},
"next_step": {
"phase": "warm",
"action": "forcemerge",
"name": "forcemerge"
}
}

==== Manually create an index that is managed by template and ILM (including rollover operations by day) ====

===== 手动创建原本应由template和ilm管控的索引，且索引名内包含日期(动态索引名) =====
⚠️ 这种索引不能直接粗暴地 <code>PUT /index-name-2022.10.23-000022</code> 以创建索引，否则手动创建出来的索引，在rollover滚动 (例如rollover-max_age:1d)的时候，创建出来的新索引名字仍然是创建索引时定义的日期，而不是当天轮滚发生时的日期(如 <code>index-name-2022.10.23-000023</code>)

这个现象可以通过判断 <code>GET /index-name/_settings</code> 中 <code>index.provided_name</code> 属性看出来

解法:
PUT %3Cindex-name-%7Bnow%2Fd%7D-000099%3E
注意在kibana-Dev Tools中不要做URL Decode，他就是这样的需要编码一下(解码后就是: <code><index-name-{now/d}-000099></code>)

ps: 如果怕创建错名字的话，可以使用 <code>GET %3Cindex-name-%7Bnow%2Fd%7D-000099%3E/_settings</code> 预览一下生成的索引名效果

对于索引最后的这个序号，[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html#increment-index-names-for-alias 无论前一个索引的名称是什么，该编号始终为 6 个字符，且为零填充]。即使手动创建的索引结尾是<code>-00001</code>，在rollover发生以后，索引后缀序号依然会变成<code>-000002</code>

==== 故障处理: 有的时候对现有的索引修改了其引用的 ilm policy 为别的 policy,或者修改了其引用的 ilm policy中的 phase 定义。会导致索引ilm故障 ====
有可能导致他的ilm处理会出问题（不记得怎么告警），没记错的话通过 <code>GET {index_name}/_ilm/explain</code> 能看到 error 信息，能看到卡在某个 phase 失败

这时候需要人为修改 index 的 ilm phase 修复，如
POST _ilm/move/insight-es-k8s-logs-dce5-aliyun-default-prd-2024.01.16
{
"current_step": {
"phase": "hot",
"action": "rollover",
"name": "ERROR"
},
"next_step": {
"phase": "cold"
}
}
或者尝试通过 <code>POST {index_name}/_ilm/retry</code> 接口重试

===Cluster setting related ES集群参数设置===
/_cluster/settings?include_defaults=true&pretty

/_cluster/settings?include_defaults=true

==== Wildcard expressions or all indices are not allowed ====

===== 允许泛匹配删除索引 =====
PUT /_cluster/settings
{
"persistent": {
"action": {
"destructive_requires_name": "false"
}
}
}

====primaries recovery settings ====
=====控制索引恢复或者relocating的并发数=====
{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 10,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_outgoing_recoveries": null,
"node_concurrent_recoveries": 20
}
}
}
}
}

{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": null,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": null
}
}
}
}
}

{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 30,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": 10
}
}
}
}
}
cluster.routing.allocation.node_concurrent_recoveries: A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>.
# PUT /_cluster/settings
{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_concurrent_recoveries": 8
}
}
}
}
}

=== recovery.max_bytes_per_sec [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/recovery.html 修改relocating时并发传输数据量] ===
加大此数值可以有效缩短es relocating index的耗时

indices.recovery.max_bytes_per_sec: Limits total inbound and outbound recovery traffic for each node. Applies to both peer recoveries as well as snapshot recoveries (i.e., restores from a snapshot). Defaults to <code>40mb</code> unless the node is a dedicated cold or frozen node, in which case the default relates to the total memory available to the node. 

===Index settings===

====modify the number of replicas in bulk====

===== 批量/单个设置索引副本数 =====
PUT /index_name*/_settings
{
"index": {
"number_of_replicas": 1
}
}

=== Search Documents ===

==== match_all 搜索 ====
GET /sw_segment-20230914/_search
{
"query": {
"match_all": {}
},
"size": 1
}

==== 单字段排序匹配搜索(match) ====
GET /sw_segment-20230914/_search
{
"query": {
"match": {
"segment_id": "b7bb26fae59e4f45b101346cb83ff796.69.16946808855979526"
}
},
"sort": [
{
"start_time": {
"order": "desc"
}
}
],
"size": 1
}

==== 对某个字段的值进行聚合(查询该字段有多少种类型的值) ====
GET /.monitoring-es-7-2024.12.13/_search
{
"size": 0,
"aggs": {
"unique_types": {
"terms": {
"field": "type",
"size": 10
}
}
}
}

=== Elastic Cloud on Kubernetes (ECK / Elastic operator) ===
ECK operator下管理的Elasticsearch如果要修改<code>cluster.routing.allocation.exclude</code> 的参数配置，需要先为 elasticsearch 实例配置annotation: 'eck.k8s.elastic.co/managed=false'，不然会配置一会就会被刷回原状

===Other===

====对于有大量索引的刚重启的es集群====
(主分片在1w-2w)

=====加快es集群恢复速度=====
结合es节点资源监控图，观测节点cpu压力，以及cpu IO wait

适当通过update cluster settings接口动态增加node_initial_primaries_recoveries (Defaults to <code>4</code>)

和 node_concurrent_recoveries

(A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>

Defaults to <code>2</code>)数值

通过使用 cluster settings + include_defaults=true 筛选查到当前配置值

减少集群从red状态到yellow状态的耗时：增加索引副本数量，增加node_initial_primaries_recoveries值

减少集群从yellow状态到green状态的耗时：增加 node_concurrent_recoveries 值

通过访问 <code>/_cluster/allocation/explain</code> 接口查到阻碍集群 to green(yellow)的原因

=====在es集群恢复期间因节点内存压力大(node was low on resources: memory.)而被k8s Evicted=====
调整缩小 jvm 配置值，尽量不超配(requests 和 limit尽量一致或提高requests值)

=== Error ===

==== 集群分片数达到maximum错误 ====
集群分片数达到maximum错误会有如下log信息，但是集群的健康状态不会改变
2022-11-10T10:26:03.643184618Z org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [3] shards, but this cluster currently has [1999]/[2000] maximum normal shards open;
解决:

调整index ilm 策略或者调整集群的max_shards_per_node配置

临时生效配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"transient": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
永久更改性配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"persistent": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
[[分类:Elasticsearch]]
{{DEFAULTSORT:api随记}}

Java容器相关参数随记

2025-06-24T08:36:22Z

Admin：

[[Java动态参数随记|Java动态参数随记(注入运行时参数)]]

[[Java命令参数随记]]

=== 命令随记 ===
查看系统中有哪些java进程，命令 <code>jps</code>

查看指定java进程占用的内存情况，命令 <code>jmap -heap ${PID}</code>

查看系统或容器配置，命令 <code>java -XshowSettings:system -version</code>

查看JVM的相关配置，命令 <code>java -XshowSettings:vm -version</code>

查看支持的java参数列表 <code>java -XX:+PrintFlagsFinal -version</code>，视情况需要加上<code>-XX:+UnlockExperimentalVMOptions</code>和<code>-XX:+UnlockDiagnosticVMOptions</code> 参数

查看是否支持<code>UseCGroupMemoryLimitForHeap</code>参数 <code>java -XX:+UnlockDiagnosticVMOptions -XX:+UnlockExperimentalVMOptions -XX:+PrintFlagsFinal -version |grep -i UseCGroupMemoryLimitForHeap</code>

=== JVM参数 ===

==== -XX:+UnlockExperimentalVMOptions ====
用于解锁实验性参数

==== -XX:+UnlockDiagnosticVMOptions ====
用于解锁一些诊断性的JVM选项。这些选项通常用于调试和诊断JVM的问题。

当JVM运行时，某些选项可能会被禁用以提高安全性和稳定性。UnlockDiagnosticVMOptions参数可以用于解锁这些选项，以便可以在运行时使用它们来进行调试和诊断。

==== -XX:+UseCGroupMemoryLimitForHeap ====
实验性参数，用于让Xmx感知docker的memory limit

这个参数自JDK8u121加入，在java8,9,10中都可用，不过由于在JDK8u191后新增了容器支持开关-XX:UseContainerSupport，并且默认开启，建议使用UseContainerSupport参数

java11正式移除UseCGroupMemoryLimitForHeap，代码改动见8194086: Remove deprecated experimental flag UseCGroupMemoryLimitForHeap

==== 其他参数 ====

JDK8u191增加了这些参数:

* MaxRAMPercentage 堆的最大值百分比。

* InitialRAMPercentage 堆的初始化的百分比。

* MinRAMPercentage 堆的最小值的百分比。

如果使用了-Xmx参数，则直接将MaxHeapSize（最大堆大小）等同于我们设置的-Xmx。

=== cgroup 与 java ===
1. jvm当前生效感知的CPU 内存限制

* 自 openjdk 8u131 起，有参数 -XX:+UseCGroupMemoryLimitForHeap 感知内存限制，8u191 起 XX:+UseContainerSupport 参数来支持 JVM 容器化。

* 在 CGroup V2 下，需要 8u372、11.0.16、15 及以上才能正常感知到 (https://bugs.openjdk.org/browse/JDK-8230305)

2. Oracle 相关 jdk/jre (8uxxx) yum 可以在 <nowiki>https://www.oracle.com/java/technologies/javase/javase8u211-later-archive-downloads.html</nowiki> 获取

COPY FROM:

* [https://blog.kelu.org/tech/2018/05/30/running-a-jvm-in-a-container-without-getting-killed.html 防止容器因 jvm 占用资源过多被杀死而频繁重启]
* [https://jvm-argument-for-docker.teaho.net/ 容器环境下的相关JVM参数]
* [https://cloud.tencent.com/developer/article/1340356 聊聊新版JDK对docker容器的支持]
* [https://blog.csdn.net/Thousa_Ho/article/details/77278656 JVM 参数使用总结]

[[分类:Java]]

使用jq批量修改es index template的lifecycle配置

2025-06-19T08:19:04Z

Admin：

使用jq批量修改es index template的lifecycle配置

2025-06-19T08:17:57Z

Admin：

aka: use jq tool to modify es index template lifecycle setting in bulk

新版本的es index template不支持通过结合索引匹配+优先级配置的方式来叠加批量修改已存在的索引模版(legacy template 支持)，只支持通过compose实现类似用法，部分场景会带来大量的搬砖成本(例如修改skywalking的索引模板中的生命周期配置或者索引副本数配置)

引入脚本:<pre>
<nowiki>
#!/bin/bash
#set -exu
set -ue
host="127.0.0.1:9200"
username="elastic"
password="elastic"
ilm_name="sw-default-ilm"

curl_options=(
"-u" "$username:$password"
"-s"
)

es_sw_templates=`curl ${curl_options[@]} "$host/_cat/templates/sw*?s=name" | awk '{print $1}'`
# bak
curl ${curl_options[@]} "${host}/_index_template?pretty" > es-${host}-index-templates_`date +%F%T`.bak.json

for template in ${es_sw_templates[@]}
do

if [[ $template == "sw_segment" ]]
then
continue
fi
echo PUT /_index_template/${template}
new_template=""
new_template=`curl ${curl_options[@]} "${host}/_index_template/${template}" | jq '.index_templates[0].index_template | .template.settings.index.lifecycle.name |= "'${ilm_name}'"'`
curl -X PUT -H "Content-Type: application/json" ${curl_options[@]} "${host}/_index_template/${template}" -d "${new_template}"
echo
done
</nowiki></pre>
[[分类:Elasticsearch]]
[[分类:Linux]]
{{DEFAULTSORT:jq}}

Elasticsearch7在容器中运行的时候是怎么感知到内存limit然后传递给java的xms和xmx参数

2025-06-19T05:49:29Z

Admin：

首先，看一下elasticsearch容器的启动命令，避免被<code>exec</code> 迷惑
[root@gzu1 ~]# docker inspect 1.1.1.1/library/elasticsearch:7.17.5| jq '.[].ContainerConfig | {Cmd,Entrypoint} '
{
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"eswrapper\"]"
],
"Entrypoint": [
"/bin/tini",
"--",
"/usr/local/bin/docker-entrypoint.sh"
]
}
进入到 es 容器中确认，与1号进程是预期一致的
[root@gzu1 ~]# kubectl exec -it -n es-system es-data-0 -- bash
root@es-data-0:/usr/share/elasticsearch# '''ps auxwwww'''
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root '''1''' 0.0 0.0 2500 540 ? Ss Mar27 3:07 '''/bin/tini -- /usr/local/bin/docker-entrypoint.sh eswrapper'''
elastic+ 7 103 4.7 103849888 18890788 ? Sl Mar27 124591:05 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-2689440660253749261 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Des.cgroups.hierarchy.override=/ '''-Xms16384m -Xmx16384m''' -XX:MaxDirectMemorySize=8589934592 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch
elastic+ 224 0.0 0.0 108392 4636 ? Sl Mar27 0:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root 50256 0.0 0.0 4244 2324 pts/0 Ss+ 04:05 0:00 bash
root 64422 0.3 0.0 4376 2328 pts/1 Ss 05:22 0:00 bash
root 64430 0.0 0.0 5900 1492 pts/1 R+ 05:22 0:00 ps auxwwww

''' 可以看到这里实际的运行 elasticsearch 的 java 命令中，居然有Xms 和Xmx，而且是 pod 内存限制的一半'''

而在当前pod yaml中，并没有指定这些参数:
[root@gzu1 ~]# kubectl get pods -n es-system es-data-0 -o json | jq '.spec.containers[].resources'
{
"limits": {
"cpu": "8",
"memory": "'''32Gi'''"
},
"requests": {
"cpu": "4",
"memory": "'''32Gi'''"
}
}
[root@gzu1 ~]# kubectl get pods -n es-system es-data-0 -o json | jq '.spec.containers[].env'
[
{
"name": "POD_IP",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "status.podIP"
}
}
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "NODE_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "spec.nodeName"
}
}
},
{
"name": "NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "PROBE_PASSWORD_PATH",
"value": "/mnt/elastic-internal/probe-user/elastic-internal-probe"
},
{
"name": "PROBE_USERNAME",
"value": "elastic-internal-probe"
},
{
"name": "READINESS_PROBE_PROTOCOL",
"value": "http"
},
{
"name": "HEADLESS_SERVICE_NAME",
"value": "dmp-es-data"
},
{
"name": "NSS_SDB_USE_CACHE",
"value": "no"
}
]
[root@gzu1 ~]# kubectl get pods -n es-system es-data-0 -o json | jq '.spec.containers[]'|grep -i env # 确认没有通过configmap等其他形式挂入env
"env": [
[root@gzu1 ~]#

这时候打开 <code>/usr/local/bin/docker-entrypoint.sh</code> 脚本开始检查该 elasticsearch 的初始化过程，能发现

<code>/usr/local/bin/docker-entrypoint.sh</code> 经过一些简单的初始化以后，还会按需通过读取一些文件，转化为本文暂时不关注的环境变量，最后会调用 <code>/usr/share/elasticsearch/bin/elasticsearch</code>

而 <code>/usr/share/elasticsearch/bin/elasticsearch</code> ,众所周知，他还是一个脚本

在该脚本中，他会执行 <code>source "`dirname "$0"`"/elasticsearch-env</code> ，也就是通过 <code>/usr/share/elasticsearch/bin/elasticsearch-env</code> 做一些环境初始化，例如定义<code>ES_JAVA_HOME</code> 环境变量，以及决定用内置<code>jdk</code>还是通过<code>env</code>指定的<code>jdk</code>。还会按需通过读取一些文件，转化为本文暂时不关注的环境变量

然后在指定完需要用哪个 <code>java</code> 以后，回到<code>/usr/share/elasticsearch/bin/elasticsearch</code>

可以看到存在一些不起眼的代码
# CONTROLLING STARTUP:
#
# This script relies on a few environment variables to determine startup
# behavior, those variables are:
#
# ES_PATH_CONF -- Path to config directory
# ES_JAVA_OPTS -- External Java Opts on top of the defaults set
#
# '''Optionally, exact memory values can be set using the `ES_JAVA_OPTS`'''. Example
# values are "512m", and "10g".
#
# ES_JAVA_OPTS="-Xms8g -Xmx8g" ./bin/elasticsearch

........

if [ -z "$ES_TMPDIR" ]; then
ES_TMPDIR=`"$JAVA" "$XSHARE" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.TempDirectory`
fi

.......

# The JVM options parser produces the final JVM options to start Elasticsearch.
# It does this by incorporating JVM options in the following way:
# '''- first, system JVM options are applied (these are hardcoded options in the'''
# '''parser)'''
# - second, JVM options are read from jvm.options and jvm.options.d/*.options
# - third, JVM options from ES_JAVA_OPTS are applied
# - fourth, ergonomic JVM options are applied
'''ES_JAVA_OPTS=`export ES_TMPDIR; "$JAVA" "$XSHARE" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JvmOptionsParser "$ES_PATH_CONF" "$ES_HOME/plugins"`'''
我们在 elasticsearch 容器中同样执行这些代码看看效果:
root@es-data-0:/usr/share/elasticsearch# /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.tools.launchers.TempDirectory
/tmp/elasticsearch-9132815636720135492

root@es-data-0:/usr/share/elasticsearch# export ES_TMPDIR=/tmp/elasticsearch-9132815636720135492

root@es-data-0:/usr/share/elasticsearch# '''/usr/share/elasticsearch/jdk/bin/java -Xshare:auto -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.tools.launchers.JvmOptionsParser /usr/share/elasticsearch/config /usr/share/elasticsearch/plugins'''
-Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-9132815636720135492 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m '''-Xms16384m -Xmx16384m''' -XX:MaxDirectMemorySize=8589934592 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25
在这里找到了Xms 和 Xmx的定义，也就是说 <code>/usr/share/elasticsearch/bin/elasticsearch</code> 脚本会通过执行 <code>org.elasticsearch.tools.launchers.TempDirectory</code> 产生一个临时文件夹，然后继续执行 <code>org.elasticsearch.tools.launchers.JvmOptionsParser</code> 产生合适的 Xms 及 Xmx 参数

而如果我们在环境变量中预先定义了 <code>ES_JAVA_OPTS</code> ，就会有如下效果(在该脚本注释中也有提到)
root@es-data-0:/usr/share/elasticsearch# '''export ES_JAVA_OPTS="-Xms8g -Xmx8g"'''

root@es-data-0:/usr/share/elasticsearch# '''/usr/share/elasticsearch/jdk/bin/java -Xshare:auto -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.tools.launchers.JvmOptionsParser /usr/share/elasticsearch/config /usr/share/elasticsearch/plugins'''
-Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-9132815636720135492 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m '''-Xms8g -Xmx8g''' -XX:MaxDirectMemorySize=4294967296 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25
[[分类:Elasticsearch]]

Elasticsearch7在容器中运行的时候是怎么感知到内存limit然后传递给java的xms和xmx参数

2025-06-19T05:46:33Z

Admin：创建页面，内容为“首先，看一下elasticsearch容器的启动命令，避免被<code>exec</code> 迷惑 [root@gzu1 ~]# docker inspect 1.1.1.1/library/elasticsearch:7.17.5| jq '.[].ContainerConfig | {Cmd,Entrypoint} ' { "Cmd": [ "/bin/sh", "-c", "#(nop) ", "CMD [\"eswrapper\"]" ], "Entrypoint": [ "/bin/tini", "--", "/usr/local/bin/docker-entrypoint.sh" ] } 进入到 es 容器中确认，与1号进程是预期一致的 [root@gz…”

首先，看一下elasticsearch容器的启动命令，避免被<code>exec</code> 迷惑
[root@gzu1 ~]# docker inspect 1.1.1.1/library/elasticsearch:7.17.5| jq '.[].ContainerConfig | {Cmd,Entrypoint} '
{
"Cmd": [
"/bin/sh",
"-c",
"#(nop) ",
"CMD [\"eswrapper\"]"
],
"Entrypoint": [
"/bin/tini",
"--",
"/usr/local/bin/docker-entrypoint.sh"
]
}
进入到 es 容器中确认，与1号进程是预期一致的
[root@gzu1 ~]# kubectl exec -it -n es-system es-data-0 -- bash
root@es-data-0:/usr/share/elasticsearch# '''ps auxwwww'''
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root '''1''' 0.0 0.0 2500 540 ? Ss Mar27 3:07 '''/bin/tini -- /usr/local/bin/docker-entrypoint.sh eswrapper'''
elastic+ 7 103 4.7 103849888 18890788 ? Sl Mar27 124591:05 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-2689440660253749261 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Des.cgroups.hierarchy.override=/ '''-Xms16384m -Xmx16384m''' -XX:MaxDirectMemorySize=8589934592 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/usr/share/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=docker -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch
elastic+ 224 0.0 0.0 108392 4636 ? Sl Mar27 0:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root 50256 0.0 0.0 4244 2324 pts/0 Ss+ 04:05 0:00 bash
root 64422 0.3 0.0 4376 2328 pts/1 Ss 05:22 0:00 bash
root 64430 0.0 0.0 5900 1492 pts/1 R+ 05:22 0:00 ps auxwwww

可以看到这里实际的运行 elasticsearch 的java命令中，居然有Xms 和Xmx，而且是pod 内存限制的一半

而在当前pod yaml中，并没有指定这些参数:
[root@gzu1 ~]# kubectl get pods -n es-system es-data-0 -o json | jq '.spec.containers[].resources'
{
"limits": {
"cpu": "8",
"memory": "'''32Gi'''"
},
"requests": {
"cpu": "4",
"memory": "'''32Gi'''"
}
}
[root@gzu1 ~]# kubectl get pods -n es-system es-data-0 -o json | jq '.spec.containers[].env'
[
{
"name": "POD_IP",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "status.podIP"
}
}
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "NODE_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "spec.nodeName"
}
}
},
{
"name": "NAMESPACE",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
}
}
},
{
"name": "PROBE_PASSWORD_PATH",
"value": "/mnt/elastic-internal/probe-user/elastic-internal-probe"
},
{
"name": "PROBE_USERNAME",
"value": "elastic-internal-probe"
},
{
"name": "READINESS_PROBE_PROTOCOL",
"value": "http"
},
{
"name": "HEADLESS_SERVICE_NAME",
"value": "dmp-es-data"
},
{
"name": "NSS_SDB_USE_CACHE",
"value": "no"
}
]
[root@gzu1 ~]# kubectl get pods -n es-system es-data-0 -o json | jq '.spec.containers[]'|grep -i env # 确认没有通过configmap等其他形式挂入env
"env": [
[root@gzu1 ~]#

这时候打开 <code>/usr/local/bin/docker-entrypoint.sh</code> 脚本开始检查该 elasticsearch 的初始化过程，能发现

<code>/usr/local/bin/docker-entrypoint.sh</code> 经过一些简单的初始化以后，还会按需通过读取一些文件，转化为本文暂时不关注的环境变量，最后会调用 <code>/usr/share/elasticsearch/bin/elasticsearch</code>

而 <code>/usr/share/elasticsearch/bin/elasticsearch</code> 总所周知，他还是一个脚本

在该脚本中，他会执行 <code>source "`dirname "$0"`"/elasticsearch-env</code> ，也就是通过 <code>/usr/share/elasticsearch/bin/elasticsearch-env</code> 做一些环境初始化，例如定义<code>ES_JAVA_HOME</code> 环境变量，以及决定用内置<code>jdk</code>还是通过<code>env</code>指定的<code>jdk</code>。还会按需通过读取一些文件，转化为本文暂时不关注的环境变量

然后在指定完需要用哪个 <code>java</code> 以后，回到<code>/usr/share/elasticsearch/bin/elasticsearch</code>

可以看到存在一些不起眼的代码
# CONTROLLING STARTUP:
#
# This script relies on a few environment variables to determine startup
# behavior, those variables are:
#
# ES_PATH_CONF -- Path to config directory
# ES_JAVA_OPTS -- External Java Opts on top of the defaults set
#
# '''Optionally, exact memory values can be set using the `ES_JAVA_OPTS`'''. Example
# values are "512m", and "10g".
#
# ES_JAVA_OPTS="-Xms8g -Xmx8g" ./bin/elasticsearch

........

if [ -z "$ES_TMPDIR" ]; then
ES_TMPDIR=`"$JAVA" "$XSHARE" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.TempDirectory`
fi

.......

# The JVM options parser produces the final JVM options to start Elasticsearch.
# It does this by incorporating JVM options in the following way:
# '''- first, system JVM options are applied (these are hardcoded options in the'''
# '''parser)'''
# - second, JVM options are read from jvm.options and jvm.options.d/*.options
# - third, JVM options from ES_JAVA_OPTS are applied
# - fourth, ergonomic JVM options are applied
'''ES_JAVA_OPTS=`export ES_TMPDIR; "$JAVA" "$XSHARE" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JvmOptionsParser "$ES_PATH_CONF" "$ES_HOME/plugins"`'''
我们在 elasticsearch 容器中同样执行这些代码看看效果:
root@es-data-0:/usr/share/elasticsearch# /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.tools.launchers.TempDirectory
/tmp/elasticsearch-9132815636720135492

root@es-data-0:/usr/share/elasticsearch# export ES_TMPDIR=/tmp/elasticsearch-9132815636720135492

root@es-data-0:/usr/share/elasticsearch# '''/usr/share/elasticsearch/jdk/bin/java -Xshare:auto -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.tools.launchers.JvmOptionsParser /usr/share/elasticsearch/config /usr/share/elasticsearch/plugins'''
-Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-9132815636720135492 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m '''-Xms16384m -Xmx16384m''' -XX:MaxDirectMemorySize=8589934592 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25
在这里找到了Xms 和 Xmx的定义，也就是说 <code>/usr/share/elasticsearch/bin/elasticsearch</code> 脚本会通过执行 <code>org.elasticsearch.tools.launchers.TempDirectory</code> 产生一个临时文件夹，然后继续执行 <code>org.elasticsearch.tools.launchers.JvmOptionsParser</code> 产生合适的 Xms 及 Xmx 参数

而如果我们在环境变量中预先定义了 <code>ES_JAVA_OPTS</code> ，就会有如下效果(在该脚本注释中也有提到)
root@es-data-0:/usr/share/elasticsearch# '''export ES_JAVA_OPTS="-Xms8g -Xmx8g"'''

root@es-data-0:/usr/share/elasticsearch# '''/usr/share/elasticsearch/jdk/bin/java -Xshare:auto -cp "/usr/share/elasticsearch/lib/*" org.elasticsearch.tools.launchers.JvmOptionsParser /usr/share/elasticsearch/config /usr/share/elasticsearch/plugins'''
-Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -XX:+UseG1GC -Djava.io.tmpdir=/tmp/elasticsearch-9132815636720135492 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m '''-Xms8g -Xmx8g''' -XX:MaxDirectMemorySize=4294967296 -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=25
[[分类:Elasticsearch]]

Elasticsearch的一些api随记

2025-06-06T18:46:32Z

Admin：

===Health===
/_cat/health

/_cluster/health

===Indices health===
按条件查看索引状态
/_cat/indices?help
/_cat/indices?health=red&v&s=store.size:desc,index

/_cat/indices?health=yellow&v&s=store.size:desc,index

/_cat/indices?health=green&v&s=store.size:desc,index
注意在配合 * 通配符搜索/操作索引的时候，如果涉及隐藏的datastream / 隐藏的index，默认不会匹配命中，必须指定 <code>expand_wildcards=all</code> 参数

-> https://www.elastic.co/guide/en/elasticsearch/reference/7.16/multi-index.html#hidden

===Nodes===
/_cat/nodes?v
查看es各节点磁盘空间占用、分片数目等
/_cat/allocation?v

/_cat/nodeattrs

===Get master node===
/_cat/master?v

===Cluster allocation explain related===
可以用于定位分片状态以及分片为何故障
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

/_cluster/allocation/explain

===Shards===
粗略查看分片情况，特别是查看分片分布节点或大小/状态
GET /_cat/shards

GET /_cat/shards?index=index_name

GET /_cat/shards?index=index_na*
查看分片分配失败原因
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

==== Recovery API ====
Returns information about ongoing and completed shard recoveries, similar to the index recovery API.

For data streams, the API returns information about the stream’s backing indices

可以查看当前正在 relocating 的分片，也能查到各分片处理进度百分比
GET /_cat/recovery?active_only=true&s=index&v

=== Adds a data stream or index to an alias, and sets the write index or data stream for the alias ===
为别名设置可写索引或数据流

If the alias doesn’t exist, the <code>add</code> action creates it.
POST /_aliases
{
"actions": [
{
"add": {
"index": "es-k8s-logs-000020",
"alias": "es-k8s-logs-alias",
"is_write_index": true
}
}
]
}

===Thread pool related===
https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-thread-pool.html
/_cluster/settings?pretty&include_defaults=true | grep processors

====Get maximum number of threads info====
curl "127.1:9200/_cat/thread_pool?v&h=ip,node_name,id,name,max,size,queue_size,queue,active,rejected&pretty"

=== Templates 模板 ===
/_cat/templates?v
⚠️ /_template/${template_name} is legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.

新版本中使用 <code>/_index_template</code> 取代
GET/PUT /_template/${template_name}
但是 index_template 对比 legacy template有个很明显的差异就是，legacy template可以直接根据priority进行叠加覆盖，而index_template哪个template priority高，就只有哪个生效 https://www.elastic.co/guide/en/elasticsearch/reference/7.17/index-templates.html<blockquote>If a new data stream or index matches more than one index template, the index template with the highest priority is used.</blockquote>

==== Use template to change the replicas settings of all indexes (Legacy index template) ====
Multiple index templates can potentially match an index, in this case, both the settings and mappings are merged into the final configuration of the index.

The order of the merging can be controlled using the <code>order</code> parameter, with lower order being applied first, '''and higher orders overriding them.'''

legacy es template 中, 取值范围为 0 - 2^31-1 (0~2147483647)
PUT /_template/${template_name}
{
"order": 2147483647,
"index_patterns": [
"*"
],
"settings": {
"index": {
"number_of_replicas": "0"
}
}
}
[[使用jq批量修改es index template的lifecycle配置]]

=== ILM (index lifecycle policy) 索引生命周期 ===
顾名思义，ilm另外也可用于做ES集群的冷热温架构。

不同的阶段(phase)能做哪些事可以在这个 [https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html Document] 查看

比较难受的是，ilm目前没有类似 <code>_cat/templates</code> 的接口一次性只查看这个集群已配置的 ILM 策略名字，只能一次性获取全部策略具体定义 (不过可以利用浏览器的F12 json preview折叠来曲线救国)
GET /_ilm/policy

==== Get specific ilm policy detail 获取特定ILM策略定义 ====
GET /_ilm/policy/${ilm_name}

PUT /_ilm/policy/ilm-30d-delete
{
"policy": {
"phases": {
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}

==== Get index's ilm status 获取索引当前 ILM 状态 ====
GET /${index_name}/_ilm/explain

GET /${index_name}/_ilm/explain?human

==== Move index's ilm to step 修改索引的ILM阶段状态(人为触发ILM action执行) ====
https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-move-to-step.html
POST _ilm/move/my-index-000001
{
"current_step": {
"phase": "new",
"action": "complete",
"name": "complete"
},
"next_step": {
"phase": "warm",
"action": "forcemerge",
"name": "forcemerge"
}
}

==== Manually create an index that is managed by template and ILM (including rollover operations by day) ====

===== 手动创建原本应由template和ilm管控的索引，且索引名内包含日期(动态索引名) =====
⚠️ 这种索引不能直接粗暴地 <code>PUT /index-name-2022.10.23-000022</code> 以创建索引，否则手动创建出来的索引，在rollover滚动 (例如rollover-max_age:1d)的时候，创建出来的新索引名字仍然是创建索引时定义的日期，而不是当天轮滚发生时的日期(如 <code>index-name-2022.10.23-000023</code>)

这个现象可以通过判断 <code>GET /index-name/_settings</code> 中 <code>index.provided_name</code> 属性看出来

解法:
PUT %3Cindex-name-%7Bnow%2Fd%7D-000099%3E
注意在kibana-Dev Tools中不要做URL Decode，他就是这样的需要编码一下(解码后就是: <code><index-name-{now/d}-000099></code>)

ps: 如果怕创建错名字的话，可以使用 <code>GET %3Cindex-name-%7Bnow%2Fd%7D-000099%3E/_settings</code> 预览一下生成的索引名效果

对于索引最后的这个序号，[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html#increment-index-names-for-alias 无论前一个索引的名称是什么，该编号始终为 6 个字符，且为零填充]。即使手动创建的索引结尾是<code>-00001</code>，在rollover发生以后，索引后缀序号依然会变成<code>-000002</code>

==== 故障处理: 有的时候对现有的索引修改了其引用的 ilm policy 为别的 policy,或者修改了其引用的 ilm policy中的 phase 定义。会导致索引ilm故障 ====
有可能导致他的ilm处理会出问题（不记得怎么告警），没记错的话通过 <code>GET {index_name}/_ilm/explain</code> 能看到 error 信息，能看到卡在某个 phase 失败

这时候需要人为修改 index 的 ilm phase 修复，如
POST _ilm/move/insight-es-k8s-logs-dce5-aliyun-default-prd-2024.01.16
{
"current_step": {
"phase": "hot",
"action": "rollover",
"name": "ERROR"
},
"next_step": {
"phase": "cold"
}
}
或者尝试通过 <code>POST {index_name}/_ilm/retry</code> 接口重试

===Cluster setting related ES集群参数设置===
/_cluster/settings?include_defaults=true&pretty

/_cluster/settings?include_defaults=true

==== Wildcard expressions or all indices are not allowed ====

===== 允许泛匹配删除索引 =====
PUT /_cluster/settings
{
"persistent": {
"action": {
"destructive_requires_name": "false"
}
}
}

====primaries recovery settings ====
=====控制索引恢复或者relocating的并发数=====
{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 10,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_outgoing_recoveries": null,
"node_concurrent_recoveries": 20
}
}
}
}
}

{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": null,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": null
}
}
}
}
}

{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 30,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": 10
}
}
}
}
}
cluster.routing.allocation.node_concurrent_recoveries: A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>.
# PUT /_cluster/settings
{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_concurrent_recoveries": 8
}
}
}
}
}

=== recovery.max_bytes_per_sec [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/recovery.html 修改relocating时并发传输数据量] ===
加大此数值可以有效缩短es relocating index的耗时

indices.recovery.max_bytes_per_sec: Limits total inbound and outbound recovery traffic for each node. Applies to both peer recoveries as well as snapshot recoveries (i.e., restores from a snapshot). Defaults to <code>40mb</code> unless the node is a dedicated cold or frozen node, in which case the default relates to the total memory available to the node. 

===Index settings===

====modify the number of replicas in bulk====

===== 批量/单个设置索引副本数 =====
PUT /index_name*/_settings
{
"index": {
"number_of_replicas": 1
}
}

=== Search Documents ===

==== match_all 搜索 ====
GET /sw_segment-20230914/_search
{
"query": {
"match_all": {}
},
"size": 1
}

==== 单字段排序匹配搜索(match) ====
GET /sw_segment-20230914/_search
{
"query": {
"match": {
"segment_id": "b7bb26fae59e4f45b101346cb83ff796.69.16946808855979526"
}
},
"sort": [
{
"start_time": {
"order": "desc"
}
}
],
"size": 1
}

==== 对某个字段的值进行聚合(查询该字段有多少种类型的值) ====
GET /.monitoring-es-7-2024.12.13/_search
{
"size": 0,
"aggs": {
"unique_types": {
"terms": {
"field": "type",
"size": 10
}
}
}
}

=== Elastic Cloud on Kubernetes (ECK / Elastic operator) ===
ECK operator下管理的Elasticsearch如果要修改<code>cluster.routing.allocation.exclude</code> 的参数配置，需要先为 elasticsearch 实例配置annotation: 'eck.k8s.elastic.co/managed=false'，不然会配置一会就会被刷回原状

===Other===

====对于有大量索引的刚重启的es集群====
(主分片在1w-2w)

=====加快es集群恢复速度=====
结合es节点资源监控图，观测节点cpu压力，以及cpu IO wait

适当通过update cluster settings接口动态增加node_initial_primaries_recoveries (Defaults to <code>4</code>)

和 node_concurrent_recoveries

(A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>

Defaults to <code>2</code>)数值

通过使用 cluster settings + include_defaults=true 筛选查到当前配置值

减少集群从red状态到yellow状态的耗时：增加索引副本数量，增加node_initial_primaries_recoveries值

减少集群从yellow状态到green状态的耗时：增加 node_concurrent_recoveries 值

通过访问 <code>/_cluster/allocation/explain</code> 接口查到阻碍集群 to green(yellow)的原因

=====在es集群恢复期间因节点内存压力大(node was low on resources: memory.)而被k8s Evicted=====
调整缩小 jvm 配置值，尽量不超配(requests 和 limit尽量一致或提高requests值)

=== Error ===

==== 集群分片数达到maximum错误 ====
集群分片数达到maximum错误会有如下log信息，但是集群的健康状态不会改变
2022-11-10T10:26:03.643184618Z org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [3] shards, but this cluster currently has [1999]/[2000] maximum normal shards open;
解决:

调整index ilm 策略或者调整集群的max_shards_per_node配置

临时生效配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"transient": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
永久更改性配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"persistent": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
[[分类:Elasticsearch]]
{{DEFAULTSORT:api随记}}

Elasticsearch的一些api随记

2025-05-08T09:35:05Z

Admin：

===Health===
/_cat/health

/_cluster/health

===Indices health===
按条件查看索引状态
/_cat/indices?help
/_cat/indices?health=red&v&s=store.size:desc,index

/_cat/indices?health=yellow&v&s=store.size:desc,index

/_cat/indices?health=green&v&s=store.size:desc,index
注意在配合 * 通配符搜索/操作索引的时候，如果涉及隐藏的datastream / 隐藏的index，默认不会匹配命中，必须指定 <code>expand_wildcards=all</code> 参数

-> https://www.elastic.co/guide/en/elasticsearch/reference/7.16/multi-index.html#hidden

===Nodes===
/_cat/nodes?v
查看es各节点磁盘空间占用、分片数目等
/_cat/allocation?v

/_cat/nodeattrs

===Get master node===
/_cat/master?v

===Cluster allocation explain related===
可以用于定位分片状态以及分片为何故障
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

/_cluster/allocation/explain

===Shards===
粗略查看分片情况，特别是查看分片分布节点或大小/状态
GET /_cat/shards

GET /_cat/shards?index=index_name

GET /_cat/shards?index=index_na*
查看分片分配失败原因
/_cat/shards/index_name-*?v&s=state,index&h=index,shard,prirep,state,docs,store,ip,node,unassigned.reason

==== Recovery API ====
Returns information about ongoing and completed shard recoveries, similar to the index recovery API.

For data streams, the API returns information about the stream’s backing indices

可以查看当前正在 relocating 的分片，也能查到各分片处理进度百分比
GET /_cat/recovery?active_only=true&s=index&v

=== Adds a data stream or index to an alias, and sets the write index or data stream for the alias ===
为别名设置可写索引或数据流

If the alias doesn’t exist, the <code>add</code> action creates it.
POST /_aliases
{
"actions": [
{
"add": {
"index": "es-k8s-logs-000020",
"alias": "es-k8s-logs-alias",
"is_write_index": true
}
}
]
}

===Thread pool related===
https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-thread-pool.html
/_cluster/settings?pretty&include_defaults=true | grep processors

====Get maximum number of threads info====
curl "127.1:9200/_cat/thread_pool?v&h=ip,node_name,id,name,max,size,queue_size,queue,active,rejected&pretty"

=== Templates 模板 ===
/_cat/templates?v
⚠️ /_template/${template_name} is legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.

新版本中使用 <code>/_index_template</code> 取代
GET/PUT /_template/${template_name}

==== Use template to change the replicas settings of all indexes (Legacy index template) ====
Multiple index templates can potentially match an index, in this case, both the settings and mappings are merged into the final configuration of the index.

The order of the merging can be controlled using the <code>order</code> parameter, with lower order being applied first, '''and higher orders overriding them.'''

legacy es template 中, 取值范围为 0 - 2^31-1 (0~2147483647)
PUT /_template/${template_name}
{
"order": 2147483647,
"index_patterns": [
"*"
],
"settings": {
"index": {
"number_of_replicas": "0"
}
}
}
[[使用jq批量修改es index template的lifecycle配置]]

=== ILM (index lifecycle policy) 索引生命周期 ===
顾名思义，ilm另外也可用于做ES集群的冷热温架构。

不同的阶段(phase)能做哪些事可以在这个 [https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html Document] 查看

比较难受的是，ilm目前没有类似 <code>_cat/templates</code> 的接口一次性只查看这个集群已配置的 ILM 策略名字，只能一次性获取全部策略具体定义 (不过可以利用浏览器的F12 json preview折叠来曲线救国)
GET /_ilm/policy

==== Get specific ilm policy detail 获取特定ILM策略定义 ====
GET /_ilm/policy/${ilm_name}

PUT /_ilm/policy/ilm-30d-delete
{
"policy": {
"phases": {
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}

==== Get index's ilm status 获取索引当前 ILM 状态 ====
GET /${index_name}/_ilm/explain

GET /${index_name}/_ilm/explain?human

==== Move index's ilm to step 修改索引的ILM阶段状态(人为触发ILM action执行) ====
https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-move-to-step.html
POST _ilm/move/my-index-000001
{
"current_step": {
"phase": "new",
"action": "complete",
"name": "complete"
},
"next_step": {
"phase": "warm",
"action": "forcemerge",
"name": "forcemerge"
}
}

==== Manually create an index that is managed by template and ILM (including rollover operations by day) ====

===== 手动创建原本应由template和ilm管控的索引，且索引名内包含日期(动态索引名) =====
⚠️ 这种索引不能直接粗暴地 <code>PUT /index-name-2022.10.23-000022</code> 以创建索引，否则手动创建出来的索引，在rollover滚动 (例如rollover-max_age:1d)的时候，创建出来的新索引名字仍然是创建索引时定义的日期，而不是当天轮滚发生时的日期(如 <code>index-name-2022.10.23-000023</code>)

这个现象可以通过判断 <code>GET /index-name/_settings</code> 中 <code>index.provided_name</code> 属性看出来

解法:
PUT %3Cindex-name-%7Bnow%2Fd%7D-000099%3E
注意在kibana-Dev Tools中不要做URL Decode，他就是这样的需要编码一下(解码后就是: <code><index-name-{now/d}-000099></code>)

ps: 如果怕创建错名字的话，可以使用 <code>GET %3Cindex-name-%7Bnow%2Fd%7D-000099%3E/_settings</code> 预览一下生成的索引名效果

对于索引最后的这个序号，[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html#increment-index-names-for-alias 无论前一个索引的名称是什么，该编号始终为 6 个字符，且为零填充]。即使手动创建的索引结尾是<code>-00001</code>，在rollover发生以后，索引后缀序号依然会变成<code>-000002</code>

==== 故障处理: 有的时候对现有的索引修改了其引用的 ilm policy 为别的 policy,或者修改了其引用的 ilm policy中的 phase 定义。会导致索引ilm故障 ====
有可能导致他的ilm处理会出问题（不记得怎么告警），没记错的话通过 <code>GET {index_name}/_ilm/explain</code> 能看到 error 信息，能看到卡在某个 phase 失败

这时候需要人为修改 index 的 ilm phase 修复，如
POST _ilm/move/insight-es-k8s-logs-dce5-aliyun-default-prd-2024.01.16
{
"current_step": {
"phase": "hot",
"action": "rollover",
"name": "ERROR"
},
"next_step": {
"phase": "cold"
}
}
或者尝试通过 <code>POST {index_name}/_ilm/retry</code> 接口重试

===Cluster setting related ES集群参数设置===
/_cluster/settings?include_defaults=true&pretty

/_cluster/settings?include_defaults=true

==== Wildcard expressions or all indices are not allowed ====

===== 允许泛匹配删除索引 =====
PUT /_cluster/settings
{
"persistent": {
"action": {
"destructive_requires_name": "false"
}
}
}

====primaries recovery settings ====
=====控制索引恢复或者relocating的并发数=====
{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 10,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_outgoing_recoveries": null,
"node_concurrent_recoveries": 20
}
}
}
}
}

{
"transient": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": null,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": null
}
}
}
}
}

{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_initial_primaries_recoveries": 30,
"node_concurrent_incoming_recoveries": null,
"node_concurrent_recoveries": 10
}
}
}
}
}
cluster.routing.allocation.node_concurrent_recoveries: A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>.
# PUT /_cluster/settings
{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"node_concurrent_recoveries": 8
}
}
}
}
}

=== recovery.max_bytes_per_sec [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/recovery.html 修改relocating时并发传输数据量] ===
加大此数值可以有效缩短es relocating index的耗时

indices.recovery.max_bytes_per_sec: Limits total inbound and outbound recovery traffic for each node. Applies to both peer recoveries as well as snapshot recoveries (i.e., restores from a snapshot). Defaults to <code>40mb</code> unless the node is a dedicated cold or frozen node, in which case the default relates to the total memory available to the node. 

===Index settings===

====modify the number of replicas in bulk====

===== 批量/单个设置索引副本数 =====
PUT /index_name*/_settings
{
"index": {
"number_of_replicas": 1
}
}

=== Search Documents ===

==== match_all 搜索 ====
GET /sw_segment-20230914/_search
{
"query": {
"match_all": {}
},
"size": 1
}

==== 单字段排序匹配搜索(match) ====
GET /sw_segment-20230914/_search
{
"query": {
"match": {
"segment_id": "b7bb26fae59e4f45b101346cb83ff796.69.16946808855979526"
}
},
"sort": [
{
"start_time": {
"order": "desc"
}
}
],
"size": 1
}

==== 对某个字段的值进行聚合(查询该字段有多少种类型的值) ====
GET /.monitoring-es-7-2024.12.13/_search
{
"size": 0,
"aggs": {
"unique_types": {
"terms": {
"field": "type",
"size": 10
}
}
}
}

=== Elastic Cloud on Kubernetes (ECK / Elastic operator) ===
ECK operator下管理的Elasticsearch如果要修改<code>cluster.routing.allocation.exclude</code> 的参数配置，需要先为 elasticsearch 实例配置annotation: 'eck.k8s.elastic.co/managed=false'，不然会配置一会就会被刷回原状

===Other===

====对于有大量索引的刚重启的es集群====
(主分片在1w-2w)

=====加快es集群恢复速度=====
结合es节点资源监控图，观测节点cpu压力，以及cpu IO wait

适当通过update cluster settings接口动态增加node_initial_primaries_recoveries (Defaults to <code>4</code>)

和 node_concurrent_recoveries

(A shortcut to set both <code>cluster.routing.allocation.node_concurrent_incoming_recoveries</code> and <code>cluster.routing.allocation.node_concurrent_outgoing_recoveries</code>

Defaults to <code>2</code>)数值

通过使用 cluster settings + include_defaults=true 筛选查到当前配置值

减少集群从red状态到yellow状态的耗时：增加索引副本数量，增加node_initial_primaries_recoveries值

减少集群从yellow状态到green状态的耗时：增加 node_concurrent_recoveries 值

通过访问 <code>/_cluster/allocation/explain</code> 接口查到阻碍集群 to green(yellow)的原因

=====在es集群恢复期间因节点内存压力大(node was low on resources: memory.)而被k8s Evicted=====
调整缩小 jvm 配置值，尽量不超配(requests 和 limit尽量一致或提高requests值)

=== Error ===

==== 集群分片数达到maximum错误 ====
集群分片数达到maximum错误会有如下log信息，但是集群的健康状态不会改变
2022-11-10T10:26:03.643184618Z org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [3] shards, but this cluster currently has [1999]/[2000] maximum normal shards open;
解决:

调整index ilm 策略或者调整集群的max_shards_per_node配置

临时生效配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"transient": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
永久更改性配置:
<nowiki>curl -H "content-type: application/json" -X PUT "127.0.0.1:9200/_cluster/settings" -d '{"persistent": {"cluster.max_shards_per_node": "5000"}}'</nowiki>
[[分类:Elasticsearch]]
{{DEFAULTSORT:api随记}}

在使用kubectl apply操作workload产生的非预期行为

2025-05-07T10:44:20Z

Admin：

=== 简述 ===

* kubectl apply 对于之前本身就是 apply 或者 create --save-config 产生的resources，可能存在非预期行为( kubectl.kubernetes.io/last-applied-configuration )，例如修改原有的workload probe settings(bug at 1.18.x), 或者修改hostAlias(修改原有的 hostalias 的 ip)

=== hostalias复现: ===
先通过<code>kubectl create -f</code> 创建一个普通的deployment(不要用 <code>kubectl apply -f</code> 来创建，不然就绕过这个bug了~)
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
sit.k8s.io/app: yaml-test
name: yaml-test
spec:
replicas:0
selector:
matchLabels:
sit.k8s.io/app: yaml-test
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
sit.k8s.io/app: yaml-test
name: yaml-test
spec:
hostAliases:
- hostnames:
- testaaaa.com
- testbbb.com
ip: 1.1.1.7
containers:
- image: 192.168.150.181/test/nginx-2048:latest
imagePullPolicy: IfNotPresent
name: yaml-test
readinessProbe:
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
resources:
limits:
cpu: "1"
memory: "64Mi"
requests:
cpu: 64m
memory: "64Mi"
dnsPolicy: ClusterFirst
restartPolicy: Always

然后修改一下hostAliases的值(例如下面这样)，执行<code>kubectl apply -f xxx.yaml --dry-run=server -oyaml</code>
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
sit.k8s.io/app: yaml-test
name: yaml-test
spec:
replicas: 0
selector:
matchLabels:
sit.k8s.io/app: yaml-test
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
sit.k8s.io/app: yaml-test
name: yaml-test
spec:
hostAliases:
- hostnames:
- testaaaa.com
- testbbb.com
ip: 1.1.1.8
containers:
- image: 192.168.150.181/test/nginx-2048:latest
imagePullPolicy: IfNotPresent
name: yaml-test
readinessProbe:
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 1
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
resources:
limits:
cpu: "1"
memory: "64Mi"
requests:
cpu: 64m
memory: "64Mi"
dnsPolicy: ClusterFirst
restartPolicy: Always

会发现apply以后的结果hostAliases字段非预期（1.1.1.8直接被追加了进去，而不是替换）

[[文件:Kubectl apply merge.png|无框|400x400像素]]

或者一开始使用apply创建资源，然后删除kubectl.kubernetes.io/last-applied-configuration: ，再修改ip，再apply

如果一开始是用<code>kubectl apply -f xxxx</code>创建资源，然后用<code>apply -f</code>更新资源，则不会复现

==== 这个也是跟k8s apply的实现方式有关系 ====
主要跟 kubectl.kubernetes.io/last-applied-configuration 这个annotation有关系

[https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#how-apply-calculates-differences-and-merges-changes How apply calculates differences and merges changes]

[[文件:Kubectl apply patch strategy.png|无框|900x900像素]]

Related article: [[K8s的一些小坑或者bug简要记录]]

[[分类:K8s]]
{{DEFAULTSORT:apply}}

文件:Kubectl apply patch strategy.png

2025-05-07T10:42:11Z

Admin：

kubectl_apply_patch_strategy

文件:Kubectl apply merge.png

2025-05-07T10:39:49Z

Admin：

kubectl_apply_merge

Keycloak设置http代理proxy

2025-04-29T09:56:49Z

Admin：

=== keycloak 开启log debug: ===
环境变量 KC_LOG_LEVEL debug

如果不生效，还可以考虑 ROOT_LOGLEVEL debug

=== 为keycloak发起的访问请求设置http代理(Proxy settings for keycloak outgoing requests) ===

==== Official document ====
https://www.keycloak.org/docs/latest/server_installation/#_proxymappings
 

==== Related configuration file path ====
<blockquote>There are some things you’ll need to configure in <code>standalone.xml</code>, <code>standalone-ha.xml</code>, or <code>domain.xml</code>. The location of this file depends on your operating mode.</blockquote>/opt/jboss/keycloak/standalone/configuration/****

eg: /opt/jboss/keycloak/standalone/configuration/standalone-ha.xml
 
<spi name="connectionsHttpClient">
<provider name="default" enabled="true">
<properties>
<property name="proxy-mappings" value="[&quot;.*\\.(microsoft|microsoftonline)\\.com;<nowiki>http://your-proxy-address.net:8080</nowiki>&quot;]"/>
</properties>
</provider>
</spi>

====ps====
需要注意的是 &quot; 是 " 的实体字符
Note that one needs to encode <code>"</code> characters with <code>&quot;</code>.
[[分类:Softwares]]

[[分类:Other]]

Elasticsearch7系列下同索引主分片分布不均衡思考

2025-04-28T16:23:05Z

Admin：

es版本: 7.17.5

现象:
> GET /_cat/shards/sw_segment-20250429?v&s=shard,prirep > es-shard-status-sw_segment-20250428

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-0|wc -l
26

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-1|wc -l
27

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-2|wc -l
72

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-3|wc -l
29

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-4|wc -l
26
可以看到 sw_segment-20250428 这个索引在不同节点上，主分片分布非常不均衡，如果这个索引写入量/数据量很大的话，很容易导致 IO 相对其他实例有很大的差异

但是经过查询后发现，在整个集群所有分片中，主分片在所有es 实例中分布较为均衡

而es7系列原生自带的 balance 行为不支持针对同 index 的 primary shard 进行干预 (8.12 有 <code>cluster.routing.allocation.balance.write_load</code> 的权重配置)

这时候可以利用index <code>total_shards_per_node</code>选项来进行较为'''十分不灵活不优雅不可靠'''的干预，先将索引副本数设置为0，然后在索引初始化前设置该配置，以强制约束每一个es实例分布一样数量的分片数量。等到全部主分片都调度初始化好了，再根据需要把副本数调整回来，把 <code>total_shards_per_node</code> 也根据<code>(该索引的分片数 * 副本数)</code>进行合适的调整。

也可以为了长期打算，将<code>total_shards_per_node</code>数量配置为 <code>(该索引的分片数 * 副本数) / (集群中可调度该索引分片的es实例数 - 1)</code>

当然，也可以像社区issue中提到那样，用现成的工具 / 开发工具，利用<code>reroute</code>等api接口进行实时干预，并将相关shard进行统计监控

相关社区issue: https://github.com/elastic/elasticsearch/issues/41543
[[分类:Elasticsearch]]

Elasticsearch7系列下同索引主分片分布不均衡思考

2025-04-28T16:22:49Z

Admin：

es版本: 7.17.5

现象:
> GET /_cat/shards/sw_segment-20250429?v&s=shard,prirep > es-shard-status-sw_segment-20250428

> cat es-shard-status-sw_segment-20250428 | grep " p " | greep es-data-ssd-0|wc -l
26

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-1|wc -l
27

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-2|wc -l
72

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-3|wc -l
29

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-4|wc -l
26
可以看到 sw_segment-20250428 这个索引在不同节点上，主分片分布非常不均衡，如果这个索引写入量/数据量很大的话，很容易导致 IO 相对其他实例有很大的差异

但是经过查询后发现，在整个集群所有分片中，主分片在所有es 实例中分布较为均衡

而es7系列原生自带的 balance 行为不支持针对同 index 的 primary shard 进行干预 (8.12 有 <code>cluster.routing.allocation.balance.write_load</code> 的权重配置)

这时候可以利用index <code>total_shards_per_node</code>选项来进行较为'''十分不灵活不优雅不可靠'''的干预，先将索引副本数设置为0，然后在索引初始化前设置该配置，以强制约束每一个es实例分布一样数量的分片数量。等到全部主分片都调度初始化好了，再根据需要把副本数调整回来，把 <code>total_shards_per_node</code> 也根据<code>(该索引的分片数 * 副本数)</code>进行合适的调整。

也可以为了长期打算，将<code>total_shards_per_node</code>数量配置为 <code>(该索引的分片数 * 副本数) / (集群中可调度该索引分片的es实例数 - 1)</code>

当然，也可以像社区issue中提到那样，用现成的工具 / 开发工具，利用<code>reroute</code>等api接口进行实时干预，并将相关shard进行统计监控

相关社区issue: https://github.com/elastic/elasticsearch/issues/41543
[[分类:Elasticsearch]]

Elasticsearch7系列下同索引主分片分布不均衡思考

2025-04-28T16:22:37Z

Admin：

es版本: 7.17.5

现象:
> GET /_cat/shards/sw_segment-20250429?v&s=shard,prirep > es-shard-status-sw_segment-20250428

> cat es-shard-status-sw_segment-20250428 | grep " p " | greep es-data-ssd-0|wc -l
26

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-1|wc -l
27

> cat es-shard-status-sw_segment-20250428 | grep " p " | grep es-data-ssd-2|wc -l
72

> cat es-shard-status-sw_segment-20250428 | grep " p " |grep es-data-ssd-3|wc -l
29

> cat es-shard-status-sw_segment-20250428 | grep " p " |grep es-data-ssd-4|wc -l
26
可以看到 sw_segment-20250428 这个索引在不同节点上，主分片分布非常不均衡，如果这个索引写入量/数据量很大的话，很容易导致 IO 相对其他实例有很大的差异

但是经过查询后发现，在整个集群所有分片中，主分片在所有es 实例中分布较为均衡

而es7系列原生自带的 balance 行为不支持针对同 index 的 primary shard 进行干预 (8.12 有 <code>cluster.routing.allocation.balance.write_load</code> 的权重配置)

这时候可以利用index <code>total_shards_per_node</code>选项来进行较为'''十分不灵活不优雅不可靠'''的干预，先将索引副本数设置为0，然后在索引初始化前设置该配置，以强制约束每一个es实例分布一样数量的分片数量。等到全部主分片都调度初始化好了，再根据需要把副本数调整回来，把 <code>total_shards_per_node</code> 也根据<code>(该索引的分片数 * 副本数)</code>进行合适的调整。

也可以为了长期打算，将<code>total_shards_per_node</code>数量配置为 <code>(该索引的分片数 * 副本数) / (集群中可调度该索引分片的es实例数 - 1)</code>

当然，也可以像社区issue中提到那样，用现成的工具 / 开发工具，利用<code>reroute</code>等api接口进行实时干预，并将相关shard进行统计监控

相关社区issue: https://github.com/elastic/elasticsearch/issues/41543
[[分类:Elasticsearch]]