AWS-CNI路由模式测试和验证
目录
1. 测试资源
- 测试主机
sh-saas-k8stest-node-dev-02 10.19.0.22
- 同网段主机：
sh-saas-k8stest-master-dev-01 10.19.0.11
sh-saas-k8stest-master-dev-02 10.19.0.12
- 子网网段：
10.19.16.0/20
10.19.0.0/20
- 弹性网卡
| 网卡 | 网卡类型 | IP |
|---|---|---|
| Primary ENI eni-jmdk5zda | 主网卡 | 10.19.0.22 主IP |
| saas-k8stest-01 eni-e5l4vlhe | 辅助网卡 | 10.19.16.17 主IP10.19.16.6 辅助IP10.19.16.9 辅助IP |
| saas-k8stest-02 eni-4vdxgc7g | 辅助网卡 | 10.19.0.24 主IP10.19.0.17 辅助IP10.19.0.32 辅助IP |
2. 测试环境准备
2.1 停止有影响的一些服务
systemctl status kubelet
systemctl status kube-proxy.service
2.2 清空ipvs和iptables的规则
ipvsadm -C
iptables -F #(flush 清除所有的已定规则)
iptables -X #(delete 删除所有用户“自定义”的链（tables）)
iptables -Z #（zero 将所有的chain的计数与流量统计都归零）
iptables -F -t mangle
iptables -F -t nat
iptables -X -t mangle
iptables -X -t nat
清完后应该像下面这样：
[root@sh-saas-k8stest-node-dev-02 ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@sh-saas-k8stest-node-dev-02 ~]# iptables-save
# Generated by iptables-save v1.4.21 on Mon May 9 18:31:58 2022
*mangle
:PREROUTING ACCEPT [291:18765]
:INPUT ACCEPT [193:14509]
:FORWARD ACCEPT [62:3224]
:OUTPUT ACCEPT [174:17125]
:POSTROUTING ACCEPT [236:20349]
COMMIT
# Completed on Mon May 9 18:31:58 2022
# Generated by iptables-save v1.4.21 on Mon May 9 18:31:58 2022
*filter
:INPUT ACCEPT [3336:249912]
:FORWARD ACCEPT [930:48360]
:OUTPUT ACCEPT [3106:325671]
COMMIT
# Completed on Mon May 9 18:31:58 2022
# Generated by iptables-save v1.4.21 on Mon May 9 18:31:58 2022
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Mon May 9 18:31:58 2022
2.3 清理网卡
ip link delete kube-ipvs0
ip link delete dummy0
清理完后如下：
[root@sh-saas-k8stest-node-dev-02 nic-hotplug]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 52:54:00:9b:34:09 brd ff:ff:ff:ff:ff:ff
inet 10.19.0.22/20 brd 10.19.15.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fe9b:3409/64 scope link
valid_lft forever preferred_lft forever
3. 与主网卡不同子网的弹性网卡测试
3.1 弹性网卡主IP测试
进入腾讯云控制台，找到sh-saas-k8stest-node-dev-02，绑定saas-k8stest-01 | eni-e5l4vlhe(辅助网卡)。
绑定后，我们将看到：
[root@sh-saas-k8stest-node-dev-02 nic-hotplug]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 52:54:00:9b:34:09 brd ff:ff:ff:ff:ff:ff
inet 10.19.0.22/20 brd 10.19.15.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fe9b:3409/64 scope link
valid_lft forever preferred_lft forever
260: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 20:90:6f:ff:7b:f1 brd ff:ff:ff:ff:ff:ff
在腾讯云控制台，我们可以看到此弹性网卡的MAC地址为：20:90:6F:FF:7B:F1，与上面的eth1网卡的mac地址一样，说明此网卡为刚新增加的弹性网卡。
下面按照AWS CNI agent的setupENINetwork内的方法来配置网卡，添加路由：
# 设置eni网卡的mtu
ip link set dev eth1 mtu 1500
# 让eni网卡up
ip link set dev eth1 up
# 删除eni网卡上所有已经存在的ip地址
ip add del 10.19.16.17 dev eth1
# 设置eni网卡的ip
# eniIP： primary IP of that ENI
ip add add 10.19.16.17 dev eth1
[root@sh-saas-k8stest-node-dev-02 nic-hotplug]# ip route
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
169.254.0.0/16 dev eth0 scope link metric 1002
[root@sh-saas-k8stest-node-dev-02 nic-hotplug]# ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
# 在deviceNumber + 1路由表里，删除已经存在默认路由，添加默认路由指向eni子网的网关
# tableNumber := deviceNumber + 1
# gw: 网关为Eni IP子网的第二个IP地址
# Add a direct link route for the host is ENI IP only
ip route add 10.19.16.1/32 scope link dev eth1 table 2
# Route all other traffic via the host is ENI IP
# tableNumber := deviceNumber + 1
# gw: 网关为Eni IP子网的第二个IP地址
ip route add 0.0.0.0/0 scope 0 via 10.19.16.1 dev eth1 table 2
# 在main路由表里删除源地址为eniIP，目标地址为eniSubnetCIDR的路由
ip route del 10.19.16.0/20 src 10.19.16.17 scope link table main
[root@sh-saas-k8stest-node-dev-02 nic-hotplug]# ip route show table 2
default via 10.19.16.1 dev eth1
10.19.16.1 dev eth1 scope link
[root@sh-saas-k8stest-node-dev-02 nic-hotplug]# ip route
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
169.254.0.0/16 dev eth0 scope link metric 1002
ping下其它主机试一下：
[root@sh-saas-k8stest-node-dev-02 nic-hotplug]# ping -I eth1 -c 1 10.19.0.11
PING 10.19.0.11 (10.19.0.11) from 10.19.16.17 eth1: 56(84) bytes of data.
发现ping不通
再按照AWS CNI 二进制文件的方法来添加相关路由：
ip route add 10.19.16.17/32 dev eth1 scope link
# https://blog.clanzx.net/network/ip-neigh.html
ip neigh add 10.19.16.17 lladdr 20:90:6f:ff:7b:f1 nud permanent dev eth1
ip rule add to 10.19.16.17/32 pref 512 table main
ip rule add from 10.19.16.17/32 pref 1536 table 2
现在可以通了
[root@sh-saas-k8stest-node-dev-02 nic-hotplug]# ping -I eth1 -c 1 10.19.0.11
PING 10.19.0.11 (10.19.0.11) from 10.19.16.17 eth1: 56(84) bytes of data.
64 bytes from 10.19.0.11: icmp_seq=1 ttl=64 time=0.163 ms
--- 10.19.0.11 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.163/0.163/0.163/0.000 ms
现在完整的路由规则如下：
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule
0: from all lookup local
512: from all to 10.19.16.17 lookup main
1536: from 10.19.16.17 lookup 2
32766: from all lookup main
32767: from all lookup default
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table local
broadcast 10.19.0.0 dev eth0 proto kernel scope link src 10.19.0.22
local 10.19.0.22 dev eth0 proto kernel scope host src 10.19.0.22
broadcast 10.19.15.255 dev eth0 proto kernel scope link src 10.19.0.22
local 10.19.16.17 dev eth1 proto kernel scope host src 10.19.16.17
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table main
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
10.19.16.17 dev eth1 scope link
169.254.0.0/16 dev eth0 scope link metric 1002
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table 2
default via 10.19.16.1 dev eth1
10.19.16.1 dev eth1 scope link
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table main
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
10.19.16.17 dev eth1 scope link
169.254.0.0/16 dev eth0 scope link metric 1002
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table default
[root@sh-saas-k8stest-node-dev-02 ~]#
稍为解释一下，这里用到了策略路由。ip rule和ip route分别的作用是：ip rule用于管理路由规则（策略），ip route是管理路由表的（路由）。 简单理解就是一个管策略，一个管路由，数据包通过策略（ip rule）决定走哪张路由表(ip route)，再通过这张路由表的路由来决定发到哪个目的地。
如下面的ip rule list：
[root@sh-saas-k8stest-node-dev-02 nic-hotplug]# ip rule
0: from all lookup local
512: from all to 10.19.16.17 lookup main
1536: from 10.19.16.17 lookup 2
32766: from all lookup main
32767: from all lookup default
第一列数字表示规则的优先级，优先级别越高的规则越先匹配（数值越小优先级越高）。from表示来源，to表示目的地，lookup表示查哪张路由表。linux可以自定义从1－252个路由表，linux系统本身维护了4个路由表，详细如下：
| 路由表号 | 路由表名 | 备注 |
|---|---|---|
| 0 | unspec | 系统保留表 |
| 1~252 | 用户自定义 | 用户可自定义表 |
| 253 | defult | 没特别指定的默认路由都放在改表 |
| 254 | main | 没指明路由表的所有路由放在该表 |
| 255 | local | 保存本地接口地址，广播地址、NAT地址 由系统维护，用户不得更改 |
用户可编辑/etc/iproute2/rt_tables文件来自定义路由表名，也可直接使用数字的路由表号
[root@sh-saas-k8stest-node-dev-02 etc]# cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
257 eth1
258 eth2
3.2 弹性网卡辅助IP模拟POD网络测试
每个pod的网络都是一个独立的network namespace,通过新建network space 配置网卡来模拟POD的网络环境
下面开始新建network namespace模拟测试POD网络：
主IP 10.19.16.17已经绑在了网卡上，我们使用10.19.16.6做为POD IP，veth网卡对host端网卡名为veth1,容器端为eth0，network namespace为netns1。
- 先新建network namespace
ip netns add netns1
- 新建网卡对
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip link add eth0 type veth peer name veth1
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip link list
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: veth1@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether fa:e4:56:6f:92:8d brd ff:ff:ff:ff:ff:ff
3: eth0@veth1: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether e6:74:22:f7:b6:93 brd ff:ff:ff:ff:ff:ff
- 启动网卡
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip link set dev veth1 up
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip link set dev eth0 up
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip link set dev lo up
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: veth1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether fa:e4:56:6f:92:8d brd ff:ff:ff:ff:ff:ff
3: eth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether e6:74:22:f7:b6:93 brd ff:ff:ff:ff:ff:ff
- 添加默认路由
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip route add 169.254.1.1 scope link dev eth0
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip route add 0.0.0.0/0 scope 0 via 169.254.1.1 dev eth0
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip route list
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
- 给容器里的网卡绑定ip
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip addr add 10.19.16.6 dev eth0
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip addr add 127.0.0.1 dev lo
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 127.0.0.1/32 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: veth1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether fa:e4:56:6f:92:8d brd ff:ff:ff:ff:ff:ff
inet6 fe80::f8e4:56ff:fe6f:928d/64 scope link
valid_lft forever preferred_lft forever
3: eth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e6:74:22:f7:b6:93 brd ff:ff:ff:ff:ff:ff
inet 10.19.16.6/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::e474:22ff:fef7:b693/64 scope link
valid_lft forever preferred_lft forever
- 添加网关169.254.1.1的arp表
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip neigh add 169.254.1.1 lladdr fa:e4:56:6f:92:8d nud permanent dev eth0
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip neigh list
169.254.1.1 dev eth0 lladdr 26:b0:af:76:4f:c9 PERMANENT
- 将host端网卡移动到host命名空间
host命名空间默认是没有的，可以使用进程ID做为参数，我们使用linux的init进行，也就是PID 1来使用，这样就到主机的命名空间了。 参考：https://segmentfault.com/q/1010000018494978
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip link set veth1 netns 1
[root@sh-saas-k8stest-node-dev-02 ~]# ip link list veth1
267: veth1@if3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether fa:e4:56:6f:92:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 127.0.0.1/32 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if267: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
link/ether e6:74:22:f7:b6:93 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.19.16.6/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::e474:22ff:fef7:b693/64 scope link
valid_lft forever preferred_lft forever
- 启动host网卡
host网卡移到host命名空间后，状态会变成down,需要再次启动。
[root@sh-saas-k8stest-node-dev-02 ~]# ip link set dev veth1 up
[root@sh-saas-k8stest-node-dev-02 ~]# ip link list veth1
267: veth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether fa:e4:56:6f:92:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0
- 添加用于host上访问pod的路由
[root@sh-saas-k8stest-node-dev-02 ~]# ip route add 10.19.16.6/32 dev veth1 scope link
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
10.19.16.6 dev veth1 scope link
169.254.0.0/16 dev eth0 scope link metric 1002
- 添加用于访问pod流量的路由策略
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule add to 10.19.16.6/32 pref 512 table main
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule add from 10.19.16.6/32 pref 1536 table 2
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule list
0: from all lookup local
512: from all to 10.19.16.17 lookup main
512: from all to 10.19.16.6 lookup main
1536: from 10.19.16.17 lookup 2
1536: from 10.19.16.6 lookup 2
32766: from all lookup main
32767: from all lookup default
最终的策略路由如下：
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule list
0: from all lookup local
512: from all to 10.19.16.17 lookup main
512: from all to 10.19.16.6 lookup main
1536: from 10.19.16.17 lookup 2
1536: from 10.19.16.6 lookup 2
32766: from all lookup main
32767: from all lookup default
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table local
broadcast 10.19.0.0 dev eth0 proto kernel scope link src 10.19.0.22
local 10.19.0.22 dev eth0 proto kernel scope host src 10.19.0.22
broadcast 10.19.15.255 dev eth0 proto kernel scope link src 10.19.0.22
local 10.19.16.17 dev eth1 proto kernel scope host src 10.19.16.17
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table main
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
10.19.16.6 dev veth1 scope link
10.19.16.17 dev eth1 scope link
169.254.0.0/16 dev eth0 scope link metric 1002
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table default
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table 2
default via 10.19.16.1 dev eth1
10.19.16.1 dev eth1 scope link
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
现在看看网络是否能ping通？
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ping -c 1 10.19.0.11
PING 10.19.0.11 (10.19.0.11) 56(84) bytes of data.
64 bytes from 10.19.0.11: icmp_seq=1 ttl=63 time=0.213 ms
--- 10.19.0.11 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.213/0.213/0.213/0.000 ms
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 ping -c 1 10.19.16.17
PING 10.19.16.17 (10.19.16.17) 56(84) bytes of data.
64 bytes from 10.19.16.17: icmp_seq=1 ttl=64 time=0.034 ms
--- 10.19.16.17 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.034/0.034/0.034/0.000 ms
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 tracepath -n 10.19.0.11
1?: [LOCALHOST] pmtu 1500
1: 10.19.0.22 0.057ms
1: 10.19.0.22 0.019ms
2: 10.19.0.11 0.229ms reached
Resume: pmtu 1500 hops 2 back 2
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns1 tracepath -n 10.19.16.17
1?: [LOCALHOST] pmtu 1500
1: 10.19.16.17 0.038ms reached
1: 10.19.16.17 0.013ms reached
Resume: pmtu 1500 hops 1 back 1
可以看到是可以ping通的，说明以上的配置是没有问题的。模拟的POD网络可以正常和外部通信。
4. 与主网卡同子网的弹性网卡测试
4.1 弹性网卡主IP测试
进入腾讯云控制台，找到sh-saas-k8stest-node-dev-02，绑定saas-k8stest-02 eni-4vdxgc7g(辅助网卡)。
绑定后，我们将看到：
[root@sh-saas-k8stest-node-dev-02 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 52:54:00:9b:34:09 brd ff:ff:ff:ff:ff:ff
inet 10.19.0.22/20 brd 10.19.15.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fe9b:3409/64 scope link
valid_lft forever preferred_lft forever
260: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 20:90:6f:ff:7b:f1 brd ff:ff:ff:ff:ff:ff
inet 10.19.16.17/32 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::2290:6fff:feff:7bf1/64 scope link
valid_lft forever preferred_lft forever
267: veth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether fa:e4:56:6f:92:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::f8e4:56ff:fe6f:928d/64 scope link
valid_lft forever preferred_lft forever
268: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 20:90:6f:e2:01:48 brd ff:ff:ff:ff:ff:ff
在腾讯云控制台，我们可以看到此弹性网卡的MAC地址为：20:90:6f:e2:01:48，与上面的eth2网卡的mac地址一样，说明此网卡为刚新增加的弹性网卡。
下面按照AWS CNI agent的setupENINetwork内的方法来配置网卡，添加路由：
- 设置eni网卡的mtu
ip link set dev eth2 mtu 1500
- 让eni网卡up
ip link set dev eth2 up
- 删除eni网卡上所有已经存在的ip地址
ip add del 10.19.0.24 dev eth2
- 设置eni网卡的ip
[root@sh-saas-k8stest-node-dev-02 ~]# ip addr add 10.19.0.24 dev eth2
[root@sh-saas-k8stest-node-dev-02 ~]# ip addr list eth2
268: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 20:90:6f:e2:01:48 brd ff:ff:ff:ff:ff:ff
inet 10.19.0.24/32 scope global eth2
valid_lft forever preferred_lft forever
inet6 fe80::2290:6fff:fee2:148/64 scope link
valid_lft forever preferred_lft forever
- 在deviceNumber + 1路由表里，删除已经存在默认路由，添加默认路由指向eni子网的网关
[root@sh-saas-k8stest-node-dev-02 ~]# ip route
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
10.19.16.6 dev veth1 scope link
10.19.16.17 dev eth1 scope link
169.254.0.0/16 dev eth0 scope link metric 1002
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule
0: from all lookup local
512: from all to 10.19.16.17 lookup main
512: from all to 10.19.16.6 lookup main
1536: from 10.19.16.17 lookup 2
1536: from 10.19.16.6 lookup 2
32766: from all lookup main
32767: from all lookup default
[root@sh-saas-k8stest-node-dev-02 ~]# ip route add 10.19.0.1/32 scope link dev eth2 table 3
[root@sh-saas-k8stest-node-dev-02 ~]# ip route add 0.0.0.0/0 scope 0 via 10.19.0.1 dev eth2 table 3
[root@sh-saas-k8stest-node-dev-02 ~]# ip route del 10.19.0.0/20 src 10.19.0.24 scope link table main
RTNETLINK answers: No such process
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table 3
default via 10.19.0.1 dev eth2
10.19.0.1 dev eth2 scope link
[root@sh-saas-k8stest-node-dev-02 ~]# ip route
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
10.19.16.6 dev veth1 scope link
10.19.16.17 dev eth1 scope link
169.254.0.0/16 dev eth0 scope link metric 1002
- 按照AWS CNI 二进制文件的方法来添加相关路由
增加策略路由：
[root@sh-saas-k8stest-node-dev-02 ~]# ip route add 10.19.0.24/32 dev eth2 scope link
[root@sh-saas-k8stest-node-dev-02 ~]# ip neigh add 10.19.0.24 lladdr 20:90:6f:e2:01:48 nud permanent dev eth2
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule add to 10.19.0.24/32 pref 512 table main
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule add from 10.19.0.24/32 pref 1536 table 3
增加完后完整的路由信息如下：
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule list
0: from all lookup local
512: from all to 10.19.16.17 lookup main
512: from all to 10.19.16.6 lookup main
512: from all to 10.19.0.24 lookup main
1536: from 10.19.16.17 lookup 2
1536: from 10.19.16.6 lookup 2
1536: from 10.19.0.24 lookup 3
32766: from all lookup main
32767: from all lookup default
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table main
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
10.19.0.24 dev eth2 scope link
10.19.16.6 dev veth1 scope link
10.19.16.17 dev eth1 scope link
169.254.0.0/16 dev eth0 scope link metric 1002
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table 3
default via 10.19.0.1 dev eth2
10.19.0.1 dev eth2 scope link
[root@sh-saas-k8stest-node-dev-02 ~]# ip neigh list | grep eth2
10.19.0.24 dev eth2 lladdr 20:90:6f:e2:01:48 PERMANENT
- 验证是否能与外部通信
最后我们来验证是否能与外部通信，可以看到是能正常与外部ping通的。
[root@sh-saas-k8stest-node-dev-02 ~]# ping -I eth2 -c 1 10.19.0.11
PING 10.19.0.11 (10.19.0.11) from 10.19.0.24 eth2: 56(84) bytes of data.
64 bytes from 10.19.0.11: icmp_seq=1 ttl=64 time=0.191 ms
--- 10.19.0.11 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.191/0.191/0.191/0.000 ms
4.2 弹性网卡辅助IP模拟POD网络测试
下面像不同子网辅助IP模拟POD网络一样开始新建network namespace模拟测试POD网络：
主IP 10.19.0.24已经绑在了网卡上，我们使用10.19.0.32做为POD IP，network namespace为netns2， veth网卡对host端网卡名为veth2,容器端为eth0，。
- 先新建network namespace
ip netns add netns2
- 新建网卡对
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip link add eth0 type veth peer name veth2
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip link list
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: veth2@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 26:e3:d4:cd:8b:e4 brd ff:ff:ff:ff:ff:ff
3: eth0@veth2: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether b6:04:ab:66:ec:c5 brd ff:ff:ff:ff:ff:ff
- 启动网卡
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip link set dev veth2 up
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip link set dev eth0 up
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip link set dev lo up
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: veth2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 26:e3:d4:cd:8b:e4 brd ff:ff:ff:ff:ff:ff
3: eth0@veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether b6:04:ab:66:ec:c5 brd ff:ff:ff:ff:ff:ff
- 添加默认路由
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip route add 169.254.1.1 scope link dev eth0
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip route add 0.0.0.0/0 scope 0 via 169.254.1.1 dev eth0
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip route list
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
- 给容器里的网卡绑定ip
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip addr add 10.19.0.32 dev eth0
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip addr add 127.0.0.1 dev lo
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 127.0.0.1/32 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: veth2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 26:e3:d4:cd:8b:e4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::24e3:d4ff:fecd:8be4/64 scope link
valid_lft forever preferred_lft forever
3: eth0@veth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b6:04:ab:66:ec:c5 brd ff:ff:ff:ff:ff:ff
inet 10.19.0.32/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::b404:abff:fe66:ecc5/64 scope link
valid_lft forever preferred_lft forever
- 添加网关169.254.1.1的arp表
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip neigh add 169.254.1.1 lladdr 26:e3:d4:cd:8b:e4 nud permanent dev eth0
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip neigh list
169.254.1.1 dev eth0 lladdr 26:e3:d4:cd:8b:e4 PERMANENT
- 将host端网卡移动到host命名空间
host命名空间默认是没有的，可以使用进程ID做为参数，我们使用linux的init进行，也就是PID 1来使用，这样就到主机的命名空间了。 参考：https://segmentfault.com/q/1010000018494978
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip link set veth2 netns 1
[root@sh-saas-k8stest-node-dev-02 ~]# ip link list veth2
269: veth2@if3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 26:e3:d4:cd:8b:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 1
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 127.0.0.1/32 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if269: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
link/ether b6:04:ab:66:ec:c5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.19.0.32/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::b404:abff:fe66:ecc5/64 scope link
valid_lft forever preferred_lft forever
- 启动host网卡
host网卡移到host命名空间后，状态会变成down,需要再次启动。
[root@sh-saas-k8stest-node-dev-02 ~]# ip link set dev veth2 up
[root@sh-saas-k8stest-node-dev-02 ~]# ip link list veth2
269: veth2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 26:e3:d4:cd:8b:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 1
- 添加用于host上访问pod的路由
[root@sh-saas-k8stest-node-dev-02 ~]# ip route add 10.19.0.32/32 dev veth2 scope link
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
10.19.0.24 dev eth2 scope link
10.19.0.32 dev veth2 scope link
10.19.16.6 dev veth1 scope link
10.19.16.17 dev eth1 scope link
169.254.0.0/16 dev eth0 scope link metric 1002
- 添加用于访问pod流量的路由策略
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule
0: from all lookup local
512: from all to 10.19.16.17 lookup main
512: from all to 10.19.16.6 lookup main
512: from all to 10.19.0.24 lookup main
1536: from 10.19.16.17 lookup 2
1536: from 10.19.16.6 lookup 2
1536: from 10.19.0.24 lookup 3
32766: from all lookup main
32767: from all lookup default
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule add to 10.19.0.32/32 pref 512 table main
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule add from 10.19.0.32/32 pref 1536 table 3
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule list
0: from all lookup local
512: from all to 10.19.16.17 lookup main
512: from all to 10.19.16.6 lookup main
512: from all to 10.19.0.24 lookup main
512: from all to 10.19.0.32 lookup main
1536: from 10.19.16.17 lookup 2
1536: from 10.19.16.6 lookup 2
1536: from 10.19.0.24 lookup 3
1536: from 10.19.0.32 lookup 3
32766: from all lookup main
32767: from all lookup default
最终的策略路由如下：
[root@sh-saas-k8stest-node-dev-02 ~]# ip rule list
0: from all lookup local
512: from all to 10.19.16.17 lookup main
512: from all to 10.19.16.6 lookup main
512: from all to 10.19.0.24 lookup main
512: from all to 10.19.0.32 lookup main
1536: from 10.19.16.17 lookup 2
1536: from 10.19.16.6 lookup 2
1536: from 10.19.0.24 lookup 3
1536: from 10.19.0.32 lookup 3
32766: from all lookup main
32767: from all lookup default
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table local
broadcast 10.19.0.0 dev eth0 proto kernel scope link src 10.19.0.22
local 10.19.0.22 dev eth0 proto kernel scope host src 10.19.0.22
local 10.19.0.24 dev eth2 proto kernel scope host src 10.19.0.24
broadcast 10.19.15.255 dev eth0 proto kernel scope link src 10.19.0.22
local 10.19.16.17 dev eth1 proto kernel scope host src 10.19.16.17
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table main
default via 10.19.0.1 dev eth0
10.19.0.0/20 dev eth0 proto kernel scope link src 10.19.0.22
10.19.0.24 dev eth2 scope link
10.19.0.32 dev veth2 scope link
10.19.16.6 dev veth1 scope link
10.19.16.17 dev eth1 scope link
169.254.0.0/16 dev eth0 scope link metric 1002
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table default
[root@sh-saas-k8stest-node-dev-02 ~]# ip route list table 3
default via 10.19.0.1 dev eth2
10.19.0.1 dev eth2 scope link
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
现在看看网络是否能ping通？
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ping -c 1 10.19.0.11
PING 10.19.0.11 (10.19.0.11) 56(84) bytes of data.
64 bytes from 10.19.0.11: icmp_seq=1 ttl=63 time=0.275 ms
--- 10.19.0.11 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.275/0.275/0.275/0.000 ms
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ping -c 1 10.19.16.17
PING 10.19.16.17 (10.19.16.17) 56(84) bytes of data.
64 bytes from 10.19.16.17: icmp_seq=1 ttl=64 time=0.029 ms
--- 10.19.16.17 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.029/0.029/0.029/0.000 ms
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ping -c 1 10.19.16.6
PING 10.19.16.6 (10.19.16.6) 56(84) bytes of data.
64 bytes from 10.19.16.6: icmp_seq=1 ttl=63 time=0.043 ms
--- 10.19.16.6 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.043/0.043/0.043/0.000 ms
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 ping -c 1 10.19.0.22
PING 10.19.0.22 (10.19.0.22) 56(84) bytes of data.
64 bytes from 10.19.0.22: icmp_seq=1 ttl=64 time=0.037 ms
--- 10.19.0.22 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.037/0.037/0.037/0.000 ms
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 tracepath -n 10.19.0.11
1?: [LOCALHOST] pmtu 1500
1: 10.19.0.22 0.046ms
1: 10.19.0.22 0.014ms
2: 10.19.0.11 0.280ms reached
Resume: pmtu 1500 hops 2 back 2
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 tracepath -n 10.19.16.6
1?: [LOCALHOST] pmtu 1500
1: 10.19.0.22 0.038ms
1: 10.19.0.22 0.013ms
2: 10.19.16.6 0.024ms reached
Resume: pmtu 1500 hops 2 back 2
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 tracepath -n 10.19.16.17
1?: [LOCALHOST] pmtu 1500
1: 10.19.16.17 0.047ms reached
1: 10.19.16.17 0.025ms reached
Resume: pmtu 1500 hops 1 back 1
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 tracepath -n 10.19.0.22
1?: [LOCALHOST] pmtu 1500
1: 10.19.0.22 0.048ms reached
1: 10.19.0.22 0.027ms reached
Resume: pmtu 1500 hops 1 back 1
可以看到是可以ping通的，说明以上的配置是没有问题的。模拟的POD网络可以正常和外部通信。
5. 弹性网卡流量转发路径测试
下面的traepath测试结果可以看到，当弹性网卡的IP到外部时，会经过10.19.0.22，也就是CVM机器的主网卡IP上。
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 tracepath -n 10.19.0.11
1?: [LOCALHOST] pmtu 1500
1: 10.19.0.22 0.046ms
1: 10.19.0.22 0.014ms
2: 10.19.0.11 0.280ms reached
Resume: pmtu 1500 hops 2 back 2
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 tracepath -n 10.19.16.6
1?: [LOCALHOST] pmtu 1500
1: 10.19.0.22 0.038ms
1: 10.19.0.22 0.013ms
2: 10.19.16.6 0.024ms reached
Resume: pmtu 1500 hops 2 back 2
这是不是表示在某一台CVM主机上，所有弹性网卡的流量进出都需要经过主网卡呢？
我们做一个测试验证，在10.19.0.11主机上ping 10.19.0.32，然后在主网卡eth0及弹性网卡eth2上分别抓包。如果在主网卡eth0也能抓到ping包，说明弹性网卡的流量进出都需要经过主网卡的，反之则不是，弹性网卡对CVM来说就像物理网卡一样。
在10.19.0.11主机上ping 10.19.0.32：
[root@sh-saas-k8stest-master-dev-01 ~]# ping -c 3 10.19.0.32
PING 10.19.0.32 (10.19.0.32) 56(84) bytes of data.
64 bytes from 10.19.0.32: icmp_seq=1 ttl=63 time=0.209 ms
64 bytes from 10.19.0.32: icmp_seq=2 ttl=63 time=0.191 ms
64 bytes from 10.19.0.32: icmp_seq=3 ttl=63 time=0.191 ms
--- 10.19.0.32 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.191/0.197/0.209/0.008 ms
[root@sh-saas-k8stest-master-dev-01 ~]#
通过下面的抓包结果我们可以看到，主网卡eth0上并没有抓到ping包，只在辅助网卡eth2上抓到了。
[root@sh-saas-k8stest-node-dev-02 ~]# tcpdump -i eth0 -n icmp | grep 10.19.0.11
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C52 packets captured
52 packets received by filter
0 packets dropped by kernel
[root@sh-saas-k8stest-node-dev-02 ops]# tcpdump -i eth2 -n icmp | grep 10.19.0.11
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
17:06:19.949603 IP 10.19.0.11 > 10.19.0.32: ICMP echo request, id 44314, seq 1, length 64
17:06:19.949659 IP 10.19.0.32 > 10.19.0.11: ICMP echo reply, id 44314, seq 1, length 64
17:06:20.949770 IP 10.19.0.11 > 10.19.0.32: ICMP echo request, id 44314, seq 2, length 64
17:06:20.949823 IP 10.19.0.32 > 10.19.0.11: ICMP echo reply, id 44314, seq 2, length 64
17:06:21.949766 IP 10.19.0.11 > 10.19.0.32: ICMP echo request, id 44314, seq 3, length 64
17:06:21.949824 IP 10.19.0.32 > 10.19.0.11: ICMP echo reply, id 44314, seq 3, length 64
^C130 packets captured
130 packets received by filter
0 packets dropped by kernel
tracepath的结果可能是显示误差，发现mtr也是这样：
[root@sh-saas-k8stest-node-dev-02 ~]# ip netns exec netns2 mtr -r -c 3 -n 10.19.0.11
Start: Sun May 15 17:15:03 2022
HOST: sh-saas-k8stest-node-dev-02 Loss% Snt Last Avg Best Wrst StDev
1.|-- 10.19.0.22 0.0% 3 0.1 0.1 0.0 0.1 0.0
2.|-- 10.19.0.11 0.0% 3 0.2 0.2 0.2 0.3 0.0
[root@sh-saas-k8stest-master-dev-01 ~]# mtr -r -c 3 -n 10.19.0.32
Start: Sun May 15 17:13:51 2022
HOST: sh-saas-k8stest-master-dev- Loss% Snt Last Avg Best Wrst StDev
1.|-- 10.19.0.22 0.0% 3 0.2 0.2 0.2 0.2 0.0
2.|-- 10.19.0.32 0.0% 3 0.2 0.2 0.2 0.2 0.0
6. 测试结论
- AWS CNI采用的路由策略在腾讯云环境下的弹性网卡同样是适用的，原理一致
- 腾讯云弹性网卡只能绑定一个子网，子网是基于区域（如上海4区）的
- 弹性网卡本身对于CVM来说跟物理机和物理网卡一样，流量进入都是通过弹性网卡本身，而不是CVM的主网卡转发。
7. 参考
路由相关参考：
https://www.jianshu.com/p/8499b53eb0a5
http://ruanhao.cc/blog/2020-03-04-ip-rule-route.html
https://blog.clanzx.net/network/ip-neigh.html
https://blog.csdn.net/chengxuyuanyonghu/article/details/39558643
