VM to VM in single network on two compute nodes

2.4.2.Setting up tunnels and sending to underlay

root@overcloud-novacompute0-vli5de2egecg:~# ovs-vsctl show

...

Bridge br-tun

Interface patch-int

type: patch

options: {peer=patch-tun}

Port br-tun

Interface br-tun

type: internal

Port "vxlan-0a000a17"

Interface "vxlan-0a000a17"

type: vxlan

options: {df_default="false", in_key=flow, local_ip="10.0.10.14", out_key=flow, remote_ip="10.0.10.23"}

Port "vxlan-0a000a0a"

Interface "vxlan-0a000a0a"

type: vxlan

options: {df_default="false", in_key=flow, local_ip="10.0.10.14", out_key=flow, remote_ip="10.0.10.10"}

ovs_version: "2.3.0"

There is patch port, but also virtual ports for VXLAN tunnels. In our case remote IP 10.0.10.23 is our second compute node while 10.0.10.10 is controller (network node). Tunnel ID is set by OpenFlow rules as we will see.

Let’s gather port IDs:

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl show br-tun | grep '('

OFPT_FEATURES_REPLY (xid=0x2): dpid:00009e4ffab46e48

1(patch-int): addr:7a:c7:3a:cf:90:5e

2(vxlan-0a000a0a): addr:ba:0c:97:69:99:7f

5(vxlan-0a000a17): addr:8a:30:a7:83:71:08

LOCAL(br-tun): addr:9e:4f:fa:b4:6e:48

OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

2.4.3.OpenFlow rules in br-tun

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl dump-flows br-tun table=0

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=2979679.382s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop

cookie=0x0, duration=2137891.215s, table=0, n_packets=1540418, n_bytes=159465418, idle_age=0, hard_age=65534, priority=1,in_port=5 actions=resubmit(,4)

cookie=0x0, duration=2979677.434s, table=0, n_packets=9757051, n_bytes=701535045, idle_age=0, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)

cookie=0x0, duration=2979663.060s, table=0, n_packets=36460, n_bytes=2664832, idle_age=1, hard_age=65534, priority=1,in_port=2 actions=resubmit(,4)

Highlighted item matches our packets – packets coming from patch-int interface, so from our local VMs (br-int). Next processing is instructed to be in table 1.

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl dump-flows br-tun table=1

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3022441.256s, table=1, n_packets=9965883, n_bytes=717582013, idle_age=0, hard_age=65534, priority=0 actions=resubmit(,2)

cookie=0x0, duration=129202.321s, table=1, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,dl_vlan=70,dl_src=fa:16:3e:42:d7:50 actions=mod_dl_src:fa:16:3f:9e:30:0c,resubmit(,2)

cookie=0x0, duration=1920675.743s, table=1, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,dl_vlan=49,dl_src=fa:16:3e:72:b4:78 actions=mod_dl_src:fa:16:3f:9e:30:0c,resubmit(,2)

…

There are rules for different situations (routing) which does not match our packet. Highlighted item is matched so we continue processing in table 2.

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3022697.188s, table=2, n_packets=2839090, n_bytes=250753467, idle_age=0, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)

cookie=0x0, duration=3022697.094s, table=2, n_packets=7130151, n_bytes=467126446, idle_age=0, hard_age=65534, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)

Simple rule – if it is unicast, continue in table 20, is it is broadcast/multicast, process table 22.

So let’s see table 20. There are plenty of rules there so we will filter specifically rules containing our VLAN:

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl dump-flows br-tun table=20,dl_vlan=69

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=129582.625s, table=20, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,dl_vlan=69,dl_dst=fa:16:3e:1f:87:98 actions=strip_vlan,set_tunnel:0x3f2,output:2

cookie=0x0, duration=129582.725s, table=20, n_packets=37, n_bytes=3140, idle_age=42689, hard_age=65534, priority=2,dl_vlan=69,dl_dst=fa:16:3e:b2:3d:19 actions=strip_vlan,set_tunnel:0x3f2,output:2

cookie=0x0, duration=84689.592s, table=20, n_packets=27959, n_bytes=2694902, idle_age=17753, hard_age=65534, priority=2,dl_vlan=69,dl_dst=fa:16:3e:fd:7f:88 actions=strip_vlan,set_tunnel:0x3f2,output:5

We have found our destination MAC which is our destination VM! According to rules we are going to strip VLAN tag, set VXLAN VNI 0x3f2 and send to port 5. As we know already this is virtual port representing VXLAN tunnel to second compute node:

Port "vxlan-0a000a17"

Interface "vxlan-0a000a17"

type: vxlan

options: {df_default="false", in_key=flow, local_ip="10.0.10.14", out_key=flow, remote_ip="10.0.10.23"}

...

5(vxlan-0a000a17): addr:8a:30:a7:83:71:08

2.4.4.Going out of compute node

root@overcloud-novacompute0-vli5de2egecg:~# tcpdump -e -i eth0 -c 100 | grep -B1 192.168.10.9

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

18:40:53.191267 14:58:d0:d3:00:ee (oui Unknown) > fc:15:b4:84:12:98 (oui Unknown), ethertype IPv4 (0x0800), length 148: overcloud-NovaCompute1-c4ia2jfbd75d.46874 > overcloud-NovaCompute0-vli5de2egecg.4789: VXLAN, flags [I] (0x08), vni 1010

fa:16:3e:fd:7f:88 (oui Unknown) > fa:16:3e:21:cf:75 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.9 > 192.168.10.8: ICMP echo reply, id 2569, seq 916, length 64

Investigate its content. There is VXLAN VNI (1010 – hexa 3F2). Outer header addresses are first and second compute node. Internal MAC and IP are not visible to underlay.

2.4.5.Receiving traffic on second compute node

Do we see packet entering second compute node?

root@overcloud-novacompute1-c4ia2jfbd75d:~# tcpdump -e -i eth0 -c 100 | grep -B1 192.168.10.9

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

18:50:18.235754 fc:15:b4:84:12:98 (oui Unknown) > 14:58:d0:d3:00:ee (oui Unknown), ethertype IPv4 (0x0800), length 148: overcloud-NovaCompute0-vli5de2egecg.33236 > overcloud-NovaCompute1-c4ia2jfbd75d.4789: VXLAN, flags [I] (0x08), vni 1010

fa:16:3e:21:cf:75 (oui Unknown) > fa:16:3e:fd:7f:88 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.8 > 192.168.10.9: ICMP echo request, id 2569, seq 1481, length 64

18:50:18.236188 14:58:d0:d3:00:ee (oui Unknown) > fc:15:b4:84:12:98 (oui Unknown), ethertype IPv4 (0x0800), length 148: overcloud-NovaCompute1-c4ia2jfbd75d.46874 > overcloud-NovaCompute0-vli5de2egecg.4789: VXLAN, flags [I] (0x08), vni 1010

fa:16:3e:fd:7f:88 (oui Unknown) > fa:16:3e:21:cf:75 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.9 > 192.168.10.8: ICMP echo reply, id 2569, seq 1481, length 64

We do and it is within VXLAN tunnel.

2.4.6.Entering vSwitch br-tun

First we are going to hit br-tun. Let see what ports we have in this.

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-vsctl show

c8132496-677e-4596-b869-db490bbde09a

Bridge br-tun

Port "vxlan-0a000a0e"

Interface "vxlan-0a000a0e"

type: vxlan

options: {df_default="false", in_key=flow, local_ip="10.0.10.23", out_key=flow, remote_ip="10.0.10.14"}

Port br-tun

Interface br-tun

type: internal

Port "vxlan-0a000a0a"

Interface "vxlan-0a000a0a"

type: vxlan

options: {df_default="false", in_key=flow, local_ip="10.0.10.23", out_key=flow, remote_ip="10.0.10.10"}

Port patch-int

Interface patch-int

type: patch

options: {peer=patch-tun}

…

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-ofctl show br-tun | grep '('

OFPT_FEATURES_REPLY (xid=0x2): dpid:0000bea8033fc743

1(patch-int): addr:12:bb:16:22:89:94

9(vxlan-0a000a0a): addr:ae:d1:7b:25:4a:9f

10(vxlan-0a000a0e): addr:86:5f:dd:f8:52:34

LOCAL(br-tun): addr:be:a8:03:3f:c7:43

OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

From VXLAN packet we know it is coming from compute node 10.0.10.14 so we are interested in virtual vxlan-0a000a0e with ID 10. Next interesting port is patch-int (ID 1) which connects br-tun with br-int, where our VMs are.

2.4.7.OpenFlow rules in receiving br-tun

As always start with table 0.

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-ofctl dump-flows br-tun table=0

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3217627.895s, table=0, n_packets=1, n_bytes=70, idle_age=65534, hard_age=65534, priority=0 actions=drop

cookie=0x0, duration=2334908.443s, table=0, n_packets=3086659, n_bytes=273190971, idle_age=0, hard_age=65534, priority=1,in_port=10 actions=resubmit(,4)

cookie=0x0, duration=3217625.968s, table=0, n_packets=7881046, n_bytes=574586131, idle_age=0, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)

cookie=0x0, duration=2338482.854s, table=0, n_packets=824, n_bytes=118581, idle_age=23628, hard_age=65534, priority=1,in_port=9 actions=resubmit(,4)

Our packet is coming from virtual port 10 (VXLAN tunnel from first compute node) so we are processed in table 4.

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-ofctl dump-flows br-tun table=4

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=2323822.652s, table=4, n_packets=194, n_bytes=30402, idle_age=41121, hard_age=65534, priority=1,tun_id=0x3f8 actions=mod_vlan_vid:13,resubmit(,9)

cookie=0x0, duration=2323818.137s, table=4, n_packets=43, n_bytes=2918, idle_age=1246, hard_age=65534, priority=1,tun_id=0x3f7 actions=mod_vlan_vid:14,resubmit(,9)

cookie=0x0, duration=283537.088s, table=4, n_packets=30573, n_bytes=2956761, idle_age=0, hard_age=65534, priority=1,tun_id=0x3f2 actions=mod_vlan_vid:32,resubmit(,9)

cookie=0x0, duration=1571703.627s, table=4, n_packets=2783687, n_bytes=243107744, idle_age=0, hard_age=65534, priority=1,tun_id=0x3fb actions=mod_vlan_vid:22,resubmit(,9)

cookie=0x0, duration=283531.497s, table=4, n_packets=24, n_bytes=2088, idle_age=647, hard_age=65534, priority=1,tun_id=0x3f3 actions=mod_vlan_vid:33,resubmit(,9)

cookie=0x0, duration=1578175.630s, table=4, n_packets=962, n_bytes=107040, idle_age=904, hard_age=65534, priority=1,tun_id=0x3f6 actions=mod_vlan_vid:19,resubmit(,9)

cookie=0x0, duration=3217700.707s, table=4, n_packets=152, n_bytes=10872, idle_age=65534, hard_age=65534, priority=0 actions=drop

What is happening here is that globally significant VXLAN VNIs are mapped to locally significant VLAN IDs. In our case our traffic will be tagged with VLAN 32 and continue processing in table.

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-ofctl dump-flows br-tun table=9

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3217847.919s, table=9, n_packets=3087116, n_bytes=273274962, idle_age=0, hard_age=65534, priority=0 actions=resubmit(,10)

cookie=0x0, duration=3217848.293s, table=9, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,dl_src=fa:16:3f:5d:a5:3f actions=output:1

cookie=0x0, duration=3217848.106s, table=9, n_packets=994, n_bytes=117609, idle_age=742, hard_age=65534, priority=1,dl_src=fa:16:3f:9e:30:0c actions=output:1

In our situation here we simply jump to 10.

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-ofctl dump-flows br-tun table=10

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3217976.164s, table=10, n_packets=3087488, n_bytes=273308532, idle_age=1, hard_age=65534, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

So we will investigate packet and learn it (information will be stored in table 20). Then we go to port 1 which is patch between br-tun and br-int.

2.4.8.Receiving br-int vSwitch

We are pretty close to destination VM now. Let’s investigate ports on our br-int and especially look what ports are access in VLAN 32.

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-vsctl show | grep -A1 'tag: 32'

tag: 32

Interface "qvoaeee0c10-2e"

--

tag: 32

Interface "qr-9ab15d1e-3d"

OK, we have two ports here. VMs ports start with QVO so that is what we are looking for. Let’s get IDs:

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-ofctl show br-int | grep '('

OFPT_FEATURES_REPLY (xid=0x2): dpid:000032cb98c22142

1(int-br-svc): addr:2e:5b:31:74:67:44

2(patch-tun): addr:36:64:57:50:fd:d1

27(qvoba52a47a-2c): addr:3e:ae:5e:13:d9:9c

28(qr-0d6c7979-4e): addr:00:00:00:00:00:00

29(qr-2ce50678-96): addr:00:00:00:00:00:00

30(qvo4de718f9-e1): addr:92:eb:02:e5:28:36

42(qr-eb3e9a50-e0): addr:00:00:00:00:00:00

43(qvo4882de2e-f0): addr:be:e5:e3:81:ca:3f

47(qr-1da629e3-59): addr:00:00:00:00:00:00

48(qvo24cb5e56-2e): addr:62:0d:de:77:41:ee

49(qvo9f070d74-37): addr:9e:62:f8:31:71:f2

69(qr-9ab15d1e-3d): addr:00:00:00:00:00:00

70(qr-f01425f2-58): addr:00:00:00:00:00:00

71(qvoaeee0c10-2e): addr:e6:be:fe:d4:c2:45

LOCAL(br-int): addr:32:cb:98:c2:21:42

OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

Check OpenFlow rules and start in table 0.

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-ofctl dump-flows br-int table=0

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3235615.981s, table=0, n_packets=1016, n_bytes=121842, idle_age=608, hard_age=65534, priority=2,in_port=2,dl_src=fa:16:3f:9e:30:0c actions=resubmit(,1)

cookie=0x0, duration=3235616.167s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=2,dl_src=fa:16:3f:5d:a5:3f actions=resubmit(,1)

cookie=0x0, duration=3235616.280s, table=0, n_packets=12456333, n_bytes=1552895815, idle_age=0, hard_age=65534, priority=1 actions=NORMAL

cookie=0x0, duration=2201144.250s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=31 actions=drop

cookie=0x0, duration=1589613.591s, table=0, n_packets=3928327, n_bytes=730610264, idle_age=0, hard_age=65534, priority=3,in_port=1,vlan_tci=0x0000 actions=mod_vlan_vid:23,NORMAL

Once again we will hit forward NORMAL so we gonna use simple switching. Let’s list forwarding.

root@overcloud-novacompute1-c4ia2jfbd75d:~# ovs-appctl fdb/show br-int

port VLAN MAC Age

1 23 38:22:d6:e9:92:35 1

49 23 fa:16:3e:b3:43:cd 0

1 23 00:25:6e:0a:f6:07 0

2 22 fa:16:3e:da:9d:c7 0

2 32 fa:16:3e:21:cf:75 0

48 22 fa:16:3e:d1:ab:81 0

47 22 fa:16:3e:03:42:5e 0

71 32 fa:16:3e:fd:7f:88 0

OK – packet will go to port 71, which is qvoaeee0c10-2e.

2.4.9.Final step

From beginning we know local name of our destination VM and also we know that packet comes to qvoaeee0c10-2e which enters iptables bridge as qvbaeee0c10-2e. Let’s find match here.

root@overcloud-novacompute1-c4ia2jfbd75d:~# virsh dumpxml instance-0000010e | grep -A 7 "





'qbraeee0c10-2e'/>

root@overcloud-novacompute1-c4ia2jfbd75d:~# brctl show qbraeee0c10-2e

bridge name bridge id STP enabled interfaces

qbraeee0c10-2e 8000.1268fea17497 no qvbaeee0c10-2e

tapaeee0c10-2e

Download 467.98 Kb.

Share with your friends: