Hp helion openstack

North-South routing with SNAT

Download 467.98 Kb.

Page	6/8
Date	28.01.2017
Size	467.98 Kb.
	#8870

1 2 3 4 5 6 7 8

2.7.North-South routing with SNAT

If we want instance to be accessible from external world (such as for web servers), we might assign Floating IP (external world IP) as we discussed previously. That scenario use distributed routing and static NAT so it is handled directly in compute node. Helion OpenStack also supports situation when VMs are sharing single external IP and dynamic NAT (PAT) is used so VMs are able to access external resources. Typical use is to have ability to download patches and packages from Internet or internal repositories. Currently this type of traffic is centralized (this comes with performance limitations so front end systems should always use rather Floating IP). Traffic is sent to network node (by default hosted on Helion OpenStack controller node).

Let’s choose one of our instances with no Floating IP assigned:

root@helion-ProLiant-DL380-Gen9:~# nova list --all-tenants 1 --tenant baa7096fe1d54571900c3758397e0939 --fields name,OS-EXT-SRV-ATTR:host,OS-EXT-SRV-ATTR:instance_name,Networks

+--------------------------------------+------------+-------------------------------------+--------------------------------+-------------------------------+

+--------------------------------------+------------+-------------------------------------+--------------------------------+-------------------------------+

+--------------------------------------+------------+-------------------------------------+--------------------------------+-------------------------------+

2.7.1.Traffic leaving VM

Find out VM tap interface and let’s listen to traffic

root@overcloud-novacompute0-vli5de2egecg:~# virsh dumpxml instance-0000010e | grep -A 7 "

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on tapaeee0c10-2e, link-type EN10MB (Ethernet), capture size 262144 bytes

03:52:23.952227 fa:16:3e:fd:7f:88 (oui Unknown) > fa:16:3e:07:de:20 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.9 > 10.0.98.1: ICMP echo request, id 5015, seq 70, length 64

03:52:23.972099 fa:16:3e:1f:87:98 (oui Unknown) > fa:16:3e:fd:7f:88 (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.98.1 > 192.168.10.9: ICMP echo reply, id 5015, seq 70, length 64

03:52:24.953261 fa:16:3e:fd:7f:88 (oui Unknown) > fa:16:3e:07:de:20 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.9 > 10.0.98.1: ICMP echo request, id 5015, seq 71, length 64

03:52:24.960822 fa:16:3e:1f:87:98 (oui Unknown) > fa:16:3e:fd:7f:88 (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.98.1 > 192.168.10.9: ICMP echo reply, id 5015, seq 71, length 64

We can see packets going out of VM subnet – destination MAC is gateway MAC (distributed MAC of DVR) and IP destination is somewhere in external world (our external network is 172.16.0.0/16 and from that physical router id routing to 10.0.98.0/24).

2.7.2.Entering vSwitch br-int

As with previous examples we first find input port to br-int:

root@overcloud-novacompute0-vli5de2egecg:~# brctl show | grep -B1 tapaeee0c10-2e

qbraeee0c10-2e 8000.9efc47859395 no qvbaeee0c10-2e

tapaeee0c10-2e

From iptables related bridge we are leaving via qvbaeee0c10-2e, therefore veth pair and port that connects to br-int qvbaeee0c10-2e.

What internal VLAN tag is applied?

root@overcloud-novacompute0-vli5de2egecg:~# ovs-vsctl show | grep -A3 qvoaeee0c10-2e

Port "qvoaeee0c10-2e"

tag: 69

Interface "qvoaeee0c10-2e"

What is port ID?

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl show br-int | grep qvoaeee0c10-2e

235(qvoaeee0c10-2e): addr:4a:2b:f6:9f:8d:19

Print OpenFlow rules in table 0:

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl dump-flows br-int table=0

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3847209.754s, table=0, n_packets=2, n_bytes=220, idle_age=65534, hard_age=65534, priority=2,in_port=87,dl_src=fa:16:3f:5d:a5:3f actions=resubmit(,1)

cookie=0x0, duration=3847209.944s, table=0, n_packets=2341, n_bytes=235364, idle_age=0, hard_age=65534, priority=2,in_port=87,dl_src=fa:16:3f:4d:1f:fb actions=resubmit(,1)

cookie=0x0, duration=3847210.061s, table=0, n_packets=88100169, n_bytes=13792305674, idle_age=0, hard_age=65534, priority=1 actions=NORMAL

cookie=0x0, duration=2242139.596s, table=0, n_packets=16314557, n_bytes=4001739751, idle_age=0, hard_age=65534, priority=3,in_port=86,vlan_tci=0x0000 actions=mod_vlan_vid:57,NORMAL

We are using NORMAL switching behavior so let’s look into forwarding table and search for our destination MAC:

root@overcloud-novacompute0-vli5de2egecg:~# ovs-appctl fdb/show br-int | grep fa:16:3e:07:de:20

212 69 fa:16:3e:07:de:20 0

What is name of this port?

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl show br-int | grep '212('

212(qr-9ab15d1e-3d): addr:00:00:00:00:00:00

This means we are leaving to router

2.7.3.Router

As with previous examples find out router ID.

root@helion-ProLiant-DL380-Gen9:~# neutron router-list

+--------------------------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| id | name | external_gateway_info |

| 2e791c6d-b0ed-45b4-b04b-68a712b118ac | router1 | {"network_id": "3a5b5cd4-0c4b-4bc3-b44e-826c7b19556e", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "e3be37fb-1ced-432f-950c-99b887bb52c2", "ip_address": "172.16.2.157"}]} |

Tenant IP addresses can be overlapping between tenants therefore every router sits in its own name space. Name contains router ID so we can find it:

root@overcloud-novacompute0-vli5de2egecg:~# ip netns | grep 2e791c6d-b0ed-45b4-b04b-68a712b118ac

qrouter-2e791c6d-b0ed-45b4-b04b-68a712b118ac

Name space is string starting with „qrouter-“ followed by router ID. What interfaces are in

root@overcloud-novacompute0-vli5de2egecg:~# ip netns exec qrouter-2e791c6d-b0ed-45b4-b04b-68a712b118ac ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: rfp-2e791c6d-b: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

link/ether 8a:26:b3:8e:eb:68 brd ff:ff:ff:ff:ff:ff

inet 169.254.30.210/31 scope global rfp-2e791c6d-b

valid_lft forever preferred_lft forever

inet 172.16.2.3/32 brd 172.16.2.3 scope global rfp-2e791c6d-b

valid_lft forever preferred_lft forever

inet6 fe80::8826:b3ff:fe8e:eb68/64 scope link

valid_lft forever preferred_lft forever

632: qr-9ab15d1e-3d: mtu 1500 qdisc noqueue state UNKNOWN group default

link/ether fa:16:3e:07:de:20 brd ff:ff:ff:ff:ff:ff

inet 192.168.10.1/24 brd 192.168.10.255 scope global qr-9ab15d1e-3d

valid_lft forever preferred_lft forever

inet6 fe80::f816:3eff:fe07:de20/64 scope link

valid_lft forever preferred_lft forever

634: qr-f01425f2-58: mtu 1500 qdisc noqueue state UNKNOWN group default

link/ether fa:16:3e:42:d7:50 brd ff:ff:ff:ff:ff:ff

inet 192.168.20.1/24 brd 192.168.20.255 scope global qr-f01425f2-58

valid_lft forever preferred_lft forever

inet6 fe80::f816:3eff:fe42:d750/64 scope link

valid_lft forever preferred_lft forever

In our scenario (SNAT) we are not able to route and NAT traffic in distributed way so router does not route and traffic is sent back from very same interface it has come.

root@overcloud-novacompute0-vli5de2egecg:~# ip netns exec qrouter-2e791c6d-b0ed-45b4-b04b-68a712b118ac ip rule list

0: from all lookup local

32766: from all lookup main

32767: from all lookup default

32797: from 192.168.10.8 lookup 16

3232238081: from 192.168.10.1/24 lookup 3232238081

3232240641: from 192.168.20.1/24 lookup 3232240641

root@overcloud-novacompute0-vli5de2egecg:~# ip netns exec qrouter-2e791c6d-b0ed-45b4-b04b-68a712b118ac ip route show table 3232238081

default via 192.168.10.7 dev qr-9ab15d1e-3d

So you might wonder why bother going throw router. Our current destination MAC is VM default gateway which is distributed router MAC – and that is shared between compute nodes. We do not want to use this when talking via underlay as in some cases (VLAN isolation instead of our VXLAN solution here) this would lead to same MAC in multiple locations. We are going to swap this address to something else.

root@overcloud-novacompute0-vli5de2egecg:~# ip netns exec qrouter-2e791c6d-b0ed-45b4-b04b-68a712b118ac tcpdump icmp -e -l -i qr-9ab15d1e-3d

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on qr-9ab15d1e-3d, link-type EN10MB (Ethernet), capture size 262144 bytes

03:53:31.052919 fa:16:3e:fd:7f:88 (oui Unknown) > fa:16:3e:07:de:20 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.9 > 10.0.98.1: ICMP echo request, id 5015, seq 137, length 64

03:53:31.052963 fa:16:3e:07:de:20 (oui Unknown) > fa:16:3e:1f:87:98 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.9 > 10.0.98.1: ICMP echo request, id 5015, seq 137, length 64

03:53:32.054817 fa:16:3e:fd:7f:88 (oui Unknown) > fa:16:3e:07:de:20 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.9 > 10.0.98.1: ICMP echo request, id 5015, seq 138, length 64

03:53:32.054846 fa:16:3e:07:de:20 (oui Unknown) > fa:16:3e:1f:87:98 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.9 > 10.0.98.1: ICMP echo request, id 5015, seq 138, length 64

2.7.4.Back from router to br-int

Now we are back in br-int and without any VLAN

root@overcloud-novacompute0-vli5de2egecg:~# ovs-vsctl show | grep -A3 qr-9ab15d1e-3d

Port "qr-9ab15d1e-3d"

tag: 69

Interface "qr-9ab15d1e-3d"

type: internal

But this time destination MAC address is different so let’s do lookup in forwarding table.

root@overcloud-novacompute0-vli5de2egecg:~# ovs-appctl fdb/show br-int | grep fa:16:3e:1f:87:98

87 69 fa:16:3e:1f:87:98 0

What is port 87?

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl show br-int | grep '87('

87(patch-tun): addr:b2:5f:40:f0:2a:4f

It is patch between br-int and br-tun.

2.7.5.Going out of compute node

What ports we have on br-tun and where they are pointing to?

root@overcloud-novacompute0-vli5de2egecg:~# ovs-vsctl show | grep -A100 br-tun

Bridge br-tun

Port patch-int

Interface patch-int

type: patch

options: {peer=patch-tun}

Port br-tun

Interface br-tun

type: internal

Port "vxlan-0a000a17"

Interface "vxlan-0a000a17"

type: vxlan

options: {df_default="false", in_key=flow, local_ip="10.0.10.14", out_key=flow, remote_ip="10.0.10.23"}

Port "vxlan-0a000a0a"

Interface "vxlan-0a000a0a"

type: vxlan

options: {df_default="false", in_key=flow, local_ip="10.0.10.14", out_key=flow, remote_ip="10.0.10.10"}

ovs_version: "2.3.0"

This time we are interested in highlighted virtual port which is VXLAN tunnel towards network node. Let’s find out IDs.

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl show br-tun | grep '('

OFPT_FEATURES_REPLY (xid=0x2): dpid:00009e4ffab46e48

1(patch-int): addr:7a:c7:3a:cf:90:5e

2(vxlan-0a000a0a): addr:ba:0c:97:69:99:7f

5(vxlan-0a000a17): addr:8a:30:a7:83:71:08

LOCAL(br-tun): addr:9e:4f:fa:b4:6e:48

OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

We can start going throw OpenFlow table 0.

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl dump-flows br-tun table=0

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3923903.494s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop

cookie=0x0, duration=3082115.327s, table=0, n_packets=2385586, n_bytes=253280753, idle_age=0, hard_age=65534, priority=1,in_port=5 actions=resubmit(,4)

cookie=0x0, duration=3923901.546s, table=0, n_packets=13713693, n_bytes=993529751, idle_age=0, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)

cookie=0x0, duration=3923887.172s, table=0, n_packets=446801, n_bytes=87876656, idle_age=1, hard_age=65534, priority=1,in_port=2 actions=resubmit(,4)

Next processing is in table 1 and as there are quite a few records there we will look specifically for mentions of VLAN 69.

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl dump-flows br-tun table=1,dl_vlan=69

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=1030733.343s, table=1, n_packets=2527, n_bytes=247390, idle_age=1, hard_age=65534, priority=1,dl_vlan=69,dl_src=fa:16:3e:07:de:20 actions=mod_dl_src:fa:16:3f:9e:30:0c,resubmit(,2)

cookie=0x0, duration=1030733.441s, table=1, n_packets=2, n_bytes=276, idle_age=65534, hard_age=65534, priority=2,dl_vlan=69,dl_dst=fa:16:3e:07:de:20 actions=drop

cookie=0x0, duration=1030733.539s, table=1, n_packets=15, n_bytes=630, idle_age=2352, hard_age=65534, priority=3,arp,dl_vlan=69,arp_tpa=192.168.10.1 actions=drop

We have no match here, our packet is matched by something more generic. Let’s find out.

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl dump-flows br-tun table=1

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3924083.843s, table=1, n_packets=13673850, n_bytes=990360128, idle_age=0, hard_age=65534, priority=0 actions=resubmit(,2)

cookie=0x0, duration=1030844.908s, table=1, n_packets=12959, n_bytes=1269982, idle_age=55489, hard_age=65534, priority=1,dl_vlan=70,dl_src=fa:16:3e:42:d7:50 actions=mod_dl_src:fa:16:3f:9e:30:0c,resubmit(,2)

...

So we need to continue in table 2.

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl dump-flows br-tun table=2

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3924131.804s, table=2, n_packets=4736846, n_bytes=410971859, idle_age=1, hard_age=65534, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)

cookie=0x0, duration=3924131.710s, table=2, n_packets=8976032, n_bytes=582566690, idle_age=0, hard_age=65534, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)

Our packet is unicast therefore we go to table 20.

root@overcloud-novacompute0-vli5de2egecg:~# ovs-ofctl dump-flows br-tun table=20,dl_vlan=69

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=1030945.736s, table=20, n_packets=2805, n_bytes=270522, idle_age=0, hard_age=65534, priority=2,dl_vlan=69,dl_dst=fa:16:3e:1f:87:98 actions=strip_vlan,set_tunnel:0x3f2,output:2

cookie=0x0, duration=1030945.836s, table=20, n_packets=170, n_bytes=16275, idle_age=2565, hard_age=65534, priority=2,dl_vlan=69,dl_dst=fa:16:3e:b2:3d:19 actions=strip_vlan,set_tunnel:0x3f2,output:2

cookie=0x0, duration=986052.703s, table=20, n_packets=43248, n_bytes=4170488, idle_age=65534, hard_age=65534, priority=2,dl_vlan=69,dl_dst=fa:16:3e:fd:7f:88 actions=strip_vlan,set_tunnel:0x3f2,output:5

Here we go, there is match with destination MAC. We are going to strip internal VLAN tag, add VXLAN VNI 3F2 and send packet via port 2, which is tunnel to network node.

Lastly let’s check how packets going out of compute node look.

root@overcloud-novacompute0-vli5de2egecg:~# tcpdump -e -i eth0 -c 200 | grep -B1 192.168.10.9

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

03:55:02.198288 fc:15:b4:84:12:98 (oui Unknown) > 00:25:6e:0a:f6:07 (oui Unknown), ethertype IPv4 (0x0800), length 148: overcloud-NovaCompute0-vli5de2egecg.35509 > overcloud-controller0-sujhw52cufku.4789: VXLAN, flags [I] (0x08), vni 1010

fa:16:3f:9e:30:0c (oui Unknown) > fa:16:3e:1f:87:98 (oui Unknown), ethertype IPv4 (0x0800), length 98: 192.168.10.9 > 10.0.98.1: ICMP echo request, id 5015, seq 228, length 64

03:55:02.202161 00:25:6e:0a:f6:07 (oui Unknown) > fc:15:b4:84:12:98 (oui Unknown), ethertype IPv4 (0x0800), length 148: overcloud-controller0-sujhw52cufku.46717 > overcloud-NovaCompute0-vli5de2egecg.4789: VXLAN, flags [I] (0x08), vni 1010

fa:16:3e:1f:87:98 (oui Unknown) > fa:16:3e:fd:7f:88 (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.98.1 > 192.168.10.9: ICMP echo reply, id 5015, seq 228, length 64

2.7.6.Receiving in Network Node

What ports network node use?

root@overcloud-controller0-sujhw52cufku:~# ovs-vsctl show

1718fb09-77ba-4171-80a8-86b1dcdfe4bb

Bridge br-tun

Port "vxlan-0a000a17"

Interface "vxlan-0a000a17"

type: vxlan

options: {df_default="false", in_key=flow, local_ip="10.0.10.10", out_key=flow, remote_ip="10.0.10.23"}

Port br-tun

Interface br-tun

type: internal

Port "vxlan-0a000a0e"

Interface "vxlan-0a000a0e"

type: vxlan

options: {df_default="false", in_key=flow, local_ip="10.0.10.10", out_key=flow, remote_ip="10.0.10.14"}

Port patch-int

Interface patch-int

type: patch

options: {peer=patch-tun}

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl show br-tun | grep '('

OFPT_FEATURES_REPLY (xid=0x2): dpid:00003602a443274e

1(patch-int): addr:b2:84:3b:2c:07:d6

2(vxlan-0a000a17): addr:16:32:ca:3a:50:06

3(vxlan-0a000a0e): addr:a2:6d:15:1d:b2:cb

LOCAL(br-tun): addr:36:02:a4:43:27:4e

OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

2.7.7.OpenFlow rules in Network Node br-tun

Start in table 0

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl dump-flows br-tun table=0

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3101202.673s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=0 actions=drop

cookie=0x0, duration=3101194.652s, table=0, n_packets=864713, n_bytes=49231013, idle_age=0, hard_age=65534, priority=1,in_port=3 actions=resubmit(,4)

cookie=0x0, duration=3101200.743s, table=0, n_packets=7468912, n_bytes=1230567666, idle_age=0, hard_age=65534, priority=1,in_port=1 actions=resubmit(,1)

cookie=0x0, duration=3101195.474s, table=0, n_packets=10997, n_bytes=892718, idle_age=1043, hard_age=65534, priority=1,in_port=2 actions=resubmit(,4)

As we are coming from port 3 we are instructed to continue in table 4. We expect a lot of rules there so let’s look specifically for our VXLAN VNI.

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl dump-flows br-tun table=4,tun_id=0x3f2

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3101233.824s, table=4, n_packets=11274, n_bytes=1107027, idle_age=0, hard_age=65534, priority=1,tun_id=0x3f2 actions=mod_vlan_vid:14,resubmit(,9)

We assigned VLAN 14 to packet and continue in table 9.

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl dump-flows br-tun table=9

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3101343.824s, table=9, n_packets=833201, n_bytes=46768187, idle_age=1, hard_age=65534, priority=0 actions=resubmit(,10)

cookie=0x0, duration=3101344.201s, table=9, n_packets=9411, n_bytes=700956, idle_age=1186, hard_age=65534, priority=1,dl_src=fa:16:3f:4d:1f:fb actions=output:1

cookie=0x0, duration=3101344.022s, table=9, n_packets=33285, n_bytes=2671458, idle_age=1, hard_age=65534, priority=1,dl_src=fa:16:3f:9e:30:0c actions=output:1

Next to table 10.

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl dump-flows br-tun table=10

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3101409.350s, table=10, n_packets=833226, n_bytes=46769741, idle_age=0, hard_age=65534, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1

We will learn information from packet and store it in table 20. We are now leaving br-tun via patch-int, so we are heading towards br-int.

2.7.8.OpenFlow rules Network Node br-int

We expect quite a lot of ports in Network Node, so let’s find specifically patch port where traffic from br-tun enters br-int.

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl show br-int | grep patch

127(patch-tun): addr:2a:75:6e:b7:0e:10

List rules in table 0.

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl dump-flows br-int table=0

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3106124.326s, table=0, n_packets=9411, n_bytes=700956, idle_age=5966, hard_age=65534, priority=2,in_port=127,dl_src=fa:16:3f:4d:1f:fb actions=resubmit(,1)

cookie=0x0, duration=3106124.168s, table=0, n_packets=38061, n_bytes=3139506, idle_age=0, hard_age=65534, priority=2,in_port=127,dl_src=fa:16:3f:9e:30:0c actions=resubmit(,1)

cookie=0x0, duration=3106124.440s, table=0, n_packets=1614948, n_bytes=176983979, idle_age=0, hard_age=65534, priority=1 actions=NORMAL

cookie=0x0, duration=2406992.930s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=172 actions=drop

cookie=0x0, duration=1908738.959s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=183 actions=drop

cookie=0x0, duration=2941285.074s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=160 actions=drop

cookie=0x0, duration=1908743.232s, table=0, n_packets=1, n_bytes=42, idle_age=65534, hard_age=65534, priority=2,in_port=182 actions=drop

cookie=0x0, duration=3106113.776s, table=0, n_packets=7341436, n_bytes=1148437924, idle_age=1, hard_age=65534, priority=3,in_port=126,vlan_tci=0x0000 actions=mod_vlan_vid:6,NORMAL

We have match for packets from br-tun and source MAC address and continue processing in table 1. Let’s look on rules there specifically for VLAN 14.

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl dump-flows br-int table=1,dl_vlan=14

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=1164110.409s, table=1, n_packets=154, n_bytes=42546, idle_age=401, hard_age=65534, priority=2,ip,dl_vlan=14,nw_dst=192.168.10.0/24 actions=strip_vlan,mod_dl_src:fa:16:3e:07:de:20,output:227,output:121

cookie=0x0, duration=1164114.182s, table=1, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=4,dl_vlan=14,dl_dst=fa:16:3e:b2:3d:19 actions=strip_vlan,mod_dl_src:fa:16:3e:07:de:20,output:121

cookie=0x0, duration=1164110.550s, table=1, n_packets=21661, n_bytes=2122778, idle_age=0, hard_age=65534, priority=4,dl_vlan=14,dl_dst=fa:16:3e:1f:87:98 actions=strip_vlan,mod_dl_src:fa:16:3e:07:de:20,output:227

There is match for destination MAC. We are going to strip vLAN tag a modify source MAC to put back distributed router MAC address and go out of port 227. What it is?

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl show br-int | grep '227('

227(sg-ff1a1932-74): addr:00:00:00:00:00:00

2.7.9.SNAT namespace

Port sg-ff1a1932-74 belongs to SNAT name space which ID we find based on router ID.

root@overcloud-controller0-sujhw52cufku:~# ip netns | grep 2e791c6d-b0ed-45b4-b04b-68a712b118ac

snat-2e791c6d-b0ed-45b4-b04b-68a712b118ac

qrouter-2e791c6d-b0ed-45b4-b04b-68a712b118ac

List interfaces.

root@overcloud-controller0-sujhw52cufku:~# ip netns exec snat-2e791c6d-b0ed-45b4-b04b-68a712b118ac ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

407: qg-b2712c4a-2b: mtu 1500 qdisc noqueue state UNKNOWN group default

link/ether fa:16:3e:c3:f1:8d brd ff:ff:ff:ff:ff:ff

inet 172.16.2.157/16 brd 172.16.255.255 scope global qg-b2712c4a-2b

valid_lft forever preferred_lft forever

inet6 fe80::f816:3eff:fec3:f18d/64 scope link

valid_lft forever preferred_lft forever

409: sg-ff1a1932-74: mtu 1500 qdisc noqueue state UNKNOWN group default

link/ether fa:16:3e:1f:87:98 brd ff:ff:ff:ff:ff:ff

inet 192.168.10.7/24 brd 192.168.10.255 scope global sg-ff1a1932-74

valid_lft forever preferred_lft forever

inet6 fe80::f816:3eff:fe1f:8798/64 scope link

valid_lft forever preferred_lft forever

411: sg-f9e28eef-bc: mtu 1500 qdisc noqueue state UNKNOWN group default

link/ether fa:16:3e:c7:6e:85 brd ff:ff:ff:ff:ff:ff

inet 192.168.20.2/24 brd 192.168.20.255 scope global sg-f9e28eef-bc

valid_lft forever preferred_lft forever

inet6 fe80::f816:3eff:fec7:6e85/64 scope link

valid_lft forever preferred_lft forever

What interface will packet exit? Investigate routing table.

root@overcloud-controller0-sujhw52cufku:~# ip netns exec snat-2e791c6d-b0ed-45b4-b04b-68a712b118ac ip route

default via 172.16.0.1 dev qg-b2712c4a-2b

172.16.0.0/16 dev qg-b2712c4a-2b proto kernel scope link src 172.16.2.157

192.168.10.0/24 dev sg-ff1a1932-74 proto kernel scope link src 192.168.10.7

192.168.20.0/24 dev sg-f9e28eef-bc proto kernel scope link src 192.168.20.2

Before packet leaves we are going to do dynamic source NAT. This is implemented via iptables.

root@overcloud-controller0-sujhw52cufku:~# ip netns exec snat-2e791c6d-b0ed-45b4-b04b-68a712b118ac iptables --table nat --list

Chain PREROUTING (policy ACCEPT)

target prot opt source destination

neutron-l3-agent-PREROUTING all -- anywhere anywhere

Chain INPUT (policy ACCEPT)

target prot opt source destination

Chain OUTPUT (policy ACCEPT)

target prot opt source destination

neutron-l3-agent-OUTPUT all -- anywhere anywhere

Chain POSTROUTING (policy ACCEPT)

target prot opt source destination

neutron-l3-agent-POSTROUTING all -- anywhere anywhere

neutron-postrouting-bottom all -- anywhere anywhere

Chain neutron-l3-agent-OUTPUT (1 references)

target prot opt source destination

Chain neutron-l3-agent-POSTROUTING (1 references)

target prot opt source destination

ACCEPT all -- anywhere anywhere ! ctstate DNAT

Chain neutron-l3-agent-PREROUTING (1 references)

target prot opt source destination

Chain neutron-l3-agent-float-snat (0 references)

target prot opt source destination

Chain neutron-l3-agent-snat (1 references)

target prot opt source destination

SNAT all -- 192.168.10.0/24 anywhere to:172.16.2.157

SNAT all -- 192.168.20.0/24 anywhere to:172.16.2.157

Chain neutron-postrouting-bottom (1 references)

target prot opt source destination

neutron-l3-agent-snat all -- anywhere anywhere

We can now capture traffic leaving SNAT space to see whether source IP has changed.

root@overcloud-controller0-sujhw52cufku:~# ip netns exec snat-2e791c6d-b0ed-45b4-b04b-68a712b118ac tcpdump icmp -e -l -i qg-b2712c4a-2b

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on qg-b2712c4a-2b, link-type EN10MB (Ethernet), capture size 262144 bytes

04:09:17.521252 fa:16:3e:c3:f1:8d (oui Unknown) > 38:22:d6:e9:92:23 (oui Unknown), ethertype IPv4 (0x0800), length 98: 172.16.2.157 > 10.0.98.1: ICMP echo request, id 5015, seq 1082, length 64

04:09:17.526868 38:22:d6:e9:92:23 (oui Unknown) > fa:16:3e:c3:f1:8d (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.98.1 > 172.16.2.157: ICMP echo reply, id 5015, seq 1082, length 64

04:09:18.522571 fa:16:3e:c3:f1:8d (oui Unknown) > 38:22:d6:e9:92:23 (oui Unknown), ethertype IPv4 (0x0800), length 98: 172.16.2.157 > 10.0.98.1: ICMP echo request, id 5015, seq 1083, length 64

04:09:18.528884 38:22:d6:e9:92:23 (oui Unknown) > fa:16:3e:c3:f1:8d (oui Unknown), ethertype IPv4 (0x0800), length 98: 10.0.98.1 > 172.16.2.157: ICMP echo reply, id 5015, seq 1083, length 64

OK – both routing and NAT happened.

2.7.10.Final step

Modified packet now enters br-ex – do we have some VLAN tag here? What is ID of this port?

root@overcloud-controller0-sujhw52cufku:~# ovs-vsctl show | grep -A3 qg-b2712c4a-2b

Port "qg-b2712c4a-2b"

Interface "qg-b2712c4a-2b"

type: internal

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl show br-ex | grep '('

OFPT_FEATURES_REPLY (xid=0x2): dpid:000000256e0af607

1(vlan172): addr:00:25:6e:0a:f6:07

10(qg-e9c2c154-9c): addr:26:01:00:00:00:00

13(qg-92be0f9c-e8): addr:26:01:00:00:00:00

14(qg-ef8abb09-76): addr:26:01:00:00:00:00

15(qg-cf26a386-18): addr:26:01:00:00:00:00

19(qg-15fc384e-a5): addr:00:00:00:00:00:00

22(qg-da9fa75a-7b): addr:00:00:00:00:00:00

25(qg-ddb6b311-95): addr:00:00:00:00:00:00

31(qg-6a140e68-f6): addr:00:00:00:00:00:00

34(qg-88e27db1-6b): addr:00:00:00:00:00:00

42(qg-27387ea9-a9): addr:00:00:00:00:00:00

43(qg-b89620d8-3a): addr:00:00:00:00:00:00

44(qg-b2712c4a-2b): addr:00:00:00:00:00:00

49(qg-184f9c03-d9): addr:00:00:00:00:00:00

LOCAL(br-ex): addr:00:25:6e:0a:f6:07

OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

Highlighted is port 1 which goes to external network (in our case this is on the same physical NIC, but it is tagged in external VLAN 172.

List OpenFlow rules on br-ex.

root@overcloud-controller0-sujhw52cufku:~# ovs-ofctl dump-flows br-ex table=0

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=3722047.411s, table=0, n_packets=199158, n_bytes=88736522, idle_age=0, hard_age=65534, priority=0 actions=NORMAL

We are going to use NORMAL forwarding so let’s look into forwarding table.

root@overcloud-controller0-sujhw52cufku:~# ovs-appctl fdb/show br-ex

port VLAN MAC Age

49 0 fa:16:3e:03:44:e7 127

1 0 38:22:d6:e9:92:23 103

34 0 fa:16:3e:3b:0b:76 103

15 0 fa:16:3e:81:c5:ee 0

44 0 fa:16:3e:c3:f1:8d 0

We have match here for destination MAC so we switch packet to port 1 which is our external VLAN 172 network.

Download 467.98 Kb.

Share with your friends:

1 2 3 4 5 6 7 8