Neutron分析（5）—— neutron-l3-agent中的iptables

一.iptables簡介

1.iptables數據包處理流程

以本機爲目的的包，由上至下，走左邊的路
本機產生的包，從local process開始走左邊的路
本機轉發的包，由上至下走右邊的路

簡化流程以下：

 

2.iptables表結構

在neutron中主要用到filter表和nat表
filter表:
Chain INPUT
Chain FORWARD
Chain OUTPUT
filter表用於信息包過濾，它包含INPUT、OUTPUT和FORWARD 鏈。

nat表:
Chain PREROUTING
Chain OUTPUT
Chain POSTROUTING
nat表用於網絡地址轉換，PREROUTING鏈由指定信息包一到達防火牆就改變它們的規則所組成，而 POSTROUTING 鏈由指定正當信息包打算離開防火牆時改變它們的規則所組成。

More：
Traversing of tables and chains
Linux Firewalls Using iptables

二.l3 agent消息處理

_rpc_loop  ---  _process_router            ---  _router_added
 
                                           ---  process_router
                                 
                                           ---  _router_removed
           
           ---  _process_router_delete     ---  _router_removed

 

在上面幾個方法中，會涉及到iptables的處理。

三.iptables_manager初始化

iptables_manager的初始化是在class IptablesManager中完成的，它對iptables的鏈進行了包裝。

源碼目錄：neutron/neutron/agent/linux/iptables_manager.py

主要操做：

新建一個neutron-filter-top鏈，這個是沒有包裝的，加在原生的FORWARD和OUTPUT鏈上。
對filter表的INPUT，OUTPUT，FORWARD鏈進行包裝，將到達原鏈的數據包轉發到包裝鏈，還增長一個包裝的local鏈。
對於nat表，PREROUTING，OUTPUT，POSTROUTING鏈進行包裝，另外在POSTROUTING鏈以後加了snat鏈。

代碼分析：

對於l3 agent，binary_name是neturon-l3-agent。

filter表的操做：
增長一個鏈neutron-filter-top，增長規則：
-A FORWARD -j neutron-filter-top
-A OUTPUT -j neutron-filter-top

增長一個包裝鏈neutron-l3-agent-local，增長規則：
-A neutron-filter-top -j neutron-l3-agent-local

        # Add a neutron-filter-top chain. It's intended to be shared
        # among the various nova components. It sits at the very top
        # of FORWARD and OUTPUT.
        for tables in [self.ipv4, self.ipv6]:
            tables['filter'].add_chain('neutron-filter-top', wrap=False)
            tables['filter'].add_rule('FORWARD', '-j neutron-filter-top',
                                      wrap=False, top=True)
            tables['filter'].add_rule('OUTPUT', '-j neutron-filter-top',
                                      wrap=False, top=True)

            tables['filter'].add_chain('local')
            tables['filter'].add_rule('neutron-filter-top', '-j $local',
                                      wrap=False)

 

包裝IPv4和IPv6 filter表的INPUT，OUTPUT，FORWARD鏈，以及IPv4 nat表的PREROUTING，OUTPUT，POSTROUTING鏈。

將到達原鏈的數據包轉發到包裝鏈：

        # Wrap the built-in chains
        builtin_chains = {4: {'filter': ['INPUT', 'OUTPUT', 'FORWARD']},
                          6: {'filter': ['INPUT', 'OUTPUT', 'FORWARD']}}

        if not state_less:
            self.ipv4.update(
                {'nat': IptablesTable(binary_name=self.wrap_name)})
            builtin_chains[4].update({'nat': ['PREROUTING',
                                      'OUTPUT', 'POSTROUTING']})

        for ip_version in builtin_chains:
            if ip_version == 4:
                tables = self.ipv4
            elif ip_version == 6:
                tables = self.ipv6

            for table, chains in builtin_chains[ip_version].iteritems():
                for chain in chains:
                    tables[table].add_chain(chain)
                    tables[table].add_rule(chain, '-j $%s' %
                                           (chain), wrap=False)

 

包裝鏈neutron-l3-agent-INPUT，neutron-l3-agent-OUTPUT，neutron-l3-agent-FORWARD，添加規則：
-A INPUT -j neutron-l3-agent-INPUT
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A FORWARD -j neutron-l3-agent-FORWARD

nat表的操做：
（承上面的代碼）
包裝鏈neutron-l3-agent-PREROUTING，neutron-l3-agent-OUTPUT，neutron-l3-agent-POSTROUTING，添加規則：
-A PREROUTING -j neutron-l3-agent-PREROUTING
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A POSTROUTING -j neutron-l3-agent-POSTROUTING

nat表中添加neutron-postrouting-bottom鏈，增長規則：
-A POSTROUTING -j neutron-postrouting-bottom

nat表中添加包裝鏈neutron-l3-agent-snat，增長規則：
-A neutron-postrouting-bottom -j neutron-l3-agent-snat

nat表中添加包裝鏈neutron-l3-agent-float-snat，增長規則：
-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat

代碼以下：

        if not state_less:
            # Add a neutron-postrouting-bottom chain. It's intended to be
            # shared among the various nova components. We set it as the last
            # chain of POSTROUTING chain.
            self.ipv4['nat'].add_chain('neutron-postrouting-bottom',
                                       wrap=False)
            self.ipv4['nat'].add_rule('POSTROUTING',
                                      '-j neutron-postrouting-bottom',
                                      wrap=False)

            # We add a snat chain to the shared neutron-postrouting-bottom
            # chain so that it's applied last.
            self.ipv4['nat'].add_chain('snat')
            self.ipv4['nat'].add_rule('neutron-postrouting-bottom',
                                      '-j $snat', wrap=False)

            # And then we add a float-snat chain and jump to first thing in
            # the snat chain.
            self.ipv4['nat'].add_chain('float-snat')
            self.ipv4['nat'].add_rule('snat', '-j $float-snat')

 

四.l3 agent代碼中關於iptables的處理

1._router_added

_router_added方法，建立和metadata相關的iptables規則：

    def _router_added(self, router_id, router):
        ri = RouterInfo(router_id, self.root_helper,
                        self.conf.use_namespaces, router)
        self.router_info[router_id] = ri
        if self.conf.use_namespaces:
            self._create_router_namespace(ri)
        for c, r in self.metadata_filter_rules():
            ri.iptables_manager.ipv4['filter'].add_rule(c, r)
        for c, r in self.metadata_nat_rules():
            ri.iptables_manager.ipv4['nat'].add_rule(c, r)
        ri.iptables_manager.apply()
        super(L3NATAgent, self).process_router_add(ri)
        if self.conf.enable_metadata_proxy:
            self._spawn_metadata_proxy(ri.router_id, ri.ns_name)

 

1.metadata_filter_rules方法中，若是enable_metadata_proxy爲True，增長規則

    def metadata_filter_rules(self):
        rules = []
        if self.conf.enable_metadata_proxy:
            rules.append(('INPUT', '-s 0.0.0.0/0 -d 127.0.0.1 '
                          '-p tcp -m tcp --dport %s '
                          '-j ACCEPT' % self.conf.metadata_port))
        return rules

 

而後在filter表中增長這條規則，接受全部從外面進來到達metadata_port端口的數據包：
-A neutron-l3-agent-INPUT -s 0.0.0.0/0 -d 127.0.0.1 -p tcp -m tcp –dport 9697 -j ACCEPT

2.metadata_nat_rules方法，若是enable_metadata_proxy爲True，增長規則

    def metadata_nat_rules(self):
        rules = []
        if self.conf.enable_metadata_proxy:
            rules.append(('PREROUTING', '-s 0.0.0.0/0 -d 169.254.169.254/32 '
                          '-p tcp -m tcp --dport 80 -j REDIRECT '
                          '--to-port %s' % self.conf.metadata_port))
        return rules

 

而後在nat表中增長這條規則作DNAT轉換，在route以前，將虛擬機訪問169.254.169.254端口80的數據包重定向到metadat_port端口：
-A neutron-l3-agent-PREROUTING -s 0.0.0.0/0 -d 169.254.169.254/32 -p tcp -m tcp –dport 80 -j REDIRECT –to-port 9697

再調用iptables_manager.apply()方法，應用規則：
iptables-save -c ，獲取當前全部iptables信息；
iptables-restore -c ，應用最新的iptables配置；

2.process_router

process_router方法：

1.perform_snat_action，爲external gateway處理SNAT規則

    def _handle_router_snat_rules(self, ri, ex_gw_port, internal_cidrs,
                                  interface_name, action):
        # Remove all the rules
        # This is safe because if use_namespaces is set as False
        # then the agent can only configure one router, otherwise
        # each router's SNAT rules will be in their own namespace
        ri.iptables_manager.ipv4['nat'].empty_chain('POSTROUTING')
        ri.iptables_manager.ipv4['nat'].empty_chain('snat')

        # Add back the jump to float-snat
        ri.iptables_manager.ipv4['nat'].add_rule('snat', '-j $float-snat')

        # And add them back if the action if add_rules
        if action == 'add_rules' and ex_gw_port:
            # ex_gw_port should not be None in this case
            # NAT rules are added only if ex_gw_port has an IPv4 address
            for ip_addr in ex_gw_port['fixed_ips']:
                ex_gw_ip = ip_addr['ip_address']
                if netaddr.IPAddress(ex_gw_ip).version == 4:
                    rules = self.external_gateway_nat_rules(ex_gw_ip,
                                                            internal_cidrs,
                                                            interface_name)
                    for rule in rules:
                        ri.iptables_manager.ipv4['nat'].add_rule(*rule)
                    break
        ri.iptables_manager.apply()

 

先清空nat表的neutron-l3-agent-POSTROUTING鏈和neutron-l3-agent-snat鏈；

再在nat表的neutron-l3-agent-snat鏈添加規則：

-A neutron-l3-agent-snat -j neutron-l3-agent-float-snat

而後對應add_rules操做，則處理external_gateway_nat_rules，處理完後在nat表中添加規則：

    def external_gateway_nat_rules(self, ex_gw_ip, internal_cidrs,
                                   interface_name):
        rules = [('POSTROUTING', '! -i %(interface_name)s '
                  '! -o %(interface_name)s -m conntrack ! '
                  '--ctstate DNAT -j ACCEPT' %
                  {'interface_name': interface_name})]
        for cidr in internal_cidrs:
            rules.extend(self.internal_network_nat_rules(ex_gw_ip, cidr))
        return rules

 

規則命令以下：

-A neutron-l3-agent-POSTROUTING ! -i qg-XXX ! -o qg-XXX -m conntrack ! –ctstate DNAT -j ACCEPT

這條命令的意思是除了出口和入口都爲qg-XXX，（qg便是router上的外部網關接口）匹配除了DNAT以外的其餘狀態。

而後處理internal_network_nat_rules：

    def internal_network_nat_rules(self, ex_gw_ip, internal_cidr):
        rules = [('snat', '-s %s -j SNAT --to-source %s' %
                 (internal_cidr, ex_gw_ip))]
        return rules

 

規則命令以下：

-A neutron-l3-agent-snat -s internal_cidr -j SNAT –to-source ex_gw_ip

2.process_router_floating_ip_nat_rules方法，處理floating ip，做SNAT/DNAT轉換。

    def process_router_floating_ip_nat_rules(self, ri):
        """Configure NAT rules for the router's floating IPs.

        Configures iptables rules for the floating ips of the given router
        """
        # Clear out all iptables rules for floating ips
        ri.iptables_manager.ipv4['nat'].clear_rules_by_tag('floating_ip')

        # Loop once to ensure that floating ips are configured.
        for fip in ri.router.get(l3_constants.FLOATINGIP_KEY, []):
            # Rebuild iptables rules for the floating ip.
            fixed = fip['fixed_ip_address']
            fip_ip = fip['floating_ip_address']
            for chain, rule in self.floating_forward_rules(fip_ip, fixed):
                ri.iptables_manager.ipv4['nat'].add_rule(chain, rule,
                                                         tag='floating_ip')

        ri.iptables_manager.apply()

   def floating_forward_rules(self, floating_ip, fixed_ip):
        return [('PREROUTING', '-d %s -j DNAT --to %s' %
                 (floating_ip, fixed_ip)),
                ('OUTPUT', '-d %s -j DNAT --to %s' %
                 (floating_ip, fixed_ip)),
                ('float-snat', '-s %s -j SNAT --to %s' %
                 (fixed_ip, floating_ip))]

 

先清理nat表全部的floationg ip規則；而後floating_forward_rules方法,在nat表中處理floating ip和fixed ip的NAT轉換：

 

具體規則以下：
-A neutron-l3-agent-PREROUTING -d floating_ip -j DNAT –to fixed_ip
-A neutron-l3-agent-OUTPUT -d floating_ip -j DNAT –to fixed_ip
-A neutron-l3-agent-float-snat -s fixed_ip -j SNAT –to floating_ip

3._router_removed

_router_removed方法，刪除和metadata相關的規則：

    def _router_removed(self, router_id):
        ri = self.router_info.get(router_id)
        if ri is None:
            LOG.warn(_("Info for router %s were not found. "
                       "Skipping router removal"), router_id)
            return
        ri.router['gw_port'] = None
        ri.router[l3_constants.INTERFACE_KEY] = []
        ri.router[l3_constants.FLOATINGIP_KEY] = []
        self.process_router(ri)
        for c, r in self.metadata_filter_rules():
            ri.iptables_manager.ipv4['filter'].remove_rule(c, r)
        for c, r in self.metadata_nat_rules():
            ri.iptables_manager.ipv4['nat'].remove_rule(c, r)
        ri.iptables_manager.apply()
        if self.conf.enable_metadata_proxy:
            self._destroy_metadata_proxy(ri.router_id, ri.ns_name)
        del self.router_info[router_id]
        self._destroy_router_namespace(ri.ns_name)

 

五.總結

l3 agent初始化完成後，iptables處理流程以下：

感謝春祥提供圖片！

Reference：
Neutron中的iptables

本文轉自http://squarey.me/cloud-virtualization/iptables_usage_in_l3_agent.html