Thermalcircle

climbing the thermals

User Tools

Site Tools

blog:linux:nftables_ipsec_packet_flow

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
blog:linux:nftables_ipsec_packet_flow [2021-12-05] – tiny cosmetics Andrej Stenderblog:linux:nftables_ipsec_packet_flow [2025-01-05] (current) – showing encaps more clearly, no linebreaks Andrej Stender
Line 1: Line 1:
-{{tag>linux netfilter nftables ipsec strongswan charon swanctl xfrm}}+{{tag>linux kernel netfilter nftables ipsec strongswan charon swanctl xfrm}}
 ====== Nftables - Netfilter and VPN/IPsec packet flow ====== ====== Nftables - Netfilter and VPN/IPsec packet flow ======
 ~~META: ~~META:
 date created = 2020-05-30 date created = 2020-05-30
 ~~ ~~
- 
-~~NOTOC~~ 
  
 In this article I like to explain how the packet flow through  In this article I like to explain how the packet flow through 
Line 38: Line 36:
 packets can travel through it. In case of //tunnel-mode// the IP packets which shall travel through the VPN tunnel are being encrypted and then encapsulated in packets of the so-called ESP protocol((Yes, there is also the AH protocol, which can be used as alternative to ESP, but AH only provides authentication and data integrity and no encryption. Thus, it is neither relevant for this article nor widely used in practice.)). The whole thing is then further encapsulated into another (an "outer") IP packet. The reason is that the VPN tunnel itself is merely a point-to-point connection between two VPN endpoints (one source and one destination IP address), but those endpoints are in that case VPN gateways which are used to connect entire subnets on both ends of the tunnel. Thus, the source and destination IP addresses of the "payload" packets which travel through the VPN gateway need to be kept independent from the source and destination IP addresses of the "outer" IP packets. Encapsulation then looks like this (example for a TCP connection): packets can travel through it. In case of //tunnel-mode// the IP packets which shall travel through the VPN tunnel are being encrypted and then encapsulated in packets of the so-called ESP protocol((Yes, there is also the AH protocol, which can be used as alternative to ESP, but AH only provides authentication and data integrity and no encryption. Thus, it is neither relevant for this article nor widely used in practice.)). The whole thing is then further encapsulated into another (an "outer") IP packet. The reason is that the VPN tunnel itself is merely a point-to-point connection between two VPN endpoints (one source and one destination IP address), but those endpoints are in that case VPN gateways which are used to connect entire subnets on both ends of the tunnel. Thus, the source and destination IP addresses of the "payload" packets which travel through the VPN gateway need to be kept independent from the source and destination IP addresses of the "outer" IP packets. Encapsulation then looks like this (example for a TCP connection):
  
-| <html><code>|eth|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|<span style="color: navy;">ip</span>|tcp|payload|</code></html>((Of course, in the actual packet there is no "blank space" between the ''eth'' (ethernet) and the ''ip'' header. I just show it like this here to emphasize WHERE in the packet the ESP header and the outer IP header are being inserted.)) | A "normal" packet which shall travel through the VPN tunnel, is encrypted and encapsulated like this while traversing the VPN gateway. | +| A "normal" packet which shall travel through the VPN tunnel, is encrypted and encapsulated like this while traversing the VPN gateway: | 
-| <html><code>|eth|<span style="color: red;">ip</span>|esp<span style="background-color:#E5E4E2;">|<span style="color: navy;">ip</span>|tcp|payload|</span></code></html>((The darker grey background here shall indicate that this is the part of the whole packet which gets encrypted.))((There is one further detail which I intentionally omit here because it is not relevant for the topic at hand: The ESP protocol in reality does not only possess a //header// but also a //trailer// part which comes after the payload. So, to be completely accurate, the ESP encapsulation in reality looks like this: ''|eth|ip|esp-header|ip|tcp|payload|esp-trailer|''.)) | :::  |+| <html><code>|eth|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|<span style="color: navy;">ip</span>|tcp|payload|</code></html>((Of course, in the actual packet there is no "blank space" between the ''eth'' (ethernet) and the ''ip'' header. I just show it like this here to emphasize WHERE in the packet the ESP header and the outer IP header are being inserted.)) | 
 +| <html><code>|eth|<span style="color: red;">ip</span>|esp<span style="background-color:#E5E4E2;">|<span style="color: navy;">ip</span>|tcp|payload|</span></code></html>((The darker grey background here shall indicate that this is the part of the whole packet which gets encrypted.))((There is one further detail which I intentionally omit here because it is not relevant for the topic at hand: The ESP protocol in reality does not only possess a //header// but also a //trailer// part which comes after the payload. So, to be completely accurate, the ESP encapsulation in reality looks like this: ''|eth|ip|esp-header|ip|tcp|payload|esp-trailer|''.)) |
  
 If //Nat-traversal// is active, then ESP is additionally encapsulated in UDP: If //Nat-traversal// is active, then ESP is additionally encapsulated in UDP:
  
-| <html><code>|eth|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|<span style="color: navy;">ip</span>|tcp|payload|</code></html> |In case of //Nat-traversal// additional encapsulation in UDP (same port ''4500'' as for IKE is then used here). | +| In case of //Nat-traversal// additional encapsulation in UDP (same port ''4500'' as for IKE is then used here): | 
-| <html><code>|eth|<span style="color: red;">ip</span>|udp|esp<span style="background-color:#E5E4E2;">|<span style="color: navy;">ip</span>|tcp|payload|</span></code></html> | :::  |+| <html><code>|eth|&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;|<span style="color: navy;">ip</span>|tcp|payload|</code></html> | 
 +| <html><code>|eth|<span style="color: red;">ip</span>|udp|esp<span style="background-color:#E5E4E2;">|<span style="color: navy;">ip</span>|tcp|payload|</span></code></html> |
  
  
Line 147: Line 147:
  
 | <WRAP>{{:linux:routing-step.png?nolink |}} The routing lookup is performed for incoming as well as local outgoing packets, see Figure {{ref>nfhooksxfrm1}}. Function ''[[https://elixir.bootlin.com/linux/v5.10.46/source/include/net/ip_fib.h#L363|fib_lookup()]]'' performs the actual lookup into the policy routing rules and routing tables. The routing decision resulting from this lookup is attached to the traversing network packet (skb). | <WRAP>{{:linux:routing-step.png?nolink |}} The routing lookup is performed for incoming as well as local outgoing packets, see Figure {{ref>nfhooksxfrm1}}. Function ''[[https://elixir.bootlin.com/linux/v5.10.46/source/include/net/ip_fib.h#L363|fib_lookup()]]'' performs the actual lookup into the policy routing rules and routing tables. The routing decision resulting from this lookup is attached to the traversing network packet (skb).
-It is an instance of two combined structs, the outer ''[[https://elixir.bootlin.com/linux/v5.10.46/source/include/net/route.h#L49|struct rtable]]'' and the inner ''[[https://elixir.bootlin.com/linux/v5.10.46/source/include/net/dst.h#L24|struct dst_entry]]'', Together, both contain information like the output network interface, the ip address of the next hop gateway (if existing), function pointers which determine the path this packet takes through the remaining part of the kernel network stack, and more. At first glance, routing has nothing to do with the Xfrm framework, but it is relevant in this context as you will see below.</WRAP> | +It is an instance of two combined structs, the outer ''[[https://elixir.bootlin.com/linux/v5.10.46/source/include/net/route.h#L49|struct rtable]]'' and the inner ''[[https://elixir.bootlin.com/linux/v5.10.46/source/include/net/dst.h#L24|struct dst_entry]]'', Together, both contain information like the output network interface, the ip address of the next hop gateway (if existing), function pointers which determine the path this packet takes through the remaining part of the kernel network stack, and more. My [[routing_decisions_in_the_linux_kernel_1_lookup_packet_flow|article series on routing]] explains that in detail. At first glance, routing has nothing to do with the Xfrm framework, but it is relevant in this context as you will see below.</WRAP> | 
-| <WRAP>{{:linux:xfrm-action-policy-out.png?nolink |}} This action is performed for forwarded as well as for local outgoing packets after the routing lookup, see Figure {{ref>nfhooksxfrm1}} and function ''[[https://elixir.bootlin.com/linux/v5.10.46/source/net/xfrm/xfrm_policy.c#L3183|xfrm_lookup()]]''. The Xfrm framework performs a lookup into the IPsec SPD, searching for a matching output policy (''dir out'' SP). If no matching policy is found, the network packet stays unchanged and simply continues on its way. If a matching policy is found, a lookup into the SAD is performed to resolve an SA which corresponds to the matching SP (shown as an attached magenta box named //Xfrm lookup state//). If the resolved SA specifies tunnel-mode, then yet another routing lookup is performed, this time for the (future) outer IPv4 packet which will later encapsulate the current packet. The actual packet transformation does not yet happen at this point. Instead, a "bundle" of transformation instructions for this packet is assembled. The term "bundle" stems from the kernel source code and refers to a bunch of struct instances pointing to each other. Among those are the original routing decision of this packet, the SP, the SA((there can be more than one SA being applied to a packet, but that is a less common case)), the routing decision for the future outer IP packet and more. Those are usually assembled around one or several instances of ''[[https://elixir.bootlin.com/linux/v5.10.46/source/include/net/xfrm.h#L925|struct xfrm_dst]]''. The bundle is attached to the network packet (skb), replacing the originally attached routing decision. Function pointers within the bundle ensure, that the packet takes a different path through the remaining part of the kernel network stack.</WRAP> |+| <WRAP>{{:linux:xfrm-action-policy-out.png?nolink |}} This action is performed for forwarded as well as for local outgoing packets after the routing lookup, see Figure {{ref>nfhooksxfrm1}} and function ''[[https://elixir.bootlin.com/linux/v5.10.46/source/net/xfrm/xfrm_policy.c#L3183|xfrm_lookup()]]''. The Xfrm framework performs a lookup into the IPsec SPD, searching for a matching output policy (''dir out'' SP). If no matching policy is found, the network packet stays unchanged and simply continues on its way. If a matching policy is found, a lookup into the SAD is performed to resolve an SA which corresponds to the matching SP (shown as an attached magenta box named //Xfrm lookup state//). If the resolved SA specifies tunnel-mode, then yet another routing lookup is performed, this time for the (future) outer IPv4 packet which will later encapsulate the current packet. The actual packet transformation does not yet happen at this point. Instead, a "bundle" of transformation instructions for this packet is assembled. The term "bundle" stems from the kernel source code and refers to a bunch of struct instances pointing to each other. Among those are the original routing decision of this packet, the SP, the SA((there can be more than one SA being applied to a packet, but that is a less common case)), the routing decision for the future outer IP packet and more. Those are usually assembled around one or several instances of ''[[https://elixir.bootlin.com/linux/v5.10.46/source/include/net/xfrm.h#L925|struct xfrm_dst]]''. The bundle is attached to the network packet (skb), replacing the originally attached routing decision. Function pointers within the bundle ensure, that the packet takes a different path through the remaining part of the kernel network stack. Figure {{ref>xfrm_dst}} shows how a bundle would actually look like. 
 +</WRAP> |
 | <WRAP>{{:linux:xfrm-action-encode.png?nolink |}} This is where packets which shall travel through the VPN tunnel are being encrypted and encapsulated. The Xfrm framework transforms a packet according to the instructions in the "bundle" attached to it. A function pointer within the bundle makes sure, that the packet takes a "detour" into this transformation code after traversing the Netfilter //Postrouting// hook. For IPv4 packets, the entry function which leads the packet on this path is ''[[https://elixir.bootlin.com/linux/v5.10.46/source/net/ipv4/xfrm4_output.c#L31|xfrm4_output()]]''. In case of tunnel-mode this transformation means encapsulating the IP packet into a new outer IP packet and then encapsulating the inner IP packet into ESP protocol and encrypting it and its payload. After completion of the transformation, the xfrm components/instructions are being removed from the "bundle", leaving only the routing decision for the outer IP packet attached to the packet.</WRAP> | | <WRAP>{{:linux:xfrm-action-encode.png?nolink |}} This is where packets which shall travel through the VPN tunnel are being encrypted and encapsulated. The Xfrm framework transforms a packet according to the instructions in the "bundle" attached to it. A function pointer within the bundle makes sure, that the packet takes a "detour" into this transformation code after traversing the Netfilter //Postrouting// hook. For IPv4 packets, the entry function which leads the packet on this path is ''[[https://elixir.bootlin.com/linux/v5.10.46/source/net/ipv4/xfrm4_output.c#L31|xfrm4_output()]]''. In case of tunnel-mode this transformation means encapsulating the IP packet into a new outer IP packet and then encapsulating the inner IP packet into ESP protocol and encrypting it and its payload. After completion of the transformation, the xfrm components/instructions are being removed from the "bundle", leaving only the routing decision for the outer IP packet attached to the packet.</WRAP> |
 | <WRAP>{{:linux:xfrm-action-decode.png?nolink |}} This is where packets which have been received through the VPN tunnel are being decrypted and decapsulated. If an IP packet on the local input path contains an ESP packet, then the Xfrm framework performs a lookup into the SAD (//Xfrm lookup state//), based on the SPI((SPI is an integer value in the unencrypted part of the ESP header)) and the destination IP address. If no matching SA is found, the packet is dropped. If a matching SA is found, the ESP packet is decrypted and decapsulated. In case of tunnel-mode the outer IP packet is decapsulated. The remaining inner IP packet then is re-inserted into the receive path on OSI layer 2. Packets remember which SA has been used on them((in an skb extension named ''sec_path'')). That becomes relevant when they later traverse the //Xfrm lookup in policy// or //Xfrm lookup fwd policy// action.\\ \\ In case of //Nat-traversal// mode, both IKE and ESP packets arrive on the local input path being encapsulated in UDP on port 4500 and the kernel must distinguish between both. This is done based on the so-called //Non-ESP Marker//((4 zero bytes ''0x00000000'' at beginning of UDP payload, defined in [[https://datatracker.ietf.org/doc/html/rfc3948|RFC3948]])). IKE packets are given to a UDP socket where userspace application Strongswan is listening ((There is more to that. Strongswan is required to set a special socket option called ''UDP_ENCAP'' or else it won't receive any IKE packets on port 4500. But that is an implementation detail.)), while ESP packets are decrypted and decapsulated as described above.</WRAP> | | <WRAP>{{:linux:xfrm-action-decode.png?nolink |}} This is where packets which have been received through the VPN tunnel are being decrypted and decapsulated. If an IP packet on the local input path contains an ESP packet, then the Xfrm framework performs a lookup into the SAD (//Xfrm lookup state//), based on the SPI((SPI is an integer value in the unencrypted part of the ESP header)) and the destination IP address. If no matching SA is found, the packet is dropped. If a matching SA is found, the ESP packet is decrypted and decapsulated. In case of tunnel-mode the outer IP packet is decapsulated. The remaining inner IP packet then is re-inserted into the receive path on OSI layer 2. Packets remember which SA has been used on them((in an skb extension named ''sec_path'')). That becomes relevant when they later traverse the //Xfrm lookup in policy// or //Xfrm lookup fwd policy// action.\\ \\ In case of //Nat-traversal// mode, both IKE and ESP packets arrive on the local input path being encapsulated in UDP on port 4500 and the kernel must distinguish between both. This is done based on the so-called //Non-ESP Marker//((4 zero bytes ''0x00000000'' at beginning of UDP payload, defined in [[https://datatracker.ietf.org/doc/html/rfc3948|RFC3948]])). IKE packets are given to a UDP socket where userspace application Strongswan is listening ((There is more to that. Strongswan is required to set a special socket option called ''UDP_ENCAP'' or else it won't receive any IKE packets on port 4500. But that is an implementation detail.)), while ESP packets are decrypted and decapsulated as described above.</WRAP> |
Line 166: Line 167:
 are optional to use and never became the default. The Strongswan documentation calls VPN setups based on those virtual network interfaces [[https://wiki.strongswan.org/projects/strongswan/wiki/RouteBasedVPN|"Route-based VPNs"]]. It seems, essentially two types of virtual interfaces have been introduced in this context over the years: The older ''vti'' interfaces and the newer ''xfrm'' interfaces((Additional kinds of virtual network interfaces exist in this context, like e.g. the ''gre'' interfaces, but they represent an entirely different concept and protocol (''GRE'' protocol) which is e.g. used to build DMVPN setups. That is an advanced topic which works a little different than the "normal" IPsec VPN which I describe here.)). In the remaining part of this article I will describe how the IPsec-based VPN looks like from Netfilter point-of-view in the "normal" case where NO virtual network interfaces are used. are optional to use and never became the default. The Strongswan documentation calls VPN setups based on those virtual network interfaces [[https://wiki.strongswan.org/projects/strongswan/wiki/RouteBasedVPN|"Route-based VPNs"]]. It seems, essentially two types of virtual interfaces have been introduced in this context over the years: The older ''vti'' interfaces and the newer ''xfrm'' interfaces((Additional kinds of virtual network interfaces exist in this context, like e.g. the ''gre'' interfaces, but they represent an entirely different concept and protocol (''GRE'' protocol) which is e.g. used to build DMVPN setups. That is an advanced topic which works a little different than the "normal" IPsec VPN which I describe here.)). In the remaining part of this article I will describe how the IPsec-based VPN looks like from Netfilter point-of-view in the "normal" case where NO virtual network interfaces are used.
  
 +<figure xfrm_dst>
 +{{:linux:xfrm_dst.png?direct&700|}}
 +<caption>Simplified illustration of an Xfrm bundle, attached to a network packet
 +(click to enlarge). In IPsec tunnel-mode, the bundle contains two //routing decisions//,
 +references to IPsec SA and SP and function pointers to lead the packet
 +on the Xfrm encrypt+encapsulate path. Compare it to a normal
 +//routing decision// object, which I described in my
 +[[routing_decisions_in_the_linux_kernel_1_lookup_packet_flow#the_routing_decision_object|article series on routing]].
 +</caption>
 +</figure>
  
 ===== Example Site-to-site VPN ===== ===== Example Site-to-site VPN =====
Line 350: Line 361:
 <figure echo_request_r1_traversal> <figure echo_request_r1_traversal>
 {{:linux:packet-flow-ipsec-tunnel-encrypt.png?direct&700|}} {{:linux:packet-flow-ipsec-tunnel-encrypt.png?direct&700|}}
-<caption> ICMP echo-request ''h1'' -> ''h2'', ''r1'' traversal (click to enlarge)</caption>+<caption> ICMP echo-request ''h1'' -> ''h2'', ''r1'' traversal, encrypt+encapsulate (click to enlarge)</caption>
 </figure> </figure>
  
Line 422: Line 433:
 <figure echo_reply_r1_traversal> <figure echo_reply_r1_traversal>
 {{:linux:packet-flow-ipsec-tunnel-decrypt.png?direct&700|}} {{:linux:packet-flow-ipsec-tunnel-decrypt.png?direct&700|}}
-<caption> ICMP echo-reply ''h2'' -> ''h1'', ''r1'' traversal (click to enlarge)</caption>+<caption> ICMP echo-reply ''h2'' -> ''h1'', ''r1'' traversal, decrypt+decapsulate (click to enlarge)</caption>
 </figure> </figure>
  
Line 551: Line 562:
 Several means have been implemented to address those kind of issues: Several means have been implemented to address those kind of issues:
   * Strongswan provides an optional ''_updown'' script which is called each time when the VPN tunnel comes up or goes down. You can use it to dynamically set/remove the Nftables rules you require. The default version of that script already sets some Iptables rules for you, but depending on your system (kernel/Nftables version) this can create more problems than it solves. And who says that these default rules do fit well with your intended behavior? Thus, if you use this, you probably need to replace the default script with your own version.   * Strongswan provides an optional ''_updown'' script which is called each time when the VPN tunnel comes up or goes down. You can use it to dynamically set/remove the Nftables rules you require. The default version of that script already sets some Iptables rules for you, but depending on your system (kernel/Nftables version) this can create more problems than it solves. And who says that these default rules do fit well with your intended behavior? Thus, if you use this, you probably need to replace the default script with your own version.
-  * Nftables offers IPSEC EXPRESSIONS (syntax ''ipsec {in | out} [ spnum NUM ] ...'') which make it possible to make determine whether a packet is part of the VPN context or not. However, you require a recent version of Nftables and of the Linux kernel for that. Check your man page ''man 8 nft''. See also section [[#Context]] below.+  * Nftables offers IPSEC EXPRESSIONS (syntax ''ipsec {in | out} [ spnum NUM ] ...'') which make it possible to determine whether a packet is part of the VPN context or not. I published a follow-up article which covers that topic in detail: [[nftables_demystifying_ipsec_expressions|Nftables - Demystifying IPsec expressions]].
   * Nftables offers "markers" which you can set and read on traversing packets (syntax ''meta mark'') which can help you to mark packets (set the mark) as being part of the VPN context in one hook/chain and in a later hook/chain you can read the mark again.   * Nftables offers "markers" which you can set and read on traversing packets (syntax ''meta mark'') which can help you to mark packets (set the mark) as being part of the VPN context in one hook/chain and in a later hook/chain you can read the mark again.
   * So-called ''vti'' or ''xfrm'' virtual network interfaces can optionally be used "on top" of the default Xfrm framework behavior.   * So-called ''vti'' or ''xfrm'' virtual network interfaces can optionally be used "on top" of the default Xfrm framework behavior.
- 
-I am planning to describe some of those means in more detail in another article, however I still need to write that one. ;-) I'll place a link here once I find the time to write it. 
  
  
Line 569: Line 578:
  
 ===== Feedback ===== ===== Feedback =====
-[[:feedback|Feedback]] to this article is very welcome!+[[:feedback|Feedback]] to this article is very welcome! Please be aware that I did not develop or contribute to any of the software components described here. I'm merely some developer who took a look at the source code and did some practical experimenting. If you find something which I might have misunderstood or described incorrectly here, then I would be very grateful, if you bring this to my attention and of course I'll then fix my content asap accordingly. 
  
 ===== References ===== ===== References =====
Line 577: Line 586:
   * [[https://ramirose.wixsite.com/ramirosen|Linux Kernel Networking - Implementation and Theory (Rami Rosen, Apress, 2014)]]   * [[https://ramirose.wixsite.com/ramirosen|Linux Kernel Networking - Implementation and Theory (Rami Rosen, Apress, 2014)]]
  
-//published 2020-05-30//, //last modified 2021-12-05//+//published 2020-05-30//, //last modified 2025-01-05//
  
blog/linux/nftables_ipsec_packet_flow.1638735429.txt.gz · Last modified: 2021-12-05 by Andrej Stender