View all posts
Server Admin

14th November 2021

NGINX tuning with Kernel TLS and SSL_sendfile( )

nginx ktls

TLS (Transport Layer Security) is one of the most widely used cryptographic protocols. When TLS is implemented in the kernel (kTLS), performance is improved by reducing the number of copying operations between user and kernel space.

When kTLS and sendfile() are combined, data is encrypted in kernel space before being passed on to the network stack for transmission. As a result, there is no need to copy data into user space to be encrypted by TLS libraries, and then back into kernel space to be transmitted. A key feature of kTLS is the ability to offload TLS processing to hardware, including TLS symmetric crypto processing to network devices.

TLS offloading has been supported in modern Linux and FreeBSD kernels, and now NGINX Open Source is too! When serving static files with SSL_sendfile(), NGINX 1.21.4 introduces support for kTLS, which can improve performance dramatically. In order for NGINX to use SSL_sendfile(), both the kernel and OpenSSL need to be built with kTLS.

A lot has been happening with kTLS implementations recently, and the technology is constantly evolving. The information and instructions here describe support for kTLS as of November 2021, but keep an eye on nginx.org and the NGINX blog for any changes.

General Requirements

  • Operating system – Either of:
    • FreeBSD 13.0+. The only OS which supports kTLS in NGINX is FreeBSD 13.0+ without requiring a manual build of NGINX with OpenSSL 3.0.0+. Please see enable kTLS for NGINX on FreeBSD.
    • Linux distributions built on Linux kernel versions 4.17 or later, though we recommend those built on version 5.2 or later whenever possible. However, OpenSSL 3.0.0 requires kernel header version 4.17 or later (kTLS support is actually available in version 4.13).
  • OpenSSL – Version 3.0.0 or later
  • NGINX – Version 1.21.4 or later (mainline)

Operating System Support

OSs That Support kTLS

kTLS and the indicated ciphers are supported by the following OSs as of November 2021. See TLS Protocol and Cipher Support for details about cipher support.

TLSv1.2 ciphers TLSv1.3
ciphers		 Cipher

TLS_CHACHA20_

POLY1305_SHA256

 Linux kernels
Amazon Linux 2* ✅ ✅ ❌ 5.10
CentOS 8** ✅ ❌ ❌ 4.18
FreeBSD 13.0 ✅ ✅ ❌ *** N/A
RHEL 8 ✅ ❌ ❌ 4.18
SLES 15 SP2 ✅ ✅ ✅ 5.3
Ubuntu 20.04 LTS ✅ ❌​​ ❌ 5.4
Ubuntu 21.04 ✅ ✅ ✅ 5.11
Ubuntu 21.10 ✅ ✅ ✅ 5.13

* Kernel version must be 5.10, not 4.14; see OSs That Do Not Support kTLS and the Amazon Linux 2 FAQ
** Inherits its kTLS support status from RHEL 8 as its upstream source
*** See the FreeBSD commit log

OSs That Do Not Support kTLS

The following OSs do not support kTLS, for the indicated reason:

  • Alpine Linux 3.11–3.14 – With the CONFIG_TLS=n option, kTLS is not built as a module or as part of the kernel.
  • Amazon Linux 2 – According to the Amazon Linux 2 FAQ, the Amazon Linux 2 AMI uses a Linux kernel version of 4.14.
  • CentOS 7.4+ – This kernel inherits the status of support for kTLS from RHEL 7.4+ as its upstream source.
  • Debian 10 and 11 – CONFIG_TLS=n is specified in the kernel configuration (see Debian bug report logs).
  • RHEL 7.4+ – Linux kernel version is 3.10.
  • SLES 12 SP5+ – Linux kernel version is 4.12.
  • Ubuntu 18.04 LTS – Linux kernel version is 4.15.

TLS Protocol and Cipher Support

There is a wide range of support for TLS protocols and ciphers among OSs that support kTLS.

With TLSv1.2, the kTLS module supports these ciphers:

  • AES128-GCM-SHA256
  • AES256-GCM-SHA384
  • ECDHE-RSA-AES128-GCM-SHA256
  • ECDHE-RSA-AES256-GCM-SHA384

With TLSv1.3, the kTLS module supports these cipher suites:

  • TLS_AES_128_GCM_SHA256
  • TLS_AES_256_GCM_SHA384
  • TLS_CHACHA20_POLY1305_SHA256 (only some OSs, as specified in OSs That Support kTLS)

In the directory where NGINX was built (for example, your home directory), run the openssl-3.0.0/.openssl/bin/openssl ciphers command to verify which OpenSSL ciphers are selected.

Enabling kTLS in NGINX

Due to the fact that kTLS handles all encryption and decryption in the kernel, kTLS improves NGINX’s performance. It eliminates the need to copy data into user space to be encrypted by TLS libraries and then back into kernel space for transmission by encrypting it directly in kernel space before passing it to the network stack.

ktls topology

Loading kTLS in the Kernel

Modern FreeBSD and Linux distributions usually include kTLS as a module (with the CONFIG_TLS=m option). Before you start NGINX, you must explicitly load the kTLS module.

On FreeBSD, run these commands as the root user:

kldload ktls ocf
sysctl kern.ipc.tls.enable=1

For details about the FreeBSD command options, see the man page for ktls(4).

On Linux distributions, run this command as the root user:

modprobe tls

Enabling NGINX with kTLS on FreeBSD

You can enable kTLS support in NGINX on FreeBSD by following the same instructions as for Linux distributions. The following steps should be followed to utilize the builds of NGINX with kTLS in the openssl-devel tree in FreeBSD Ports Collection. See TLS Offload in the Kernel on the FreeBSD website for more information, including an overview of kTLS.

Choose the appropriate options in the config menu to build OpenSSL 3.0 with kTLS support:

cd /usr/ports/security/openssl-devel && make config && make install

Modify /etc/make.conf to use openssl-devel as the default SSL library:

echo "DEFAULT_VERSIONS+=ssl=openssl-devel >> /etc/make.conf

Build NGINX:

cd /usr/ports/security/openssl-devel && make install

Building NGINX with kTLS on Linux Distributions

Linux distributions typically include an OpenSSL version earlier than 3.0.0 (usually version 1.1). You must therefore build NGINX from source with OpenSSL 3.0.0.

When the configure command is run, two options are crucial to enabling kTLS support:

  • –with-openssl=../openssl-3.0.0
  • –with-openssl-opt=enable-ktls

Additional configuration options are available for the NGINX modules included in the official binary packages available at nginx.org. A custom set of modules may be specified instead. You can see what build options were used for your current NGINX binary by running nginx -V.

Run the following commands to build NGINX with OpenSSL 3.0.0:

wget https://nginx.org/download/nginx-1.21.4.tar.gz
wget https://www.openssl.org/source/openssl-3.0.0.tar.gz
tar xzf openssl-3.0.0.tar.gz
cd nginx-1.21.4
./configure \
--with-debug \
--prefix=/usr/local \
--conf-path=/usr/local/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--user=nginx \
--group=nginx \
--with-compat \
--with-file-aio \
--with-threads \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_mp4_module \
--with-http_random_index_module \
--with-http_realip_module \
--with-http_secure_link_module \
--with-http_slice_module \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_v2_module \
--with-mail \
--with-mail_ssl_module \
--with-stream \
--with-stream_realip_module \
--with-stream_ssl_module \
--with-stream_ssl_preread_module \
--with-openssl=../openssl-3.0.0 \
--with-openssl-opt=enable-ktls \
--with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' \
-with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
make –j4
make install

It should be noted that NGINX is statically linked with OpenSSL 3.0.0 libraries. Changing OpenSSL later requires downloading and unpacking the new source archive, and then restarting NGINX with those commands.

Configuring NGINX

kTLS is enabled by including the ssl_conf_command directive with the Options KTLS parameter in the server{} context, as in this sample configuration used for our testing:

worker_processes auto;
error_log /var/log/nginx/error.log debug;

events {}

http {
    sendfile on;

    server {
        listen 443 ssl;
        ssl_certificate ssl/example.crt;
        ssl_certificate_key ssl/example.key;
        ssl_conf_command Options KTLS;
        ssl_protocols TLSv1.3;

        location / {
            root /data;
    	}
    }
}

Verifying kTLS is Enabled

Verify that NGINX is using kTLS by enabling debugging mode and checking the error log for BIO_get_ktls_send() and SSL_sendfile().

grep BIO /var/log/nginx/error.log
2021/11/14 06:01:47 [debug] 274541#274541: *2 BIO_get_ktls_send(): 1
2021/11/14 06:01:50 [debug] 274541#274541: *3 BIO_get_ktls_send(): 1
 
grep SSL_sendfile /var/log/nginx/error.log
2021/11/14 06:01:47 [debug] 274541#274541: *2 SSL_sendfile: 1047576
2021/11/14 06:01:50 [debug] 274541#274541: *3 SSL_sendfile: 1047576

It is recommended that you disable debugging mode after making these checks, especially in production environments. The large number of write operations associated with debug logging results in a performance penalty; also, debug logs can be huge and quickly exhaust available space on a disk partition.

Performance Improvement with kTLS

Under heavy load, SSL_sendfile() can increase throughput by up to 2x compared to userspace TLS, but the size of the performance boost may depend on a number of factors (disk performance, system load, etc). Your network card may also support TLS offload, which may reduce CPU usage.

Testing Performance

Follow these instructions to run a simple one thread test to measure the performance boost on your setup.

Make a large file that fits completely in the disk cache:

truncate -s 1g /data/1G

Check the throughput with this command; the base command will be repeated multiple times for the best results. For a basic statistical analysis, run the output through Ministat [FreeBSD] [Ubuntu].

for i in 'seq 1 100'; do curl -k -s -o /dev/null -w '%{speed_download}\n' https://localhost/1G | ministat

Results of Performance Testing
Our results are presented as output from ministat, where each value is the download speed in kBytes/second.

Throughput for Ubuntu without kTLS:

    N           Min           Max        Median           Avg        Stddev
x  10        529199        705720        662354      654321.6     48025.103

Throughput for Ubuntu with kTLS:

    N           Min           Max        Median           Avg        Stddev
x  10        619105        760208        756278      741848.3     43255.246

The 1.21.4 release of NGINX adds support for kTLS in SSL_sendfile(). Depending on the operating system, performance is typically improved by 8% to 29%.

Related Articles...

Install LAMP

How to install LAMP Stack on Ubuntu 20.04

What is LAMP? The LAMP acronym stands for Linux, Apache, MariaDB/MySQL, and PHP, all of which are free and open source. It is the most commonly used software stack for dynamic websites and web applications. The operating system is Linux, the... Read more

This website uses cookies

We use cookies for the analysis of our visitor data, to improve our website, and to give you a great website experience. For more information about the cookies we use, please see our cookie policy.