← Blog

A lock-up in the Zephyr USB network driver, and a two-line fix

Many small Zephyr boards have no Ethernet connector. They connect to a PC with Ethernet over the USB cable, through the USB CDC-NCM driver of Zephyr. Under load, this driver could stop the whole USB device: no network, no serial console, until a reset. We found the cause while we tested an encrypted network client on a Teensy 4.1. The fix changes two lines. We offer it to the Zephyr project: pull request #123515.

The symptom

An open Zephyr issue, #102857, shows the same symptom on a Raspberry Pi Pico after an nmap scan.

How we found the cause

A stopped board cannot tell you what is wrong, because its console is also stopped. We added a small watchdog thread to our test firmware. Every second, it read how many free network packets were available. When no packets were free for 8 seconds, it wrote the state of all threads into a RAM area that a reset does not clear. Then it reset the board. After the reset, the firmware printed this record.

The record showed that all threads were waiting, also the USB thread. From there, we read the driver code.

The cause

Zephyr runs all USB work of a device in one thread. When the PC sends Ethernet data, this thread copies the data into network packets. The driver asked for these packets like this:

Under load, the network stack had no free packets. So the USB thread waited. But the stack frees packets only after it sends its replies, and in our Zephyr version the send function waited for the same USB thread. Each part waited for the other, so the device stopped until a reset.

The fix

The receive function must never wait. If there is no free packet, it now drops the received data and continues. A normal Ethernet driver does the same when its receive buffers are full. The temporary packet now comes from the pool for received data, which is the correct pool.

-	src = net_pkt_alloc(K_MSEC(NET_PKT_ALLOC_TIMEOUT));
+	src = net_pkt_rx_alloc(K_NO_WAIT);
...
-		pkt = net_pkt_rx_alloc_with_buffer(data->iface, len, NET_AF_UNSPEC, 0, K_FOREVER);
+		pkt = net_pkt_rx_alloc_with_buffer(data->iface, len, NET_AF_UNSPEC, 0, K_NO_WAIT);

The change also removes one constant that is no longer used. Under a flood, the board now drops packets and logs a message for each drop. It does not stop.

Test result

Notes for other users

Next

This is our first change to the core of Zephyr. We thank the maintainers of the USB device stack for code that was easy to read and to test. We will follow the review of the pull request and report the result here.


Made with a notepad.