Postmortem on the Framework Router
It's kind of crazy, I haven't even gotten a chance to write anything about how I set up a router using a Framework mainboard, and I'm already writing a postmortem about my investigation into how it died. Please, don't let this scare you off from attempting to do something similar (although there are valid reasons why it's not the best approach if you have other options; more to come on that soon). I'm pretty sure it didn't fail because of the specific hardware involved.
On the morning of September 21st, I woke up to a prolonged power outage. The electricity came back on after a couple of hours, but my OPNsense router/firewall that I just finished upgrading a month earlier was completely off the network and unresponsive. After a few attempts at restarting it and re-plugging cables, I brought it to my desk and plugged it into a monitor and keyboard.
The Proxmox VE installation was perfectly fine, and the OPNsense VM was starting automatically. I noticed that the lights on the network adapters weren't lighting up, so I focused in on those as the culprit right away. I found these dmesg messages which seemed to confirm something was wrong there:
Cannot enable. Maybe the USB cable is bad?unable to enumerate USB device
I tried the adapters in different ports and found that they worked fine in the ports on the right side of the mainboard, but not either of the ports on the left side, which is where they had been plugged in. Other devices would sometimes work in the left ports, but not consistently. Other things started going wrong too though: the UEFI BIOS settings wouldn't persist after reboots, and at one point the board failed to boot altogether for a while, needing a CMOS battery reset to limp back to life.
I concluded that the most likely explanation was a power surge that could have happened as part of the outage. All my network equipment is surge-protected and backed up by a UPS, at least for power, but the upstream equipment owned by the ISP is not since it's in a tiny closet on the other side of the apartment. Based on the symptoms I'm seeing, I suspect that there was a power surge which traveled along the incoming Ethernet cable from there and damaged the left-side USB controller on the mainboard.
With only two functioning USB-C ports, it's impractical to keep using this hardware as a router. I need at least three ports for it to make sense, one for power and one each for the two network adapters. I could try to get a USB hub that would let me plug more into the remaining ports, but given how contrived the setup already was, I think it's best to switch to a different strategy. I also don't know what else might be subtly damaged and waiting to fail. The mainboard may still be useful as a non-critical server, but its routing days are definitely over.
Next time, I'm going to close the final gap in my anti-power-surge defenses and put a surge protector on the WAN line. I just learned today that, not only do they exist for network connections (and from reputable vendors too), but they can actually be pretty reasonably priced. That's pretty neat.