Some SBCs only boot from said SD card though, while some do support more robust media. However, too many images are presuming you boot from SD which is a pita.
With or without Das Uboot, they still rely on board specific firmware (even Uboot is customised for many boards to make it work). OSes that state they do support aarch64, often require to have UEFI on your system so no way they are gonna boot on e.g. your Raspberry Pi.
Add to that, that is unlikely that browsers compiled for arm64 will have feature parity with their x86-64 counterparts. Goodbye Digitale Rights Management, and with that goodbye services like Tidal or Spotify (unless you run an OS that is still supported by their apps).
You forgot about ditching more of the chipset etc. in favour of integrating everything into the CPU die.