{"type":"ScholarlyArticle","title":"Going Beyond the Edge: Distributed Inference of Transformer Models on Ultra-Low-Power Wireless Devices","authors":[{"name":"Alexander Gräfe","slug":"alexander-grafe","given":"Alexander","family":"Gräfe","affiliation":""},{"name":"Ding Huo","slug":"ding-huo","given":"Ding","family":"Huo","affiliation":""},{"name":"Vincent de Bakker","slug":"vincent-de-bakker","given":"Vincent","family":"de Bakker","affiliation":""},{"name":"Johannes Berger","slug":"johannes-berger","given":"Johannes","family":"Berger","affiliation":""},{"name":"Marco Zimmerling","slug":"marco-zimmerling","given":"Marco","family":"Zimmerling","affiliation":""},{"name":"Sebastian Trimpe","slug":"sebastian-trimpe","given":"Sebastian","family":"Trimpe","affiliation":""}],"abstract":"Transformer models are rapidly becoming a cornerstone of modern Internet of Things (IoT) applications, yet their computational and memory demands far exceed the capabilities of a single typical ultra-low-power IoT device. We present CATS, a framework for distributed transformer inference on ultra-low-power wireless devices, enabling multiple devices to collaboratively execute models far larger than what a single device can sustain. At its core, CATS is a communication-aware distributed transformer inference scheme co-designed across transformer partitioning, wireless communication and training. It employs SomeGather, a new pruned communication primitive that selectively broadcasts activation columns to reduce communication bandwidth and RAM usage without sacrificing model accuracy. Building on SomeGather, we design a partitioning method that exploits this primitive for efficient model parallelism. To cope with unreliable wireless communication, CATS employs message-dropout during training, which mimics packet losses and yields models that are robust to message loss during inference. In real-world experiments, we show that CATS brings distributed transformer inference to ultra-low-power wireless devices for the first time, with deployments on up to 16 devices that collaboratively execute transformer models up to 14 times larger than what a single device can run.","doi":"10.24963/ijcai.2026/14","url":"https://ijcai.org/proceedings/2026/14","pdf":"https://ijcai.org/proceedings/2026/0014.pdf","markdown":"https://ijcai.org/proceedings/2026/14.md","bibtex":"https://ijcai.org/proceedings/2026/bibtex/14","volume":{"slug":"2026","title":"Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2026","url":"https://ijcai.org/proceedings/2026/","year":2026,"location":"Bremen, Germany","dates":"15-21 August 2026","editors":"Diego Calvanese","isbn":"978-1-956792-09-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2026"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":14,"pages":{"first":118,"last":126},"keywords":["Agent-based and Multi-agent Systems: Agent communication","Agent-based and Multi-agent Systems: Engineering methods, platforms, languages and tools"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}