Untether AI’s speedAI 240 Slim: Advancing Energy-Efficient AI Solutions for Edge Applications

A few days ago Untether AI announced the availability for shipping of its speedAI 240 Slim AI inference accelerator cards, which they claim is the world’s fastest and most energy-efficient for applications ranging from cloud to edge. This assertion is backed by achieving industry-leading performance in the MLPerf Inference 4.1 benchmarks. Verified results demonstrate that these cards offer up to six times the energy efficiency and three times lower latency compared to other participants. Specifically, the speedAI 240 Preview accelerator card achieved the highest throughput of any single PCIe card in both Datacenter and Edge categories for the ResNet-50 image classification benchmark, recording 70,348 samples per second.

What is the MLPerf? It’s the industry-standard measure of AI performance, developed by MLCommons, a consortium of researchers and AI companies, including Nvidia, Google, and Intel, among others.

What’s the problem Untheter AI is addressing? The data movement bottleneck in AI inference, where moving data between memory and compute units in traditional architectures leads to high latency and energy consumption.

What’s Untether AI’s Approach to address the problem of data movement bottleneck? They use At-Memory Compute (AMC) architecture, which places compute elements adjacent to memory cells. This architecture is distinguished by its short, direct, parallel connections between memory and processing elements, which optimize both data throughput and energy efficiency. By bringing computation to the data, AMC minimizes latency and power consumption associated with data movement, making it ideal for real-time AI inference with high data volumes and diverse workloads.

How Other Players are Addressing This? Other companies, like NVIDIA, AMD, Intel, and IBM address the same problems with innovations in memory-compute integration and in-memory computing (IMC).

• NVIDIA uses high-bandwidth memory (HBM3e) and NVLink interconnects in its GPUs to accelerate data transfer across memory layers, reducing latency and boosting AI efficiency. For instance, the NVIDIA Blackwell architecture combines HBM3e with large L2 cache to keep data close to compute elements, minimizing off-chip data transfers and enhancing throughput.
• AMD addresses the problem with extensive HBM3 memory in its MI300X GPUs, allowing AI models to be stored close to compute units and reducing the energy cost of moving data on and off the chip. This architecture is designed to handle large language models (LLMs) and other high-performance AI applications, where data proximity to compute is critical.
• IBM’s NorthPole processor takes a modular approach to in-memory computing, co-locating memory and processing on the same chip, an architectural approach also known as on-chip memory. This design allows data to be processed within the memory plane itself, minimizing data transfer needs and achieving high efficiency for low-latency AI inference tasks.
• Intel has explored in-memory computing in its research with analog computation, leveraging mixed-signal circuits in memory cells to perform essential operations like multiply-accumulate (MAC) directly in memory. This IMC approach, significantly cuts down on latency and energy consumption, especially for edge AI applications requiring ultra-low power.

Why does Untether AI’s announcement matters? It is significant as the AI chip market, valued at USD 61.45 billion in 2023 and projected to reach USD 621.15 billion by 2032 with a 29.4% CAGR, is seeing rapid expansion, with AI inference chips poised to lead this growth (as per SNSInsider research). The inference segment alone, is anticipated to grow at a CAGR of 30.38%, driven by demand for low-latency, energy-efficient solutions in edge and on-prem data centers. This segment has become a highly coveted space for the established players like NVIDIA, AMD, and IBM, who see these markets as critical to their future growth strategies.

Competitive Impact. Untether AI’s speedAI 240 Slim, with its emphasis on energy efficiency and low latency, enters this competitive landscape as a focused solution for industries that require high performance under power constraints, offering a new alternative for edge AI and industrial applications. Competing directly with products from major players, the speedAI 240 Slim addresses critical challenges in AI inference at the edge and in data centers, delivering low power consumption and high throughput with its At-Memory Compute architecture. These features make it suitable for applications such as automotive, agriculture and computer vision. Notably, companies like Mercedes-Benz are already working with Untether AI to deploy this technology in its autonomous vehicle programs. Additionally, Untether AI recently announced a collaboration with Arm to integrate its inference acceleration technology with Arm’s latest-generation Automotive Enhanced (AE) technology, targeting performance, power, safety, and security needs for advanced Driver Assistance Systems (ADAS) and autonomous vehicle applications.

However, mainstream players still maintain a lead in several areas. Companies like NVIDIA and AMD have invested significantly in high-bandwidth memory solutions (e.g., NVIDIA’s HBM3e and AMD’s HBM3) and advanced interconnects (e.g., NVIDIA’s NVLink and AMD’s Infinity Fabric)), enabling robust scalability and integration into broader AI infrastructure. More importantly, these established solutions are deeply embedded across hyperscaler platforms and are optimized for a wide array of high-demand AI applications, from large-scale cloud environments to LLM training. Additionally, NVIDIA and AMD bring massive software ecosystems, well established for years, and industry-standard compatibility, which give them an advantage in versatility and market penetration.

While Untether AI introduces significant architectural innovation and improved efficiency for specific use cases, scaling and integrating at the level of established players, which dominate in comprehensive infrastructure and software support, remains a challenge. Nevertheless, Untether AI’s approach could push them to explore more energy-efficient designs, potentially catalyzing broader innovation in the market.

Is Untether AI a Disruptor? Untether AI does exhibit some characteristics of a disruptor, as outlined by Clayton Christensen in The Innovator’s Dilemma. While the AI hardware market is highly advanced and complex, Untether AI’s approach aligns with Christensen’s description in key ways:

• Serving Underserved or Emerging Markets: Untether AI is positioned to meet specific needs in edge AI inference, a segment often underserved by traditional GPU-based solutions focused on large data center and cloud AI tasks. By targeting applications like automotive, agriculture, and machine vision with its energy-efficient, low-latency designs, Untether AI appeals to markets that require high performance in constrained environments, which is a space not always fully addressed by mainstream players.
• Prioritizing Distinct Performance Metrics: Unlike established competitors that prioritize maximum performance and scalability, Untether AI focuses on minimizing power consumption and latency, which are critical for edge AI applications. This targeted emphasis on energy efficiency and compact form factors provides value where traditional AI hardware may be too power-hungry or costly, aligning well with the needs of edge deployment. By focusing on these metrics, Untether AI can offer a competitive alternative that may evolve to appeal to broader AI markets.
• Gradual Improvement and Market Encroachment: Although it currently targets niche edge applications, Untether AI’s technology holds potential for incremental improvements and broader applications over time. As it continues to enhance its At-Memory Compute architecture and prove its energy-efficient approach, Untether AI could expand into larger segments, including more comprehensive data center applications, potentially challenging established players. If successful, this incremental growth could attract mainstream customers, pressuring incumbents to adapt.

While not a traditional disruptor in the sense of “low-end” market entry, Untether AI is a contender with disruptive potential by providing an alternative approach tailored to specific performance needs. This positions Untether AI as a unique player capable of influencing the trajectory of AI hardware design and competition.