Qualcomm is well known for its position in the smartphone market and in 5G modems, but it is less known for its role in AI, specifically at the edge and on-device. To address this, the company recently held an AI Workshop for about 50 press members, analysts, and influencers at its headquarters in San Diego. I believe that Qualcomm’s goal for the event was to better communicate the breadth of the company’s AI capabilities, both in hardware and software, beyond the smartphone.
Durga Malladi is Qualcomm’s senior vice president and general manager of technology planning and edge solutions. He kicked off the day with a high-level view of AI, starting with the cyclical nature of AI’s evolution starting with the booms of the 1950s and 1960s, the 1980s, and the last 10 years—and the slumps between them. He also touted the potential growth opportunity for Gen AI and the challenges of scaling it in the face of increasing cost per inference due to infrastructure shortages. These challenges also come with increases in power consumption, as Google demonstrated in its 2023 environmental report that shows the company missing its power reduction goals because of AI.
Malladi also introduced the concept of hybrid AI, which is a balanced approach that blends on-device and cloud AI compute. This is possible thanks to improvements in smaller models such as Llama 3B achieving higher model quality compared to older, much larger models. Malladi views Qualcomm as a leader in AI with strengths in R&D, AI hardware, and software design as well as a sizeable install base and ecosystem enablement. I believe that Qualcomm’s approach to AI has been measured and has scaled appropriately with the demand from OEMs and developers. As such, it definitely deserves credit for how it has grown its AI capabilities.
Qualcomm’s Approach to AI Research
Qualcomm’s vice president of engineering for AI research, Jilei Hou, talked about the company’s long history of AI research going back to 2007. The timeline of this work seen below really shows how much Qualcomm has gradually advanced to arrive where it is today in AI. Its steady investments and iterative approach to AI have prepared it to meet the new need for on-device AI capabilities.
Hou also talked about how Qualcomm’s approach to AI research is a blend of fundamental research, platform research, and applied research. This is necessary because Qualcomm is both a software and hardware provider and needs to do a variety of research (and plenty of it) to support its OEM customers’ and developers’ needs. The fruits of this research are significant and include impressive displays such as the demo at MWC this year that showed a 7-billion-parameter multimodal model running on a smartphone.
Qualcomm, like many other chipmakers in the AI space, has also worked on quantization of 32-bit floating point models down to 16-, 8-, and 4-bit sizes to increase performance and improve power consumption. Its researchers have also done work in the realms of knowledge distillation, speculative decoding, stable diffusion, and low-rank adaptation.
Qualcomm believes that this research will enable it to evolve its on-device AI capabilities to enable embodied AI robots that run mostly locally. I believe that we are still quite a ways away from embodied AI in humanoid robots, but I do believe that we will see only increased demand for on-device AI—and Qualcomm clearly has significant infrastructure in place to serve those needs.
Software and Tools
Qualcomm believes that it has most of the software pieces in place with its AI Stack and AI Hub to enable AI frameworks and runtimes that developers and OEMs are most likely to use. At the AI Workshop, Qualcomm brought on developers from Fulltrack AI and Lantronix to demonstrate how Qualcomm’s software solutions have enabled faster time to market and better scale and performance.
There are some questions about whether Qualcomm’s stack is open-source enough for developers to want to approach it, but that seems to be a common problem with most Qualcomm software. Many developers express to me that they’d like to see Qualcomm be more transparent and more open-source to make development easier. That said, I do think that working in AI has forced Qualcomm to be more open and flexible with its software solutions. Still, the software and tools that Qualcomm provides are ultimately designed to take advantage of the hardware that Qualcomm designs for its OEM customers.
AI Hardware and Ecosystem
The approach Qualcomm has taken to AI hardware has mostly been dependent on the use cases for AI at the time. This means that in the beginning Qualcomm’s hardware only had scalar and vector processing for simple convolutional neural networks. This eventually changed with the addition of tensor cores, which enabled transformers and more types of neural networks. This changed the component that Qualcomm refers to as “the Hexagon” from a DSP into an NPU, which created a new core for developers to target but also created a processor for advanced matrix multiplication. I’ve written extensively about the NPU inside Qualcomm’s latest Snapdragon X Elite and how it has evolved over the years.
Qualcomm has developed AI processing capabilities on its GPU as well as its CPU, meaning that different workloads may run on different cores, which is where the software and tools come into play. This has also driven the industry, including Qualcomm, to gravitate toward heterogenous computing to maximize power, bandwidth, and performance. Qualcomm’s own tests show its NPU delivering 5x higher performance at lower thermals against competitors such as Intel. Qualcomm also cites the reach of its AI hardware across smartphones, PCs, and automotive systems as enabling scale for software developers and OEMs.
The biggest challenge that I see Qualcomm having in this space is convincing both OEMs and developers to adopt its software solutions to optimize for Snapdragon hardware. There are challenges on the Android front with Google having its own ideas for AI and frameworks at odds with what OEMs want to do to claim differentiation. On the Windows front, similarly, you have Microsoft with Copilot+ in tension with its OEMs, which want to differentiate between each other on software and AI. I do believe that right now it makes sense for developers to optimize for Snapdragon, but if more Arm vendors are indeed coming into the market, then that seems like a less attractive proposition.
AI Demos Galore
At the event, there were multiple rooms with AI demos representing the work of the many different divisions of Qualcomm. I believe these demos did the best job of demonstrating the breadth of the AI applications that the company enables with its hardware and software solutions. That said, the most exciting applications were in the most exciting new businesses from Qualcomm: PC and automotive.
Since I didn’t attend CES 2024, I didn’t get to see Qualcomm’s AI-enhanced demo vehicle then, but I can now tell you that the visuals and AI experiences from this demo were compelling. Likewise, for the PC, applications in security and identity seemed like natural and easy ways to enhance the user experience by making systems less intrusive but more secure. McAfee’s AI-enhanced deepfake detection tool could also become a potent helper for inexperienced PC users.
Qualcomm Goes Broad For AI
If Qualcomm’s goal for the AI Workshop was to help the press, analysts, and influencers understand the scope of its AI capabilities, it succeeded. The company did a great job of communicating its AI research and hardware and software capabilities. I think the demos also did a great job of connecting hardware and software together and making the application of AI feel more realistic.
In the bigger picture, I also believe Qualcomm did a great job of showing the breadth of its business and how much of that is now accelerated by AI. One place I felt like Qualcomm forgot to talk about AI was as it pertains to 5G and modems; there was very little said about that, which makes me think this event was much more focused on third-party AI applications rather than in-house applications of AI. Even so, 5G modems are Qualcomm’s bread and butter and would still be a great place to demonstrate AI. With the growth of hybrid AI and the increasing importance of on-device AI, Qualcomm has positioned itself well as a leader in the space, both in software and hardware.
