You’re Not Paying Attention - Wake Up!
Unless you’ve been living under a rock, you’re aware of the extent to which AI has completely taken over the discourse within the tech industry in the past couple of years. Every startup is now an AI startup. In fact, in this week’s YC Demo Day, it appears there are 91 AI companies (up from 58 in the last batch and only 12 as of W21). Venture funding for AI isn’t slowing down either - last year, $50B was invested into AI companies. AI’s domination as a topic of interest in the tech industry is steadily growing, with many companies increasingly integrating AI into their products and services. Frequent high-profile news cycles around OpenAI continue to feed into the AI frenzy.
Not that it’s not warranted. I think we all appreciate how fundamentally AI is going to transform almost every aspect of society in the coming decades.
However, with that said, there are some foundational operational challenges that we will have to solve for in order to see the this societal and global transformation take effect, and most of you are paying zero attention to it.
I know, I know, it’s boring. It’s totally un-sexy. But you know all those cool AI products you are building, investing in, using, including ChatGPT? Yeah, those things have no future without what I’m going to tell you next. So wake up. :)
AI’s Operating Needs Are Demanding, Voracious - and Unmet
The International Data Corporation (IDC) predicts that the global datasphere will grow to 175 zettabytes by 2025, which will require significant increases in computing power and infrastructure to handle AI workloads efficiently. In fact, according to a study by OpenAI, the computational power required for large AI models has been doubling every 3.4 months since 2012, putting pressure on data centers to keep up with increasing demands.
Moreover, with the proliferation of real-time AI applications like autonomous vehicles and augmented reality, there’s a growing need for low-latency data processing, requiring data centers to invest in faster hardware and optimized architectures. If we examine cloud adoption trends as well, we are seeing that the migration of AI workloads to the cloud is accelerating, with Gartner predicting that by 2024, 80% of enterprises will have shut down their traditional data centers, emphasizing the need for scalable and efficient cloud infrastructure to support AI initiatives.
However, current hardware architectures are reaching their limits in terms of power efficiency and performance scalability, highlighting the urgency for innovation in semiconductor technologies to unlock new levels of computational power for AI tasks. In addition, the cost of building and maintaining data centers is increasing, driven by factors such as rising energy costs and the need for specialized cooling solutions to manage heat generated by high-performance computing hardware. We are also seeing an increase in data security concerns. As AI models become more complex and valuable, there’s a heightened need for secure data processing and storage infrastructure, prompting investments in hardware security features and encryption technologies. These are just some of the issues we are seeing as the pants are getting too tight around an ever expanding belly. I’m not even getting into the needs of edge computing here, for instance.
With all of the above said, it’s clear that the full potential of AI hinges very squarely on our ability to build the underlying infrastructure to power it and all its applications. The semiconductor industry, while stagnant in growth for years, is suddenly poised for an explosion of growth, thanks to AI. And it isn’t just about NVIDIA. NVIDIA is just the tip of the iceberg.
So what else is needed to power the impending AI cambrian explosion, beyond NVIDIA? What advancements are needed for AI deployment in data centers, particularly for tasks such as training large language models in data centers. In other words, for the AI dream to become a reality, what underlying infrastructure has to be built? Let’s get into it.
Cyphr’s Investment Thesis: Infrastructural Excellence
To give you a sense of the objective of this post, this post is part 1 of a two part series that outlines Cyphr’s investment thesis around “infrastructural excellence.” Part 2 is on computing power needed for consumer electronics as well as on AI-based IP that will be sold by companies as library components to be integrated within other chips.
My view, as founder of the venture fund Cyphr, is that we are in the midst of witnessing the Intelligence Renaissance, which will usher in an age of startups focused on exponential learning and infrastructural excellence. We aim to support startups in the Intelligence Renaissance with resources, regulatory guidance, and business and strategic support. I outline below what excites us within the space of computing (essentially semiconductors).
But first, before we dive in, if you don’t know much about this space and want a primer on how data centers are relevant for AI, I will be writing a primer on the fundamentals next, so stay tuned.
Power Efficiency
The development of advanced process technologies, such as 5nm, 3nm, and beyond, in semiconductor manufacturing is crucial for powering the future of AI. These advancements enable more transistors to be packed into the same space, resulting in increased energy efficiency, lower power consumption, and reduced heat generation in AI chips. Additionally, innovations in low-power chip architectures, like dynamic voltage and frequency scaling (DVFS), further contribute to minimizing power consumption during AI computations. This underlying infrastructure is essential for supporting the growing demands of AI technology, ensuring enhanced performance and sustainability in AI applications.
In plain English: Building better and smaller computer chips is super important for the future of AI. When we make these chips smaller, we can fit more tiny switches called transistors onto them. This means they use less power, produce less heat, and work more efficiently. We're also coming up with clever ways to make the chips use less power when they're doing AI tasks. All these improvements are important because they help AI technology work better and last longer, making it more useful for all sorts of things.
Heat Dissipation
Integrated cooling solutions and materials advances are essential for the future of AI development. Semiconductor devices with integrated cooling technologies, like embedded microfluidic cooling channels, are crucial for efficiently dissipating heat directly from the source, thus preventing overheating and ensuring optimal performance. Furthermore, research into new materials with higher thermal conductivity, such as graphene or advanced composites, holds promise in enhancing the heat dissipation capabilities of semiconductors. These advancements not only improve the efficiency of cooling systems but also enable higher operational frequencies, crucial for handling the increasingly complex computations required by AI technologies. In essence, the development of integrated cooling solutions and materials advances is vital for sustaining the growth and performance of AI systems in the future.
In plain English: Cooling and better materials are important for making AI technology work well in the future. Chips that run AI can get really hot, so we need smart ways to keep them cool. One way is by building cooling channels right into the chip itself, which helps take away the heat quickly and stops the chip from getting too hot. Also, scientists are looking for new materials like graphene that can help chips stay cool better. When chips stay cooler, they can work faster and handle more complex tasks, which is super important for AI technology to keep improving in the future.
Latency Reduction
For the future of AI, advancements in on-chip memory technologies and the adoption of optical interconnects are imperative. Enhancing on-chip memory, such as SRAM, DRAM, and emerging technologies like MRAM or ReRAM, is crucial for reducing data access latency, ultimately improving overall system performance. Faster and higher-capacity memory options enable quicker access to data, which is essential for AI algorithms that rely heavily on data processing. Additionally, the integration of optical interconnects instead of traditional metal ones can significantly reduce latency and power consumption in chip-to-chip communication. This transition facilitates faster data transfers between processors, which is particularly beneficial for distributed AI deployments where efficient communication between multiple processing units is essential for seamless operation. In essence, the development of on-chip memory innovations and optical interconnects is essential for advancing the capabilities and efficiency of AI systems in the future.
In plain English: For AI to improve in the future, we need to make some important upgrades to the way computer chips work. One thing we need to do is improve the memory inside the chips. This memory helps the chips access data quickly, which is important for AI algorithms that do a lot of data processing. We're also thinking about using light instead of metal wires to connect different chips together. This can make communication between chips much faster and use less power. These upgrades will make AI systems work faster and more efficiently, especially when they're spread out over different parts, helping them work together seamlessly.
Enhanced Parallel Processing
In addition, to realize the potential of AI, the development of heterogeneous integration technology is crucial. By employing techniques like chiplets and advanced packaging (such as 2.5D and 3D stacking), heterogeneous integration enables the assembly of chips with specialized functions into a single package. This approach optimizes processing paths for different AI workloads, enhancing overall performance and efficiency. Additionally, the integration of dedicated AI accelerator cores directly into CPU and GPU chips offers specialized processing power tailored for AI tasks. This integration not only offloads the workload from general-purpose cores but also improves power efficiency, making AI computations more streamlined and effective. Overall, the advancement of heterogeneous integration and dedicated AI accelerators is essential for driving the future of AI technology, enabling higher performance and energy-efficient computing solutions.
In plain English: To help AI technology reach its full potential, we need to put different kinds of computer chips together in a smart way. We can do this by using techniques like stacking chips on top of each other. By doing this, we can make one powerful package with chips that each do different things for AI. This makes AI work better and faster because each chip can focus on its own task. Also, we can add special chips just for AI tasks onto the main chips in computers. This helps the main chips handle AI tasks more efficiently, saving power and making AI run smoother. So, by using these methods, we can make AI technology work even better and use less energy in the future.
Software and Hardware Co-Design
The development of co-optimized AI hardware and software is imperative as well. This approach involves designing AI hardware alongside software frameworks and compilers that are tailored to leverage the specific architectural features of the hardware. By doing so, it maximizes performance and efficiency, ensuring that AI models operate optimally while minimizing power consumption and heat generation. Co-optimization ensures synergy between hardware and software, leading to more streamlined and effective AI computations. Ultimately, this advancement is essential for driving the progress of AI technology, enabling enhanced performance, energy efficiency, and scalability in AI applications.
In plain English: It's important to make both the hardware and software for AI work together perfectly. This means designing computer chips and the programs that run on them in a way that makes them work best together. By doing this, we can make sure that AI models run as fast and as efficiently as possible, while also using less power and producing less heat. When hardware and software work together well, it makes AI computations run smoother and more effectively. This is a big step forward for AI technology, making it work better, use less energy, and be able to handle more tasks in the future.
Energy Harvesting and Management
To unlock the full potential of AI, the development of smart power management and energy harvesting technologies is imperative. Smart power management involves implementing advanced techniques to intelligently adjust power usage according to workload demands, potentially incorporating machine learning algorithms for real-time adaptation. This ensures optimal power utilization, enhancing efficiency and performance in AI systems. Additionally, exploring energy harvesting technologies that convert environmental energy into electrical energy can supplement semiconductor power supplies, potentially offsetting some power demands and increasing sustainability. By integrating these advancements, AI systems can operate more efficiently, with reduced environmental impact, thereby realizing their full potential in revolutionizing various domains.
In plain English: To make AI work at its best, we need to devise clever techniques to adjust how much power AI systems use based on what they're doing. This helps them work more efficiently and perform better. Also, we're looking into ways to turn natural energy sources, like sunlight or heat, into electricity to power AI systems. This can help reduce the amount of power they need from traditional sources, making them more sustainable. By doing these things, AI can work better and have a smaller impact on the environment.
Call to Action
In short, dealing with the challenges of AI in data centers needs a bunch of different improvements in computer chips. We need better ways to make chips, design them, keep them cool, and find new materials for them. We also need to add special chips just for AI tasks and make sure the software and hardware work together well.
If you are a startup solving any of the above problems, contact us. If you are building the infrastructural excellence needed for the Intelligence Renaissance, contact us. We’d love to work with you.