I Built a CoPilot+ AI PC (without Windows)

Jeff Geerling
4 Jun 202412:49

TLDRIn this video, the creator unveils a custom Raspberry Pi AI PC, the 'CoPilot,' which challenges Microsoft's Copilot with 13 TOPS of compute, later boosted to 51 TOPS. The $70 Raspberry Pi AI kit, featuring an M.2 HAT and Hailo NPU, is highlighted for its efficiency and potential in machine vision applications. Despite the AI hype, the kit offers practical solutions for real-world problems like traffic monitoring and quality assurance. The video explores the kit's capabilities, compares it with GPUs and other AI devices, and concludes with an ambitious, if not entirely successful, attempt to maximize neural compute power.

Takeaways

  • 🤖 The CoPilot PC is a custom Raspberry Pi AI PC, not to be confused with Microsoft's Copilot.
  • 🚀 It aims to achieve over 40 TOPS of neural compute, which is more than Apple's M4 or Snapdragon X.
  • 💾 The $70 Raspberry Pi AI kit includes an M.2 HAT and a 13 TOPS Hailo NPU for machine learning tasks.
  • 🔎 The kit is efficient with 8 TOPS per watt, making it suitable for tasks like object detection and image segmentation.
  • 📈 The AI kit is designed for practical applications such as machine vision and real-time video processing.
  • 🌟 Raspberry Pi is focusing on real-time video applications like selective recording and traffic monitoring.
  • 🔩 The AI kit is a PCI Express device and can be used on other computers besides the Raspberry Pi.
  • 🚀 The video demonstrates the potential of combining multiple AI accelerators to achieve over 50 TOPS.
  • 🔋 There are limitations, such as power requirements and the complexity of using multiple devices.
  • 🛠️ The Raspberry Pi AI kit offers an alternative to the Coral TPU and is expected to evolve with more exposure.
  • 👨‍💻 The presenter, Jeff Geerling, suggests that while the AI kit is niche, it will be loved by those in machine vision and robotics.

Q & A

  • What is the main difference between the CoPilot PC and Microsoft's Copilot?

    -The CoPilot PC is a custom AI PC built using a Raspberry Pi with an AI kit, whereas Microsoft's Copilot is a software development tool. The narrator built the CoPilot PC because they didn't like the way Microsoft marketed their Copilot.

  • What is the minimum neural compute required for Microsoft's Copilot?

    -Microsoft states that you need at least 40 TOPS of neural compute for Copilot.

  • What is included in the $70 Raspberry Pi AI kit?

    -The $70 Raspberry Pi AI kit includes an M.2 HAT and a 13 TOPS Hailo NPU, which is like a machine learning jetpack for the Raspberry Pi.

  • How does the efficiency of the Hailo-8L NPU compare to the Coral TPU?

    -The Hailo-8L NPU runs at 13 TOPS and 8 TOPS per watt, which is more efficient than the Coral TPU that provides 2 TOPS at around 2 TOPS per watt.

  • What are some practical applications of the Raspberry Pi AI kit?

    -The Raspberry Pi AI kit is geared towards machine vision applications such as object detection, pose estimation, image segmentation, and can be used in robotics, safety monitoring, and quality assurance.

  • Why did the narrator choose not to integrate an NPU into the main Raspberry Pi chip?

    -Raspberry Pi doesn't build their own main chips; they rely on Broadcom. The decision to not integrate an NPU into the main chip would be up to Broadcom.

  • What is the difference between using an NPU and a GPU for AI tasks?

    -GPUs are great for training and running models but have higher power requirements. NPUs like the Hailo are better for low-power edge devices where efficiency is crucial.

  • What limitations does the Raspberry Pi AI kit have in terms of RAM and training models?

    -The AI kit doesn't offer a lot of RAM, limiting the size of the models that can be run on it. Training models on the Pi is possible but slow, taking days instead of minutes or hours on a modern GPU.

  • What is the purpose of the Pineboard's HatBrick! Commander mentioned in the script?

    -The Pineboard's HatBrick! Commander is a PCI Express board that adapts one port into two, allowing the connection of multiple AI accelerator boards to the Raspberry Pi.

  • What is the maximum neural compute the narrator achieved with the Raspberry Pi AI kit and additional PCI Express devices?

    -The narrator achieved a maximum of 47 TOPS (actually 51) by connecting multiple AI accelerator boards to the Raspberry Pi, although not all were usable due to power limitations.

  • What alternative options does the narrator suggest for those who need more neural compute than the AI kit provides?

    -For those who need more neural compute, the narrator suggests buying a more powerful NPU, such as Hailo's line of bigger cards with 52 to 208 TOPS of performance, starting at $250.

Outlines

00:00

🤖 Custom Raspberry Pi AI PC Introduction

The speaker introduces a custom AI PC built using a Raspberry Pi, which they've dubbed 'CoPilot PC'. Unlike Microsoft's Copilot, this is a DIY project. They critique Microsoft's marketing of Copilot and highlight the Raspberry Pi AI kit's 13 TOPS of neural compute, which is less than Microsoft's requirement but more efficient than Apple's M4 or Snapdragon X. The kit includes an M.2 HAT and a Hailo NPU, offering 8 TOPS per watt. The speaker compares it to the Coral TPU used in their Frigate server, which detects objects at 2 TOPS per watt. The AI kit is geared towards practical applications like machine vision and is expected to be popular with Pi cameras. Raspberry Pi's focus is on real-time video applications, which can solve real-world problems in various sectors.

05:04

🚀 Enhancing Raspberry Pi with Multiple AI Accelerators

The speaker demonstrates how to enhance the Raspberry Pi's AI capabilities by adding multiple AI accelerators, including the Hailo NPU and Coral TPUs. They use Pineboard's HatBrick! Commander to expand the number of available PCI Express ports. Despite the complexity of the setup, they manage to get the system to recognize the accelerators, achieving a total of 47 TOPS [51 actually] of neural compute. However, they encounter issues with power supply and initialization, suggesting that the system might be drawing too much power for the Pi's PCI Express port to handle. The speaker concludes that while the AI kit is powerful, it may be more practical to use a more robust NPU for high-performance needs.

10:09

🔍 Evaluating the Raspberry Pi AI Kit's Practicality

The speaker reflects on the practicality of the Raspberry Pi AI kit, noting that while it's a niche product, it serves a specific audience well, particularly those interested in machine vision and robotics. They compare it favorably to the Coral as an alternative AI solution. The speaker also speculates on future developments in the field, such as the potential for more integrated AI solutions and the emergence of new products that combine AI with camera modules. They conclude by reiterating their appreciation for the AI kit, despite acknowledging that it may not be suitable for everyone.

Mindmap

Keywords

💡CoPilot

CoPilot in the video refers to a custom AI PC built by the creator, which is distinct from Microsoft's Copilot. It is a play on words, highlighting the creator's dissatisfaction with Microsoft's marketing approach. The CoPilot PC is a Raspberry Pi-based system designed for AI applications, showcasing the potential of using Raspberry Pi for machine learning tasks.

💡TOPS

TOPS, or Tera Operations Per Second, is a unit of measurement for the processing capability of AI hardware. The video discusses the AI capabilities of the CoPilot PC in terms of TOPS, comparing it to other devices like Apple's M4 and Snapdragon X. The creator aims to achieve more than 13 TOPS with the CoPilot PC, which is a key performance metric in AI and machine learning applications.

💡Raspberry Pi AI kit

The Raspberry Pi AI kit is a $70 kit that includes an M.2 HAT and a 13 TOPS Hailo NPU. It is described as a machine learning 'jetpack' for the Raspberry Pi, enabling the Pi to perform AI tasks more efficiently. The kit is central to the CoPilot PC's capabilities, demonstrating how Raspberry Pi can be adapted for AI applications.

💡Hailo NPU

The Hailo NPU, or Neural Processing Unit, is a component of the Raspberry Pi AI kit that provides 13 TOPS of compute power. It is likened to a 'little machine learning jetpack' and is crucial for the CoPilot PC's ability to perform AI tasks. The Hailo NPU's efficiency and performance are highlighted as advantages over other AI hardware.

💡Coral TPU

The Coral TPU, or Tensor Processing Unit, is mentioned as a comparison to the Hailo NPU. It provides 2 TOPS of compute power and is used in the creator's Frigate server for tasks like object detection. The Coral TPU is an example of how AI hardware can be integrated into various projects, including the CoPilot PC.

💡Machine Vision

Machine vision is a key application area for the CoPilot PC, as discussed in the video. It involves using AI to analyze visual information from cameras or images, with potential uses in robotics, safety monitoring, and quality assurance. The video suggests that the Raspberry Pi AI kit, with its focus on machine vision, is well-suited for these types of applications.

💡AI Hype Train

The term 'AI Hype Train' is used in the video to describe the current trend of companies heavily promoting AI capabilities. The creator expresses skepticism about the hype surrounding AI, preferring practical applications that solve real-world problems. This concept is central to the video's narrative, as the creator seeks to demonstrate the CoPilot PC's practical uses.

💡Neural Compute

Neural compute refers to the computational power required for processing neural networks, a subset of AI. In the video, the creator aims to surpass Microsoft's stated requirement of 40 TOPS for Copilot with the CoPilot PC. Neural compute is a key factor in determining the capabilities of AI hardware, such as the Raspberry Pi AI kit.

💡HatBrick! Commander

The HatBrick! Commander is a PCI Express board mentioned in the video, used to expand the number of available ports for connecting AI hardware to the Raspberry Pi. It is part of the creator's attempt to increase the CoPilot PC's neural compute power by connecting multiple AI accelerators.

💡Model Zoo

The Model Zoo is a collection of pre-built AI models provided by Hailo, which can be used with the Raspberry Pi AI kit. These models cover various applications like upscaling and face recognition, offering a starting point for users to explore AI capabilities without having to build models from scratch.

💡Machine Learning

Machine learning is a subset of AI that involves teaching computers to learn from data. In the video, machine learning is the core function of the CoPilot PC, with the Raspberry Pi AI kit enabling the Pi to perform tasks like object detection and image segmentation. The video emphasizes the practical applications of machine learning in real-world scenarios.

Highlights

Introduction to the custom CoPilot PC, a Raspberry Pi AI PC.

Comparison with Microsoft's Copilot, highlighting differences and personal views.

Microsoft's requirement of at least 40 TOPS of neural compute for Copilot.

The CoPilot PC's capability of reaching 47 TOPS [actually 51], surpassing Apple's M4 and Snapdragon X.

The $70 Raspberry Pi AI kit includes an M.2 HAT and a 13 TOPS Hailo NPU.

Comparison of the Coral TPU and Hailo-8L NPU in terms of TOPS and efficiency.

The focus on machine vision and real-world applications of the AI kit.

Raspberry Pi's preannouncement of an AI camera at Embedded World.

The versatility of the AI kit as a PCI Express device for various computers.

Examples of real-world applications like factory inspections and traffic planning.

The potential of the AI kit for robotics and safety monitoring.

The question of why Raspberry Pi doesn't integrate an NPU into their main chip.

Comparison of different approaches to AI hardware by various companies.

The limitations of the AI kit in terms of RAM and model training capabilities.

Demonstration of the YOLOv5 object identification model running on the Pi 5's CPU and Hailo NPU.

The benefits of using an AI coprocessor for power efficiency and CPU usage.

The potential of the AI kit for pose estimation and gesture-based applications.

The use of image segmentation for applications like iPhone's portrait mode.

The challenge of achieving more than 13 TOPS of neural compute to beat Microsoft's requirement.

The attempt to connect multiple AI accelerators to the Pi and the issues encountered.

The conclusion that a beefier NPU might be a better solution for high TOPS requirements.

The potential future of AI in microcontrollers and camera modules.

Final thoughts on the AI kit's niche appeal and its value for machine vision and robotics.