🏆 Winner: Best Overall / Most Impressive Project

Ladybug Robotics - Reading Robot

What is a useful but not proven thing a robot arm can do utilizing physical AI?

Ladybug Robotics - Reading Robot

The Problem

How might we create a screen-free, highly engaging storytelling experience that makes reading more accessible and interactive using physical AI?

The Solution

Diagram explaining UX Engineering: Robot Infrastructure The Challenge The Goal: Bridge the gap between raw hardware constraints and high-volume ML training pipelines. The Problem: Traditional UI frameworks lack established design patterns for 3D spatial interfaces and real-time sensor streams. Architectural Impact Dual-Perception Vision: Designed an optimized visual hierarchy mapping split camera feeds (gripper vs. overview) to maximize operator situational awareness. Telemetry & Kinematics: Integrated live 6-DoF joint state data directly into a low-latency debugging layout. ML Data Pipeline: Abracted complex Hugging Face LeRobot dataset recording into an intuitive, human-in-the-loop teleoperation workflow. Infrastructure & DX (Developer Experience) Hardware-Aware UI: Built interface logic grounded in physical system specs (payload limits, joint velocities, and latency windows). Zero-Precedent Design: Engineered a custom, minimalist design system token architecture optimized for high-cognitive-load lab environments.
The Goal is to bridge the gap between raw hardware constraints and high-volume ML training pipelines. The Problem is that traditional UI frameworks lack established design patterns for 3D spatial interfaces and real-time sensor streams.
A system architecture diagram titled "How the Reading Robot Works," detailing a five-step process for a screen-free storytelling experience. The steps are: 1. Accessible Entry for users with limited mobility; 2. Physical Actuation using an SO-101 Robotic Arm to open the book; 3. Mechanical Navigation for turning pages with tactile precision; 4. Vision & Synthesis using Claude and ElevenLabs to identify text and generate AI voice; and 5. Guided Experience, resulting in a harmonious and predictable integration of hardware and AI.
Breaking down the physical AI interaction loop: How the Ladybug system seamlessly bridges hardware actuation and digital synthesis to deliver an accessible, screen-free storytelling experience.
RoleUX Designer, Strategist & Vibecoder
Timeline48hrs
Tech Stack
  • • Antigravity IDE
  • • SO-101 Robotic Arm (Solo-CLI)
  • • Claude Vision API
  • • ElevenLabs Audio
  • • GitHub
The TeamShola + 8 Hackathon Attendees
300Accepted
1Best Overall/Most Impressive Project Award
45Projects

Design Process & Strategy

1

This or That?

Which robot for our task?:
Diagram picture of choosing robot arm over humanoid robot
Diagram picture of choosing robot arm over humanoid robot

There were a bunch of robots to choose from at the hackathon including a humanoid robot that we really wanted to play with however for purposes we decided that the SO-101 Robotic arm was better. Why? Because it was more easily approachable by people and because the code would be easier to integrate into an arm than the full humanoid robot.

2

Define User Interaction

AI Integration & Feedback:

Mapped out how a user intuitively approaches, initiates, and stops the robotic arm without complex interfaces.

Scrappy Figjam diagram used with the engineering team to map out the flow.
Scrappy Figjam diagram used with the engineering team to map out the flow.
Safety & Trust:

Designed physical constraints and visual/auditory cues so the user felt safe interacting with moving hardware.

Safety and Trust principles communicated with the team.
Safety and Trust principles communicated with the team.
3

AI Integration & Feedback

Auditory Experience:

Selected and tuned the ElevenLabs voice model to sound engaging and natural, matched the pacing of physical page turns.

System Status:

Communicated the state of the robot (scanning the page, processing text, reading, turning) without breaking the magic of the storytelling experience.

4

Rapid Prototyping

Build it!:

Tested the interactions in real-time under extreme 48-hour constraints. We validated the timing of the mechanical page turn against the audio playback, ensuring the visual classification (distinguishing content from covers/blank pages) felt seamless to the user.

  1. Engaging storytelling experience for blind or limited mobility of the hands, arms or fingers.
  2. Opens the book
gif of robot opening gif
robot opens the book

3. Turns pages

gif of robot arm turning pages.
robot arm turning pages

4. Reads aloud using computer vision and AI voice generation.

gif of robot finishing book and closing it.
robot finishes the book and closes it.

5. The result feels like a guided story time, where the hardware and AI behave predictably and the user always knows what will happen next.

See it in action

Github