🏆 Winner: Best Overall / Most Impressive Project

Physical AI - Reading Robot

What is a useful but not proven thing a robot arm can do utilizing physical AI?

Physical AI - Reading Robot

The Problem

Hackathon prompt: Make a robot do a task that is currently challenging in the space…


 My take on it: How might we create a screen-free, highly engaging storytelling experience that makes reading more accessible and interactive using physical AI? :)

The Solution

Diagram explaining UX Engineering: Robot Infrastructure The Challenge The Goal: Bridge the gap between raw hardware constraints and high-volume ML training pipelines. The Problem: Traditional UI frameworks lack established design patterns for 3D spatial interfaces and real-time sensor streams. Architectural Impact Dual-Perception Vision: Designed an optimized visual hierarchy mapping split camera feeds (gripper vs. overview) to maximize operator situational awareness. Telemetry & Kinematics: Integrated live 6-DoF joint state data directly into a low-latency debugging layout. ML Data Pipeline: Abracted complex Hugging Face LeRobot dataset recording into an intuitive, human-in-the-loop teleoperation workflow. Infrastructure & DX (Developer Experience) Hardware-Aware UI: Built interface logic grounded in physical system specs (payload limits, joint velocities, and latency windows). Zero-Precedent Design: Engineered a custom, minimalist design system token architecture optimized for high-cognitive-load lab environments.
The Goal is to bridge the gap between raw hardware constraints and high-volume ML training pipelines. The Problem is that traditional UI frameworks lack established design patterns for 3D spatial interfaces and real-time sensor streams.


Robot reading demo:

we actually sampled a teammate’s voice for this. This was done to again to reinforce human comfortability.

RoleUX Designer, Strategist & Vibecoder
Timeline48hrs
Tech Stack
  • • GitHub
  • • Antigravity IDE
  • • SO-101 Robotic Arm (Solo-CLI)
  • • Claude Vision API
  • • ElevenLabs Audio
The TeamShola + 8 Hackathon Attendees (see linkedin post)
#1 🏆Best Overall/Most Impressive Project awarded to my team!

Design Process & Strategy

1

This or That?

Which robot for our task?:
Diagram picture of choosing robot arm over humanoid robot
Diagram picture of choosing robot arm over humanoid robot

There were a bunch of robots to choose from at the hackathon including a humanoid robot that we really wanted to play with however for purposes we decided that the SO-101 Robotic arm was better. Why? Because it was more easily approachable by people and because the code would be easier to integrate into an arm than the full humanoid robot.

2

Define User Interaction

Map what to do:

Mapped out how a user intuitively approaches, initiates, and stops the robotic arm without complex interfaces.

Scrappy Figjam diagram used with the engineering team to map out the flow.
Scrappy Figjam diagram used with the engineering team to map out the flow.
Ensuring Safety & Trust:

Designed physical constraints and visual/auditory cues so the user felt safe interacting with moving hardware.

Safety and Trust principles communicated with the team.
Safety and Trust principles communicated with the team.
Systems Thinking:
systems diagram: Breaking down the physical AI interaction loop: How the Ladybug system seamlessly bridges hardware actuation and digital synthesis to deliver an accessible, screen-free storytelling experience.
Breaking down the physical AI interaction loop: How the Ladybug system seamlessly bridges hardware actuation and digital synthesis to deliver an accessible, screen-free storytelling experience.
3

AI Integration & Feedback

Auditory Experience:

Selected and tuned the ElevenLabs voice model to sound engaging and natural, matched the pacing of physical page turns.

System Status:

Communicated the state of the robot (scanning the page, processing text, reading, turning) without breaking the magic of the storytelling experience.

4

Rapid Prototyping

Build it!:

Tested the interactions in real-time under extreme 48-hour constraints. We validated the timing of the mechanical page turn against the audio playback, ensuring the visual classification (distinguishing content from covers/blank pages) felt seamless to the user.

  1. Engaging storytelling experience for blind or limited mobility of the hands, arms or fingers.
  2. Opens the book
gif of robot opening gif
robot opens the book

3. Turns pages

gif of robot arm turning pages.
robot arm turning pages

4. Reads aloud using computer vision and AI voice generation.

gif of robot finishing book and closing it.
robot finishes the book and closes it.

5. The result feels like a guided story time, where the hardware and AI behave predictably and the user always knows what will happen next.

5

What I learned & More.

What I Learned:

I was one of the few designers in a room full of robotics engineers. Staying human-centered made the difference: the engineering made the robot work, and the design made people want to use it. Physical AI needs both.

My Role:

I chose the robot we built on (the SO-101 arm over a humanoid), defined the task it would attempt, and brought the physical parts that helped the final demo come together. I also designed how the robot responds to human error, like a page turned wrong or a reader losing their place.

What Worked:

Designing for human error was my idea, and it carried the demo. The robot learned not only what to do but how to do it: when something went wrong, it recovered calmly instead of stalling. That is what made a moving arm feel safe to sit next to.

Outcome:

Out of 45 projects and 300 participants, Ladybug won Best Overall. Judges responded to how friendly the robot felt, the clarity of the idea, and how well it ran live.

See it in action

GithubLinkedin post