Brain Waves Could Solve Physical AI's Data Scarcity Problem

{ "title": "Brain waves and Jenga: Inside Encord's plan to break the physical AI data bottleneck", "body": "In a warehouse in San Leandro, California, a pilot named Andrew Ceja sits down to play Jenga. He wears a headset that tracks what his eyes see and, crucially, what his brain feels. The headset, supplied by German neuroscience startup Zander Labs, measures brain activity to deduce mental states like error, intent, and surprise. The goal is to generate richer training data for physical AI — the humanoid robots and warehouse automation systems that the industry believes are the next frontier.\n\nEncord, the AI data tooling company that operates the facility, is trialing the brain wave headset to address what it sees as the biggest bottleneck in robotics: a scarcity of real-world physical training data. The trial aims to build an initial brain wave-tagged dataset, run it through customer robotics models, evaluate performance improvement, and then decide whether to scale. The company, founded in 2019, has grown from a tooling provider for machine-vision applications into a full-service data producer, operating its own warehouse and employing pilots to generate the physical-world data that robots need to learn.\n\n## The data that does not exist\n\nEncord was founded to help companies building machine-vision applications annotate data and evaluate models. Now, it produces training data itself. Vineeth Velmurugan, head of robot learning at Encord and a veteran of OpenAI’s robot lab and the warehouse automation firm Berkshire Grey, put it bluntly: “The data simply does not exist.”\n\nSelf-driving car companies collect physical-world data themselves, but that approach is hard to scale. Training from video can work but lacks the fidelity of real-world data. Velmurugan estimates that it will take a dataset about five times the size of YouTube’s video corpus to break through the physical AI data bottleneck. That comparison underscores the sheer volume of data needed: YouTube users upload hundreds of hours of video every minute, and Encord believes physical AI will require an order of magnitude more.\n\nCompanies building robot brains currently rely on two main sources: egocentric video from workers wearing cameras, augmented with other angles and metrics, and data from remotely operated robots. Encord draws egocentric data from several factories globally and uses its San Leandro facility to experiment with new modalities — like brain waves — or collect datasets for specific skills that require fine-tuning. The warehouse has about a dozen pilots. Both Ceja and Sofia Infante, another pilot, previously worked at Scale, the AI data annotation firm, before joining Encord. Ceja also worked at a waste management company maintaining a robotic trash sorter. Infante now maneuvers robotic arms to plug and unplug ethernet cables from a server back — a task data center operators want automated.\n\nThe pilots are not just data collectors; they are problem-solvers. Ceja described the work as a constant puzzle: how to break down a human action into steps a robot can learn. Infante’s work on ethernet cables, for instance, requires precise force control — a skill that current robotic arms struggle with. The data they generate is annotated with physical descriptions like “right hand tightens bolt” to aid LLM-based models. Velmurugan said dense annotation is worth 100 times as much as “junky ego data” for training specific tasks, and costs only 20 times more to produce. “It’s a good trade, on paper,” he said.\n\n## Brain waves as a data modality\n\nThe headset Ceja wears combines a camera tracking what he sees with brain wave sensors. Lukas Gehrke, a neuroscientist at Zander Labs who supervises the work, said the amount of brain activity during a task offers clues for when to deploy the highest-effort models. The idea is that a brain wave-tagged dataset can create a more useful training set for models, helping them understand not just what happened, but what the human operator was thinking or feeling at the time.\n\nGehrke’s team measures brain activity to deduce whether the pilot detected an error, formed an intent, or was surprised by an outcome. Those signals, layered onto the physical actions recorded by the camera and robotic arms, could help models learn more efficiently. For example, if a pilot’s brain shows a surprise signal when a Jenga tower wobbles, a model can learn that the outcome was unexpected — a cue that might be invisible in video alone.\n\nVelmurugan called this the “bleeding edge” of the effort to solve the robotics data bottleneck. “Every humanoid company has asked us for these pieces,” he said, referring to the specific data collection techniques Encord is developing. The brain wave modality is still experimental, but early results suggest it can reduce the amount of data needed to train a model for a given task. Gehrke noted that the headset is non-invasive and can be worn for hours, making it practical for warehouse settings.\n\n## Manufacturing physical training data\n\nThe economics of building physical AI models differ sharply from those of large language models. LLM makers scraped text off the internet — from sources like Stack Overflow — at near-zero cost. Physical data, by contrast, has to be manufactured, not just collected. That changes the economics of building models. Encord must pay pilots, maintain a warehouse, and operate robotic rigs to generate each data point.\n\nEncord’s pilots use leader-follower rigs — paired robotic arms where one is controlled by a human and the other mimics the motion — to create data for tasks like pouring coffee, which involves sloshy liquids, and stacking poker chips. Storage racks in the warehouse hold cartons of fake flowers, books, plastic vegetables, kitty litter trays, scoops, bags, and wires — all props for training manipulators for household tasks. The variety of objects is deliberate: robots must learn to handle items of different shapes, weights, and textures.\n\nThe pincers on the robotic arms are far less dexterous than human fingers and lack the degrees of freedom of human arms. To compensate, Encord is developing another data modality: sensors strapped to the forearm to detect electrical signals in muscles. The goal is to build a 3D depiction of hand position based on those arm sensors for a more robust understanding of human motion. This technique, called electromyography, could allow robots to infer hand posture even when the camera view is blocked.\n\nEncord’s datasets are annotated with physical descriptions like “right hand tightens bolt” to aid LLM-based models. Velmurugan said dense annotation is worth 100 times as much as “junky ego data” for training specific tasks, and costs only 20 times more to produce. “It’s a good trade, on paper,” he said. The company is also experimenting with synthetic data, generated by simulating tasks in software, but Velmurugan stressed that real-world data remains essential for tasks involving physical contact or unpredictable materials.\n\n## The business of data scarcity\n\nEncord is one of a growing number of startups betting that the next real constraint on humanoid and warehouse robots will be scarcity of real-world physical training data. The bet that generative AI can do for robots what it has done for chatbots keeps running into this same wall. Data generation itself has become a business, not just a research problem. Competitors include companies like Scale AI and Sama, but Encord differentiates itself by operating its own physical facility and employing pilots full-time.\n\nEncord’s vantage point — sitting between many robotics companies — is part of its pitch. Velmurugan said the company’s visibility across the industry lets it see which data techniques gain traction before any single customer does. Progress is being made, he said, even if the scale of the challenge is enormous. He pointed to recent advances in robot dexterity, such as the ability to fold laundry or assemble furniture, as evidence that the data approach is working.\n\nCeja, who enjoys the challenge of solving training tasks for robots, said, “It’s something new every day!” For now, that means playing Jenga with a brain wave headset on, while a neuroscientist watches the signals and a robotic arm learns from every pull. The long-term goal is to reduce the amount of human effort required to train robots, but for the foreseeable future, humans like Ceja and Infante remain essential. As Velmurugan put it, “The data simply does not exist” — and until it does, someone has to make it.\n\n## Related on Neura Market\n\n- AI and Robotics Market Analysis\n- Data Annotation and Tooling Sector\n- Neuroscience and Brain-Computer Interfaces" }