Singulos Research introduces Perceptus AI for AR that understands the real world

Singles is revealing Perceptus, a space technology that enables augmented reality experiences with greater fidelity to the real world.

Vancouver, Canada-based Singulos believes it can enable a new class of compelling augmented reality applications using its Perceptus platform, which enables real-time understanding and tracking of physical objects in arbitrary 3D environments.

I interviewed CEO Brad Quinton, an artificial intelligence professor and researcher at the University of British Columbia, and he showed me a demo where we played a round of chess together.

“Perceptus is really about giving augmented reality an understanding of objects in the environment,” said Quinton. “There are many augmented reality applications, but they don’t do much. You can put a vase on a table, or pretend you have a piece of furniture in the corner. But apps don’t react to the world around you. And that’s why we believe that giving AR the ability to understand these objects allows it to understand their context. And using that context, you can be proactive and help you engage with the world.”

To show what he meant, he used an iPad to capture a real-world view of his chessboard. He moved a piece on his actual chessboard, and then I looked at my screen and saw the piece move on my screen. I used my mouse to move a pawn and he could see the move on his computer screen. So the app was smart enough to see the board pieces from him and integrate them into the digital game we were playing together remotely.

This sounds simple enough, but it requires other computer vision technology to create an augmented reality experience.

The technology gives AR a visual understanding of the world, allowing apps to create AR experiences that are engaging, accessible, and useful. Through the Perceptus platform, smartphones, tablets, and AR/VR headsets can not only see, but also continuously understand physical objects in 3D, all while operating at the edge to protect user privacy.

This ability to dynamically identify objects and their context, as people move freely through the real world, forms the basis for a new generation of AR and mixed reality apps and experiences that could change the way humans can interact. with computers, Quinton said.

As of today, the Perceptus platform is available via licenses to interested companies.

Perceptus was developed by a team of serial entrepreneurs from Singlellos Research. Led by Quinton, the team has previously founded high-tech start-ups that were quickly acquired by industry leaders such as Qualcomm and Tektronix. Together, the Singles team has developed the semiconductor technologies and software at the core of most modern smartphones and mobile processors today.

I’m sure we all have futuristic movies or concept videos from big tech companies. Of those, you can expect AR to already provide a number of useful features, but in reality, AR today lacks a visual understanding of the world around it, he said.

In a demo, he showed a video that used Perceptus to identify all the parts needed to put together a piece of furniture. One by one, the iPad camera identified and color-coded each piece. A piece was missing and the camera moves in one direction and finds the missing piece.

In another demo, the iPad’s camera hovers over a bunch of Lego pieces and identifies them individually. You can watch it identify and categorize each piece. And then find out what you can build with all those pieces. The company calls this dynamic understanding, and it works with any lighting conditions, head movements, or object rotations.

“We have an understanding of that Lego piece in front of you, and we’re helping you come up with ideas for things you could build,” Quinton said, proceeding to demonstrate. “I’m going to start detecting all these pieces. So one of the important things to note here is that Perceptus is understanding where these pieces are in three dimensions, even when they’re not in camera view. So if I move the camera, it rediscovers pieces that I already know. It adds the sum to inventory when it finds new ones.”

You can see some of the potential as the combination of device and app gets smarter.

“Now Perceptus says that you could build that item with this specific inventory,” Quinton said. “I could say I could build something, or I could say I don’t have all the right colors. It is adapting to the environment. We believe that app developers can create all kinds of apps that interact with the real world.”

The Lego app might have a database of designs that you can access to determine what you can build, like a plastic duck.

“The good thing is that it will only show you things that work with what’s in front of you,” Quinton said.

Eventually, it would be great if Perceptus could tell you step by step how to build something. But he doesn’t do that on the show today, Quinton said. If anything, customers would be the ones to figure out how to build such things with their applications.

“If you’re a furniture maker, of course you already have the instructions,” he said. “So it’s about the machine learning all the different parts.”

It’s similar to the Osmo AI technology that uses a camera on an iPad to recognize items in a kids’ learning app.

Quinton said that understanding objects, in real time, is extremely challenging. In fact, that’s one of the reasons much of the metaverse’s early focus was limited to entirely virtual worlds. It is much simpler for a computer to understand the artificial world it has created. But Singulos believes that for most applications people prefer to exist in the real world.

More specifically, Perceptus’ core technology uses an entirely new, proprietary approach to artificial intelligence and computer vision that enables the platform to continuously identify, understand and track objects. This technological advancement is what enables a new class of compelling and useful AR experiences.

Singles is also built on existing mobile processor technology: any smartphone, tablet and glasses can deliver intelligent and physically aware AR experiences, making AR experiences more accessible and mainstream.

And it has the ability to control locally, without the cloud. Processing is done entirely at the edge, just like on the iPad, giving users direct control of their data. This is, of course, attractive to any company that takes privacy seriously.

The Singles research team is currently organizing discussions with technology companies that are interested in licensing deals to use Perceptus and embed the platform in their own apps and devices.

When you move an item on the chessboard, “what I think is exciting about this is the idea that you’re now manipulating my reality, right?” Quinton said.

The company is talking to potential licensees who can develop their own apps. Quinton said early customers can walk away with something for the summer. He said the technology targets Qualcomm’s Snapdragon processors, Apple’s M1 processor and one of Google’s processors.

“What we need for this to work are neural accelerators,” he said. “All modern mobile phones have neural accelerators, like Apple’s Neural Engine and the Hexagon in Qualcomm chipsets. And that is, that is really what is required. So we can be Android, we can be iOS, that’s not a problem. But it has to be a fairly modern mobile processor.”

Over time, the technology will improve with better and faster AI hardware. The company exists for three years and has six people.