Facebook announced that it would implement a new artificial intelligence technology—known as GrokNet—as part of its platform. The technology, which began as GrokStyle, was innovated by CS Professor and Dean of Computing and Information Science Kavita Bala and Sean Bell (M.S. ’15, Ph.D., ’16). Bell, a Research Scientist at Facebook, will now lead GrokNet at Facebook.

In an article in the Cornell Chronicle by Melanie Lefkowitz, we learn that GrokNet “builds on the GrokStyle technology and automatically suggests attributes such as colors and materials when sellers upload photos of products for sale […]. The technology allows buyers to conduct detailed searches when looking for specific items, even if the sellers didn’t add those details to the descriptions when they posted the photos.” As Bala explains: “When you take a picture of something, we can tell you not only if it’s a chair or a table, which anyone can do, but we can also tell you exactly which chair or table it is—an Ikea Mammut table or an Eames chair—as well as what style and material they appear to be.”

GrokNet now powers Facebook Marketplace “by predicting color, style and material attributes, and by matching uploaded photos to clean catalog images.” The technology “uses a combination of deep learning (a kind of artificial intelligence) and a very large database of images.” Bell says: “It can be tricky to market a product or dedicate time to creating listings. GrokNet does the hard work for you.” 

TechHQ has run a profile of the new technology, where Jia Jen Low writes that the new GrokNet “system [is] twice as accurate as Facebook’s prior systems at recognizing products” and “improved coverage for its Home and Garden category from 33% to 90%. Trained on a diverse data set, it even works for items that may look different depending on what part of the world you’re in.” This insight became evident during the start-up’s origins, when, back in 2015, GrokStyle collaborated with Swedish design company, Ikea. As Bala observes: “Augmented reality is a particularly compelling experience in the context of furniture, shopping, and interior design. And so rolling out with Ikea made a lot of sense for us, and indeed, they felt that we did for them too.”

As Low concludes: “The research and development, and subsequent sale of GrokStyle demonstrates the appetite in industry for technology that can enhance user experience, and ultimately drive engagement in new or augmented applications—optimizing the buying and selling experience of Facebook Marketplace is just one example of computer vision’s potential, and there’s a long way to go yet.”

Bala—the outgoing Chair of Computer Science and incoming Dean of Computing and Information Science (CIS)—was recently awarded the ACM SIGGRAPH 2020 Computer Graphics Achievement Award. She is also an ACM Fellow.

At the latest Women in Computer Vision Workshop (WiCV), in conjunction with CVPR Virtual, Bala gave an invited talk “Understanding Visual Appearance from Micron to Global Scale.” After noting that “augmented reality/mixed reality (AR/MR) is poised to create compelling and immersive user experiences by combining computer vision and computer graphics,” Bala described her “group’s research on these complementary areas: graphics models for realistic visual appearance, and visual search and fine-grained recognition for scene understanding.”