Facebook is working on systems that will allow you to type by thought alone and speak through touch

Regina Duncan

Facebook yesterday shared insights from a secretive unit headed by a former chief of the Pentagon’s research arm, disclosing that the social media company is studying ways for people to communicate by thought and touch. It launched the unit, called Building 8, last year to conduct long-term work that might lead to hardware products. In charge is Regina Dugan, who led a similar group at Alphabet Inc’s Google and was previously director of the US Government’s Advanced Research Projects Agency, or DARPA.

Dugan told software developers at Facebook’s annual F8 conference that the company was modeling Building 8 after DARPA, a government office founded in the 1950s that devised gave the Internet and GPS receivers used in consumer devices. Any hardware rollouts are years away, Dugan said in a speech. Potential products could, if successful, be a way for Facebook to diversify beyond its heavy reliance on advertising revenue. One example of Building 8’s work so far, Dugan said, was an attempt to improve technology that allows people to type words using their minds. “It sounds impossible, but it’s closer than you may realise,” Dugan said.

Using brain implants, people can already type eight words a minute, she said. Facebook’s goal, working with researchers at several US universities, is to make the system non-invasive, as well as fast enough so that people can type 100 words a minute just by thinking. Possible uses include helping disabled people and “the ability to text your friend without taking out your phone,” she said. Another Building 8 project, she said, was trying to advance the ability to communicate through touch only, an idea with roots in Braille, a writing system for the blind and visually impaired. “We are building the hardware and software necessary to deliver language through the skin.” A video played at the conference showed two Facebook employees talking to each other through touch. As one employee, Frances, wore an electronic device on her arm, the other, Freddy, used a computer programme to send pressure changes to her arm. “If you ask Frances what she feels,” Dugan said, “she’ll tell you that she has learned to feel the acoustic shape of a word on her arm.” In December, Facebook signed a deal with 17 universities including Harvard and Princeton to allow swifter collaboration on projects with Dugan’s team.

Day Two Highlights from F8

In the opening keynote, Facebook chief technology officer Mike Schroepfer talked about its goal to develop technology that will help everyone “build global community. To do that, we’re investing in a number of foundational technologies over the next 10 years, including connectivity, artificial intelligence, and virtual and augmented reality.” Schroepfer and other speakers — director of connectivity programmes, Yael Maguire; director of applied machine learning, Joaquin Quiñonero Candela; chief scientist of Oculus Research, Michael Abrash; and vice president of engineering and Building 8, Regina Dugan — shared updates and visions for some of the company’s long-term focus areas.

Connectivity

Rather than look for a one-size-fits-all connectivity solution, Facebook is investing in a building-block strategy — designing different technologies for different use cases, which are then used together to create flexible and extensible networks. It highlighted new milestones in its efforts to reach people who are unconnected and increase capacity and performance for everyone else. Facebook said itset three new records in wireless data transfer: 36 gigabits per second over 13 kilometers point-to-point using millimeter-wave (MMW) technology; 80 gigabits per second between those same points using its optical cross-link technology; and 16 gigabits per second from a location on the ground to a circling Cessna aircraft over 7 kilometers away using MMW. Additionally, its Terragraph system being tested with San Jose in the city’s downtown corridor has become the first city-scale mesh millimeter-wave system capable of delivering fiber-like multi-gigabits/s of performance and reliability. It also announced Tether-tenna, a new kind of “insta-infrastructure” where a small helicopter tethered to a wire containing fiber and power can be deployed immediately to bring back connectivity in case of emergency.

Artificial Intelligence

Facebook is building “visual experiences for people” — including an AI-infused camera across Facebook, Instagram and Messenger. With the ability to run AI and computer vision algorithms on the device, this camera can now understand your surroundings, recognise people, places and things. It can annotate and enhance images and video. The new Camera Effects Platform gives developers a way to build new tools for creative expression, and it shared a few demos of ideas that have come out of its research. In a presentation yesterday, applied machine learning director Joaquin Quiñonero Candela talked about how AI has revolutionised the ability of computers to process and understand images and videos. “It’s easy to forget that only five years ago, computers saw images as just a collection of numbers, with no particular meaning to them. Now computers can understand every single individual pixel of an image. These advancements enable new experiences, like adding digital objects and effects to a real world scene. We believe AI belongs to everyone. That’s why in addition to opening the Camera Effects Platform, we announced that we are open sourcing Caffe2 — a framework to build and run AI algorithms on a phone — and building partnerships with Amazon, Intel, Microsoft, NVIDIA, Qualcomm, and others.”

Virtual Reality

Facebook is investing in VR across mobile and PC hardware, software and content — from Oculus Rift and Gear VR to Facebook Spaces. Yesterday, it introduced the newest designs for the Surround 360 technology that allows people to produce high quality videos for VR. The new x24 and its smaller counterpart, x6, create some of the most immersive and engaging content ever shot for VR. The new camera technology lets you move around within the video scene and experience the content from different viewing angles. This means you can move your head around in the world and see it from different angles — what’s known as six degrees of freedom, or 6DoF — bringing the feeling of immersion and depth to a whole new level.

Augmented Reality

On day one of F8, Mark Zuckerberg talked about how the camera is the first augmented reality platform. Yesterday, chief scientist of Oculus Research Michael Abrash shared a vision for the path to full AR — where augmentation enhances your vision and hearing seamlessly while being light, comfortable, power-efficient and socially acceptable enough to accompany you everywhere. He talked about the rise of virtual computing — which encompasses both virtual and augmented reality — as the next great wave after personal computing. “Virtual computing is just starting to form, but it will give us the ability to transcend time and space to connect with one another in new ways. In order to make virtual computing as much a part of everyday life as the smartphone is today, we’re going to need see-through augmented reality, which will likely be transparent glasses that can show virtual images overlaid on the real world. The set of technologies needed to reach full AR doesn’t exist yet. This is a decade-long investment and it will require major advances in material science, perception, graphics and many other areas. But once that’s achieved, AR has the potential to enhance almost every aspect of our lives, revolutionizing how we work, play and interact.

For more details on the final day’s topics, see Facebook’s Developer Blog and Engineering Blog. You can also watch all F8 keynotes on the Facebook for Developers Page.