The future of AI is neuromorphic. Meet the scientists building digital ‘brains’ for your phone

By Hugo Angel,

Neuromorphic chips are being designed to specifically mimic the human brain – and they could soon replace CPUs
Neuroscape Lab
AI services like Apple’s Siri and others operate by sending your queries to faraway data centers, which send back responses. The reason they rely on cloud-based computing is that today’s electronics don’t come with enough computing power to run the processing-heavy algorithms needed for machine learning. The typical CPUs most smartphones use could never handle a system like Siri on the device. But Dr. Chris Eliasmith, a theoretical neuroscientist and co-CEO of Canadian AI startup Applied Brain Research, is confident that a new type of chip is about to change that.
Many have suggested Moore’s law is ending and that means we won’t get ‘more compute’ cheaper using the same methods,” Eliasmith says. He’s betting on the proliferation of ‘neuromorphics’ — a type of computer chip that is not yet widely known but already being developed by several major chip makers.
Traditional CPUs process instructions based on “clocked time” – information is transmitted at regular intervals, as if managed by a metronome. By packing in digital equivalents of neurons, neuromorphics communicate in parallel (and without the rigidity of clocked time) using “spikes” – bursts of electric current that can be sent whenever needed. Just like our own brains, the chip’s neurons communicate by processing incoming flows of electricity – each neuron able to determine from the incoming spike whether to send current out to the next neuron.
What makes this a big deal is that these chips require far less power to process AI algorithms. For example, one neuromorphic chip made by IBM contains five times as many transistors as a standard Intel processor, yet consumes only 70 milliwatts of power. An Intel processor would use anywhere from 35 to 140 watts, or up to 2000 times more power.
Eliasmith points out that neuromorphics aren’t new and that their designs have been around since the 80s. Back then, however, the designs required specific algorithms be baked directly into the chip. That meant you’d need one chip for detecting motion, and a different one for detecting sound. None of the chips acted as a general processor in the way that our own cortex does.
This was partly because there hasn’t been any way for programmers to design algorithms that can do much with a general purpose chip. So even as these brain-like chips were being developed, building algorithms for them has remained a challenge.
Eliasmith and his team are keenly focused on building tools that would allow a community of programmers to deploy AI algorithms on these new cortical chips.
Central to these efforts is Nengo, a compiler that developers can use to build their own algorithms for AI applications that will operate on general purpose neuromorphic hardware. Compilers are a software tool that programmers use to write code, and that translate that code into the complex instructions that get hardware to actually do something. What makes Nengo useful is its use of the familiar Python programming language – known for it’s intuitive syntax – and its ability to put the algorithms on many different hardware platforms, including neuromorphic chips. Pretty soon, anyone with an understanding of Python could be building sophisticated neural nets made for neuromorphic hardware.
Things like vision systems, speech systems, motion control, and adaptive robotic controllers have already been built with Nengo,Peter Suma, a trained computer scientist and the other CEO of Applied Brain Research, tells me.
Perhaps the most impressive system built using the compiler is Spaun, a project that in 2012 earned international praise for being the most complex brain model ever simulated on a computer. Spaun demonstrated that computers could be made to interact fluidly with the environment, and perform human-like cognitive tasks like recognizing images and controlling a robot arm that writes down what it’s sees. The machine wasn’t perfect, but it was a stunning demonstration that computers could one day blur the line between human and machine cognition. Recently, by using neuromorphics, most of Spaun has been run 9000x faster, using less energy than it would on conventional CPUs – and by the end of 2017, all of Spaun will be running on Neuromorphic hardware.
Eliasmith won NSERC’s John C. Polyani award for that project — Canada’s highest recognition for a breakthrough scientific achievement – and once Suma came across the research, the pair joined forces to commercialize these tools.
While Spaun shows us a way towards one day building fluidly intelligent reasoning systems, in the nearer term neuromorphics will enable many types of context aware AIs,” says Suma. Suma points out that while today’s AIs like Siri remain offline until explicitly called into action, we’ll soon have artificial agents that are ‘always on’ and ever-present in our lives.
Imagine a SIRI that listens and sees all of your conversations and interactions. You’ll be able to ask it for things like – “Who did I have that conversation about doing the launch for our new product in Tokyo?” or “What was that idea for my wife’s birthday gift that Melissa suggested?,” he says.
When I raised concerns that some company might then have an uninterrupted window into even the most intimate parts of my life, I’m reminded that because the AI would be processed locally on the device, there’s no need for that information to touch a server owned by a big company. And for Eliasmith, this ‘always on’ component is a necessary step towards true machine cognition. “The most fundamental difference between most available AI systems of today and the biological intelligent systems we are used to, is the fact that the latter always operate in real-time. Bodies and brains are built to work with the physics of the world,” he says.
Already, major efforts across the IT industry are heating up to get their AI services into the hands of users. Companies like Apple, Facebook, Amazon, and even Samsung, are developing conversational assistants they hope will one day become digital helpers.
Monday 6 March 2017

A Giant Neuron Has Been Found Wrapped Around the Entire Circumference of the Brain

By Hugo Angel,

Allen Institute for Brain Science

This could be where consciousness forms. For the first time, scientists have detected a giant neuron wrapped around the entire circumference of a mouse’s brain, and it’s so densely connected across both hemispheres, it could finally explain the origins of consciousness.

Using a new imaging technique, the team detected the giant neuron emanating from one of the best-connected regions in the brain, and say it could be coordinating signals from different areas to create conscious thought.

This recently discovered neuron is one of three that have been detected for the first time in a mammal’s brain, and the new imaging technique could help us figure out if similar structures have gone undetected in our own brains for centuries.

At a recent meeting of the Brain Research through Advancing Innovative Neurotechnologies initiative in Maryland, a team from the Allen Institute for Brain Science described how all three neurons stretch across both hemispheres of the brain, but the largest one wraps around the organ’s circumference like a “crown of thorns”.
You can see them highlighted in the image at the top of the page.

Lead researcher Christof Koch told Sara Reardon at Nature that they’ve never seen neurons extend so far across both regions of the brain before.
Oddly enough, all three giant neurons happen to emanate from a part of the brain that’s shown intriguing connections to human consciousness in the past – the claustrum, a thin sheet of grey matter that could be the most connected structure in the entire brain, based on volume.

This relatively small region is hidden between the inner surface of the neocortex in the centre of the brain, and communicates with almost all regions of cortex to achieve many higher cognitive functions such as

  • language,
  • long-term planning, and
  • advanced sensory tasks such as
  • seeing and
  • hearing.

Advanced brain-imaging techniques that look at the white matter fibres coursing to and from the claustrum reveal that it is a neural Grand Central Station,Koch wrote for Scientific American back in 2014. “Almost every region of the cortex sends fibres to the claustrum.”

The claustrum is so densely connected to several crucial areas in the brain that Francis Crick of DNA double helix fame referred to it a “conductor of consciousnessin a 2005 paper co-written with Koch.

They suggested that it connects all of our external and internal perceptions together into a single unifying experience, like a conductor synchronises an orchestra, and strange medical cases in the past few years have only made their case stronger.

Back in 2014, a 54-year-old woman checked into the George Washington University Medical Faculty Associates in Washington, DC, for epilepsy treatment.

This involved gently probing various regions of her brain with electrodes to narrow down the potential source of her epileptic seizures, but when the team started stimulating the woman’s claustrum, they found they could effectively ‘switch’ her consciousness off and on again.

Helen Thomson reported for New Scientist at the time:
When the team zapped the area with high frequency electrical impulses, the woman lost consciousness. She stopped reading and stared blankly into space, she didn’t respond to auditory or visual commands and her breathing slowed.

As soon as the stimulation stopped, she immediately regained consciousness with no memory of the event. The same thing happened every time the area was stimulated during two days of experiments.”

According to Koch, who was not involved in the study, this kind of abrupt and specific ‘stopping and starting‘ of consciousness had never been seen before.

Another experiment in 2015 examined the effects of claustrum lesions on the consciousness of 171 combat veterans with traumatic brain injuries.

They found that claustrum damage was associated with the duration, but not frequency, of loss of consciousness, suggesting that it could play an important role in the switching on and off of conscious thought, but another region could be involved in maintaining it.

And now Koch and his team have discovered extensive neurons in mouse brains emanating from this mysterious region.

In order to map neurons, researchers usually have to inject individual nerve cells with a dye, cut the brain into thin sections, and then trace the neuron’s path by hand.

It’s a surprisingly rudimentary technique for a neuroscientist to have to perform, and given that they have to destroy the brain in the process, it’s not one that can be done regularly on human organs.

Koch and his team wanted to come up with a technique that was less invasive, and engineered mice that could have specific genes in their claustrum neurons activated by a specific drug.

When the researchers fed the mice a small amount of the drug, only a handful of neurons received enough of it to switch on these genes,Reardon reports for Nature.

That resulted in production of a green fluorescent protein that spread throughout the entire neuron. The team then took 10,000 cross-sectional images of the mouse brain, and used a computer program to create a 3D reconstruction of just three glowing cells.

We should keep in mind that just because these new giant neurons are connected to the claustrum doesn’t mean that Koch’s hypothesis about consciousness is correct – we’re a long way from proving that yet.

It’s also important to note that these neurons have only been detected in mice so far, and the research has yet to be published in a peer-reviewed journal, so we need to wait for further confirmation before we can really delve into what this discovery could mean for humans.

But the discovery is an intriguing piece of the puzzle that could help up make sense of this crucial, but enigmatic region of the brain, and how it could relate to the human experience of conscious thought.

The research was presented at the 15 February meeting of the Brain Research through Advancing Innovative Neurotechnologies initiative in Bethesda, Maryland.

ORIGINAL: ScienceAlert

28 FEB 2017

Google Unveils Neural Network with “Superhuman” Ability to Determine the Location of Almost Any Image

By Hugo Angel,

Guessing the location of a randomly chosen Street View image is hard, even for well-traveled humans. But Google’s latest artificial-intelligence machine manages it with relative ease.
Here’s a tricky task. Pick a photograph from the Web at random. Now try to work out where it was taken using only the image itself. If the image shows a famous building or landmark, such as the Eiffel Tower or Niagara Falls, the task is straightforward. But the job becomes significantly harder when the image lacks specific location cues or is taken indoors or shows a pet or food or some other detail.Nevertheless, humans are surprisingly good at this task. To help, they bring to bear all kinds of knowledge about the world such as the type and language of signs on display, the types of vegetation, architectural styles, the direction of traffic, and so on. Humans spend a lifetime picking up these kinds of geolocation cues.So it’s easy to think that machines would struggle with this task. And indeed, they have.

Today, that changes thanks to the work of Tobias Weyand, a computer vision specialist at Google, and a couple of pals. These guys have trained a deep-learning machine to work out the location of almost any photo using only the pixels it contains.

Their new machine significantly outperforms humans and can even use a clever trick to determine the location of indoor images and pictures of specific things such as pets, food, and so on that have no location cues.

Their approach is straightforward, at least in the world of machine learning.

  • Weyand and co begin by dividing the world into a grid consisting of over 26,000 squares of varying size that depend on the number of images taken in that location.
    So big cities, which are the subjects of many images, have a more fine-grained grid structure than more remote regions where photographs are less common. Indeed, the Google team ignored areas like oceans and the polar regions, where few photographs have been taken.


  • Next, the team created a database of geolocated images from the Web and used the location data to determine the grid square in which each image was taken. This data set is huge, consisting of 126 million images along with their accompanying Exif location data.
  • Weyand and co used 91 million of these images to teach a powerful neural network to work out the grid location using only the image itself. Their idea is to input an image into this neural net and get as the output a particular grid location or a set of likely candidates. 
  • They then validated the neural network using the remaining 34 million images in the data set.
  • Finally they tested the network—which they call PlaNet—in a number of different ways to see how well it works.

The results make for interesting reading. To measure the accuracy of their machine, they fed it 2.3 million geotagged images from Flickr to see whether it could correctly determine their location. “PlaNet is able to localize 3.6 percent of the images at street-level accuracy and 10.1 percent at city-level accuracy,” say Weyand and co. What’s more, the machine determines the country of origin in a further 28.4 percent of the photos and the continent in 48.0 percent of them.

That’s pretty good. But to show just how good, Weyand and co put PlaNet through its paces in a test against 10 well-traveled humans. For the test, they used an online game that presents a player with a random view taken from Google Street View and asks him or her to pinpoint its location on a map of the world.

Anyone can play at Give it a try—it’s a lot of fun and more tricky than it sounds.

GeoGuesser Screen Capture Example

Needless to say, PlaNet trounced the humans. “In total, PlaNet won 28 of the 50 rounds with a median localization error of 1131.7 km, while the median human localization error was 2320.75 km,” say Weyand and co. “[This] small-scale experiment shows that PlaNet reaches superhuman performance at the task of geolocating Street View scenes.

An interesting question is how PlaNet performs so well without being able to use the cues that humans rely on, such as vegetation, architectural style, and so on. But Weyand and co say they know why: “We think PlaNet has an advantage over humans because it has seen many more places than any human can ever visit and has learned subtle cues of different scenes that are even hard for a well-traveled human to distinguish.

They go further and use the machine to locate images that do not have location cues, such as those taken indoors or of specific items. This is possible when images are part of albums that have all been taken at the same place. The machine simply looks through other images in the album to work out where they were taken and assumes the more specific image was taken in the same place.

That’s impressive work that shows deep neural nets flexing their muscles once again. Perhaps more impressive still is that the model uses a relatively small amount of memory unlike other approaches that use gigabytes of the stuff. “Our model uses only 377 MB, which even fits into the memory of a smartphone,” say Weyand and co.

That’s a tantalizing idea—the power of a superhuman neural network on a smartphone. It surely won’t be long now!

Ref: : PlaNet—Photo Geolocation with Convolutional Neural Networks

ORIGINAL: TechnoplogyReview
by Emerging Technology from the arXiv
February 24, 2016

JPMorgan Software Does in Seconds What Took Lawyers 360,000 Hours

By Hugo Angel,

New software does in seconds what took staff 360,000 hours Bank seeking to streamline systems, avoid redundancies

At JPMorgan Chase & Co., a learning machine is parsing financial deals that once kept legal teams busy for thousands of hours.

The program, called COIN, for Contract Intelligence, does the mind-numbing job of interpreting commercial-loan agreements that, until the project went online in June, consumed 360,000 hours of work each year by lawyers and loan officers. The software reviews documents in seconds, is less error-prone and never asks for vacation.

Attendees discuss software on Feb. 27, the eve of JPMorgan’s Investor Day.
Photographer: Kholood Eid/Bloomberg

While the financial industry has long touted its technological innovations, a new era of automation is now in overdrive as cheap computing power converges with fears of losing customers to startups. Made possible by investments in machine learning and a new private cloud network, COIN is just the start for the biggest U.S. bank. The firm recently set up technology hubs for teams specializing in big data, robotics and cloud infrastructure to find new sources of revenue, while reducing expenses and risks.

The push to automate mundane tasks and create new tools for bankers and clients — a growing part of the firm’s $9.6 billion technology budget — is a core theme as the company hosts its annual investor day on Tuesday.

Behind the strategy, overseen by Chief Operating Operating Officer Matt Zames and Chief Information Officer Dana Deasy, is an undercurrent of anxiety: Though JPMorgan emerged from the financial crisis as one of few big winners, its dominance is at risk unless it aggressively pursues new technologies, according to interviews with a half-dozen bank executives.

Redundant Software

That was the message Zames had for Deasy when he joined the firm from BP Plc in late 2013. The New York-based bank’s internal systems, an amalgam from decades of mergers, had too many redundant software programs that didn’t work together seamlessly.“Matt said, ‘Remember one thing above all else: We absolutely need to be the leaders in technology across financial services,’” Deasy said last week in an interview. “Everything we’ve done from that day forward stems from that meeting.

After visiting companies including Apple Inc. and Facebook Inc. three years ago to understand how their developers worked, the bank set out to create its own computing cloud called Gaia that went online last year. Machine learning and big-data efforts now reside on the private platform, which effectively has limitless capacity to support their thirst for processing power. The system already is helping the bank automate some coding activities and making its 20,000 developers more productive, saving money, Zames said. When needed, the firm can also tap into outside cloud services from Inc., Microsoft Corp. and International Business Machines Corp.

Tech SpendingJPMorgan will make some of its cloud-backed technology available to institutional clients later this year, allowing firms like BlackRock Inc. to access balances, research and trading tools. The move, which lets clients bypass salespeople and support staff for routine information, is similar to one Goldman Sachs Group Inc. announced in 2015.JPMorgan’s total technology budget for this year amounts to 9 percent of its projected revenue — double the industry average, according to Morgan Stanley analyst Betsy Graseck. The dollar figure has inched higher as JPMorgan bolsters cyber defenses after a 2014 data breach, which exposed the information of 83 million customers.

We have invested heavily in technology and marketing — and we are seeing strong returns,” JPMorgan said in a presentation Tuesday ahead of its investor day, noting that technology spending in its consumer bank totaled about $1 billion over the past two years.

Attendees inspect JPMorgan Markets software kiosk for Investors Day.
Photographer: Kholood Eid/Bloomberg

One-third of the company’s budget is for new initiatives, a figure Zames wants to take to 40 percent in a few years. He expects savings from automation and retiring old technology will let him plow even more money into new innovations.

Not all of those bets, which include several projects based on a distributed ledger, like blockchain, will pay off, which JPMorgan says is OK. One example executives are fond of mentioning: The firm built an electronic platform to help trade credit-default swaps that sits unused.

‘Can’t Wait’We’re willing to invest to stay ahead of the curve, even if in the final analysis some of that money will go to product or a service that wasn’t needed,Marianne Lake, the lender’s finance chief, told a conference audience in June. That’s “because we can’t wait to know what the outcome, the endgame, really looks like, because the environment is moving so fast.”As for COIN, the program has helped JPMorgan cut down on loan-servicing mistakes, most of which stemmed from human error in interpreting 12,000 new wholesale contracts per year, according to its designers.

JPMorgan is scouring for more ways to deploy the technology, which learns by ingesting data to identify patterns and relationships. The bank plans to use it for other types of complex legal filings like credit-default swaps and custody agreements. Someday, the firm may use it to help interpret regulations and analyze corporate communications.

Another program called X-Connect, which went into use in January, examines e-mails to help employees find colleagues who have the closest relationships with potential prospects and can arrange introductions.

Creating Bots
For simpler tasks, the bank has created bots to perform functions like granting access to software systems and responding to IT requests, such as resetting an employee’s password, Zames said. Bots are expected to handle 1.7 million access requests this year, doing the work of 140 people.

Matt Zames
Photographer: Kholood Eid/Bloomberg

While growing numbers of people in the industry worry such advancements might someday take their jobs, many Wall Street personnel are more focused on benefits. A survey of more than 3,200 financial professionals by recruiting firm Options Group last year found a majority expect new technology will improve their careers, for example by improving workplace performance.

Anything where you have back-office operations and humans kind of moving information from point A to point B that’s not automated is ripe for that,” Deasy said. “People always talk about this stuff as displacement. I talk about it as freeing people to work on higher-value things, which is why it’s such a terrific opportunity for the firm.

To help spur internal disruption, the company keeps tabs on 2,000 technology ventures, using about 100 in pilot programs that will eventually join the firm’s growing ecosystem of partners. For instance, the bank’s machine-learning software was built with Cloudera Inc., a software firm that JPMorgan first encountered in 2009.

We’re starting to see the real fruits of our labor,” Zames said. “This is not pie-in-the-sky stuff.


by Hugh Son
27 de febrero de 2017

Scientists Just Found Evidence That Neurons Can Communicate in a Way We Never Anticipated

By Hugo Angel,

Andrii Vodolazhskyi/

A new brain mechanism hiding in plain sight. Researchers have discovered a brand new mechanism that controls the way nerve cells in our brain communicate with each other to regulate learning and long-term memory.

The fact that a new brain mechanism has been hiding in plain sight is a reminder of how much we have yet to learn about how the human brain works, and what goes wrong in neurodegenerative disorders such as Alzheimer’s and epilepsy.

These discoveries represent a significant advance and will have far-reaching implications for the understanding of 

  • memory, 
  • cognition, 
  • developmental plasticity, and 
  • neuronal network formation and stabilisation,”  

said lead researcher Jeremy Henley from the University of Bristol in the UK.

We believe that this is a groundbreaking study that opens new lines of inquiry which will increase understanding of the molecular details of synaptic function in health and disease.

The human brain contains around 100 billion nerve cells, and each of those makes about 10,000 connections – known as synapses – with other cells.

That’s a whole lot of connections, and each of them is strengthened or weakened depending on different brain mechanisms that scientists have spent decades trying to understand.

Until now, one of the best known mechanisms to increase the strength of information flow across synapses was known as LTP, or long-term potentiation.

LTP intensifies the connection between cells to make information transfer more efficient, and it plays a role in a wide range of neurodegenerative conditions –  

  • too much LTP, and you risk disorders such as epilepsy,  
  • too little, and it could cause dementia or Alzheimer’s disease.
As far as researchers were aware, LTP is usually controlled by the activation of special proteins called NMDA receptors.

But now the UK team has discovered a brand new type of LTP that’s regulated in an entirely different way.

After investigating the formation of synapses in the lab, the team showed that this new LTP mechanism is controlled by molecules known as kainate receptors, instead of NMDA receptors.

These data reveal a new and, to our knowledge, previously unsuspected role for postsynaptic kainate receptors in the induction of functional and structural plasticity in the hippocampus,the researchers write in Nature Neuroscience.

This means we’ve now uncovered a previously unexplored mechanism that could control learning and memory.

Untangling the interactions between the signal receptors in the brain not only tells us more about the inner workings of a healthy brain, but also provides a practical insight into what happens when we form new memories,said one of the researchers, Milos Petrovic from the University of Central Lancashire.

If we can preserve these signals it may help protect against brain diseases.

Not only does this open up a new research pathway that could lead to a better understanding of how our brains work, but if researchers can find a way to target these new pathways, it could lead to more effective treatments for a range of neurodegenerative disorders.

It’s still early days, and the discovery will now need to be verified by independent researchers, but it’s a promising new field of research.

This is certainly an extremely exciting discovery and something that could potentially impact the global population,said Petrovic.

The research has been published in Nature Neuroscience.


20 FEB 2017

End-to-End Deep Learning for Self-Driving Cars

By Hugo Angel,

In a new automotive application, we have used convolutional neural networks (CNNs) to map the raw pixels from a front-facing camera to the steering commands for a self-driving car. This powerful end-to-end approach means that with minimum training data from humans, the system learns to steer, with or without lane markings, on both local roads and highways. The system can also operate in areas with unclear visual guidance such as parking lots or unpaved roads.
Figure 1: NVIDIA’s self-driving car in action.
We designed the end-to-end learning system using an NVIDIA DevBox running Torch 7 for training. An NVIDIA DRIVETM PX self-driving car computer, also with Torch 7, was used to determine where to drive—while operating at 30 frames per second (FPS). The system is trained to automatically learn the internal representations of necessary processing steps, such as detecting useful road features, with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. In contrast to methods using explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously.
We believe that end-to-end learning leads to better performance and smaller systems. Better performance results because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e. g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn’t automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps.
This blog post is based on the NVIDIA paper End to End Learning for Self-Driving Cars. Please see the original paper for full details.
Convolutional Neural Networks to Process Visual Data
CNNs[1] have revolutionized the computational pattern recognition process[2]. Prior to the widespread adoption of CNNs, most pattern recognition tasks were performed using an initial stage of hand-crafted feature extraction followed by a classifier. The important breakthrough of CNNs is that features are now learned automatically from training examples. The CNN approach is especially powerful when applied to image recognition tasks because the convolution operation captures the 2D nature of images. By using the convolution kernels to scan an entire image, relatively few parameters need to be learned compared to the total number of operations.

While CNNs with learned features have been used commercially for over twenty years [3], their adoption has exploded in recent years because of two important developments.

  • First, large, labeled data sets such as the ImageNet Large Scale Visual Recognition Challenge (ILSVRC)[4] are now widely available for training and validation.
  • Second, CNN learning algorithms are now implemented on massively parallel graphics processing units (GPUs), tremendously accelerating learning and inference ability.
The CNNs that we describe here go beyond basic pattern recognition. We developed a system that learns the entire processing pipeline needed to steer an automobile. The groundwork for this project was actually done over 10 years ago in a Defense Advanced Research Projects Agency (DARPA) seedling project known as DARPA Autonomous Vehicle (DAVE)[5], in which a sub-scale radio control (RC) car drove through a junk-filled alley way. DAVE was trained on hours of human driving in similar, but not identical, environments. The training data included video from two cameras and the steering commands sent by a human operator.
In many ways, DAVE was inspired by the pioneering work of Pomerleau[6], who in 1989 built the Autonomous Land Vehicle in a Neural Network (ALVINN) system. ALVINN is a precursor to DAVE, and it provided the initial proof of concept that an end-to-end trained neural network might one day be capable of steering a car on public roads. DAVE demonstrated the potential of end-to-end learning, and indeed was used to justify starting the DARPA Learning Applied to Ground Robots (LAGR) program[7], but DAVE’s performance was not sufficiently reliable to provide a full alternative to the more modular approaches to off-road driving. (DAVE’s mean distance between crashes was about 20 meters in complex environments.)

About a year ago we started a new effort to improve on the original DAVE, and create a robust system for driving on public roads. The primary motivation for this work is to avoid the need to recognize specific human-designated features, such as lane markings, guard rails, or other cars, and to avoid having to create a collection of “if, then, else” rules, based on observation of these features. We are excited to share the preliminary results of this new effort, which is aptly named: DAVE–2.

The DAVE-2 System
Figure 2: High-level view of the data collection system.

Figure 2 shows a simplified block diagram of the collection system for training data of DAVE-2. Three cameras are mounted behind the windshield of the data-acquisition car, and timestamped video from the cameras is captured simultaneously with the steering angle applied by the human driver. The steering command is obtained by tapping into the vehicle’s Controller Area Network (CAN) bus. In order to make our system independent of the car geometry, we represent the steering command as 1/r, where r is the turning radius in meters. We use 1/r instead of r to prevent a singularity when driving straight (the turning radius for driving straight is infinity). 1/r smoothly transitions through zero from left turns (negative values) to right turns (positive values).

Training data contains single images sampled from the video, paired with the corresponding steering command (1/r). Training with data from only the human driver is not sufficient; the network must also learn how to recover from any mistakes, or the car will slowly drift off the road. The training data is therefore augmented with additional images that show the car in different shifts from the center of the lane and rotations from the direction of the road.
The images for two specific off-center shifts can be obtained from the left and the right cameras. Additional shifts between the cameras and all rotations are simulated through viewpoint transformation of the image from the nearest camera. Precise viewpoint transformation requires 3D scene knowledge which we don’t have, so we approximate the transformation by assuming all points below the horizon are on flat ground, and all points above the horizon are infinitely far away. This works fine for flat terrain, but for a more complete rendering it introduces distortions for objects that stick above the ground, such as cars, poles, trees, and buildings. Fortunately these distortions don’t pose a significant problem for network training. The steering label for the transformed images is quickly adjusted to one that correctly steers the vehicle back to the desired location and orientation in two seconds.
Figure 3: Training the neural network.

Figure 3 shows a block diagram of our training system. Images are fed into a CNN that then computes a proposed steering command. The proposed command is compared to the desired command for that image, and the weights of the CNN are adjusted to bring the CNN output closer to the desired output. The weight adjustment is accomplished using back propagation as implemented in the Torch 7 machine learning package.

Once trained, the network is able to generate steering commands from the video images of a single center camera. Figure 4 shows this configuration.

Figure 4: The trained network is used to generate steering commands from a single front-facing center camera.

Data Collection

Training data was collected by driving on a wide variety of roads and in a diverse set of lighting and weather conditions. We gathered surface street data in central New Jersey and highway data from Illinois, Michigan, Pennsylvania, and New York. Other road types include two-lane roads (with and without lane markings), residential roads with parked cars, tunnels, and unpaved roads. Data was collected in clear, cloudy, foggy, snowy, and rainy weather, both day and night. In some instances, the sun was low in the sky, resulting in glare reflecting from the road surface and scattering from the windshield.
The data was acquired using either our drive-by-wire test vehicle, which is a 2016 Lincoln MKZ, or using a 2013 Ford Focus with cameras placed in similar positions to those in the Lincoln. Our system has no dependencies on any particular vehicle make or model. Drivers were encouraged to maintain full attentiveness, but otherwise drive as they usually do. As of March 28, 2016, about 72 hours of driving data was collected.
Network Architecture
Figure 5: CNN architecture. The network has about 27 million connections and 250 thousand parameters.

We train the weights of our network to minimize the mean-squared error between the steering command output by the network, and either the command of the human driver or the adjusted steering command for off-center and rotated images (see “Augmentation”, later). Figure 5 shows the network architecture, which consists of 9 layers, including a normalization layer, 5 convolutional layers, and 3 fully connected layers. The input image is split into YUV planes and passed to the network.

The first layer of the network performs image normalization. The normalizer is hard-coded and is not adjusted in the learning process. Performing normalization in the network allows the normalization scheme to be altered with the network architecture, and to be accelerated via GPU processing.
The convolutional layers are designed to perform feature extraction, and are chosen empirically through a series of experiments that vary layer configurations. We then use strided convolutions in the first three convolutional layers with a 2×2 stride and a 5×5 kernel, and a non-strided convolution with a 3×3 kernel size in the final two convolutional layers.
We follow the five convolutional layers with three fully connected layers, leading to a final output control value which is the inverse-turning-radius. The fully connected layers are designed to function as a controller for steering, but we noted that by training the system end-to-end, it is not possible to make a clean break between which parts of the network function primarily as feature extractor, and which serve as controller.
Training Details
The first step to training a neural network is selecting the frames to use. Our collected data is labeled with road type, weather condition, and the driver’s activity (staying in a lane, switching lanes, turning, and so forth). To train a CNN to do lane following, we simply select data where the driver is staying in a lane, and discard the rest. We then sample that video at 10 FPS because a higher sampling rate would include images that are highly similar, and thus not provide much additional useful information. To remove a bias towards driving straight the training data includes a higher proportion of frames that represent road curves.
After selecting the final set of frames, we augment the data by adding artificial shifts and rotations to teach the network how to recover from a poor position or orientation. The magnitude of these perturbations is chosen randomly from a normal distribution. The distribution has zero mean, and the standard deviation is twice the standard deviation that we measured with human drivers. Artificially augmenting the data does add undesirable artifacts as the magnitude increases (as mentioned previously).
Before road-testing a trained CNN, we first evaluate the network’s performance in simulation. Figure 6 shows a simplified block diagram of the simulation system, and Figure 7 shows a screenshot of the simulator in interactive mode.
Figure 6: Block-diagram of the drive simulator.

The simulator takes prerecorded videos from a forward-facing on-board camera connected to a human-driven data-collection vehicle, and generates images that approximate what would appear if the CNN were instead steering the vehicle. These test videos are time-synchronized with the recorded steering commands generated by the human driver.

Since human drivers don’t drive in the center of the lane all the time, we must manually calibrate the lane’s center as it is associated with each frame in the video used by the simulator. We call this position the “ground truth”.
The simulator transforms the original images to account for departures from the ground truth. Note that this transformation also includes any discrepancy between the human driven path and the ground truth. The transformation is accomplished by the same methods as described previously.

The simulator accesses the recorded test video along with the synchronized steering commands that occurred when the video was captured. The simulator sends the first frame of the chosen test video, adjusted for any departures from the ground truth, to the input of the trained CNN, which then returns a steering command for that frame. The CNN steering commands as well as the recorded human-driver commands are fed into the dynamic model [7] of the vehicle to update the position and orientation of the simulated vehicle.

Figure 7: Screenshot of the simulator in interactive mode. See text for explanation of the performance metrics. The green area on the left is unknown because of the viewpoint transformation. The highlighted wide rectangle below the horizon is the area which is sent to the CNN.

The simulator then modifies the next frame in the test video so that the image appears as if the vehicle were at the position that resulted by following steering commands from the CNN. This new image is then fed to the CNN and the process repeats.

The simulator records the off-center distance (distance from the car to the lane center), the yaw, and the distance traveled by the virtual car. When the off-center distance exceeds one meter, a virtual human intervention is triggered, and the virtual vehicle position and orientation is reset to match the ground truth of the corresponding frame of the original test video.
We evaluate our networks in two steps: first in simulation, and then in on-road tests.
In simulation we have the networks provide steering commands in our simulator to an ensemble of prerecorded test routes that correspond to about a total of three hours and 100 miles of driving in Monmouth County, NJ. The test data was taken in diverse lighting and weather conditions and includes highways, local roads, and residential streets.
We estimate what percentage of the time the network could drive the car (autonomy) by counting the simulated human interventions that occur when the simulated vehicle departs from the center line by more than one meter. We assume that in real life an actual intervention would require a total of six seconds: this is the time required for a human to retake control of the vehicle, re-center it, and then restart the self-steering mode. We calculate the percentage autonomy by counting the number of interventions, multiplying by 6 seconds, dividing by the elapsed time of the simulated test, and then subtracting the result from 1:
Thus, if we had 10 interventions in 600 seconds, we would have an autonomy value of
After a trained network has demonstrated good performance in the simulator, the network is loaded on the DRIVE PX in our test car and taken out for a road test. For these tests we measure performance as the fraction of time during which the car performs autonomous steering. This time excludes lane changes and turns from one road to another. For a typical drive in Monmouth County NJ from our office in Holmdel to Atlantic Highlands, we are autonomous approximately 98% of the time. We also drove 10 miles on the Garden State Parkway (a multi-lane divided highway with on and off ramps) with zero intercepts.
Here is a video of our test car driving in diverse conditions.

Visualization of Internal CNN State
Figure 8: How the CNN “sees” an unpaved road. Top: subset of the camera image sent to the CNN. Bottom left: Activation of the first layer feature maps. Bottom right: Activation of the second layer feature maps. This demonstrates that the CNN learned to detect useful road features on its own, i. e., with only the human steering angle as training signal. We never explicitly trained it to detect the outlines of roads.
Figures 8 and 9 show the activations of the first two feature map layers for two different example inputs, an unpaved road and a forest. In case of the unpaved road, the feature map activations clearly show the outline of the road while in case of the forest the feature maps contain mostly noise, i. e., the CNN finds no useful information in this image.
This demonstrates that the CNN learned to detect useful road features on its own, i. e., with only the human steering angle as training signal. We never explicitly trained it to detect the outlines of roads, for example.
Figure 9: Example image with no road. The activations of the first two feature maps appear to contain mostly noise, i. e., the CNN doesn’t recognize any useful features in this image.
We have empirically demonstrated that CNNs are able to learn the entire task of lane and road following without manual decomposition into road


  • Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541–551, Winter 1989.
  • Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks.
  • In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, Inc., 2012. URL:
  • L. D. Jackel, D. Sharman, Stenard C. E., Strom B. I., , and D Zuckert. Optical character recognition for self-service banking. AT&T Technical Journal, 74(1):16–24, 1995.
  • Large scale visual recognition challenge (ILSVRC). URL:
  • Net-Scale Technologies, Inc. Autonomous off-road vehicle control using end-to-end learning, July 2004. Final technical report. URL:
  • Dean A. Pomerleau. ALVINN, an autonomous land vehicle in a neural network. Technical report, Carnegie Mellon University, 1989.
  • Danwei Wang and Feng Qi. Trajectory planning for a four-wheel-steering vehicle. In Proceedings of the 2001 IEEE International Conference on Robotics & Automation, May 21–26 2001. URL:
    rlane marking detection, semantic abstraction, path planning, and control. A small amount of training data from less than a hundred hours of driving was sufficient to train the car to operate in diverse conditions, on highways, local and residential roads in sunny, cloudy, and rainy conditions.
  • The CNN is able to learn meaningful road features from a very sparse training signal (steering alone).
  • The system learns for example to detect the outline of a road without the need of explicit labels during training.
  • More work is needed to improve the robustness of the network, to find methods to verify the robustness, and to improve visualization of the network-internal processing steps.
For full details please see the paper that this blog post is based on, and please contact us if you would like to learn more about NVIDIA’s autonomous vehicle platform!
  1. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backprop- agation applied to handwritten zip code recognition. Neural Computation, 1(4):541–551, Winter 1989. URL:
  2. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, Inc., 2012. URL: 4824-imagenet-classification-with-deep-convolutional-neural-networks. pdf.
  3. L. D. Jackel, D. Sharman, Stenard C. E., Strom B. I., , and D Zuckert. Optical character recognition for self-service banking. AT&T Technical Journal, 74(1):16–24, 1995.
  4. Large scale visual recognition challenge (ILSVRC). URL: challenges/LSVRC/.
  5. Net-Scale Technologies, Inc. Autonomous off-road vehicle control using end-to-end learning, July 2004. Final technical report. URL:
  6. Dean A. Pomerleau. ALVINN, an autonomous land vehicle in a neural network. Technical report, Carnegie Mellon University, 1989. URL: cgi?article=2874&context=compsci.
  7. Danwei Wang and Feng Qi. Trajectory planning for a four-wheel-steering vehicle. In Proceedings of the 2001 IEEE International Conference on Robotics & Automation, May 21–26 2001. URL: http: //

AI Software Juggles Probabilities to Learn from Less Data

By Hugo Angel,

Gamalon has developed a technique that lets machines learn to recognize concepts in images or text much more efficiently.
An app developed by Gamalon recognizes objects after seeing a few examples. A learning program recognizes simpler concepts such as lines and rectangles.
Machine learning is becoming extremely powerful, but it requires extreme amounts of data.
You can, for instance, train a deep-learning algorithm to recognize a cat with a cat-fancier’s level of expertise, but you’ll need to feed it tens or even hundreds of thousands of images of felines, capturing a huge amount of variation in size, shape, texture, lighting, and orientation. It would be lot more efficient if, a bit like a person, an algorithm could develop an idea about what makes a cat a cat from fewer examples.
A Boston-based startup called Gamalon has developed technology that lets computers do this in some situations, and it is releasing two products Tuesday based on the approach.
If the underlying technique can be applied to many other tasks, then it could have a big impact. The ability to learn from less data could let robots explore and understand new environments very quickly, or allow computers to learn about your preferences without sharing your data.
Gamalon uses a technique that it calls Bayesian program synthesis to build algorithms capable of learning from fewer examples. Bayesian probability, named after the 18th century mathematician Thomas Bayes, provides a mathematical framework for refining predictions about the world based on experience. Gamalon’s system uses probabilistic programming—or code that deals in probabilities rather than specific variables—to build a predictive model that explains a particular data set. From just a few examples, a probabilistic program can determine, for instance, that it’s highly probable that cats have ears, whiskers, and tails. As further examples are provided, the code behind the model is rewritten, and the probabilities tweaked. This provides an efficient way to learn the salient knowledge from the data.
Probabilistic programming techniques have been around for a while. In 2015, for example, a team from MIT and NYU used probabilistic methods to have computers learn to recognize written characters and objects after seeing just one example (see “This AI Algorithm Learns Simple Tasks as Fast as We Do”). But the approach has mostly been an academic curiosity.
There are difficult computational challenges to overcome, because the program has to consider many different possible explanations, says Brenden Lake, a research fellow at NYU who led the 2015 work.
Still, in theory, Lake says, the approach has significant potential because it can automate aspects of developing a machine-learning model.Probabilistic programming will make machine learning much easier for researchers and practitioners,” Lake says. “It has the potential to take care of the difficult [programming] parts automatically.
There are certainly significant incentives to develop easier-to-use and less data-hungry machine-learning approaches. Machine learning currently involves acquiring a large raw data set, and often then labeling it manually. The learning is then done inside large data centers, using many computer processors churning away in parallel for hours or days. “There are only a few really large companies that can really afford to do this,” says Ben Vigoda, cofounder and CEO of Gamalon.
When Machines Have Ideas | Ben Vigoda | TEDxBoston
Our CEO, Ben Vigoda, gave a talk at TEDx Boston 2016 called “When Machines Have Ideas” that describes why building “stories” (i.e. Bayesian generative models) into machine intelligence systems can be very powerful.
In theory, Gamalon’s approach could make it a lot easier for someone to build and refine a machine-learning model, too. Perfecting a deep-learning algorithm requires a great deal of mathematical and machine-learning expertise. “There’s a black art to setting these systems up,” Vigoda says. With Gamalon’s approach, a programmer could train a model by feeding in significant examples.
Vigoda showed MIT Technology Review a demo with a drawing app that uses the technique. It is similar to the one released last year by Google, which uses deep learning to recognize the object a person is trying to sketch (see “Want to Understand AI? Try Sketching a Duck for a Neural Network”). But whereas Google’s app needs to see a sketch that matches the ones it has seen previously, Gamalon’s version uses a probabilistic program to recognize the key features of an object. For instance, one program understands that a triangle sitting atop a square is most likely a house. This means even if your sketch is very different from what it has seen before, providing it has those features, it will guess correctly.
The technique could have significant near-term commercial applications, too. The company’s first products use Bayesian program synthesis to recognize concepts in text.
  • One product, called Gamalon Structure, can extract concepts from raw text more efficiently than is normally possible. For example, it can take a manufacturer’s description of a television and determine what product is being described, the brand, the product name, the resolution, the size, and other features.
  • Another product, Gamalon Match, is used to categorize the products and price in a store’s inventory. In each case, even when different acronyms or abbreviations are used for a product or feature, the system can quickly be trained to recognize them.
Vigoda believes the ability to learn will have other practical benefits.

  • A computer could learn about a user’s interests without requiring an impractical amount of data or hours of training.
  • Personal data might not need to be shared with large companies, either, if machine learning can be done efficiently on a user’s smartphone or laptop.
  • And a robot or a self-driving car could learn about a new obstacle without needing to see hundreds of thousands of examples.
February 14, 2017

An international team of scientists has come up with a blueprint for a large-scale quantum computer

By Hugo Angel,

‘It is the Holy Grail of science … we will be able to do certain things we could never even dream of before’
Courtesy Professor Winfried Hensinger
Quantum computing breakthrough could help ‘change life completely‘, say scientists
Scientists claim to have produced the first-ever blueprint for a large-scale quantum computer in a development that could bring about a technological revolution on a par with the invention of computing itself.
Until now quantum computers have had just a fraction of the processing power they are theoretically capable of producing.
But an international team of researchers believe they have finally overcome the main technical problems that have prevented the construction of more powerful machines.
They are currently building a prototype and a full-scale quantum computer – many millions of times faster than the best currently available – could be built in about a decade.
This is a modal window.
Scientists invent invisible underwater robots based on glass eels
Such devices work by utilising the almost magical properties found in the world of the very small, where an atom can apparently exist in two different places at the same time.
Professor Winfried Hensinger, head of the Ion Quantum Technology Group at Sussex University, who has been leading this research, told The Independent: “It is the Holy Grail of science, really, to build a quantum computer.
And we are now publishing the actual nuts-and-bolts construction plan for a large-scale quantum computer.
It is thought the astonishing processing power unleashed by quantum mechanics will lead to new, life-saving medicines, help solve the most intractable scientific problems, and probe the mysteries of the universe.
Life will change completely. We will be able to do certain things we could never even dream of before,” Professor Hensinger said.
You can imagine that suddenly the sky is the limit.
This is really, really exciting … it’s probably one of the most exciting times to be in this field.
He said small quantum computers had been built in the past but to test the theories.
This is not an academic study any more, it really is all the engineering required to build such a device,” he said.
Nobody has really gone ahead and drafted a full engineering plan of how you build one.
Many people questioned, because this is so hard to make this happen, that it can even be built.
We show that not only can it be built, but we provide a whole detailed plan on how to make it happen.
The problem is that existing quantum computers require lasers focused precisely on individual atoms. The larger the computer, the more lasers are required and the greater the chance of something going wrong.
But Professor Hensinger and colleagues used a different technique to monitor the atoms involving a microwave field and electricity in an ‘ion-trap’ device.

What we have is a solution that we can scale to arbitrary [computing] power,” he said.

Fig. 2. Gradient wires placed underneath each gate zone and embedded silicon photodetector.
(A) Illustration showing an isometric view of the two main gradient wires placed underneath each gate zone. Short wires are placed locally underneath each gate zone to form coils, which compensate for slowly varying magnetic fields and allow for individual addressing. The wire configuration in each zone can be seen in more detail in the inset.
(B) Silicon photodetector (marked green) embedded in the silicon substrate, transparent center segmented electrodes, and the possible detection angle are shown. VIA structures are used to prevent optical cross-talk from neighboring readout zones.
Source: Science Journals — AAAS. Blueprint for a microwave trapped ion quantum computer. Lekitsch et al. Sci. Adv. 2017;3: e1601540 1 February 2017
Fig. 4. Scalable module illustration. One module consisting of 36 × 36 junctions placed on the supporting steel frame structure: Nine wafers containing the required DACs and control electronics are placed between the wafer holding 36 × 36 junctions and the microchannel cooler (red layer) providing the cooling. X-Y-Z piezo actuators are placed in the four corners on top of the steel frame, allowing for accurate alignment of the module. Flexible electric wires supply voltages, currents, and control signals to the DACs and control electronics, such as field-programmable gate arrays (FPGAs). Coolant is supplied to the microchannel cooler layer via two flexible steel tubes placed in the center of the modules.
Source: Science Journals — AAAS. Blueprint for a microwave trapped ion quantum computer. Lekitsch et al. Sci. Adv. 2017;3: e1601540 1 February 2017
Fig. 5. Illustration of vacuum chambers. Schematic of octagonal UHV chambers connected together; each chamber is 4.5 × 4.5 m2 large and can hold >2.2 million individual X-junctions placed on steel frames.
Source: Science Journals — AAAS. Blueprint for a microwave trapped ion quantum computer. Lekitsch et al. Sci. Adv. 2017;3: e1601540 1 February 2017

We are already building it now. Within two years we think we will have completed a prototype which incorporates all the technology we state in this blueprint.

At the same time we are now looking for industry partner so we can really build a large-scale device that fills a building basically.
It’s extraordinarily expensive so we need industry partners … this will be in the 10s of millions, up to £100m.
Commenting on the research, described in a paper in the journal Science Advances, other academics praised the quality of the work but expressed caution about how quickly it could be developed.
Dr Toby Cubitt, a Royal Society research fellow in quantum information theory at University College London, said: “Many different technologies are competing to build the first large-scale quantum computer. Ion traps were one of the earliest realistic proposals. 
This work is an important step towards scaling up ion-trap quantum computing.
Though there’s still a long way to go before you’ll be making spreadsheets on your quantum computer.
And Professor Alan Woodward, of Surrey University, hailed the “tremendous step in the right direction”.
It is great work,” he said. “They have made some significant strides forward.

But he added it was “too soon to say” whether it would lead to the hoped-for technological revolution.

ORIGINAL: The Independent
Ian Johnston Science Correspondent
Thursday 2 February 2017 – Bringing Deep Learning AI to the Devices at the Edge of the Network

By Hugo Angel,


Photo  – The Team


Today we announced our funding of We are excited to be working with Ali Farhadi, Mohammad Rastegari and their team on this new company. We are also looking forward to working with Paul Allen’s team at the Allen Institute for AI and in particular our good friend and CEO of AI2, Dr. Oren Etzioni who is joining the board of Machine Learning and AI have been a key investment theme for us for the past several years and bringing deep learning capabilities such as image and speech recognition to small devices is a huge challenge.

Mohammad and Ali and their team have developed a platform that enables low resource devices to perform tasks that usually require large farms of GPUs in cloud environments. This, we believe, has the opportunity to change how we think about certain types of deep learning use cases as they get extended from the core to the edge. Image and voice recognition are great examples. These are broad areas of use cases out in the world – usually with a mobile device, but right now they require the device to be connected to the internet so those large farms of GPUs can process all the information your device is capturing/sending and having the core transmit back the answer. If you could do that on your phone (while preserving battery life) it opens up a new world of options.

It is just these kinds of inventions that put the greater Seattle area at the center of the revolution in machine learning and AI that is upon us. came out of the outstanding work the team was doing at the Allen Institute for Artificial Intelligence (AI2.) and Ali is a professor at the University of Washington. Between Microsoft, Amazon, the University of Washington and research institutes such as AI2, our region is leading the way as new types of intelligent applications takes shape. Madrona is energized to play our role as company builder and support for these amazing inventors and founders.

By Matt McIlwain

AI acceleration startup collects $2.6M in funding

I was excited by the promise of and its technique that drastically reduces the computing power necessary to perform complex operations like computer vision. Seems I wasn’t the only one: the company, just officially spun off from the Allen Institute for AI (AI2), has attracted $2.6 million in seed funding from its parent company and Madrona Venture Group.

The specifics of the product and process you can learn about in detail in my previous post, but the gist is this: machine learning models for things like object and speech recognition are notoriously computation-heavy, making them difficult to implement on smaller, less powerful devices.’s researchers use a bit of mathematical trickery to reduce that computing load by an order of magnitude or two — something it’s easy to see the benefit of.

Related Articles

McIlwain will join AI2 CEO Oren Etzioni on the board of; Ali Farhadi, who led the original project, will be the company’s CEO, and Mohammad Rastegari is CTO.
The new company aims to facilitate commercial applications of its technology (it isn’t quite plug and play yet), but the research that led up to it is, like other AI2 work, open source.


AI2 Repository:

ORIGINAL: TechCrunch

Why Apple Joined Rivals Amazon, Google, Microsoft In AI Partnership

By Hugo Angel,

Apple CEO Tim Cook (Photo credit: David Paul Morris/Bloomberg)

Apple is pushing past its famous secrecy for the sake of artificial intelligence.

In December, the Cupertino tech giant quietly published its first AI research paper. Now, it’s joining the Partnership on AI to Benefit People and Society, an industry nonprofit group founded by some of its biggest rivals, including Microsoft, Google and Amazon.

On Friday, the partnership announced that Apple’s head of advanced development for SiriTom Gruber, is joining its board. Gruber has been at Apple since 2010 when the iPhone maker bought Siri, the company he cofounded and where he served as CTO.

“We’re glad to see the industry engaging on some of the larger opportunities and concerns created with the advance of machine learning and AI,” wrote Gruber in a statement on the nonprofit’s website. “We believe it’s beneficial to Apple, our customers, and the industry to play an active role in its development and look forward to collaborating with the group to help drive discussion on how to advance AI while protecting the privacy and security of consumers.”

Other members of the board include

  • Greg Corrado from Google’s DeepMind,
  • Ralf Herbrich from Amazon,
  • Eric Horvitz from Microsoft,
  • Yann Lecun from Facebook, and
  • Francesca Rossi from IBM.

Outside of large companies, the group announced it’s also adding members from the

  • American Civil Liberties Union,
  • OpenAI,
  • MacArthur Foundation,
  • Peterson Institute of International Economics,
  • Arizona State University and the
  • University of California, Berkeley.

The group was formally announced in September.

Board member Horvitz, who is director of Microsoft Research, said the members of the group started meeting with each other at various AI conferences. They were already close colleagues in the field and they thought they could start working together to discuss emerging challenges and opportunities in AI.

 “We believed there were a lot of things companies could do together on issues and challenges in the realm of AI and society,” Horvitz said in an interview. “We don’t see these as areas for competition but for rich cooperation.

The organization will work together to develop best practices and educate the public around AI. Horvitz said the group tackle, for example, critical areas like health care and transportation. The group will look at the potential for biases in AI — after some experiments have shown that the way researchers train the AI algorithms can lead to biases in gender and race. The nonprofit will also try to develop standards around human-machine collaboration, for example, to deal with questions like when should a self-driving car hand off control to the driver.

“I think there’s a realization that AI will touch society quite deeply in the coming years in powerful and nuanced ways,” Horitz said. “We think it’s really important to involve the public as well as experts. Some of these directions has no simple answer. It can’t come from a company. We need to have multiple constituents checking in.”

The AI community has been critical of Apple’s secrecy for several years secrecy has hurt the company’s recruiting efforts for AI talent. The company has been falling behind in some of the major advancements in AI, especially as intelligent voice assistants from Amazon and Google have started taking off with consumers.

Horvitz said the group had been in discussions with Apple since before its launch in September. But Apple wasn’t ready to formally join the group until now. “My own sense is that Apple was in the middle of their iOS 10 and iPhone 7 launches” and wasn’t ready to announce, he said. “We’ve always treated Apple as a founding member of the group.

I think Apple had a realization that to do the best AI research and to have access to the top minds in the field is the expectation of engaging openly with academic research communities,” Horitz said. “Other companies like Microsoft have discovered this over the years. We can be quite competitive and be open to sharing ideas when it comes to the core foundational science.

“It’s my hope that this partnership with Apple shows that the company has a rich engagement with people, society and stakeholders,” he said.

Jan 27, 2017

A biomimetic robotic platform to study flight specializations of bats

By Hugo Angel,

Some Batty Ideas Take Flight. Bat Bot (shown) is able to imitate several flight maneuvers of bats, such as bank turns and diving flight. Such agile flight is made possible by highly malleable bones and skin in bat wings. Ramezani et al. identified and implemented the most important bat wing joints by means of a series of mechanical constraints. They then designed feedback control for their flapping wing platform and covered the structure in a flexible silicon membrane. This biomimetic robot may also shed light on the role of bat legs to modulate flight pitch. [CREDIT: ALIREZA RAMEZANI
Forget drones. Think bat-bots. Engineers have created a new autonomous flying machine that looks and maneuvers just like a bat. Weighing only 93 grams, the robot’s agility comes from its complex wings made of lightweight silicone-based membranes stretched over carbon-fiber bones, the researchers report today in Science Robotics. In addition to nine joints in each wing, it sports adjustable legs, which help it steer by deforming the membrane of its tail. Complex algorithms coordinate these components, letting the bot make batlike moves including banking turns and dives. But don’t bring out the bat-signal just yet. Remaining challenges include improving battery life and developing stronger electronic components so the device can survive minor crashes. Ultimately, though, the engineers hope this highly maneuverable alternative to quadrotor drones could serve as a helpful new sidekick—lending a wing in anything from dodging through beams as a construction surveyor to aiding in disaster relief by scouting dangerous sites. The next lesson researchers hope to teach the bat-bot? Perching upside-down.
Posted in:
DOI: 10.1126/science.aal0685

A biomimetic robotic platform to study flight specializations of bats

Alireza Ramezani1, Soon-Jo Chung2,* and
Seth Hutchinson1
+ Author Affiliations
*Corresponding author. Email: [email protected]
Science Robotics 01 Feb 2017:
Vol. 2, Issue 3,
DOI: 10.1126/scirobotics.aal2505
Bats have long captured the imaginations of scientists and engineers with their unrivaled agility and maneuvering characteristics, achieved by functionally versatile dynamic wing conformations as well as more than 40 active and passive joints on the wings. Wing flexibility and complex wing kinematics not only bring a unique perspective to research in biology and aerial robotics but also pose substantial technological challenges for robot modeling, design, and control. We have created a fully self-contained, autonomous flying robot that weighs 93 grams, called Bat Bot (B2), to mimic such morphological properties of bat wings. Instead of using a large number of distributed control actuators, we implement highly stretchable silicone-based membrane wings that are controlled at a reduced number of dominant wing joints to best match the morphological characteristics of bat flight. First, the dominant degrees of freedom (DOFs) in the bat flight mechanism are identified and incorporated in B2’s design by means of a series of mechanical constraints. These biologically meaningful DOFs include asynchronous and mediolateral movements of the armwings and dorsoventral movements of the legs. Second, the continuous surface and elastic properties of bat skin under wing morphing are realized by an ultrathin (56 micrometers) membranous skin that covers the skeleton of the morphing wings. We have successfully achieved autonomous flight of B2 using a series of virtual constraints to control the articulated, morphing wings.
Biologically inspired flying robots showcase impressive flight characteristics [e.g., robot fly (1) and bird-like robots (2, 3)]. In recent years, biomimicry of bat flight has led to the development of robots that are capable of mimicking bat morphing characteristics on either a stationary (4) or a rotational pendular platform (5). However, these attempts are limited because of the inherent complexities of bat wing morphologies and lightweight form factors.
Arguably, bats have the most sophisticated powered flight mechanism among animals, as evidenced by the morphing properties of their wings. Their flight mechanism has several types of joints (e.g., ball-and-socket and revolute joints), which interlock the bones and muscles to one another and create a metamorphic musculoskeletal system that has more than 40 degrees of freedom (DOFs), both passive and active (see Fig. 1) (6). For insects, the wing structure is not as sophisticated as bats because it is a single, unjointed structural unit. Like bat wings, bird wings have several joints that can be moved actively and independently.
Fig. 1 Functional groups in bat (photo courtesy of A. D. Rummel and S. Swartz, the Aeromechanics and Evolutionary Morphology Laboratory, Brown University).
Enumerated bat joint angles and functional groups are depicted; using these groups makes it possible to categorize the sophisticated movements of the limbs during flight and to extract dominant DOFs and incorporate them in the flight kinematics of B2. The selected DOFs are coupled by a series of mechanical and virtual constraints.
Robotics research inspired by avian flight has successfully conceptualized bird wings as a rigid structure, which is nearly planar and translates—as a whole or in two to three parts—through space; however, the wing articulation involved in bat wingbeats is very pronounced. In the mechanism of bat flight, one wingbeat cycle consists of two movements: (i) a downstroke phase, which is initiated by both left and right forelimbs expanding backward and sideways while sweeping downward and forward relative to the body, and (ii) an upstroke phase, which brings the forelimbs upward and backward and is followed by the flexion of the elbows and wrists to fold the wings. There are more aspects of flapping flight that uniquely distinguish bats. Bat wings have (i) bones that deform adaptively during each wingbeat cycle, (ii) anisotropic wing membrane skin with adjustable stiffness across the wing, and (iii) a distributed network of skin sensory organs believed to provide continuous information regarding flows over the wing surfaces (7).
The motivation for our research into bat-inspired aerial robots is twofold. First, the study of these robots will provide insight into flapping aerial robotics, and the development of these soft-winged robots will have a practical impact on robotics applications where humans and robots share a common environment. From an engineering perspective, understanding bat flight is a rich and interesting problem. Unlike birds or insects, bats exclusively use structural flexibility to generate the controlled force distribution on each membrane wing. Wing flexibility and complex wing kinematics are crucial to the unrivaled agility of bat flight (8, 9). This aspect of bat flight brings a unique perspective to research in winged aerial robotics, because most previous work on bioinspired flight is focused on insect flight (1015) or hummingbird flight (16), using robots with relatively stiff wings (17, 18).
Bat-inspired aerial robots have a number of practical advantages over current aerial robots, such as quadrotors. In the case of humans and robots co-inhabiting shared spaces, the safety of bat-inspired robots with soft wings is the most important advantage. Although quadrotor platforms can demonstrate agile maneuvers in complex environments (19, 20), quadrotors and other rotorcraft are inherently unsafe for humans; demands of aerodynamic efficiency prohibit the use of rotor blades or propellers made of flexible material, and high noise levels pose a potential hazard for humans. In contrast, the compliant wings of a bat-like flapping robot flapping at lower frequencies (7 to 10 Hz versus 100 to 300 Hz of quadrotors) are inherently safe, because their wings comprise primarily flexible materials and are able to collide with one another, or with obstacles in their environment, with little or no damage.
Versatile wing conformation
The articulated mechanism of bats has speed-dependent morphing properties (21, 22) that respond differently to various flight maneuvers. For instance, consider a half-roll (180° roll) maneuver performed by insectivorous bats (23). Flexing a wing and consequently reducing the wing area would increase wing loading on the flexed wing, thereby reducing the lift force. In addition, pronation (pitch-down) of one wing and supination (pitch-up) of the other wing result in negative and positive angles of attack, respectively, thereby producing negative and positive lift forces on the wings, causing the bat to roll sharply. Bats use this maneuver to hunt insects because at 180° roll, they can use the natural camber on their wings to maximize descending acceleration. Insectivorous bats require a high level of agility because their insect preys are also capable of swooping during pursuit. With such formidable defense strategies used by their airborne prey, these bats require sharp changes in flight direction.
In mimicking bats’ functionally versatile dynamic wing conformations, two extreme paradigms are possible. On the one hand, many active joints can be incorporated in the design. This school of thought can lead to the design and development of robots with many degrees of actuation that simply cannot fly. Apart from performance issues that may appear from overactuating a dynamic system, these approaches are not practical for bat-inspired micro aerial vehicles (MAVs) because there are technical restrictions for sensing and actuating many joints in robots with tight weight (less than 100 g) and dimension restrictions. On the other hand, oversimplifying the morphing wing kinematics to oscillatory flat surfaces, which is similar to conventional ornithopters, underestimates the complexities of the bat flight mechanism. Such simplified ornithopters with simple wing kinematics may not help answer how bats achieve their impressive agile flight.
Body dimensional complexity
A better understanding of key DOFs in bat flight kinematics may help to design a simpler flying robot with substantially fewer joints that is yet capable of mimicking its biological counterparts. A similar paradigm has led to successful replications of the human terrestrial locomotion (walking and running) by using bipedal robots that have point feet (24), suggesting that feet are a redundant element of the human locomotion system. Assigning importance to the kinematic parameters can yield a simpler mechanism with fewer kinematic parameters if those parameters with higher kinematic contribution and significance are chosen. Such kinematic characterization methods have been applied to study various biological mechanisms (6, 9, 2528).
Among these studies, Riskin et al. (6) enhance our understanding of bat aerial locomotion in particular by using the method of principal components analysis (PCA) to project bat joint movements to the subspace of eigenmodes, isolating the various components of the wing conformation. By using only the first eigenmode, 34% of biological bat flight kinematics are reproducible. By superimposing the first and second eigenmodes, more than 57% of bat flight kinematics can be replicated. These findings, which emphasize the existence of synergies (29) in bat flight kinematics to describe the sophisticated movements of the limbs during flight, suggest the possibility of mimicking bat kinematics with only a few DOFs (30).
According to these PCAs, three functional groups, shown in Fig. 1, synthesize the wing morphing: (i) when wings spread, fingers bend; (ii) when wrists pronate, elbows bend; and (iii) the medial part of the wings is morphed in collaboration with the shoulders, hips, and knees (6). These dimensional complexity analyses reveal that the flapping motion of the wings, the mediolateral motion of the forelimbs, the flexion-extension of the fingers, the pronation-supination of the carpi, and the dorsoventral movement of the legs are the major DOFs. In developing our robotic platform Bat Bot (B2) (Fig. 2A), we selected these biologically meaningful DOFs and incorporated them in the design of B2 by means of a series of mechanical constraints.
Fig. 2Bat Bot.
(A) B2 is self-sustained and self-contained; it has an onboard computer and several sensors for performing autonomous navigation in its environment. The computing, sensing, and power electronics, which are accommodated within B2, are custom-made and yield a fully self-sustained system despite weight and size restrictions. The computing unit, or main control board (MCB), hosts a microprocessor. While the navigation-and-control algorithm runs on the MCB in real time, a data acquisition unit acquires sensor data and commands the micro actuators. The sensing electronics, which are circuit boards custom-designed to achieve the smallest size possible, interface with the sensors and the MCB by collecting two kinds of measurements. First, an inertial measurement unit (IMU), which is fixed to the ribcage in such a way that the x axis points forward and the z axis points upward, reads the attitudes of the robot with respect to the inertial frame. Second, five magnetic encoders are located at the elbows, hips, and flapping joint to read the relative angles between the limbs with respect to the body. (B) Dynamic modulus analysis. Samples of membrane were mounted vertically in the dynamic modulus analyzer using tension clamps with ribbed grips to ensure that there was no slipping of the sample. Data were collected using controlled force analysis at a ramp rate of 0.05 N/min over the range 0.001 to 1.000 N. The temperature was held at 24.56°C. The estimated average modulus, ultimate tensile strength (UTS), and elongation are 0.0028 MPa, 0.81 MPa, and 439.27%, respectively. The average modulus and UTS along fiber direction are 11.33 and 17.35 MPa, respectively. (C) The custom-made silicone-based membrane and embedded carbon fibers.
System design
B2’s flight mechanism (shown in Fig. 3, A to C) consists of the left and right wings, each including a forelimb and a hindlimb mechanism. The left and right wings are coupled with a mechanical oscillator. A motor spins a crankshaft mechanism, which moves both wings synchronously dorsoventrally while each wing can move asynchronously mediolaterally. The hindlimbs that synthesize the trailing edge of the wings can move asynchronously and dorsoventrally. If it were not for mechanical couplings and constraints, the morphing mechanism of B2 would have nine DOFs. Because the physical constraints are present, four DOFs are coupled, yielding a five-DOF mechanism.
(A) B2’s flight mechanism and its DOFs. We introduced mechanical couplings in the armwing to synthesize a mechanism with a few DOFs. (B) The armwing retains only one actuated movement, which is a push-pull movement produced by a spindle mechanism hosted in the shoulder. (C) The leg mechanism. (D) B2’s electronics architecture. At the center, the microprocessor from STMicroelectronics communicates with several components, including an IMU from VectorNav Technologies, an SD card reader, five AS5048 Hall effect encoders, and two dual-port dc motor drivers. Two wireless communication devices, an eight-channel micro RC receiver (DSM2) and a Bluetooth device, make it possible to communicate with the host (Panel). The microprocessor has several peripherals, such as universal synchronous/asynchronous receiver/transmitter (USART), serial peripheral interface (SPI), pulse-width modulation (PWM), and secure digital input/output (SDIO). To test and deploy the controller on the platform, we used Hardware-in-the-Loop (HIL) simulation. In this method, a real-time computer is used as a virtual plant (model), and the flight controller, which is embedded on the physical microprocessor, responds to the state variables of the virtual model. In this way, the functionality of the controller is validated and debugged before being deployed on the vehicle.
The forelimbs (see Fig. 3B), which provide membranal mechanical support and morphing leverage, consist of nine links: the humeral (p0-p1), humeral support (p1-p2), radial (p1-p3), radial support (p4-p5), carpal (p3-p4), carpal support (p1-p5), and three digital links. Mobilizing this structure requires embedding rotation in the humerus, pronating rotation in the wrists, and abduction-adduction and flexion-extension in the digits. All of these require the active actuation of the shoulders, wrists, and finger knuckles, respectively.
A few attempts have been made to incorporate similar DOFs in an MAV. Researchers at Brown University have used string-and-pulley–based actuating mechanisms to articulate a robotic membranous wing (4). In their design, the wing is mounted on a support to avoid any installation of actuators on the robotic wing. In this support, a bundle that includes several strings is routed through the wing’s links. It is then connected to several motors incorporated in the support. This form of actuation makes it possible to realize several active joints in the robotic wing. However, such a method is not practical for a flying MAV because it requires heavy actuators to be installed in the ribcage. Unlike the robotic wing from (4), we introduced physical constraints (see Fig. 3, A to C) in B2 to synthesize a flight mechanism with a few actuated joints. These mechanical constraints follow.
Morphing wing flight apparatus
A three-link mechanism, where each link is connected to the next one with a revolute joint while one link is pivoted to a fixed support, is uniquely defined mathematically using three angles or configuration variables. Regulating the position and orientation of the end effector in the three-link mechanism implies direct control of the three revolute joints. Constraining the mechanism with three rigid links results in a one-DOF mechanism requiring only one actuator.
Each of the forelimbs is similar to this three-link mechanism, and their links are hinged to one another using rigid one-DOF revolute joints. The rotational movement of the humeral link around the fixed shoulder joint p0 is affected by linear movements of the point p2relative to the humeral shoulder joint. A linear motion of the humeral support link at the shoulder moves the radial link relative to the humeral link and results in elbow flexion-extension. Although humeral and radial links move with respect to each other, a relative motion of the outer digital link with respect to the radial link is realized as the elbow flexion-extension is projected to the carpal plate through the radial support link (see Fig. 3B).
The ball-and-socket universal joints at two ends of the support radial link facilitate the passive movements of the carpal plate in a pronating direction. In contrast to biological bats, which actively rotate their wrists, B2 has passive carpal rotations with respect to the radius.
Digital links I, II, and III are cantilevered to the carpal plate (p6, p7, and p8); they are flexible slender carbon fiber tubes that can passively flex and extend with respect to the carpal plate, meaning that they introduce passive DOFs in the flight mechanism. In addition to these passive flexion-extension movements, the digital links can passively abduct and adduct with respect to each other. The fingers have no knuckles, and their relative angle with respect to one another is predefined.
As a result, each of B2’s forelimbs has one actuated DOF that transforms the linear motion of its spindle mechanism into three active and biologically meaningful movements: (i) active humeral retraction-protraction (shoulder angle), (ii) active elbow flexion-extension (elbow angle), and (iii) active carpal abduction-adduction (wrist angle). The passive DOFs include carpal pronation, digital abduction-adduction, and flexion-extension.
In the case of the hindlimbs (legs), it is challenging to accurately quantify the aerodynamic consequences of leg absence or presence in bats and determine their influence on the produced aerodynamic lift and drag forces. This is because the movements of hindlimbs affect the membrane locally at the trailing edge of the wings, whereas at distal positions, wings are mostly influenced by forelimbs. However, legs can enhance the agility of flight by providing additional control of the left and right sides of the trailing edge of the membrane wing (31). Adjusting the vertical position of the legs with respect to the body has two major effects: (i) leg-induced wing camber and (ii) increasing the angle of attack locally at the tail. In other words, increasing the leg angle increases lift, drag, and pitching moment (31). In addition, there is another benefit to carefully controlled tail actuation: Drag notably decreases because tails prevent flow detachments and delay the onset of flow separation (32).
Benefiting from these aerodynamic effects, bats have unique mechanistic bases; the anatomical evolutions in their hindlimbs enable these mammals to actively use their hindlimbs during flight (33). In contrast to terrestrial mammals, the ball-and-socket joint that connects the femoral bone to the body is rotated in such a way that knee flexion moves the ankle dorsoventrally. This condition yields pronounced knee flexions ventrally.
From a kinematics standpoint, the sophisticated movements of ankles in bats include dorsoventral and mediolateral movements. Ankles move ventrally during the downstroke, and they start moving dorsally during the upstroke (33). Motivated by the roles of legs in bat flight, we implemented two asynchronously active legs for controlling the trailing edge of the membrane wing in the design of B2. We hinged each leg to the body by one-DOF revolute joints such that the produced dorsoventral movement happens in a plane that is tilted at an angle relative to the parasagittal plane (see Fig. 3C). Contrary to biological bats, B2’s legs have no mediolateral movements; Riskin et al. (6) suggest that such movements are less pronounced in biological bats. To map the linear movements of our actuation system to the dorsoventral movements of the legs, we used a three-bar linkage mechanism (34).
Anisotropic membranous wing
The articulated body of B2 yields a structure that cannot accommodate conventional fabric covering materials, such as unstretchable nylon films. Unstretchable materials resist the forelimb and leg movements. As a result, we covered the skeleton of our robot with a custom-made, ultrathin (56 μm), silicone-based membrane that is designed to match the elastic properties of biological bats’ membranes. In general, bat skin spans the body such that it is anchored to forelimbs, digital bones, and hindlimbs. This yields a morphing mechanism with soft wings, which is driven by the movements of the limbs. These compliant and anisotropic structures with internal tensile forces in dorsoventral and mediolateral directions have elastin fiber bundles, which provide an extensibility and self-folding (self-packing) property to the wing membrane (35).
Reverse engineering all of these characteristics is not feasible from an engineering fabrication standpoint; therefore, we focused our attention on a few properties of the membrane wing. In producing such a membranous wing, we studied the anatomical properties of bats’ biological skin and found the key features to be (i) weight per unit of area (area density), (ii) tensile modulus, and (iii) stretchability (see Fig. 2, B and C). The area density is important because high-density membranes distributed across the robot’s skeleton increase the wing’s moment of inertia along the flapping axis and the overall payload of B2. In addition, internal tensile forces introduced by the membrane to the system are important because the micro motors used in the robot have limited torque outputs. When the pretension forces become large, the stall condition emerges in the actuators. This can damage the motor as well as the power electronics. The stretchability of the membrane defines the capacity of the wing to fold and unfold mediolaterally within the range of movement of actuators so that undesirable skin wrinkles or ruptures are avoided.
To produce an ultrathin and stretchable skin, we used two ultraflat metal sheets with a 10-μm flatness precision to sandwich our silicone materials. This ensures an even and consistent pressure distribution profile on the material. We synthesized a polymer in which two components—one containing a catalyst and the other containing polyorganosiloxanes with hydride functional groups—began vulcanization in the laboratory environment. The first component is a mixture of 65 to 75% by weight polyorganosiloxanes and 20 to 25% amorphous silica, and the second component is a mixture of 75 to 85% polyorganosiloxanes, 20 to 25% amorphous silica, and less than 0.1% platinum-siloxane complex. Platinum-siloxane is a catalyst for polymer chain growth. The Si–O bond length is about 1.68 Å with a bond angle of 130°, whereas the C–C bond found in most conventional polymers is about 1.54 Å with a 112° bond angle. Because of these geometric factors, silicone polymers exhibit a greater percentage of elongation and flexibility than carbon backbone polymers. However, silica is heavier than carbon, which could potentially make the wing too heavy and too rigid for flight. To solve this problem, we added hexamethyldisiloxane, which reduces the thickness and viscosity of the silicone, in an experimentally determined ratio.
Virtual constraints and feedback control
A crucial but unseen component of B2 is its flight control supported by its onboard sensors, high-performance micromotors with encoder feedback, and a microprocessor (see Fig. 3D). B2 and conventional flying robots such as fixed-wing and rotary-wing robots are analogous in that they all rely on oscillatory modulations of the magnitude and direction of aerodynamic forces. However, their flight control schemes are different. Conventional fixed-wing MAVs are often controlled by thrust and conventional control surfaces such as elevators, ailerons, and rudders. In contrast, B2 has nine active oscillatory joints (five of which are independent) in comparison to six DOFs (attitude and position) that are actively controlled. In other words, the control design requires suitable allocation of the control efforts to the joints.
In addition, challenges in flight control synthesis for B2 have roots in the nonlinear nature of the forces that act on it. B2, similar to fruit bats in size and mass (wing span, 30 to 40 cm; mass, 50 to 150 g), is capable of achieving a flapping frequency that is lower than or equal to its natural body response; as a result, it is often affected by nonlinear inertial and aerodynamic artifacts. Such forces often appear as nonlinear and nonaffine in-control terms in the equations of motion (36). Therefore, conventional approximation methods that assume flapping frequency to be much faster than the body dynamic response, such as the celebrated method of averaging, commonly applied to insect-scale flapping flight (10, 11), fail to make accurate predictions of the system’s behavior.
The approach taken in this paper is to asymptotically impose virtual constraints (holonomic constraints) on B2’s dynamic system through closed-loop feedback. This concept has a long history, but its application in nonlinear control theory is primarily due to the work of Isidori et al. (37, 38). The advantage of imposing these constraints through closed-loop feedback (software) rather than physically (hardware) is that B2’s wing configurations can be adjusted and modified during the flight. We have tested this concept on B2 to generate cruise flights, bank turning, and sharp diving maneuvers, and we anticipate that this can potentially help reconstruct the adaptive properties of bat flight for other maneuvers. For instance, bats use tip reversal at low flight speeds (hovering) to produce thrust and weight support, and the stroke plane becomes perpendicular to the body at higher flight speeds (39).
We parameterized the morphing structure of B2 by several configuration variables. The configuration variable vector qmorph, which defines the morphology of the forelimb and hindlimb as they evolve through the action of actuated coordinates, embodies nine biologically meaningful DOFs
where describes the retraction-protraction angle, is the radial flexion-extension angle, is the abduction-adduction angle of the carpus, qFL is the flapping angle, and is the dorsoventral movement of the hindlimb (see Fig. 3, B and C). Here, the superscript i denotes the right (R) or left (L) joint angles. The mechanical constraints described earlier yield a nonlinear map from actuated joint angles
to the morphology configuration variable vector qmorph. The spindle action shown in Fig. 3B is denoted by . The nonlinear map is explained mathematically in (40), which reflects two loops made by (p0-p1-p2) and (p1-p3-p4-p5), as shown in Fig. 3B. We used these configuration variables to develop B2’s nonlinear dynamic model and predefined actuator trajectories; see Materials and Methods and (40).
Now, the virtual constraints are given by
where rdes is the time-varying desired trajectory associated with the actuated coordinates, t is time, and β is the vector of the wing kinematic parameters explained in Materials and Methods. Once the virtual constraints (N) are enforced, the posture of B2 varies because the actuated portion of the system now implicitly follows the time-varying trajectory rdes. To design rdes, we precomputed the time evolution of B2’s joint trajectories for N = 0. We applied numerically stable approaches to guarantee that these trajectory evolutions take place on a constraint manifold (see Materials and Methods). Then, we used a finite-state nonlinear optimizer to shape these constraints subject to a series of predefined conditions (40).
The stability of the designed periodic solutions can be checked by inspecting the eigenvalues of the monodromy matrix [Eq. 22 in (40)] after defining a Poincaré map P and a Poincaré section (40). We computed the monodromy matrix by using a central difference scheme. We perturbed our system states around the equilibrium point at the beginning of the flapping cycle and then integrated the system dynamics given in Eqs. 10 and 16 throughout one flapping cycle.
To stabilize the designed periodic solution, we augmented the desired trajectory rdes with a correction term ,
where δβ is computed by Eq. 7. The Poincaré return map takes the robot states qk and (the Euler angles roll, pitch, and yaw and their rates) at the beginning of the kth flapping cycle and leads to the states at the beginning of the next flapping cycle,
We linearized the map P at , resulting in a dynamic system that describes the periodic behavior of the system at the beginning of each flapping cycle
where (*) denotes the equilibrium points and denotes deviations from the equilibrium points. The changes in the kinematic parameters are denoted by δβ. Here, the stability analysis of the periodic trajectories of the bat robot is relaxed to the stability analysis of the equilibrium of the linearized Poincaré return map on [see (40)]. As a result, classical feedback design tools can be applied to stabilize the system. We computed a constant state feedback gain matrix such that the closed-loop linearized map is exponentially stable:
We used this state feedback policy at the beginning of each flapping cycle to update the kinematic parameters as follows:
In Fig. 4C, the controller architecture is shown. The controller consists of two parts: (i) the discrete controller that updates the kinematic parameters β at ≈10 Hz and (ii) the morphing controller that enforces the predefined trajectories rdes and loops at 100 Hz.
Fig. 4 Untethered flights and controller architecture.
(A) Snapshots of a zero-path straight flight. (B) Snapshots of a diving maneuver. (C) The main controller consists of the discrete (C1) and morphing controllers (C2). The discrete and morphing controllers are updated through sensor measurements H1 and H2 at 10 and 100 Hz, respectively. The subsystems S1, S2, and S3 are the underactuated, actuated, and aerodynamic parts [see Materials and Methods and (40)].
Next, we used joint movements (, , , and ) to flex (extend) the armwings or ascend (descend) the legs and reconstructed two flight maneuvers: (i) a banking turn and (ii) a swoop maneuver. These joint motions were realized by modifying the term bi in the actuator-desired trajectories (Eq. 12 in Materials and Methods).
Banking turn maneuver
We performed extensive untethered flight experiments in a large indoor space (Stock Pavilion at the University of Illinois in Champaign-Urbana) where we could use a net (30 m by 30 m) to protect the sensitive electronics of B2 at the moment of landing. The flight arena was not equipped with any motion capture system. Although the vehicle landing position was adjusted by an operator to secure landings within the area, which is covered by the net, we landed outside the net many times. The launching task was performed by a human operator, thereby adding to the degree of inconsistency of the launches.
In all of these experiments, at the launch moment, the system reached its maximum flapping speed (≈10 Hz). In Fig. 5A, the time evolution of the roll angle qx sampled at 100 Hz is shown. The hand launch introduced initial perturbations, which considerably affected the first 10 wingbeats. Despite the external perturbations of the launch moment, the vehicle stabilized the roll angle within 20 wingbeats. This time envelope is denoted by Δtstab and is shown by the red region. Then, the operator sends a turn command, which is shown by the blue region. Immediately after sending the command, the roll angle increased, indicating a turn toward the right wing. The first flight test, which is shown in a solid black line and highlighted with green, does not follow the increase trend because the turn command was not applied for comparison purposes in this experiment.
Fig. 5The time evolution of the Euler angles roll qx, pitch qy, and yaw qz for eight flight tests is shown.
(A and B) The roll and pitch angles converge to a bounded neighborhood of 0° despite perturbations at the launch moment. The red region represents the time envelope required for vehicle stabilization and is denoted by Δtstab. For all of the flight experiments except the first [denoted by S.F. (straight flight) and highlighted by the green region], a bank turn command was sent at a time within the blue range. Then, the roll and pitch angles start to increase, indicating the beginning of the bank turn. (C) The behavior of the yaw angle. In the red region, vehicle heading is stabilized (except flight tests 1 and 4). In the blue region, the vehicle starts to turn toward the right armwing (negative heading rate). This behavior is not seen in the straight flight.
In Figs. 6 and 7, the morphing joint angles , , , and for these flight tests are reported. These joint angles were recorded by the onboard Hall effect sensors and were sampled at 100 Hz. As Fig. 6 (A to D) suggests, the controller achieves positive roll angle in the blue region by flexing the right armwing and extending the left arm wing.

Fig. 6 Arm wing joint angle time evolution.


Left and right armwing angles (A and B) and (C and D) are shown for eight flight tests. (A and C) Closeup views for the stabilization time envelope. The red region represents the joint movement during the stabilization time envelope. (B and D) After the stabilization time envelope, for all of the flight experiments except the first (highlighted with green), a bank turn command was sent at a time within the blue range.
Fig. 7 Leg joint angle time evolution.
Left and right leg angles  (A and B) and  (Cand D) are shown for eight flight tests. (A and C) Closeup views for the stabilization time envelope. (B and D) After the stabilization time envelope, the dorsal movement of the legs are applied to secure a successful belly landing. This dorsal movement can cause pitch-up artifacts, which are extremely nonlinear.

In Fig. 5 (B and C), the time evolutions of the Euler angles qy and qzare shown. Like the roll angle, the pitch angle was settled within a bounded neighborhood of 0 in the red region. At the moment of the banking turn (blue region), pitch-up artifacts appeared because of extreme nonlinear couplings between the roll and pitch dynamics. In addition, these pitch-ups, to some extent, are the result of the dorsal movement of the legs, which are applied to secure a successful belly landing (see Fig. 7, A to D). The straight flight pitch angle behaved differently because there were no sharp rises in the pitch angle in the blue region. In Fig. 5C, it is easy to observe that for all of the flight tests (except the straight flight), the rate of changes in the heading angle increased after the turn command is applied, suggesting the onset of the bank turning.

Diving maneuver
Next, a sharp diving maneuver, which is performed by bats when pursuing their prey, was reconstructed. Insectivorous echolocating bats face a sophisticated array of defenses used by their airborne prey. One such insect defense is the ultrasound-triggered dive, which is a sudden, rapid drop in altitude, sometimes all the way to the ground.
We tried to reconstruct this maneuver by triggering a sharp pitch-down motion at mid-flight. After launching the robot, the operator sent the command, which resulted in a sharp ventral movement of the legs (shown in Fig. 8C). Meanwhile, the armwings are stretched (shown in Fig. 8B). In Fig. 8A, a sharp rise of the pitch angle is noticeable. The vehicle swooped and reached a peak velocity of about 14 m/s. This extreme agile maneuver testifies to the level of attitude instability in B2.

Fig. 8 Joint angle evolution during swooping down.
(A) The time evolution of the Euler angles during the diving maneuver. (B) Armwing joint angles. (C) Leg joint angles. The red region indicates the stabilization time envelope; the light blue region indicates the dive time span.

Flight characteristics

B2’s flight characteristics are compared with Rousettus aegyptiacus flight information from (41). R. aegyptiacus flight information corresponds to the flight speed U that is within the range of 3 to 5 m/s. B2’s morphological details, which are presented in table S1, are used to compute B2’s flight characteristics. According to Rosén et al. (28), the arc length traveled by the wingtip stip is given by stip = 2ψbs, where ψ and bs are the flapping stroke angle and wingspan, respectively (stip,B2 = 0.48 and stip,Rous. = 0.36). A motion capture system (shown in fig. S3) was used to register the position coordinates px and py for four untethered flight tests (see fig. S2). The flight speed was calculated by taking the time derivative of pxand py. We considered the average flight speed B2 = 5.6 m/s in the succeeding calculations.
The measure K (28), which is similar to the reduced frequency and is computed on the basis of the wingtip speed, is given by K = stip/tf/U, where tf is the time span of a single wingbeat (KB2 = 0.86 and KRous. = 0.81). Subsequently, the advance ratio J is equal to the inverse of the measure K (JB2 = 1.16 and JRous. = 1.22). The wing loading Qs is given by Qs = Mbg/Smax (41), where Mb is the total body mass, g is the gravitational constant, and Smax is the maximum wing area (Qs,B2 = 13 N/m2 and Qs,Rous. = 11 N/m2).
The Strouhal number St is given by St = Δztip/tf/U (41), where Δztipis the vertical displacement of the wingtip with respect to the shoulder (28) (StB2 = 0.43 and StRous. = 0.4 − 0.6). Last, the nominal coefficient of lift Cl is computed. The coefficient is given by from (41), where ρair is the density of dry air, Vc is the velocity of the carpus (see Fig. 3B), and Fvert is the magnitude of the vertical lift force (see fig. S4). We measured Fvert by installing the robot on top of a miniature load cell, which is inside a wind tunnel. The wind tunnel is programmed to sustain air velocity at 4 to 6 m/s (Cl,B2 = 0.8 and Cl,Rous. = 1.0).
Bats are known to demonstrate exceptionally agile maneuvers thanks to many joints that are embedded in their flight mechanism, which synthesize sophisticated and functionally versatile dynamic wing conformations. Bats represent a unique solution to the challenges of maneuverable flapping flight and provide inspiration for vehicle design at bat-length scales.
The difficulties associated with reconstructing bat-inspired flight are exacerbated by the inherent complexities associated with the design of such bat robots. Consequently, we have identified and implemented the most important wing joints by means of a series of mechanical constraints and a feedback control design to control the six-DOF flight motion of the bat robot called B2.
The main results of this study are fourfold.

  • First, for robotics, this work demonstrates the synergistic design and flight control of an aerial robot with dynamic wing conformations similar to those of biological bats. Conventional flapping wing platforms have wings with few joints, which can be conceptualized as rigid bodies. These platforms often use conventional fixed-wing airplane control surfaces (e.g., rudders, ailerons, etc.); therefore, these robots are not suitable for examining the flight mechanisms of biological counterparts with nontrivial morphologies.This work has demonstrated several autonomous flight maneuvers (zero-path flight, banking turn, and diving) of a self-contained robotic platform that has fundamentally distinguished control arrays in comparison to existing flapping robots. B2 uses a morphing skeleton array wherein the use of a silicone-based skin enables the robot to morph its articulated structure in midair without losing an effective and smooth aerodynamic surface. This morphing property will not be realized with conventional fabrics (e.g., nylon and Mylar) that are primarily used in flapping wing research.
  • Next, for dynamics and control, this work applies the notion of stable periodic orbits to study aerial locomotion of B2, whose unstable flight dynamics are aggravated by the flexibility of the wings. The technique used in the paper can simplify stability analysis by establishing equivalence between the stability of a periodic orbit and a linearized Poincaré map.
  • Third, this work introduces a design scheme (as shown in Fig. 1) to mimic the key flight mechanisms of biological counterparts. There is no well-established methodology for reverse engineering the sophisticated locomotion of biological counterparts. These animals have several active and passive joints that make it impractical to incorporate all of them in the design. The framework that is introduced in this study accommodates the key DOFs of bat wings and legs in a 93-g flying robot with tight payload and size restrictions. These DOFs include the retraction-protraction of the shoulders, flexion-extension of the elbows, abduction-adduction of the wrists, and dorsoventral movement of the legs. The design framework is staged in two steps: introducing mechanical constraints motivated by PCA of bat flight kinematics and designing virtual constraints motivated by holonomically constrained mechanical systems.
  • Last but not least, this research contributes to biological studies on bat flight. The existing methods for biology rely on vision-based motion capture systems that use high-speed imaging sensors to record the trajectory of joints and limbs during bat flight. Although these approaches can effectively analyze the joint kinematics of bat wings in flight, they cannot help understand how specific DOFs or specific wing movement patterns contribute to a particular flight maneuver of a bat. B2 can be used to reconstruct flight maneuvers of bats by applying wing movement patterns observed in bat flight, thereby helping us understand the role of the dominant DOFs of bats. In this work, we have demonstrated the effectiveness of using this robot to reproduce flight maneuvers such as straight flight, banking turn, and diving flight. Motivated by previous biological studies such as that by Gardiner et al. (42), which inspects the role of the legs in modulating the pitch movement of bat flight, we have successfully implemented the dorsoventral movement control of the legs of B2 to produce a sharp diving maneuver or to maintain a straight path. Furthermore, in this work, bank turn maneuvers of bats (23) have been successfully reconstructed by controlling asymmetric wing folding of the two main wings. The self-sufficiency of an autonomous robotic platform in sensing, actuation, and computation permits extensive analysis of dynamic system responses. In other words, thorough and effective inspection of the key DOFs in bat flight is possible by selectively perturbing these joint angles of the robot and analyzing the response. It is the presence of several varying parameters in bat flight kinematics that hinders such a systematic analysis. Consequently, we envision the potential applications of our robotic platform as an important tool for studying bat flight in the context of robotic-inspired biology.
Nonlinear dynamics
The mathematical dynamic model of B2 is developed using the Lagrange method (36) after computing kinetic and potential energies. Rotary and translational kinetic energies are evaluated after defining the position and attitude of the body with respect to the inertial frame. Euler angles are used to define the attitude of the robot with respect to the inertial frame, whereas body coordinate frames, which are attached to the wings, define the wing movements with respect to the body coordinate frame.
Modeling assumptions
The following assumptions are made during the nonlinear dynamic modeling:
  1. (1) Wing inertial forces are considered because the wings are not massless.
  2. (2) There is no spanwise and chordwise flexibility in the wings; that is, it is a rigid flapping wing aircraft. Therefore, there is no flexibility-induced phase difference between flapping and feathering motions, and no degrees of underactuation are introduced as a result of passive phase difference between the flapping and feathering (pitch) motions.
  3. (3) Strip theory (43) is used for computing aerodynamic forces and moments.
  4. (4) The aerodynamic center is assumed to be located at the quarter-chord point (31), and the aerodynamic forces, which act on the aerodynamic center, include the lift and drag forces.
Method of Lagrange
During free-fall ballistic motions, B2 with its links and joints represents an open kinematic chain that evolves under the influence of gravitational and external aerodynamic forces. We used the method of Lagrange to mathematically define this dynamics. This open kinematic chain is uniquely determined with the fuselage Euler angles roll, pitch, and yaw (qx; qy; qz); fuselage center of mass (CoM) positions (px; py; pz); and morphing joint angles qmorph defined in Eq. 1. Therefore, the robot’s configuration variable vector is


where is the robot’s configuration variable space. We derived Lagrange equations after computing the total energy of the free open kinematic chain as the difference between the total kinetic energy and the total potential energy. Following Hamilton’s principle of least action, the equations of motion for the open kinematic chain with ballistic motions are given by:


where , , and denote the inertial matrix, the Coriolis matrix, and the gravity vector, respectively. The generalized forces , which reflect the role of aerodynamic forces as well the action of several morphing motors in B2, are described in (40).

Virtual constraints and offline actuator trajectory design
For wing articulations, we use a framework based on defining a set of parameterized and time-varying holonomic constraints (37, 38). This method permits shaping of the overall system dynamics through such constraints. These holonomic constraints control the posture of the articulated flight mechanism by driving the actuated portion of the system and take place through the action of the servo actuators that are embedded in the robot.
We partitioned the configuration variable vector q into the actuated coordinates qact and the remaining coordinates , which includes Euler angles and body CoM positions. The dynamics (Eq. 9) are rewritten as

In the equation above, , ,Embedded Image , , , , and are block matrices; Embedded Image and are two components of the generalized forces (40). The nonlinear system in Eq. 10 shows that the actuated and unactuated dynamics are coupled by the inertial, Coriolis, gravity, and aerodynamic terms.
The actuated dynamics represent the servo actuators in the robot. The action of these actuators is described by introducing parameterized and time-varying holonomic constraints into the dynamic system. To shape the actuated coordinates, we defined a constraint manifold, and we used numerically stable approaches to enforce the evolution of the trajectories on this manifold. Thereafter, a finite-state nonlinear optimizer shapes these constraints.
The servo actuators move the links to the desired positions. This is similar to the behavior of a holonomically constrained mechanical system and, mathematically speaking, is equivalent to the time evolution of the system dynamics given by Eq. 10 over the manifold(11)where N is the constraint equation and is given by N(t, β, qact) = qact− rdes(t, β). In the constraint equation, rdes is the vector of the desired trajectories for the actuated coordinates qact and is given by(12)where t denotes time and β = {ω, ϕi, ai, bi} parameterizes the periodic actuator trajectories that define the wing motion. These parameters are the control input to the system. Imposing the constraint equations to the system dynamics (Eq. 10) at only the acceleration level will lead to numeric problems owing to the difficulties of obtaining accurate position and velocity initial values (44). In addition, numeric discretization errors will be present during the process of integration, and the constraints will not be satisfied. Therefore, the constraints in position and velocity levels are also considered (45)

Embedded Image (13)

where κ1,2 are two constant matrices and

Substituting Eqs. 14 and 15 to Eq. 13 gives . Now, interlocking Eq. 10 and forms the following system of ordinary differential equations on a parameterized manifold:

Now, the numeric integration of the above differential-algebraic equation (DAE) is possible, and consequently, it is possible to design predefined periodic trajectories for the actuators. In (40), we have used finite-state optimization and shooting methods to design periodic solutions for the DAE.
To verify the accuracy of the proposed nonlinear dynamic model in predicting the behavior of the vehicle, we compared the trajectories from eight different flight experiments with the model-predicted trajectories. In fig. S1, the time evolution of the pitch angle qy and pitch rate angle is shown.
Supplementary Text
Fig. S1. Nonlinear model verification.
Fig. S2. Flight speed measurements.
Fig. S3. Motion capture system.
Fig. S4. Wind tunnel measurements.
Table S1. B2’s morphological details.
Movie S1. Membrane.
Movie S4. Swoop maneuver.
References (4659)
  1. K. Ma, P. Chirarattanon, S. Fulsler, R. Wood, Controlled flight of a biologically inspired, insect-scale robot. Science 340, 603–607 (2013). Abstract/FREE Full TextGoogle Scholar
  2. A. Paranjape, S.-J. Chung, J. Kim, Novel dihedral-based control of flapping-wing aircraft with application to perching. IEEE Trans. Robot. 29,1071–1084 (2013). Google Scholar
  3. J. W. Gerdes, S. K. Gupta, S. A. Wilkerson, A review of bird-inspired flapping wing miniature air vehicle designs. J. Mech. Robot. 4, 021003(2012). Google Scholar
  4. J. W. Bahlman, S. M. Swartz, K. S. Breuer, Design and characterization of a multi-articulated robotic bat wing. Bioinspir. Biomim. 8, 016009(2013). Google Scholar
  5. S.-J. Chung, M. Dorothy, Neurobiologically inspired control of engineered flapping flight. J. Guid. Control Dyn. 33, 440–453 (2010). CrossRefGoogle Scholar
  6. D. K. Riskin, D. J. Willis, J. Iriarte-Díaz, T. L. Hedrick, M. Kostandov, J.Chen, D. H. Laidlaw, K. S. Breuer, S. M. Swartz, Quantifying the complexity of bat wing kinematics. J. Theor. Biol. 254, 604–615 (2008). CrossRefPubMedWeb of ScienceGoogle Scholar
  7. S. M. Swartz, J. Iriarte-Diaz, D. K. Riskin, A. Song, X. Tian, D. J. Willis, K. S. Breuer, Wing structure and the aerodynamic basis of flight in bats, paper presented at 45th AIAA Aerospace Sciences Meeting and Exhibit, 8 to 11 January 2007, Reno, NV (2007);
  8. A. Azuma, The Biokinetics of Flying and Swimming (Springer Science & Business Media, 2012).
  9. X. Tian, J. Iriarte-Diaz, K. Middleton, R. Galvao, E. Israeli, A. Roemer, A. Sullivan, A. Song, S. Swartz, K. Breuer, Direct measurements of the kinematics and dynamics of bat flight. Bioinspir. Biomim. 1, S10–S18(2006). CrossRefPubMedGoogle Scholar
  10. X. Deng, L. Schenato, W. C. Wu, S. S. Sastry, Flapping flight for biomimetic robotic insects: Part I-system modeling. IEEE Trans. Robot.22, 776–788 (2006). CrossRefWeb of ScienceGoogle Scholar
  11. X. Deng, L. Schenato, S. S. Sastry, Flapping flight for biomimetic robotic insects: Part II-flight control design. IEEE Trans. Robot. 22,789–803 (2006). CrossRefWeb of ScienceGoogle Scholar
  12. R. J. Wood, S. Avadhanula, E. Steltz, M. Seeman, J. Entwistle, A.Bachrach, G. Barrows, S. Sanders, R. S. Fearing, Enabling technologies and subsystem integration for an autonomous palm-sized glider. IEEE Robot. Autom. Mag. 14, 82–91 (2007). Google Scholar
  13. R. J. Wood, The first takeoff of a biologically inspired at-scale robotic insect. IEEE Trans. Robot. 24, 341–347 (2008). CrossRefWeb of ScienceGoogle Scholar
  14. D. B. Doman, C. Tang, S. Regisford, Modeling interactions between flexible flapping-wing spars, mechanisms, and drive motors. J. Guid. Control Dyn.34, 1457–1473 (2011). Google Scholar
  15. I. Faruque, J. Sean Humbert, Dipteran insect flight dynamics. Part 1 Longitudinal motion about hover. J. Theor. Biol. 264, 538–552 (2010). CrossRefPubMedWeb of ScienceGoogle Scholar
  16. J. Dietsch, Air and sea robots add new perspectives to the global knowledge base. IEEE Robot. Autom. Mag. 18, 8–9 (2011). Google Scholar
  17. S. A. Combes, T. L. Daniel, Shape, flapping and flexion: Wing and fin design for forward flight. J. Exp. Biol. 204, 2073 (2001). PubMedWeb of ScienceGoogle Scholar
  18. S. A. Combes, T. L. Daniel, Into thin air: Contributions of aerodynamic and inertial-elastic forces to wing bending in the hawkmoth Manduca sexta. J. Exp. Biol. 206, 2999–3006 (2003). Abstract/FREE Full TextGoogle Scholar
  19. S. Lupashin, A. Schöllig, M. Sherback, R. D’Andrea, A simple learning strategy for high-speed quadrocopter multi-flips, in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2010), pp. 1642–1648.
  20. N. Michael, D. Mellinger, Q. Lindsey, V. Kumar, The grasp multiple micro-UAV testbed. IEEE Robot. Autom. Mag. 17, 56–65 (2010). Google Scholar
  21. H. D. Aldridge, Kinematics and aerodynamics of the greater horseshoe bat, Rhinolophus ferrumequinum, in horizontal flight at a various flight speeds. J. Exp. Biol. 126, 479–497 (1986). Abstract/FREE Full TextGoogle Scholar
  22. H. D. Aldridge, Body accelerations during the wingbeat in six bat species: The function of the upstroke in thrust generation. J. Exp. Biol.130, 275–293 (1987). Abstract/FREE Full TextGoogle Scholar
  23. U. M. Norberg, Some advanced flight manoeuvres of bats. J. Exp. Biol.64, 489–495 (1976). Abstract/FREE Full TextGoogle Scholar
  24. C. Chevallereau, G. Abba, Y. Aoustin, F. Plestan, E. R. Westervelt, C.Canudas-de-wit, J. W. Grizzle, Rabbit: A testbed for advanced control theory. IEEE Control Syst. Mag. 23, 57–79 (2003). CrossRefWeb of ScienceGoogle Scholar
  25. Y. P. Ivanenko, A. d’Avella, R. E. Poppele, F. Lacquaniti, On the origin of planar covariation of elevation angles during human locomotion. J. Neurophysiol. 99, 1890–1898 (2008). Abstract/FREE Full TextGoogle Scholar
  26. T. Chau, A review of analytical techniques for gait data. Part 1: Fuzzy, statistical and fractal methods. Gait Posture 13, 49–66 (2001). CrossRefPubMedWeb of ScienceGoogle Scholar
  27. G. Cappellini, Y. P. Ivanenko, R. E. Poppele, F. Lacquaniti, Motor patterns in human walking and running. J. Neurophysiol. 95, 3426–3437 (2006). Abstract/FREE Full TextGoogle Scholar
  28. M. Rosén, G. Spedding, A. Hedenström, The relationship between wingbeat kinematics and vortex wake of a thrush nightingale. J. Exp. Biol.207, 4255–4268 (2004). Abstract/FREE Full TextGoogle Scholar
  29. N. A. Bernstein, The Coordination and Regulation of Movements(Pergamon Press, 1967).
  30. J. Hoff, A. Ramezani, S.-J. Chung, S. Hutchinson, Synergistic design of a bio-inspired micro aerial vehicle with articulated wings. Proc. Rob. Sci. Syst. 10.15607/RSS.2016.XII.009 (2016). Google Scholar
  31. A. L. Thomas, G. K. Taylor, Animal flight dynamics I. Stability in gliding flight. J. Theor. Biol. 212, 399–424 (2001). CrossRefPubMedWeb of ScienceGoogle Scholar
  32. W. Maybury, J. Rayner, L. B. Couldrick, Lift generation by the avian tail. Proc. Biol. Sci. 268, 1443–1448 (2001). Abstract/FREE Full TextGoogle Scholar
  33. J. A. Cheney, D. Ton, N. Konow, D. K. Riskin, K. S. Breuer, S. M. Swartz,Hindlimb motion during steady flight of the lesser dog-faced fruit bat,Cynopterus brachyotis. PLOS ONE 9, e98093 (2014). CrossRefPubMedGoogle Scholar
  34. A. Ramezani, X. Shi, S.-J. Chung, S. Hutchinson, Bat Bot (B2), A biologically inspired flying machine, in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2016), pp. 3219–3226.
  35. H. Tanaka, H. Okada, Y. Shimasue, H. Liu, Flexible flapping wings with self-organized microwrinkles. Bioinspir. Biomim. 10, 046005 (2015). Google Scholar
  36. A. Ramezani, X. Shi, S.-J. Chung, S. Hutchinson, Lagrangian modeling and flight control of articulated-winged bat robot, in Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2015), pp. 2867–2874.
  37. C. I. Byrnes, A. Isidori, A frequency domain philosophy for nonlinear systems, in Proceedings of the IEEE Conference on Decision and Control(IEEE, 1984), pp. 1569–1573.
  38. A. Isidori, C. Moog, On the nonlinear equivalent of the notion of transmission zeros, in Modelling and Adaptive Control, C. I. Byrnes, A. Kurzhanski, Eds. (Springer, 1988), pp. 146–158.
  39. M. Wolf, L. C. Johansson, R. von Busse, Y. Winter, A. Hedenström,Kinematics of flight and the relationship to the vortex wake of a Pallas’ long tongued bat (Glossophaga soricina). J. Exp. Biol. 213, 2142–2153(2010). Abstract/FREE Full TextGoogle Scholar 
  40. Materials and methods are available as supplementary materials at the Science website.
  41. D. K. Riskin, J. Iriarte-Daz, K. M. Middleton, K. S. Breuer, S. M. Swartz,The effect of body size on the wing movements of pteropodid bats, with insights into thrust and lift production. J. Exp. Biol. 213, 4110–4122(2010). Abstract/FREE Full TextGoogle Scholar
  42. J. D. Gardiner, G. Dimitriadis, J. R. Codd, R. L. Nudds, A potential role for bat tail membranes in flight control. PLOS ONE 6, e18214 (2011). CrossRefPubMedGoogle Scholar
  43. J. D. DeLaurier, An aerodynamic model for flapping-wing flight.Aeronaut. J. 97, 125–130 (1993). Google Scholar
  44. U. M. Ascher, H. Chin, L. R. Petzold, S. Reich, Stabilization of constrained mechanical systems with daes and invariant manifolds. J. Struct. Mech. 23, 135–157 (1995). Google Scholar
  45. C. Führer, B. J. Leimkuhler, Numerical solution of differential-algebraic equations for constrained mechanical motion. Numer. Math. 59, 55–69(1991). CrossRefGoogle Scholar
  46. A. H. Nayfeh, Perturbation methods in nonlinear dynamics, inNonlinear Dynamics Aspects of Accelerators. Lecture Notes in Physics, J. M. Jowett, M. Month, S. Turner, Eds. (Springer, 1986), pp. 238–314.
  47. M. Goman, A. Khrabrov, State-space representation of aerodynamic characteristics of an aircraft at high angles of attack. J. Aircr. 31,1109–1115 (1994). Google Scholar
  48. A. A. Paranjape, S.-J. Chung, H. H. Hilton, A. Chakravarthy, Dynamics and performance of tailless micro aerial vehicle with flexible articulated wings.AIAA J. 50, 1177–1188 (2012). Google Scholar
  49. H. K. Khalil, J. Grizzle, Nonlinear Systems (Prentice Hall, 1996).
  50. G. Meurant, An Introduction to Differentiable Manifolds and Riemannian Geometry, vol. 120 (Academic Press, 1986).
  51. R. R. Burridge, A. A. Rizzi, D. E. Koditschek, Sequential composition of dynamically dexterous robot behaviors. Int. J. Rob. Res. 18, 534–555(1999). Google Scholar
  52. J. B. Dingwell, J. P. Cusumano, Nonlinear time series analysis of normal and pathological human walking. Chaos 10, 848–863 (2000). CrossRefPubMedWeb of ScienceGoogle Scholar
  53. M. S. Garcia, “Stability, scaling, and chaos in passive-dynamic gait models,” thesis, Cornell University, Ithaca, NY (1999).
  54. J. Guckenheimer, S. Johnson, International Hybrid Systems Workshop(Springer, 1994), pp. 202–225.
  55. Y. Hurmuzlu, C. Basdogan, J. J. Carollo, Presenting joint kinematics of human locomotion using phase plane portraits and Poincaré maps. J. Biomech. 27, 1495–1499 (1994). CrossRefPubMedGoogle Scholar
  56. S. G. Nersesov, V. Chellaboina, W. M. Haddad, A generalization of Poincaré’s theorem to hybrid and impulsive dynamical systems, inProceedings of the American Control Conference (IEEE, 2002), pp. 1240–1245.
  57. T. S. Parker, L. Chua, Practical Numerical Algorithms for Chaotic Systems(Springer Science & Business Media, 2012).
  58. B. Thuilot, A. Goswami, B. Espiau, Bifurcation and chaos in a simple passive bipedal gait, in Proceedings of the IEEE International Conference on Robotics and Automation (IEEE, 1997), pp. 792–798.
  59. E. R. Westervelt, J. W. Grizzle, C. Chevallereau, J. H. Choi, B. Morris,Feedback Control of Dynamic Bipedal Robot Locomotion (CRC Press, 2007).


We thank the team of graduate and undergraduate students from the aerospace, electrical, computer, and mechanical engineering departments for their contribution in constructing the prototype of B2 at the University of Illinois at Urbana-Champaign. In particular, we are indebted to Ph.D. students X. Shi (for hardware developments), J. Hoff (for wing kinematic analysis), and S. U. Ahmed (for helping with flight experiments). We extend our appreciation to our collaborators S. Swartz, K. S. Breuer, and H. Vejdani at Brown University for helping us to better understand the key mechanisms of bat flight.Funding: 
This work was supported by NSF (grant 1427111).

Author contributions: 
A.R., S.-J.C., and S.H. designed B2. A.R., S.-J.C., and S.H. designed control experiments, analyzed, and interpreted the data. A.R. constructed B2 and designed its controller with critical feedback from S.-J.C., and S.H. A.R. performed flight experiments. All authors prepared the manuscript.

Competing interests: 
The authors declare that they have no competing interests. Data and materials availability:Please contact S.-J.C. for data and other materials.

Copyright © 2017, American Association for the Advancement of Science

  Category: Robotics
  Comments: Comments Off on A biomimetic robotic platform to study flight specializations of bats

Top 10 Hot Artificial Intelligence (AI) Technologies

By Hugo Angel,

forrester-ai-technologiesThe market for artificial intelligence (AI) technologies is flourishing. Beyond the hype and the heightened media attention, the numerous startups and the internet giants racing to acquire them, there is a significant increase in investment and adoption by enterprises. A Narrative Science survey found last year that 38% of enterprises are already using AI, growing to 62% by 2018. Forrester Research predicted a greater than 300% increase in investment in artificial intelligence in 2017 compared with 2016. IDC estimated that the AI market will grow from $8 billion in 2016 to more than $47 billion in 2020.

Coined in 1955 to describe a new computer science sub-discipline, “Artificial Intelligence” today includes a variety of technologies and tools, some time-tested, others relatively new. To help make sense of what’s hot and what’s not, Forrester just published a TechRadar report on Artificial Intelligence (for application development professionals), a detailed analysis of 13 technologies enterprises should consider adopting to support human decision-making.

Based on Forrester’s analysis, here’s my list of the 10 hottest AI technologies:

  1. Natural Language Generation: Producing text from computer data. Currently used in customer service, report generation, and summarizing business intelligence insights. Sample vendors:
    • Attivio,
    • Automated Insights,
    • Cambridge Semantics,
    • Digital Reasoning,
    • Lucidworks,
    • Narrative Science,
    • SAS,
    • Yseop.
  2. Speech Recognition: Transcribe and transform human speech into format useful for computer applications. Currently used in interactive voice response systems and mobile applications. Sample vendors:
    • NICE,
    • Nuance Communications,
    • OpenText,
    • Verint Systems.
  3. Virtual Agents: “The current darling of the media,” says Forrester (I believe they refer to my evolving relationships with Alexa), from simple chatbots to advanced systems that can network with humans. Currently used in customer service and support and as a smart home manager. Sample vendors:
    • Amazon,
    • Apple,
    • Artificial Solutions,
    • Assist AI,
    • Creative Virtual,
    • Google,
    • IBM,
    • IPsoft,
    • Microsoft,
    • Satisfi.
  4. Machine Learning Platforms: Providing algorithms, APIs, development and training toolkits, data, as well as computing power to design, train, and deploy models into applications, processes, and other machines. Currently used in a wide range of enterprise applications, mostly `involving prediction or classification. Sample vendors:
    • Amazon,
    • Fractal Analytics,
    • Google,
    • Microsoft,
    • SAS,
    • Skytree.
  5. AI-optimized Hardware: Graphics processing units (GPU) and appliances specifically designed and architected to efficiently run AI-oriented computational jobs. Currently primarily making a difference in deep learning applications. Sample vendors:
    • Alluviate,
    • Cray,
    • Google,
    • IBM,
    • Intel,
    • Nvidia.
  6. Decision Management: Engines that insert rules and logic into AI systems and used for initial setup/training and ongoing maintenance and tuning. A mature technology, it is used in a wide variety of enterprise applications, assisting in or performing automated decision-making. Sample vendors:
    • Advanced Systems Concepts,
    • Informatica,
    • Maana,
    • Pegasystems,
    • UiPath.
  7. Deep Learning Platforms: A special type of machine learning consisting of artificial neural networks with multiple abstraction layers. Currently primarily used in pattern recognition and classification applications supported by very large data sets. Sample vendors:
    • Deep Instinct,
    • Ersatz Labs,
    • Fluid AI,
    • MathWorks,
    • Peltarion,
    • Saffron Technology,
    • Sentient Technologies.
  8. Biometrics: Enable more natural interactions between humans and machines, including but not limited to image and touch recognition, speech, and body language. Currently used primarily in market research. Sample vendors:
    • 3VR,
    • Affectiva,
    • Agnitio,
    • FaceFirst,
    • Sensory,
    • Synqera,
    • Tahzoo.
  9. Robotic Process Automation: Using scripts and other methods to automate human action to support efficient business processes. Currently used where it’s too expensive or inefficient for humans to execute a task or a process. Sample vendors:
    • Advanced Systems Concepts,
    • Automation Anywhere,
    • Blue Prism,
    • UiPath,
    • WorkFusion.
  10. Text Analytics and NLP: Natural language processing (NLP) uses and supports text analytics by facilitating the understanding of sentence structure and meaning, sentiment, and intent through statistical and machine learning methods. Currently used in fraud detection and security, a wide range of automated assistants, and applications for mining unstructured data. Sample vendors:
    • Basis Technology,
    • Coveo,
    • Expert System,
    • Indico,
    • Knime,
    • Lexalytics,
    • Linguamatics,
    • Mindbreeze,
    • Sinequa,
    • Stratifyd,
    • Synapsify.

There are certainly many business benefits gained from AI technologies today, but according to a survey Forrester conducted last year, there are also obstacles to AI adoption as expressed by companies with no plans of investing in AI:

There is no defined business case 42%
Not clear what AI can be used for 39%
Don’t have the required skills 33%
Need first to invest in modernizing data mgt platform 29%
Don’t have the budget 23%
Not certain what is needed for implementing an AI system 19%
AI systems are not proven 14%
Do not have the right processes or governance 13%
AI is a lot of hype with little substance 11%
Don’t own or have access to the required data 8%
Not sure what AI means 3%
Once enterprises overcome these obstacles, Forrester concludes, they stand to gain from AI driving accelerated transformation in customer-facing applications and developing an interconnected web of enterprise intelligence.

Follow me on Twitter @GilPress or Facebook or Google+

%d bloggers like this: