By Melanie Stone
The autonomous industry is progressing at an impressive rate, but we are still at the beginning: fully autonomous vehicles running safely on all public roads is several years away. There are still numerous challenges in the autonomous vehicle sector that researchers are trying to solve. These challenges span many areas, from in-vehicle technology to wider infrastructure communication. Cyngn specifically focuses on industrial use cases where autonomous vehicle deployments are much less complex. While fully autonomous operations on public roads are a few years away, it is important to note that industrial autonomous deployments are successfully being executed today.
Our VP of Engineering, Biao Ma, has identified three main problems that autonomous vehicle researchers are still trying to solve: occlusion, prediction, and fleet planning and control.
Just as humans require a clean line of sight to see objects, so too do sensors. Occlusion, then, occurs when an object prevents an autonomous vehicle from seeing other objects behind it.
An example of this is when you are driving on the road and there is a cyclist across the street. Let’s say, another car turns onto the road in front of you and blocks your view of the cyclist. This is occlusion, and it’s a perception problem that can occur with all combinations of objects blocking other objects. In some cases, the objects are moving. In other cases, the objects can be static like a light post, a tree, or even a building. Other forms of occlusion can occur when moving objects block each other, which often occurs in crowds.
According to a 2019 study, combating occlusion is a challenging task because “the frequency and variation of occlusion in the automotive environment is vast and can also be impacted by cultural and environmental factors.”
Like humans, most of our autonomous, sensory technology cannot see through these static or moving objects. Sensors emit beams that hit objects in the environment, which then travel back to the source to create 3D images of its surroundings. However, most sensors are bounded by traveling light. This means that when an object is blocking a second object, AV sensors are blind to what is behind this middle object. Not only do the beams lack the capacity to reach the target object, but occlusion can also result in lost data from the sensor.
This poses a problem in scenarios where it is crucial that autonomous vehicles have the capability to intuit the presence of an obstacle that isn’t necessarily visible.
An example of this is if the car in front of you slows to a stop to let a pedestrian pass. In most cases, you can intuit, without necessarily seeing the pedestrian ahead, why the car might have stopped. This intuition also tells you another important thing: you can’t “solve the obstacle” by whipping around the stopped car and plowing ahead.You might, after all, run the pedestrian over.
While there is some technology that is currently being developed to give autonomous vehicles a certain level of penetration beyond the next car. So far these technologies lack accuracy and produce minimal information.
What are researchers doing to solve the problem of occlusion?
According to Ma, there are two branches of exploration to try to push the technology forward: how AVs (1) see objects and (2) anticipate and respond without seeing a given object.
The first is about how autonomous vehicles see beyond the line of sight. Light travels in a straight line, but there are hidden objects that need to be seen to complete the entire picture. One possible solution is what’s known as collaborative perception. Think of it as a “crowd of eyes” on the street. This is essentially crowd-sourcing perception in an environment, such that vehicles, sensors, or cameras on the road can share information with each other. This ensures that each vehicle receives a complete picture of the surrounding space.
The second branch of exploration is about training a vehicle to develop an intuition about things that it can’t see. Humans naturally anticipate and respond in these scenarios because we possess intuition. For instance, if you are driving through a snowstorm, we know to slow down because while we may not be able to see it, we can imagine that black ice on the road could send our car spinning.
We can also anticipate that other cars will be driving unpredictably due to the weather and respond accordingly. Researchers are working to provide autonomous vehicles with this same sort of intuition.
The problem, Ma says, is that “there’s always a constant trade-off of responsiveness and fragility, and it is very difficult to achieve both.” By responsiveness, Ma means that vehicles should be able to quickly respond to potential danger or other objects. On the other hand, the system can’t be too fragile. If it is, then the system will be constantly responding to objects that may not be relevant, making it take forever for passengers to ever reach their destination. The system must properly balance the two aspects, which is a challenging task.
The second major obstacle is prediction. Prediction is the ability to predict the trajectory (the location and speed) of a target object in the near future. Prediction allows an autonomous vehicle to look at moving objects and guess where they will be within about three to five seconds. Typically, prediction systems will generate several possible outcomes, which are each individually assigned a likelihood of actually happening.
While driving, humans naturally make predictions all the time. In particular, humans know to take greater care around young drivers because they tend to be less predictable on the roads. Autonomous vehicles need to be able to take in this same information regarding its surroundings, run it through AI processing, and use this data to inform its decision-making and risk assessment.
There are three key technical factors for prediction: (1) the semantic, (2) physical limitations, and (3) relevancy prediction.
The first element informing prediction is the semantic. When people are first learning to drive, a major part of their study are all the symbols they’ll encounter on the road. What does a double yellow line mean? What does a blinking red light mean? Ma defines semantic as these symbols, or the meaning bounded by the environment that you are driving in.
Officially, this concept is defined as “a particular section of the driving environment having a common role that is bounded by either the traffic, social convention, or a specific area of the targeted driving space.” The semantic is how we can drive on a two-directional highway and not panic when a relatively close vehicle passes by in the other direction. This is because the dotted white line is telling us that the other vehicle is in its correct lane, heading in the correct direction.
The second kind of information that informs predictions are the physical limitations of each object. For example, it would be hard to run over a little boy that’s a half a block away, no matter how quickly he was running toward you. By contrast, a car heading in your direction a half a block away could very well crash into you. An autonomous driving system that’s good at making predictions will be good at differentiating between the capabilities of these two objects.
The third category consists of three different layers: (a) relevancy prediction, (b) predictions based on trajectory, and (c) whether there’s context around the object that requires additional caution.
First, relevancy prediction is about choosing which objects around you actually matter. But, how do you differentiate or predict what is relevant versus what is not? An example of this is when you are at an intersection, waiting to turn right so that you can get to the gas station that is on that upcoming street. If the car directly across from you is turning left onto the same street to also reach the gas station, we are aware that this car is relevant.
We, therefore, know we have to wait for them to turn first, so that we don’t hit them by turning onto the same road, at the same time. However, if we are at this same intersection and the car directly across is turning right (instead of left), we know that this car is now irrelevant.
The second layer is being able to decipher whether this object will be important for you to alter your trajectory. Is the object moving towards you in a way that could be relevant? Predictions based on trajectory use the current motion of the target object in combination with the current velocity of the object. For instance, given the previous example above, the driver’s signal may be on and signaling that they’re turning right (therefore irrelevant), but what if they change their mind and start turning towards your intended lane instead? While previously irrelevant, the object moving towards you is now relevant, presenting a scenario that a system must be able to predict.
Finally, the third layer, context considers whether there are other implications of the object by nature of its classification that forces you to respond. Consider a soccer ball that rolls in front of your car. When this happens, we have the intelligence to anticipate that there might be a little kid that follows. By knowing the context surrounding this object, we will respond by taking greater caution when approaching the ball in the road.
Ma explains how there are learning-based methods known as predictors that allow us to execute these three layers. Predictors are either algorithm-based, optimization-based, or rule-based, and help the system to predict relevancy, future trajectory, and context of a given object. There are currently learning-based methods that are working to provide predictions, along with algorithms that are based on the semantic and emotion of prediction.
“Different prediction mechanisms require the upstream systems to align and require the downstream system to use the information of the predicted trajectory,” says Ma.
Researchers study if these algorithms are getting better at prediction by comparing what really happened to what your system expected, and seeing how close these two are together. While there are significant improvements in the area of predictions, they are still in the early stages of advancement.
What are researchers doing to help improve autonomous vehicles prediction?
There are three directions that are occurring in the industry: (1) reducing the need for input, (2) improving the location of the input and output, and (3) bettering communication.
First, researchers are working to reduce the need for input. Current prediction systems are designed in a way that the prediction aspect has to be provided as input. This means the system requires knowledge of the semantic or other classifications in order to predict what another object will do. Researchers are creating new computational methods for prediction so that a system is not limited by the requirement of the input and instead, an upstream system tells the AV that the object is a pedestrian, or cyclist, for example — and that’s enough. These new methods that are being developed will make the vehicle more efficient when it comes to difficult prediction scenarios, as well as help combat corner cases that consist of unexpected behavior from other objects on the road.
The second is to improve the confidence and granularity of prediction by optimizing the input and output of information. Researchers know that if autonomous vehicles were better at making more behavior-level predictions, the vehicle would drive better. Consider a scenario where you’re trying to change lanes on a busy freeway. Here, humans know to look for a driver to wave them over. Autonomous vehicles do not understand this the same way, so researchers are trying to develop technology that will better allow AV’s to interpret these types of human-to-human communication. This will help vehicles interpret higher-level information, leading to a better understanding of what other drivers or pedestrians are trying to do.
The third direction is communication, which Ma argues is the best form of prediction. Being able to predict is efficient, but having the actual subject tell you what it is going to do is even better. By looking at the holistic view of the entire prediction stack, instead of viewing an autonomous vehicle as one subsystem, we can see that many systems can grow together. Working together allows for better communication and in turn, prediction.
3. Fleet Management
The final challenge is the coordination of a networked vehicle fleet. The first phase of autonomous vehicle development was getting individual vehicles to drive themselves; this includes developing an initial system and set of sensors. The second phase of development has been about getting systems of vehicles to coordinate and communicate with each other.
What are researchers doing to help improve AV coordination?
Algorithms are currently being processed and developed to improve fleet coordination. According to an MIT article, these various communication algorithms allow AVs to see beyond their own line of sight and improve observability and environmental awareness. Future trajectories can also be shared to improve prediction and motion coordination algorithms that can be utilized to “guarantee that decisions are jointly feasible.” This kind of communication and coordination will allow a fleet of autonomous vehicles to predict dynamic alterations in the surrounding environment. Ma says, “instead of each of the vehicles needing to perceive, track, and predict what they will do, algorithms, in a centralized and far more efficient way, will provide this information to everyone.”
The industry is continually trying to solve these three problems: occlusion, prediction, and fleet management. Solving these challenges will advance autonomy on public streets and enable autonomous vehicle technology in more complex environments. While organizations may think it will be years before this comes to their setting, Cyngn’s industrial AV technology is already working. Cyngn’s end-to-end, fully autonomous vehicle technology is available now.
Interested in learning more about how this technology can be implemented in your own environment? You can start your autonomy journey by visiting https://www.cyngn.com/services.