🏆 TSLA | Tesla Secret Weapon Spells Game Over for Car Companies

00:00
whether you're a tesla investor a fan of
00:02
their products or just a sucker for cool
00:04
technology this episode is for you first
00:06
things first i've never been more
00:08
flattered in my life it's super clear
00:10
that elon musk modeled the tesla bot
00:12
directly after me not only does it dance
00:15
like i do
00:17
but it's the same height as me runs at
00:19
my speed and even has my haircut
00:22
it's easy to get hyped about the tesla
00:24
bot especially when it's so good looking
00:26
but the things that actually hyped me up
00:28
the most from tesla's ai day were the
00:30
deep dives into the different computer
00:32
vision challenges they've been tackling
00:34
and their implications for autonomous
00:36
robo taxis so i'd like to show you why i
00:38
think it's game over for other
00:40
automakers who think that they can solve
00:42
full self driving before tesla we've
00:44
known for a long time that tesla has a
00:46
massive lead in terms of collecting real
00:48
world data to solve this problem but i
00:50
think ai day gave us some of the best
00:52
insights into how they're using all that
00:54
data when tesla thinks about full
00:56
self-driving they're thinking about
00:58
three things safety comfort and
01:00
efficiency my guess is that safety is
01:02
actually the most important then comfort
01:04
and then efficiency elon musk has
01:06
repeatedly stated that the prime
01:08
directive of the car is to not crash
01:10
safety is all about following the rules
01:12
of the road and staying the right
01:13
distance away from other vehicles
01:15
pedestrians and obstacles in my opinion
01:18
comfort is actually the second most
01:20
important because if the ride isn't
01:21
comfortable people won't want to get
01:23
back in the car comfort means planning
01:25
smooth maneuvers with high confidence so
01:27
that the tesla doesn't end up doing
01:29
things like starting a lane change it
01:30
won't finish
01:31
braking more often than necessary or
01:33
driving too aggressively or defensively
01:36
that means efficiency is really the
01:38
third priority where the focus is on
01:39
planning safe and comfortable routes
01:41
that can be completed with as little
01:43
wasted energy as possible let's check
01:45
out an example here's a tesla planning a
01:48
route to change lanes and make a
01:49
left-hand turn through traffic one plan
01:52
could be to change lanes immediately
01:53
that results in braking sharply and ends
01:55
up being pretty uncomfortable for the
01:57
passengers inside the car because of
01:59
this sharp change in velocity the next
02:02
planet considers is speeding up and
02:03
making the lane change further down the
02:05
road but doing that means it risks
02:07
missing the left-hand turn altogether
02:09
since that plan has the tesla moving
02:11
pretty fast when it gets to the position
02:13
it needs to be to make the turn
02:15
these two plans are actually extremes of
02:17
all the possible plans that the car
02:19
could consider what tesla's path
02:20
planning system actually does is
02:22
consider all the possible plans between
02:24
them and then chooses the most optimal
02:26
path that means maintaining fairly
02:28
smooth speeds as the tesla changes lanes
02:31
to make the turn the tesla full
02:33
self-driving software is constantly
02:34
optimizing its plan based on things like
02:37
how close the path would bring it to
02:38
other vehicles the total time it would
02:40
take to execute the plan how often it
02:42
would have to accelerate and decelerate
02:44
and how quickly and so on the final
02:47
result is a dynamic quarry door that
02:49
tesla is always choosing to stay inside
02:51
of and the boundaries of that corridor
02:53
shift based on what's going on around
02:55
the car it's one thing to solve this
02:57
problem on the highway where the road is
02:59
mostly flat and straight and the cars
03:01
are generally moving with the flow of
03:03
traffic it's another thing to solve this
03:05
problem for city driving where there are
03:07
way more intersections parked cars
03:09
pedestrians and things blocking the
03:11
tesla's view of its surroundings and
03:13
that brings me to the thing that really
03:15
blew my mind tesla's don't just plan
03:17
their own paths they predict the paths
03:19
of everything they can see check out
03:21
this use case they showed as to why they
03:23
need to run the planner on everything in
03:25
the scene i think this was the most
03:26
impressive part for me
03:28
the tesla is on autopilot and traveling
03:30
down a tight road crowded with cars
03:32
parked along both sides cones dumpsters
03:35
trees and a turn that prevents the
03:37
cameras from seeing very far down the
03:38
road that means that autopilot can't
03:41
plan very far into the future now watch
03:43
what happens next this may not look
03:45
super impressive on the surface but
03:47
here's what's actually going on under
03:49
the hood there was autopilot driving in
03:51
another corridor going around parked
03:52
cars cones and poles uh here there's a
03:55
3d view of the same thing
03:56
the oncoming car arrives now and
03:58
autopilot slows down a little bit but
04:00
then realizes that we cannot yield to
04:01
them because we don't have any space to
04:03
our side but the other car can yield to
04:05
us instead so instead of just blindly
04:07
breaking here all of our reasons about
04:09
that car
04:11
has low enough velocity that they can
04:12
pull over and should yield to us because
04:14
we cannot yield to them and assertively
04:15
makes progress
04:18
a second oncoming card arrives now
04:21
this vehicle has higher velocity
04:23
and like i said earlier we literally run
04:25
the autopilot planner for the other
04:26
object so in this case we run the
04:28
printer for them that object's plan now
04:30
goes around their part their sites park
04:32
cars and then after they pass the
04:34
parkours goes back to the right side of
04:36
the road for them
04:38
since we don't know what's in the mind
04:39
of the driver we actually have multiple
04:41
possible futures for this car here one
04:44
future is shown in red the other one is
04:45
shown in green
04:46
the green one is a plan that yields to
04:48
us but since this object's velocity and
04:50
acceleration are pretty high we don't
04:52
think that this person is going to yield
04:53
to us and they are actually going to go
04:54
around the spark cars so autopilot
04:56
decides that okay i have space here uh
04:59
this person is definitely going to come
05:00
so i'm going to pull over
05:02
so as autopilot is pulling over we
05:04
notice that that car has chosen to yield
05:06
to us based on their yaw rate and their
05:08
acceleration and autopilot immediately
05:10
changes his mind and continues to make
05:12
progress
05:13
this is why we need to plan for everyone
05:15
because otherwise we wouldn't know that
05:16
this person is going to go around the
05:17
other parkours
05:19
and come back to their side if you
05:20
didn't do this autopilot be too timid
05:22
and would not be a practical
05:23
self-driving car
05:24
so using the predictive path planner on
05:26
everything in the scene allows tesla's
05:28
autopilot to be aggressive enough to be
05:30
a good driver it's able to see the
05:32
oncoming car change its yaw angle that's
05:34
the angle between its nose and the road
05:37
it understands that that means that it's
05:38
pulling over and giving the right-of-way
05:40
back to the tesla so the tesla changes
05:42
its own plan based on this new
05:44
information in real time to make it
05:46
through this tight query door one of my
05:48
biggest concerns for full self driving
05:50
was moving to cameras only and away from
05:52
radar because radar typically gets
05:54
better range information and works at
05:55
night when cameras have a harder time
05:57
seeing during ai day andre carpathi
06:00
pointed out just how good their depth
06:01
and velocity detection is without radar
06:04
we also saw significant improvements in
06:06
our ability to estimate depth and of
06:07
course especially velocity so here i am
06:09
showing a clip from our remove the radar
06:11
push where we are seeing the radar depth
06:14
and velocity in green and we were trying
06:16
to
06:17
match or even surpass of course the
06:19
signal just from video networks alone
06:21
and what you're seeing here is in
06:24
orange we are seeing a single frame
06:26
performance
06:27
and in blue we are seeing again video
06:29
modules and so you see that the quality
06:30
of depth is much higher and for velocity
06:33
the orange signal of course you can't
06:34
get velocity out of a single frame
06:35
network so we use we just differentiate
06:37
depth to get that but the video module
06:39
actually is basically right on top of
06:41
the radar signal and so we found that
06:44
this worked extremely well for us this
06:46
is a massive deal because if you want to
06:48
track stuff in the present and predict
06:49
its motion in the future you need to
06:51
have a very good estimate of its range
06:53
you need to know how far away something
06:55
is to understand how fast it's actually
06:57
moving and if it's getting closer or
06:59
further away from you this is really
07:01
hard to do from single images in real
07:02
time but tesla is obviously great at
07:04
pulling that information from video in
07:07
addition to showing us that they're able
07:08
to get radar-like performance without
07:11
using radar they also showed us how they
07:13
can detect and track things that they
07:14
can't always see here's another quick
07:16
clip comparing tesla's image processing
07:18
of single frames to their video
07:20
processing when it comes to tracking
07:22
cars that they can't always see
07:25
the video networks also improved our
07:28
object detection quite a bit so in this
07:30
example i want to show you a case where
07:32
there are two cars
07:33
over there and one car is going to drive
07:35
by and occlude them briefly so look at
07:37
what's happening with the single frame
07:38
and the video predictions as the cars
07:41
pass in front of us
07:46
yeah so that makes a lot of sense so a
07:48
quick uh
07:50
playthrough through what's happening
07:51
when both of them are in view the
07:52
predictions are roughly equivalent
07:54
and you are seeing multiple orange boxes
07:56
because they're coming from different
07:57
cameras
08:00
when they are occluded the single frame
08:02
networks drop the detection but the
08:04
video module remembers it and we can
08:06
persist the cars and then when they are
08:08
only partially occluded the single frame
08:10
network is forced to make its best guess
08:12
about what it's seeing and it's forced
08:14
to make a prediction and it makes a
08:15
really terrible prediction but the video
08:17
module knows that this is not a very
08:18
easily visible
08:20
part right now and doesn't actually take
08:21
that into account
08:23
okay it's one thing to be able to do
08:25
this when the tesla is stopped on a flat
08:27
road and when it's sunny out but my
08:29
other big argument against vision only
08:31
systems was bad conditions like snow fog
08:34
and rain the way tesla solved this
08:36
problem is actually the other reason i
08:37
think it's game over for other
08:39
automakers tesla asked their fleet for
08:42
every clip where things are falling off
08:43
a vehicle and disrupt the vision of the
08:45
cameras and in one week the fleet
08:48
returned ten thousand clips of that
08:50
happening only tesla has access to so
08:52
much data of such a specific use case
08:55
because it's their cars and their
08:56
cameras and their software returning
08:58
that data for processing this is
09:00
something tesla's artificial
09:01
intelligence could only solve because
09:04
they have such a massive data advantage
09:06
and oh boy did they solve it
09:08
so once we train the networks with this
09:10
data
09:10
you can see that it's totally working
09:12
and keeps the memory that this object
09:15
was there
09:16
and provides this
09:19
my background is in computer vision and
09:21
radar signal processing and i was a
09:23
naysayer when it came to tesla moving to
09:25
vision only this whole channel is about
09:28
data-driven investing and i think it's
09:30
important to admit when i'm wrong
09:32
tesla's data just showed me that i was
09:34
wrong about needing to use radar this
09:36
data makes it clear to me that tesla
09:38
will be able to find a vision only
09:40
solution to any of these challenges i
09:43
want to point out one other reason that
09:44
vision only makes a lot of sense that
09:46
everyone definitely including me wasn't
09:48
really considering if tesla drops radar
09:51
from the story they don't need to
09:52
simulate radar data when they want more
09:54
data than their fleet can provide since
09:56
the whole system is now vision only
09:59
tesla can run a visual simulation of
10:01
road conditions using a photorealistic
10:03
physics engine and not worry about what
10:05
the radars would see in each case and
10:07
that's exactly what tesla is doing they
10:10
can use video game engines to simulate
10:12
traffic with a high degree of accuracy
10:14
not just in terms of what things look
10:15
like at different angles and how the sun
10:17
casts shadows at different times of day
10:19
but also how different cars drive and
10:21
interact in every type of road condition
10:24
and configuration the advantage of
10:26
simulated data is that it has perfect
10:28
labels perfect measurements and ideal
10:30
geometries and boundaries but why
10:32
simulate data in the first place you may
10:34
want to simulate lots of different
10:36
variations of a rare case that could be
10:38
difficult to source for example people
10:40
and animals running in the middle of the
10:42
highway during high-speed traffic is
10:44
this use case a little silly yes but it
10:46
could happen and an autonomous tesla
10:48
would need to handle it if and when it
10:50
does simulation also helps when data is
10:53
difficult to label like when there are
10:54
hundreds of moving objects in the scene
10:56
that are blocking each other crossing
10:58
paths changing angles and so on this is
11:01
really hard because of the labeling
11:02
steps specifically but it's super easy
11:05
to do with simulations because every
11:07
asset comes with the correct labels and
11:08
bounding boxes already the third case is
11:11
interesting as well closed loop cases
11:13
are when tesla wants to see what a car
11:15
would do in a very specific situation
11:17
after certain actions have already been
11:19
taken instead of asking humans to do a
11:22
specific set of actions to help with
11:23
these use cases they can just simulate
11:25
it and see what the full self-driving
11:27
software would do this simulation engine
11:29
is the second reason why it's game over
11:31
for other automakers remember tesla's
11:34
solution is now vision only so other
11:36
automakers would not only have to
11:37
translate all of this to their own
11:39
cameras and where those cameras are
11:41
placed on their own vehicles but if they
11:42
use radar or lidar they'd need to
11:44
simulate what everything would look like
11:46
to those sensors as well the first thing
11:48
tesla does is simulate everything as the
11:50
tesla cameras would see them that means
11:53
the noise profile exposure settings
11:55
distortions and motion blur effects are
11:57
all based on tesla's cameras and
11:59
calibrations specifically after
12:02
simulating what their cameras would
12:03
return the data goes through a
12:04
photorealistic rendering step with the
12:06
goal of being visually indistinguishable
12:08
from reality this includes reflections
12:11
from surfaces halo effects from lights
12:13
and anything else that could trip up a
12:15
camera they have a huge library of
12:17
assets including thousands of unique
12:19
vehicles pedestrians props like trees
12:22
buildings benches fences trash cans and
12:25
all the other items that you would
12:26
reasonably expect to be on or near the
12:28
road again the goal here is to quickly
12:30
be able to build thousands of usable
12:32
data sets to help train tesla's vision
12:34
only full self-driving algorithms for
12:36
cases where there isn't enough real
12:38
camera data they can make as many of
12:40
these variations for a given scenario as
12:42
they need with high confidence that what
12:44
they're simulating is very close to what
12:46
the real cameras would actually see in
12:48
real life
12:50
here's a direct comparison between an
12:51
original clip and one that was rendered
12:54
from the simulator
12:56
imagine how cool it would be to play the
12:57
next grand theft auto game if it looked
12:59
as real as this
13:01
so there are two things that would spell
13:02
game over for other automakers solving
13:04
full self-driving before tesla the
13:07
staggering amount of real world camera
13:08
only data that tesla can draw from their
13:11
fleet and the infinite amount of tesla
13:13
specific simulated data that they can
13:15
create whenever the fleet is somehow not
13:17
enough they use all that data to train
13:20
the full self-driving software to plan
13:22
optimized routes not only for the tesla
13:24
cars themselves but for everything the
13:26
tesla cars can see they do that so they
13:28
can make moment-by-moment decisions on
13:30
how aggressive or defensive they should
13:32
be in any given situation under any road
13:35
condition and of course if all else
13:37
fails they can just throw a sweet
13:39
looking clone of me in the car and have
13:40
it do all the driving instead comment
13:43
below or tweet me your thoughts on
13:44
tesla's ai day and any of the details
13:47
they shared i barely scratched the
13:49
surface with this episode so let me know
13:51
if there's something else that you want
13:52
me to cover either way i hope this
13:54
episode helped you understand how
13:56
tesla's massive real world data
13:58
advantage and their full self-driving
14:00
data simulator combined to spell game
14:02
over for other automakers that are
14:04
relying on less scalable and more
14:06
expensive sensors like radar and lidar
14:08
to solve this problem if it did consider
14:11
investing in the like button and
14:13
subscribing to the channel with all
14:14
notifications turned on that's a great
14:16
way to invest in the channel that
14:18
invests in you and get me into a sweet
14:21
tesla robot suit seriously how do i get
14:23
one of these bad boys until next time
14:25
this is ticker symbol you my name is
14:27
alex reminding you that the best
14:30
investment you can make
14:31
is in you
14:45
you