Oliver Cameron explains how Voyage is building self-driving cars for retirement communities
Lex Fridman interviews Oliver Cameron, co-founder and CEO of Voyage, in a guest lecture for MIT's deep learning for self-driving cars course.
Summary
Oliver Cameron, co-founder and CEO of Voyage, delivers a guest lecture at MIT's 6.S094 deep learning for self-driving cars course, hosted by Lex Fridman. Cameron traces his unconventional path from teenage software developer through Y Combinator, to Udacity's self-driving car program, and ultimately to founding Voyage. He describes how Voyage deliberately chose retirement communities as its first deployment market — a strategy originally suggested by Sebastian Thrun to Google in 2009 but rejected — arguing that slower speeds, simpler roadways, private community ownership, and favorable weather create ideal conditions for achieving Level 4 autonomy faster than competitors targeting major cities. Cameron also outlines Voyage's high-resolution sensor strategy, its approach to unsolved perception problems such as foliage detection, and the key lessons he has learned building a startup in this space.
Key Takeaways
FULL TRANSCRIPT
Introduction and Oliver Cameron's background
Lex Fridman: Welcome back to 6.S094, deep learning for self-driving cars. Today we have Oliver Cameron, co-founder and CEO of Voyage. Before that, he was the lead of the Udacity self-driving car program, which made autonomous vehicle research and development accessible to the entire world. He has a passion for the topic and a genuine open nature that makes him one of my favorite people in general and one of my favorite people working in this space. I think thousands of people agree with that, so please give Oliver a warm welcome.
Oliver Cameron: Thank you very much, Lex, and thank you all for having me here today. I'm super excited to speak all about Voyage, but in reality the thing I want to share today is — as the title says — how to start a self-driving car startup. Rarely do you get an inside scoop of how a startup is formed. You hear all the PR, all the very lovey-dovey press releases. I want to share the inside of how Voyage came to be, which was a little unconventional compared to your average self-driving car startup. They always tell you that the path from a startup to the goal you want is a zigzag. Ours was a pretty insane zigzag as well. We'll go through all of that.
Let's talk about my background — also a little unconventional. I'm not very good at learning in a classroom. For me, learning by doing, by building, has always been the thing that works best. Going all the way back to when I was a teenager, software in general was my passion — this idea that you can make something out of absolutely nothing and then all of a sudden millions, and in Facebook's case billions, of people can be using that thing. After building lots of crazy stuff, and perhaps not being too popular in high school because that's all I did, I started a company. I won't bore you with all the details, but I learned a lot during the experience, joined Y Combinator — which started right here in Cambridge, which is very cool — and then this very pivotal moment happened to me.
I heard about this online class which was generating a whole bunch of scandal and controversy. It was from a guy called Sebastian Thrun. He'd taken this Stanford class he taught in artificial intelligence and just said, "Screw it, we're going to put the whole thing online." Back then — this was around 2011 — this was a very controversial thing to do. Today MIT and many others do this all the time, but back then there was a hell of a lot of controversy around doing something like this. But this learning format really appealed to me. Being able to sit in front of my laptop, learn at my own pace, build, build, build — that really resonated with me.
I took that class in 2013 — artificial intelligence for robotics — and this was just another pivotal moment. My head exploded. All the enthusiasm I'd had for software kind of transferred to artificial intelligence and robotics, and I just became addicted to the format of what are now called MOOCs — massively open online courses. I loved them so much I decided I wanted to go do this and help others learn this stuff. So I joined Udacity and built more classes like that. I did that for four years, leading machine learning, robotics, and eventually the self-driving car curriculum, which was a lot of fun.
I got to learn directly from two great company builders — truly great company builders. One was Vishal Makhijani. He was the operator extraordinaire at Udacity — understood how to build a company, how to build a culture, how to incentivize people, and how to do all those things we don't often talk about. And Sebastian Thrun, who of course founded the Google self-driving car project in its early days, and right now I believe he's building flying cars. I learned so much from him, but this idea that you are literally in control of your destiny — that you can build absolutely anything if you put your mind to it — was always pretty inspirational.
Today, of course, I drive cars at Voyage, and we'll talk more about what makes us special compared to the other self-driving car companies you may have heard of in this class and beyond.
The Udacity self-driving car program
Oliver Cameron: Let's talk about Udacity. Raise your hand if you've heard of Udacity — I'm very curious. There you go, that's most of the room. Udacity, like I said, was founded by Sebastian Thrun. He took this class online and it just exploded, and he built a company around it. Udacity's real focus is on increasing the world's GDP — this idea that talent is everywhere, that it isn't now just constrained to the best schools in the world, that because of this proliferation of content there are talented students all over the world and all they need is the content with which to build crazy, cool, world-changing things. What I see as my job today is to go out into the world and find these ridiculously talented people and then put them to work on the hardest problems that exist. Udacity felt like the perfect place to do this as a kind of prelude to that.
About three years into Udacity, we had had this real focus on machine learning and robotics, but we really wanted to take it to the next step. We came up with this concept internally that we called "only at Udacity" — what if we taught the things that other places weren't teaching? What if people all around the world could come and learn from what may appear to be niche topics, but were just being taught at the right time because that industry was about to blow up? The first one we did of this — and we've done some after, including flying cars and a much more in-depth curriculum on artificial intelligence — was self-driving cars.
Our goal was to accelerate the deployment of self-driving cars. As Sebastian says, there are a number of reasons why self-driving cars are transformational. At the time, around 2016, it felt like self-driving cars were just taking a little bit too long. If we rewind to that particular spot in time, Google was really the only main effort going on, and what we believed is that it needed to happen faster — and that one of the reasons it wasn't happening fast enough is because there wasn't enough talent in the space. So what we decided to do was build something quite special: pair up a world-class curriculum, an actual self-driving car, and what we called our open-source challenges. All of that would come together to build this quite special curriculum.
One of our beliefs was that partnering with industry was the right way to go. That was because it felt — and I believe this — that the knowledge of how to build a self-driving car was not necessarily trapped in academia; it was trapped in industry. So we had to go straight to industry, work with engineers that were already challenging themselves with these problems, get them on camera, have them teach the concepts they know and build day in and day out, and have that be transplanted to thousands of minds around the world.
We also built an incredibly talented curriculum team. Sebastian Thrun was a big part of this curriculum. When I told folks that I'd gotten the chance to work with him specifically on self-driving cars, someone likened it to getting basketball lessons from Michael Jordan, which I thought was pretty fun.
Here's a quick photo of the first lecture recordings with eventual Voyage co-founders Eric and Mac. Eric, who's on the left, hates this picture — and here's why. He still has that box on his desk. We built a whole twelve-month curriculum to take an intermediate software engineer who may be in consumer software or some other part of the software world and take them into self-driving cars. We wanted to cover perception, prediction, planning, localization, controls — the whole breadth of the industry. The reason we wanted to do that is because we saw the best fit for a Udacity student not necessarily being a specialist in a niche — for example, just perception, although there have been a whole bunch of folks doing that as well — but that the skills of a Udacity student tend to pair themselves well with being a generalist, someone who can contribute all across the stack. So we tried to give these folks that breadth of knowledge.
The impact of this curriculum was bigger than we thought it would be. When we pitched this idea as a small team to Sebastian and Vishal at Udacity, there was a lot of skepticism that something like this was going to be successful. The reason for that skepticism is that one of the formulas Udacity looked at to determine the impact of building a certain type of content was the number of open jobs available. If there were millions of jobs open in, say, web development or mobile development, it felt like there was a massive opportunity to impact that area. But if you were to search in 2016 for self-driving car engineers or the different disciplines within it, it was kind of just Google. So it was very interesting to see the instantaneous reaction we had to launching this curriculum.
Today, over 14,000 successful students from all around the world. The most exciting thing is to see what students have done with this. I learned recently that a set of our students are building a self-driving truck startup in India. Another set of students in South Korea are building a perception engine for self-driving cars. A whole bunch of folks are building truly amazing things, and many have gotten jobs at Cruise, Zoox, Waymo, Argo — all the big names — and are actively impacting those companies today.
Building an actual self-driving car at Udacity
Oliver Cameron: Now for the fun stuff. We also decided to make the curriculum extra special by building an actual self-driving car. Whenever I talked about this internally at Udacity, people asked me why. Isn't the curriculum just enough? Why go to the length of building an actual self-driving car? Selfishly, some of it was just a personal want to build a self-driving car. But the reasoning I used is: what better way to prove to these students who are putting their faith in us that we know what we're doing than to build our own self-driving car? And also, what better way to collaborate with these students on an area that is really in its infancy than by having this platform that students could actually run code on a real car?
So we decided to buy a car. We set ourselves a milestone: to drive from Mountain View to San Francisco — 32 miles — with zero disengagements. It should be repeatable. It won't be zero disengagements every single time, because otherwise we'd have an actual self-driving car. But in a short period of time, how much progress could we make towards this stated goal?
The car we bought was a Lincoln MKZ — you're probably very familiar with it if you follow self-driving cars. They're everywhere, and there's a reason for that in terms of the drive-by-wire nature of the vehicle and other things. We outfitted it with a whole bunch of sensors — some cameras, some lidars, all that good stuff. We also tried to build our own mount, which we affectionately called the periscope. That was not our final design. We built it all from parts at Home Depot — truly an MVP.
Then we got to work. The goal was to accomplish that milestone within six months. We assembled a dream team of folks I'd worked with on different projects at Udacity — people who had worked on the machine learning curriculum, the robotics curriculum, and so on. This was one of our first days testing, and we did this at the Shoreline Amphitheatre parking lot, which is now a very popular place to test self-driving cars in the Bay Area because Google used to do it there in the past. We saw a lot of weird stuff — for example, what I believe to be a motorcycle gang.
We made progress. We kept iterating, kept building, and it started to come together. In fact, some stuff that we thought wouldn't work surprisingly just started to work. This is on El Camino Real — I'm in the backseat here. Mac discovered that we shouldn't have stopped at that traffic light, but we did. We resolved the mystery later.
We learned a lot by going on that route — the different behaviors of drivers, one of the things we were worried about being vehicles cutting us off, meaning a vehicle pulling out in front of us even a few hundred feet ahead. We drove a little slow at 25. Pretty soon it got quite boring — the car was doing very well driving itself. We built some cool algorithms to change lanes when necessary, similar to what you see with Tesla Autopilot these days. We collaborated with some students on a traffic light classifier which was integrated into ROS. Pretty boring stuff — you can tell Eric was surprised that it was just fine.
We also had a penchant for recording themed videos. Eventually we became pretty confident, but we always wanted to test most of the day just to get the most learnings out of everything. One video was made at 2:30 a.m., driving from Mountain View to San Francisco — all 32 miles. That didn't count as the milestone, just to be clear. You'll see that we eventually hit the 32 miles and Mac, in the driver's seat, was pretty excited about that. Of course that didn't count because it was in the middle of the night, and that's not going to be a very useful route. But it was an awesome accomplishment just to make it 32 miles with no disengagements — traffic lights, lane changes, all that good stuff.
After four months — this is in the daytime, beginning at around 7 a.m. — we accomplished it. That small team had come together and built something pretty cool. It could handle multi-lane roadways, varying speed limits, traffic lights, objects — all that good stuff. The thing that really brought this home to me is that the industry was now ready. It felt like this feeling I'd had in software — where someone in their bedroom can go and build something and launch it almost overnight — could now, not quite the same but close, happen in self-driving cars.
Open-source challenges and the steering angle prediction competition
Oliver Cameron: Let's talk about open-source challenges. We got the same question: why do this? It was clear to me that for something like self-driving cars, which was so formative, we had to collaborate with students to figure out the best stuff, because even the folks at Udacity were not necessarily the world's leading experts in these topics. We wanted to use this hivemind of activity from around the world to teach the best stuff.
Just through a period of a year, these are all the different challenges we launched. There were prizes, leaderboards, and all that fun stuff. The one I'll focus most on today is using deep learning to predict steering angles. The challenge was clear: given a single camera frame, you have to predict the appropriate steering angle of the vehicle. If anyone had been reading papers in 2016, this stuff was all the rage and it felt like one of those areas that was just begging for more exploration. Let's use all these students from around the world to do it.
We did have students from all around the world — over a hundred teams, people self-organized into these little groups to go and build this. Over the course of about four months we had a whole bunch of submissions, all taking incredibly different approaches to the problem. We released two sets of validation data and all that good stuff.
The winning model — I later found out that the author of this model went on to lead the self-driving car team at Yandex, which if you've been following the space is doing some pretty cool stuff in self-driving cars today. You'll see this is on a route from the Bay Area to Half Moon Bay, a very winding road, and the prediction matches pretty closely to the actual, which is nice. If you read his description of his solution, it's a pretty cool solution. The most exciting thing was just the number of different approaches to the problem, all resulting in some awesome stuff.
In true Voyage fashion, we recorded a video of what this model performed like on our car. It wasn't perfect, as any first model isn't. The general approach of camera-only driving had its faults. One of the main ones we realized after trying all this stuff out is that a car, when steered by such an input, performs differently in a real car than it does on your desk in a simulator through pre-recorded camera frames. Adjusting for those corrections was something that students added after the fact, which was pretty cool.
Founding Voyage
Oliver Cameron: After all of these things — building that curriculum, building a self-driving car, launching these challenges — it felt like it was time for something new. It was awesome to go and collaborate with all these students, and it felt like I had to go build something. So I gathered that same team that had built this curriculum and said, "We're going to go build a self-driving car."
Voyage is a new kind of taxi service. Our pitch has changed somewhat through time, but that's still pretty accurate. We started what is now called Voyage, and our goal really was to build a self-driving car but to do it differently. We didn't want to follow the same formula that we felt we'd seen from some of the other folks in the field. The reason is that those folks have real advantages. When you think about Google's project — which I'm a big fan of — they have this massive engineering pipeline of folks that want to go build a self-driving car at Waymo, but they also have a cash balance of billions of dollars that is hard to match. They also have the brand recognition of getting to work with Google and all that good stuff. So we just knew we had to think about this problem quite differently.
What motivated me is that today, as we all know, we have this incredibly broken transportation system. You step outside onto the roads today and I don't feel particularly safe when I jump into my car. We all know the stats — over a million people die in road fatalities today. That doesn't include folks that break necks, break bones, all that horrific stuff. It's also incredibly inefficient. We've all observed this as we go about our day — just the number of lanes that exist on a road today to account for peak traffic, the number of vehicles which have enough room for eight people but usually have just one person in the front seat. I read a stat recently that only 7% of the average vehicle's energy usage is going towards moving the things that are actually in the car. The rest is waste. So an incredibly inefficient system. It's also expensive — the reason we see a lot of old cars on the road today is because that's the most affordable way for a lot of folks to get around. And it's inaccessible — and you'll see why this matters to us in particular.
Our goal is to introduce a new way to explore our communities. This is kind of our mission.
Why now — the technology readiness argument
Oliver Cameron: Why now? Why is it possible to build a self-driving car now? A number of factors that we learned during the Udacity experience, and some new ones as well. It feels from everything we see that sensors are now in a position where they are capable of Level 4 self-driving cars. The resolution, the range, the reliability — all those things that were necessary for a self-driving car are today ready. That didn't used to be the case. If you rewind to 2007 and look at the cars participating in the DARPA challenges, you'll see a lot of single-channel lasers, the relic of the Velodyne HDL-64 — the spinning bucket, as it's called today. No one would have claimed those sensors were ready. But today you've got this enormous breadth of sensors that can take you there.
Compute is there. When we think about the recent rise in GPUs and whatnot, finally being able to have enough performance in the back of a car with the power constraints that you have — it's there. And talent — this is not just Google today. You've got all of these great minds from all around the world building this technology, so you're able to recruit those folks and put them to work on problems they've solved in many cases beforehand.
The reason I have yellow for computer vision — which is not a knock against computer vision — is because it's not quite there yet for a fully driverless self-driving car. If you rewound three, four, five years, this would have been a red. But today, with all the community and activity around computer vision, this is steadily getting to a green state. Pretty soon it'll be green, and then you'll have that perfect formula for Level 4 driving.
The ride-sharing market and Voyage's strategic positioning
Oliver Cameron: What we're after is ride-sharing. We believe that the optimal way for people to move around is to be able to summon a car. But the thing that's suboptimal today is that you have to have a human driving you whenever you want to move around. That prevents the cost from being lower, prevents some safety issues, prevents some quality issues. We think solving that will mean these next-generation ways of moving around will come to fruition.
But what we also see is that if we never remove the driver from the car — if a ride-hailing network always has a human driver — you are inherently limited by the number of miles you can drive. That means it will never replace personal car ownership, will never fix that fatality number I talked about, all of those things. We must solve this. So we think that by having a self-driving car, these next-generation transportation networks will come to fruition.
Our lead VC is Vinod Khosla, the founder of Khosla Ventures — an awesome guy who has done some truly world-changing things. He has this quote which I'm a big fan of: "Your market entry strategy is often different from your market disruption strategy. Start where you find a gap in the market and push your way through." This better communicates what I mentioned at the very beginning — that we should build a self-driving car but do it in a different way, because if we don't do that we're going to fall into the same traps as many of the others that have died along the way. We have to find a way to do something different that we own and that we are really, really good at.
Retirement communities as the first deployment market
Oliver Cameron: For us, that was retirement communities. Hands up if you've ever visited a retirement community. Way less than I expected — you've got to get out to one, Lex. These are just amazing places. The reasons we chose retirement communities first to deploy our self-driving technology are these four: they are slower — the speed limits in these communities tend to be far slower than you'd see on public roads; much calmer roadways — when you visit these locations, I liken it to listening to a podcast at 0.75x speed, very constrained, very slow, and a little boring from time to time; but you've also got these deeply felt transportation challenges.
We hear from these residents all the time about how transportation is a pain point and that their only option is a personally owned vehicle. These folks know in many cases they shouldn't be driving, but because they don't have an alternative, they still drive. We hear from folks that put off much-needed surgeries — hip replacements, things like that — because they don't have a friend in town who's going to be able to move them around. We hear from folks with vision degeneration who just don't see a way they'll be able to move around and keep the quality of life they've had. Folks gripping steering wheels for extended periods of time — all these challenges that felt like the best first place for a self-driving car to begin.
And there's a clear path to customers. We see that on the roads today, ride-sharing on public streets is a particularly brutal battle — a race to the bottom in terms of cost. If we owned every retirement community in the country, meaning the transportation networks there, that would in and of itself be a very valuable business.
The Villages — Voyage's first community deployment
Oliver Cameron: Let's talk about our first community — The Villages. Whenever I show this slide, people are astounded by the number of residents in a community like this: over 125,000 and growing, over 750 miles of road. What we have in this location is an exclusive license to operate an autonomous vehicle service.
This is one of our other beliefs: by partnering very deeply with the community, it means we're able to deliver a better service and grow a more reliable business. We won't have entrance from competitors — from all the other self-driving car companies — in our communities. What we actually do in exchange for that exclusive license is grant these communities equity, because if we win, it's probably — in fact, highly likely — as a result of those communities.
The addressable market for transportation in these regions is massive. These residents tend to be, as a lot of seniors tend to be, quite affluent, which means they have some disposable income when it comes to paying for ride-sharing services and other things like that. So we find that the reciprocity is absolutely perfect here. We're launching and have launched passenger services to these residents. We've gotten awesome feedback and learned a lot about the needs of providing ride-sharing for senior citizens.
Just some quick stats from my Series A fundraising deck — about the size of the senior market. Again, this is the first place we go, but you can get a feel for just how large this transportation market is. Today there are 4 to 7 million seniors, growing by 2060 to over a hundred million seniors in the US. The total addressable market for just seniors is incredibly large — 2,500-plus communities, all that good stuff.
This is how we see the landscape of potential deployments. A lot of the big guys are focusing on that bottom-left quadrant — large cities — and it makes sense because it plays to their unique strengths. It plays to their ability to deploy thousands of cars, tens of thousands of cars. It plays to the strengths they have, including some patience or ability to have more extended timelines when it comes to building this technology. But a startup like us that fights for survival every single day means we have to do things differently. So we focus on that top-right quadrant — what we've coined as self-contained communities. These places are simpler and slower, but they also have this ability for us to have that exclusivity I talked about. There are some others we play in as well, whether it's the senior market or maybe even small cities and things like that.
Voyage's autonomous technology stack
Oliver Cameron: Let's talk about autonomous technology. Just to reiterate why we deploy in retirement communities: slower speeds, simpler roadway, there is a central authority — these places tend to be run by private companies, which makes for a quite unique and very positive relationship. It means we can deploy faster. It means we have the potential to have more impact in these regions. It also turns out that retirement communities tend to be located in ideal weather for self-driving cars — think Arizona, Florida, and so on.
We have a world-class team building this at Voyage from all the major programs out there, and that makes our lives infinitely easier. One thing that also makes our lives easier is the sensor configuration of our car. We've intentionally made the decision that we're not going to focus on optimizing for cost today, but to optimize for performance. We want to get to truly driverless sooner than most. One of the easiest ways you can make your life easier is by optimizing for high-resolution sensors.
At the very top of the vehicle we have the VLS-128, which is a 128-channel lidar capable of seeing 300 meters in 360 degrees. We have many other different lidars on the vehicle to cover different blind spots. Altogether we see 12.6 million points per second, and it just looks incredibly high-resolution. You'll see our car at the bottom there, and that's the raw point cloud output that we see of the world.
We run towards Level 4. For us, what that means is that if you're building a demo self-driving car — kind of like we did at the Udacity project — you may focus on just the top four items: perception, prediction, planning, and controls. It turns out you can build a very impressive demo quite quickly by just focusing on those things. But of course those things fall apart whenever edge cases are introduced, which happen all the time. So we've spent a ton of time on all the items beyond those, because our goal is to build not a demo but a truly driverless vehicle.
We also have an emphasis on partnerships, because what we've noticed in the self-driving ecosystem is that there's not just more self-driving car companies building the full stack — there are now folks going into simulation, mapping, middleware, teleoperation, routing, sensors, and much more. So we make our lives easier by partnering with companies in those spaces so that we don't have to spin up a simulation team or an operations team to go map the world. We can just work with these very cool companies.
The unsolved problem of foliage detection in perception
Oliver Cameron: Let's talk about one unsolved problem which fascinates me. It has to do with perception. You probably won't be able to notice this unsolved problem from just this picture, but maybe if I add some annotations you might — foliage, trees, bushes, whatever you want to call them.
You may have seen some quotes in the media about some popular AV programs struggling with foliage. For example: Cruise cars sometimes slow down or stop if they see a bush on the side of a street or a lane-dividing pole. Uber's self-driving car software has routinely been fooled by the shadows of tree branches, which it would sometimes mistake for real objects. And even Voyage — there's only one hard spot on the way, and the culprit is a bush two feet high that protrudes into a lane from a street median, which Voyage considers a possible threat. Voyage may have trimmed it — and we did — but we don't think that's scalable.
So at the beginning of 2018 we decided to solve this problem. This all resides in the world of perception, which is an area of particular fascination for me. These are just some of the papers and research we see going on that intend to solve those sorts of issues.
One of the reasons you've seen those programs, including ours, be particularly sensitive to foliage is because from a perception perspective, one of the most well-known ways to detect objects is to utilize the map. If you have this map and you effectively — simplifying to a certain extent — subtract objects that aren't in the map and use that as a way to understand what's dynamic around you, then you'll end up with decent representations of cars and pedestrians and whatnot. But if foliage grows — which it does — trees extend out from the map, and that particular bush is now an object in your path.
The neural networks we're looking at don't use that same technique. They don't use the map as a prior. Instead, what they do is take this 3D scan of the world and then take a learned approach to the problem. You'll have tens of thousands, hundreds of thousands of labels of cars, humans, and so on, and these networks will be able to pick those out. We're particularly fascinated by PixelNet, which came from some great researchers at Uber ATG, VoxelNet, which came from Apple, SPG, and our engineers have been talking a lot about Fast and Furious recently, which merges together perception, prediction, and tracking into a single network — which is pretty cool — and PointPillars, which I think came from the nuTonomy team recently.
The other thing these sorts of networks solve, which I also find particularly fascinating, is that if you use traditional clustering algorithms, what you might see is that if two people are standing next to each other, the traditional algorithm will cluster them as one object. When you're trying to move away from edge cases and build a truly self-driving car, that's a non-starter, because pedestrians are the most important thing you can probably detect, and detecting two things as one thing is not going to cut it. These networks are very, very good at understanding the features and perspectives of humans even if they are in crowds, and that then helps all your stack downstream. If you have accurate perception information about objects in and around you, your predictions are much better, your tracking is much better, and ultimately how you navigate the world is much safer.
Reinforcement learning, the broader ecosystem, and lessons learned
Oliver Cameron: I'm also particularly fascinated by reinforcement learning. If you've read Waymo's recent work on imitation learning, I think that's particularly cool. Another company we track quite closely, just because they do amazing stuff, is Wayve — trying to build an entirely self-driving car powered by reinforcement learning. I think about disengagements as rewards and things like that as tools to drive better performance, and also areas of learned behavior planning — ultimately fusing rules of the road with more learned behaviors.
The ecosystem is an area that is thriving today. Seeing just how many folks are diving into not just the full stack but building tools and other really important parts of the stack, the maturation of sensors — not just higher-resolution lidar but things like 3D radar — we get pitched all the time from these companies, and it's clear to see there's been a rise in volume from all these great efforts.
Lessons learned. Now that I've been building Voyage for two years, and prior to that four years at Udacity, what things have I personally learned? They're not technical in nature. These all may look like clichés, but I promise you they came from lessons which were really, really painful in the moment.
Don't be intimidated. The thing that I feel happens a lot in self-driving cars is that because it started in this very academic sense — Stanford, Carnegie Mellon, and so on — it felt like to break into the industry you had to also go through that same path. You had to get a PhD in something and really go the path that was well-trodden. But I think that only takes the industry so far. It's really important that we get folks from all different backgrounds and all different industries to come contribute to this field, because if we don't, there is no driverless. It can't happen in that isolated bubble. It needs to be extended out.
Understand your limitations. This is perhaps more of a CEO lesson for myself, but when you're building out a company from one person or five people to today with forty-four folks, you cannot do everything. It's really important you build a team around you that is able to do what you used to do but do it ten times better. I probably didn't spend enough time building out that team until we had some challenges come our way.
Be proactive versus reactive. It's really crucial when you're building a company to try and predict what's going to happen next, because if you're reactive you're constantly two steps behind what other folks are doing.
Let go. I think a lot of folks perhaps overstay their welcome in certain areas of the company when they should just say, "Okay, I've got experts now. I can just step aside and let those folks do what they do best."
And speaking of which, hire the best. It's really easy when all this pressure is on and you're building a company to kind of sacrifice when it comes to your culture and your hiring. It's really crucial that you find folks who are not just the best in their field but are the best match for your company.
And always be curious. One of the things we believe in at Voyage is that it's important that knowledge is not isolated to just one person — that knowledge should be spread throughout the company. Because even though it may feel like over-sharing or over-communicating, what that knowledge may mean for someone with a particularly unique background is that they may do something incredibly cool with it. They may build something that totally transforms our company.
Q&A
Lex Fridman: How did you identify retirement communities as the target market to prioritize?
Oliver Cameron: So retirement communities — there's actually a really long story about this. When we were starting Voyage, Sebastian Thrun was very helpful in helping us start the company. As kind of naive founders, we were like, "Well, let's just take this El Camino thing and put it everywhere else that looks like El Camino and just do that over and over again." But he cautioned against that, and very wisely so, because again you're nothing special compared to the other self-driving car companies out there by doing so.
In 2009, he had really advocated to Google leadership — Larry Page and others — that retirement communities for self-driving cars might just be the best way for Google to go about deploying their self-driving cars. I can understand why the Google folks were like, "Well, we're Google. We're not just about retirement communities — we're about the world. Level 5 or nothing." So he got some pushback. But he did some research in that process and met some folks. So when we were starting, he said, "You've got to check out these retirement communities." So we did — we went to visit — and eventually we got there. We wouldn't have gotten to that point without Sebastian pushing for that.
Audience member: Following up on the retirement communities question — do you ever think about the other collateral issues, especially that residents would have to get into a car? How exactly would they interface? If somebody wants to call for a car to come to wherever they are and they have to move from point A to point B, how do you plan to address all these issues that are very germane — it's not just a vehicle moving on its own?
Oliver Cameron: It's a good question. The way we think about this is that today we've intentionally focused on a segment of the market called active adult communities. These folks tend to be able to go into their own cars or into a taxi, open the door, sit down, without the need for any assistance. But they may have vision issues or other issues that prevent them from driving.
There is that other market you're talking about — folks that just need that helping hand towards getting to the car. One of our beliefs as a company is that the senior market is surprisingly large, and what that means to us is that we think we can own it. We think we can be that company that any senior citizen in that situation thinks, "Oh, I should call Voyage because I need to get from point A to point B," instead of thinking about Waymo or Cruise or any of the folks going after the general big market. They'll think about Voyage because we'll deliver a product to them that is meant for those folks, designed for their use cases.
It may be that if they're going on a long trip — say they're traveling 50 miles — the first mile and the last mile of that trip may involve a human helping them into the car and then dropping that human off somewhere else to go do that all over again. It may involve robots that help people from their cars. We've heard from folks at Toyota — they're building back-carrying robots and other things that may assist seniors in getting to and from vehicles. That's why that market is particularly exciting — because it feels like you can deliver these tailored products that would enable us to be the market leader. But today we focus on active adults.
Audience member: Can you talk a little bit about how you determined your final sensor suite?
Oliver Cameron: The truth is it's never final. We think about generations of vehicles. We have a first-generation vehicle which was a Ford Fusion — had a single Velodyne HDL-64 and a bunch of cameras and radar. We set some milestones based on that vehicle, accomplished those milestones, and then once we reached the max of what we were able to do with that vehicle, we said we need to bring on a second-generation vehicle. So we did that, and we said, "Okay, we have these certain goals in mind which are pretty lofty and pretty ambitious. We need incredible range and incredible resolution." What we've discovered is that in our particular communities, going at the speeds we're going at, radar isn't particularly useful. So we don't have radar on our second-generation vehicle.
But I'm sure that when we go to a third-generation vehicle there'll be other driving factors. We work backwards from the milestone to say what do we need on this vehicle — maybe cost in the third-generation vehicle. We may say we need a more affordable sensor suite than what exists in our second-generation vehicle. But they're driven by technical requirements, and that means we're able to really marry the two with the vehicle.
Audience member: I was curious — when you showed the student-led content, one of the students had developed a traffic light sensor, and you showed later that you were getting student input for deep learning models for steering angle prediction. I was wondering what your system architecture looks like in terms of the kinds of perception you take in, how modular it is, and to what extent deep learning algorithms have played a part in those different parts of the system.
Oliver Cameron: It's a good question. I really encourage folks to get familiar with ROS. ROS has always been this kind of playground for roboticists of all different types to be able to try things out on robots. ROS 1 is particularly notorious for hobbyist types of projects, but it's not meant for production. ROS 2, which is in kind of an alpha release state, is definitely meant for more production-oriented things.
The reason I mention ROS is because it has this awesome architecture which lets you plug and play what they call nodes and experiment with different approaches to the problem. For example, what was running — that deep learning model predicting steering angles — effectively replaced our more rules-based planner and perception engine. We just plugged the output of the steering angle straight to our controller to actuate the vehicle. ROS is particularly good at those sorts of architectures, and it's all open source, so you can do some cool stuff with it.
Audience member: Can you tell us how you handle the liability and insurance for passengers in your vehicles?
Oliver Cameron: We have a pretty cool deal with a company called Intact Insurance. The idea is that insurance in the autonomous age is going to be very different than insurance today for human drivers, because there are different risk assessments and whatnot. One of the ways we're able to prove to these insurers that we're good at what we do is actually sending them data. We send them data from our cars as we drive, showing that as we move through the world we accurately detected things and planned around things and all that good stuff. They use that data to inform our insurance rates.
I think the future of AV insurance will be on similar lines but perhaps more extreme — where, for example, the rates will change depending on the complexity of the environment. If we're just driving down a completely straight road with zero vehicles around us, our insurance rate should be super low. But if we enter a city center with thousands of people and cars and all that crazy stuff, our insurance rates should rise almost instantaneously. There's a lot of room for innovation there.
Audience member: Did you have any problems onboarding people initially when they were skeptical or scared? And what are the major missing pieces in computer vision to achieve Level 4 self-driving?
Oliver Cameron: So one of the more interesting insights we had about retirees is that in my kind of naive state back in 2016, my general feeling was that retirement communities might not be the first to adopt this technology — they may be slower to adopt new technology, might be scared of it, all those sorts of things. To validate that, I went to talk to some senior citizens. I talked to my own grandma — she hates self-driving cars — so that didn't seem like a good sign. But when I went to talk to folks in these sorts of locations, the really interesting thing we learned is that with traditional consumer software or devices, yes, there is definitely a lag in adoption with senior citizens. That's proven in many studies. Senior citizens are slower to adopt Facebook, Instagram, WhatsApp, all those sorts of things. But that's because they have these very well-defined processes that they've had for most of their lives. Instead of using Facebook, they call someone up and have a conversation about their day. They don't share a picture on Instagram — they physically mail a picture. To change that behavior is tough because it's fundamentally different from what they're used to.
But the difference between that and a self-driving car is that our experience is no different from the car they're used to. It just turns out it's being driven differently. They see a car — it's a similar form factor to what they're used to. They open it, they sit in the back seat. Okay, there's a button I have to press to say go, but it's pretty similar to what I'm used to. I don't have to learn a new behavior. I don't have to change something I'm used to. That was our first learning.
And then also, they actually really don't care too much that it's autonomous. They're quite curious and enthusiastic about the technology when I'm in the car and want to tell them about lidar and deep learning and perception — and they just don't want to hear any of that stuff. It kind of dawned on me that the reason is because what senior citizens have witnessed over their lifetimes is far more dramatic than what I have. Our oldest passenger was 93, and she told me a story about how when she was very young she remembers literally moving on an almost daily basis in a horse and cart. So when you talk about self-driving cars to those folks, they couldn't care less, because between that period and today they've seen the birth of flight, planes everywhere, car proliferation, scooters, subway systems. A self-driving car to them is like, "Oh, that's cool — what I just want is for it to move me." That was our biggest learning.
On the computer vision question — what needs to happen between now and Level 4 — the Holy Grail is this: if you had perfect perception, self-driving cars are solved. If we knew every object that was on the road in and around us within a reasonable distance, self-driving cars are solved. False positives are accepted today, which I think is good, but you really want to minimize false negatives. You want zero false negatives in the world. I think that's why we still have a tiny bit of work to do, because when you think about the reason for a test driver being in the vehicle — perception feeds everything downstream. If you miss an object or misidentify an object, that effect causes the whole stack downstream to become quite chaotic. That's why I'm excited about all those networks I talked about.
One of the other things we believe helps us minimize false negatives to a near-nonexistent state is that we band together multiple networks. We don't just rely on a single layer of perception. Different networks have different strengths — for example, VoxelNet is particularly good at pedestrians, but PixelNet is not so great at pedestrians because it's from a bird's-eye view where pedestrians are quite thin. So let's band those two networks together, and let's also band together some more traditional computer vision algorithms that may not be processed on the entire 360-degree scan but may be processed on a small sample — maybe at the front of the vehicle, for example. There are just lots of little bits and pieces like that to go through to minimize the worst-case scenario, which is a false negative. But it's clear when you see Waymo and others that they feel very, very close to that.
Audience member: You mentioned that weather was one of the main reasons retirement communities were a great place to start. Can you talk about hurricanes?
Oliver Cameron: I got a question recently about this — okay, in the event of a hurricane, let's not talk about the technology for a second. We've all seen those pictures of people getting on the freeways and trying to get out of the path of a hurricane. How is that going to work in a world where self-driving cars are everywhere and personally driven vehicles may be a smaller set? I don't quite have an answer to that yet, but I think it's an interesting thought problem.
From a technology perspective, the really important part of weather handling is remote operation. All of our vehicles have a cellular connection, and each of those vehicles is connected to a remote operator who's sitting in somewhat close proximity to that vehicle. That remote operator has a few jobs. One is to just ensure the safe operation of the vehicle — make sure that vehicle is doing as it's intended to do. Another is to make sure that the operational domain we are currently operating in is the one that's designed for. All these different camera feeds are being live-streamed to this remote operator, and if there is a sudden downpour of rain, that remote operator has the ability to bring that vehicle to a safe stop until that rain shower disappears — or hurricane, whatever it may be.
There are also companies I've been pitched by that are building weather forecasting on a scale that isn't really used today — really microclimates. Thinking about just this small subsection of The Villages, predicting and understanding exact weather within those regions and then having webhooks to tell Voyage that something is about to happen. There's a lot of cool stuff happening there. But remote operation is currently the eyes and ears of all our cars to prevent that sort of issue.