Interview with Larry Matthies

An interview conducted by Peter Asaro, June 3 2011

Larry Matthies: Okay, I was born in Saskatchewan, in the town of North Battleford in Canada. I was the oldest of five boys, so I did my education up through bachelor’s degree in Saskatchewan. Then I went to the University of Waterloo for a master’s in computer science and then Carnegie Mellon for a Ph.D. in computer science.

Q: And how did you get interested in computer science?

Larry Matthies: Actually, when I was in grade 12, I took the grade 11 level computer science class and up until that time I was interested in chemistry. And I just enjoyed programming, I found it a fun challenge, decided to try that in the university and never looked back.

Q: Who did you work with at Carnegie Mellon?

Larry Matthies: I started with Hans Moravec but he was not in the computer science department per se, he was in the robotics institute. So I needed to have a computer science professor as a co-advisor which was Takeo Kanade and then along the way I switched from Hans Moravec to Steve Schaffer but still with Takeo as a co-advisor. So in fact, Takeo was really my main mentor all the way through.

Q: What was your first sort of project with… as a graduate student?

Larry Matthies: I was working in Hans Moravec’s mobile robot lab on stereovision for autonomous navigation of ground robots. So there was a problem of trying to estimate the motion of the robot as it drove through the environment by using the imagery to track features in the scene and so I made improvements to that algorithm. And at the time I didn’t actually believe that that was the most important thing to be doing but it’s turned out 30 years later that that’s now called visual odometry and it’s actually very valuable and very widely used and many people have worked on the problem since.

Q: How’d you get interested in vision?

Larry Matthies: Actually as an undergrad, I got interested in artificial intelligence and then for my master’s degree I ended up studying computer graphics. I still wanted to do artificial intelligence and I thought the best places to do that were the big schools in the US that were doing that, so that’s how I ended up at Carnegie Mellon. In my first year at CMU, I got a little disillusioned with artificial intelligence but computer vision still had some of the same appeal but it had also some of the appeal that I’d found in computer graphics. So I tried that and in particular, Hans was working on computer vision for robots, so that was exciting and I enjoyed

it. So that appealed to me because it had some of the same visual gratification you get from computer graphics but it had a strong theoretical foundation which I was looking for as well.

Q: What disillusioned you about AI?

Larry Matthies: Pardon me?

Q: What about AI disillusioned you?

Larry Matthies: Well, at the time, you know, I don’t wanna be unfair to the field but at the time AI was kind of a collection of problems that we didn’t know how to solve yet and once we knew how to solve them they were no longer considered AI. And the very fact that we didn’t know how to solve them meant we didn’t have a good theoretical foundation in some cases, at least not a strong mathematical foundation. And so at least the particular things I had been looking at at the time I was not happy with the degree of theoretical basis to them. So in computer vision, it’s really founded in strongly in physics and mathematics and you know, much of the curriculum you would find in electrical engineering for example. It’s pretty cross-disciplinary in that respect but it draws on engineering and applied math disciplines quite a bit.

Q: So what year did you get to Carnegie Mellon and what kind of systems were you working with as far as the vision of computers?

Larry Matthies: I got there in the fall of 1981 and I started working on vision in the fall of 1982. When you say what kind of systems?

Q: Like what was the technology like that you were working with?

Larry Matthies: Well, as a robot to do the vision work with, we had something that looked like a little tricycle, about the size of a tricycle. And we were working with computers that – I don’t remember exactly what we had then but I think the VAX 1170 might’ve been about the right timeframe – And they didn’t have a lot of memory so there’d been a lot of work, you know, in years just before then in writing vision software packages that could swap images in and out of memory so that you could actually process something in reasonable amounts of time. So things were a lot slower. There were central mainframes. The PDP 10 that the department had that, I don’t think we did much of the research on but you know, everybody had terminals. We didn’t have, you know, your own desktop computer in those days. The kinds of cameras that we used, this was before solid state cameras really. So, I think Vitacom’s, basically a much lower quality imaging device than we have now. You know, I work on vision for robotics, so it’s multi-

sensory problem so you’re combining the vision sensors with inertial sensors. In those days, inertial sensors were big and heavy and very expensive. So you basically didn’t use anything but the cameras. That’s kind of a quick overview I guess.

Q: And where did you go then after Carnegie Mellon?

Larry Matthies: After Carnegie Mellon, I came here. So I came here in ’89, I’ve been here ever since. And I did a lot of interviews when I graduated, 12-14, including universities, federal labs and companies and what I was enjoying at CMU was robotics for outdoor applications. So I was looking for a place to do that and this was basically the best opportunity to do that. I wasn’t necessarily interested in space per se, but once you get here, it’s nice to have a ringside seat on the space program and it’s certainly exciting to work on space applications. It so happened that I dropped into a perfect niche for me. The research I’d been doing was exactly what was needed for Mars rovers at the time and it’s gone to Mars since, so it’s really panned out well in that respect.

Q: I’m curious, since you mentioned you were in field robotics, at the time at CMU did they have a field robotics lab or what kinds of labs do they have, in particular, the ones you worked in?

Larry Matthies: So I was in the computer science department. I think the robotics institute was founded a year or two before I got there, ’79 or ’80. There were some smaller labs so Hans Moravec was running the mobile robot lab. This was before the field robotics center, before the term field robotics had been popularized and Red Whittaker who started the Field Robotics Center and I think there’s two institutions there but anyway, he was still a professor of civil engineering and I think while I was there was when the Three Mile Island incident happened and that was a big thing that propelled him along in robotics and the DARPA Autonomous Land Vehicle Program happened while I was there and I think Red was involved in building up testbed vehicles for that. So basically Red grew in mobile robotics while I was there and I think FRC got set up while I was there, toward the end of my time there or maybe just after, I can’t remember now.

Q: And is that when the term started to become popular?

Larry Matthies: It probably started then. I think it grew probably through the ’90s, as, you know, Red and others were addressing field-type applications in space, military and mining and agriculture.

Q: What were the big challenges for field robotics as opposed to just mobile robotics?

Larry Matthies: Well, there’s basically a contrast of people working indoors on level floors where there’s only two types of terrain. There’s smooth level floor and then there’s discrete obstacles. You can represent the world in two dimensions and do all the path planning in two dimensions. Weather’s always good, the lighting’s always good, whereas in field applications, you’ve got uneven terrain, you’ve got ground cover, you’ve got different soil types that some of them are drivable, some of them are not. You’ve got a variety of lighting conditions, you’ve got a variety of weather conditions, you’ve got dust that can foul the sensors or you can’t see through. You have to represent the world at least in 2.5D if not in 3D. You have to make inferences about physical properties of the world that you don’t have to bother with indoors… it’s a more complex problem. It’s more expensive to work in that domain. You know, typically, you need a bigger robot, you need just a lot more logistics and infrastructure to do field applications. So, it’s not studied as much or fewer places have the wherewithal to do that.

Q: And how did those play out in your first projects at JPL?

Larry Matthies: When I got here, NASA and JPL were working on a program called Mars Rover Sample Return. So the acronym was MRSR, the vision was to do a sample return from Mars. They had a fair bit of money. Actually, CMU and JPL were funded under that program so JPL was working on wheeled robots with stereovision. We had quite a big robot that was… I’m not exactly sure of the size but it was probably 15-20 feet long and to the top of its camera mast it was probably 10 feet high. And Carnegie Mellon was working on alternatives, basically a legged vehicle using laser range finders. So we had quite a bit of money and a big team to do field work and the field work that we did was… there’s a dry river wash next to JPL called Arroyo Seco so we would load up the vehicle on the trailer and go out there and do test runs. So… I don’t know what the budgets were but that takes a lot of financing to run that kind of experiment. Compared to working indoors, you know, you buy a little robot two-feet in diameter and you know, work with one or two grad students and that’s all you need. So, you know, very much a difference in scale between what you could do in a university or at least most universities and what you could do at a federal lab or, you know one or two companies like Martin Marietta that were working on that at the time.

Q: What were the challenges of the vision system for the sample return mission?

Larry Matthies: The algorithms were very, very slow. So, you have to have 3D perception since that’s how you could, you know, determine the… where the terrain was smooth enough to drive and where the obstacles were and so we were workin g on that with stereovision and [inaudible] was working on that with laser range finders. The stereo algorithms were very slow so one of my first big breakthroughs was designing a new stereovision algorithm that had much better computational complexity and basically reduce the runtime from half an hour per image to

six seconds per image. So that was a huge breakthrough at the time and feasibility of outdoor robot navigation.

Q: Did you work a lot with CMU still at the time or?

Larry Matthies: Not directly. You know, we were funded by the same sponsor, working on the same kind of mission but we were working independently. Since then I’ve had collaborations, you know, in the 22 years I’ve been here, had collaborations with CMU people a number of times but when I first got here, it was not directly collaborative.

Q: What were some of the later collaborations?

Larry Matthies: I may not get these in the right order, or at least chronological order but CMU managed to get funded by NASA to set up an organization they called at the time the, I think it was the NASA Robotics Engineering Consortium. It’s now called the National Robotics Engineering Consortium and so that was a vehicle to try to channel NASA-funded research into commercial applications and there was a mechanism where while you could propose joint projects, you know, between say us, CMU and the company that would try to take a piece of NASA technology and spin it into personalization. So, I worked with Sanjiv Singh on a few things on that and then along the way the Army Research Lab has funded a program called the Robotics Collaborative Technology Alliance and so I’ve collaborated a little bit with Martial Hebert along the way on that. Al Kelly was a grad student… let me think. DARPA had a program called Demo II. So, we and CMU were part of the Demo II program working on different parts of the problem and so we collaborated a little bit there and I hired Al Kelly for one or two summers as a student to bring his path planning capabilities and merge it with our vision capabilities here. And after that, that was taken to… I don’t remember if it was still Martin Marietta then or Lockheed Martin but taken to Denver to put on the robots for Demo II. We worked some with Tony Stentz who does a lot of path planning work as well. I’m probably forgetting people but those for me personally have been the most direct and the most substantial collaborations.

Q: So we heard a little bit of history about the sample return sort of getting cancelled and then evolving into the other Mars builders… So, were you working on those and were any of those systems that you worked on supportive to that?

Larry Matthies: Well so the Mars Rover Sample Return program, I was working on when I first got here and I don’t remember exactly when that got cancelled but it was fairly shortly after I got here. You know, within the first two or three years. It was simply unaffordable. And then the pendulum swung to the other extreme of looking at micro robots and that was far, far less

capable but also far, far cheaper. And so we managed to sell this mission called Mars Pathfinder which was the first lander that NASA-JPL did with airbags and it was… you know, every mission has to have some science but it was as much I think a technology test and demonstration mission as it was a science mission and the same was true with the rover. So, the rover was a technology test and demonstration. It had to have some science. You know, it was fairly minimal science and it was cheap. I think it was 25 million dollars just to do the rover. So, I was involved in that a little bit. I didn’t work directly as part of the mission team but it had a sensor called a Light Striper for detecting obstacles that was having some performance problems. And just coincidentally in parallel I was working on a next generation Light Striper and so I got involved in essentially debugging what was wrong with the one that was headed for Mars and so there turned out to be a problem with the optical design that I was able to isolate and then we got an optical engineer to come in and redesign it, so.

Q: Could you tell us a little bit about what a micro robot is like compared to the previous idea?

Larry Matthies: Well, it’s behind me, I suppose we don’t wanna turn the camera, but. So the Mars pathfinder rover which was named Sojourner, weighed about 11 kilograms. I’m not sure if that’s the exact number. The Rovers that had been looked at for Mars Rover Sample Return were probably, you know, close to a ton. I don’t know the exact number but you know, we’re talking two orders of magnitude different in weight and whereas the Sojourner rover never went more than six or eight or ten meters from the lander. The original concept for the sample return rover was to go, I don’t even know, you know, hundreds of kilometers and be a self-contained mission, whereas Sojourner was dependent on the lander and the cameras on the lander and the communication system on the lander to be told what to do and to communicate with Earth. But nevertheless, that mission did firmly convince the whole Mars science community of the importance of mobility. So, essentially, the vision of how to explore Mars was never the same after pathfinder because now everybody realized you’ve gotta have mobility because there can be stuff just out of reach from the lander that is where the big discoveries are and what we found in the Mars exploration rover mission which those two rovers have covered, I don’t remember off the cuff but you know, it’s an order of 20 kilometers in seven years. You know, we’re several kilometers away from where they landed and that’s where the big discoveries happened, so. I guess I’m straying from the original question but you know, the difference between the micro rovers at 10-11 kilograms and staying with inside of the lander versus sample return at, let’s say… well, the Mars science lab mission that we’re working on now, the next rover, I think it weighs 900 kilograms and it’s supposed to drive an order of 20 kilometers… that’s the difference.

Q: And were you involved on the Mars Explorer Program?

Larry Matthies: The Mars Exploration Rovers? Yeah, so my whole career, I’ve been in research, so I’ve never worked on the flight software or been formally part of the mission team

but key elements of, you know, the capabilities in that mission came from either me personally or my group. So, those rovers use stereovision to detect obstacles, that basically is the outcome of work I did in my first year here. They use visual odometry to help keep track of where they are and more importantly to detect right when they’re slipping. That basically grew out of my Ph.D. thesis research. The path planning software that’s on there, I didn’t have any personal hand in developing but that was done by somebody in my group and I basically convinced him that he should as a career move, get involved in the mission. Later on there were some things that were added after they already landed as kind of the mission software uploads. So, there’s software on board that does basic image processing to detect when a dust devil goes by. That software was developed by somebody in my group. That mission halfway through developing the spacecraft, you know, like two years before launch, they realized that they had underestimated what the wind velocities might be and there was a concern that the horizontal velocity of the lander might be so large that the airbags might rip on impact. So, you know, some people came to us… it turned out that there were retrorockets on board that could be used to reduce the horizontal velocity if you knew what that velocity was. So then the problem became can we get a velocity sensor. And the traditional way to do that is with a Doppler radar but there wasn’t time or money to put that into the mission at that point but coincidentally, they had designed in a camera that they weren’t using and so the question was could they put that camera back in and aim it down. It was originally designed to be on the rover looking up to trap the sun but could we aim it down instead of up. Track features on the surface during the sand and estimate horizontal velocity. So, we mounted a… you know, a crash effort, [laughs] pun intended, to try to develop that capability and that was successful. And that was used for both landings and for the sphere of rover, from the… after the fact reconstruction, it may well have saved that mission because it did lead to the retrorockets being fired, kind of reduced the velocity by about a factor of two. So for the MER mission… I also had a hand in helping keep track of where it is… besides this visual odometry thing that operates kind of over short distances. I work with a photogrammetrist named Ron Lee at Ohio State University on mapping and so there’s bundle adjustment software that Ron’s group has developed that is used to keep track of the multi-kilometer traverse and is part of the whole mapping package that Ron’s group has developed to generate terrain maps as the rover goes along. I think that’s pretty much what I did in that mission. Basically, most of the autonomous navigation capabilities in that mission came out of my group.

Q: What were some of the most important breakthroughs that you feel you’ve had?

Larry Matthies: Well, realtime stereovision. So, in the 1980’s, the dominant paradigm was to try to do stereovision by matching edges because that was thought to be much faster and when I got here I could see that for off-road navigation, you know, in desert-like terrains, edges was just the wrong way to do it, so you really needed to use what was the alternative paradigm which was area-based cross correlation. So finding an algorithm that was fast enough for real time was something I did and that was a key breakthrough. This whole visual odometry business was something actually I did at CMU before I left there. I think that was not… not at the time, so I developed the first accurate visual odometry algorithm, even though we weren’t calling it visual

odometry then and you know, at the time, you know, that was a good advance over what had been done previously but it wasn’t that important. I think, you know, over the years since then we’ve recognized it is a very important capability. Along with some collaborators, some I advised with Takeo and Rick Szeliski, we did a well-known paper that was one of the first to show how to process images in an incremental fashion to build 3D models, so, you know, much better techniques of course have emerged but that was one of the first to show that you could incrementally build 3D models from image sequences. I’ve been… well the thing I just mentioned about vision systems for Mars landers, so trying to rule out a velocity estimation during landing… when I got here in the mid ’90s, I managed to get funding to work on landing hazard detection. So, essentially I built a team that has been working on various approaches to landing hazard detection with imagery and LIDARs. None of that has gone to Mars yet but over the years we’ve built up quite a bit of capability to be able to do that and I believe that some day that will go to Mars. Precision landing has been another important thing, especially for Mars and so more or less the same team that I built up here has been working on how do you recognize landmarks during descent, so that you can reduce the positional uncertainty from say, plus or minus 20 kilometers, to plus or minus 100 meters. And that would make, you know, far more of the planet accessible and if we could land, you know, 100 meters from interesting terrain instead of, you know, 20 meters from interesting terrain or 20 kilometers from interesting terrain. You know, now you can get there in a day instead of getting there in months. Other things that I’ve worked on for space application include two explorer asteroids that have a lot of craters. We’ve developed algorithms that can use the craters to navigate a space craft around asteroids. JPL has been working on balloon missions for Titan which is the big moon of Saturn that has a thick atmosphere. There’s a question of, how do you know where the balloon is? So we’ve been working on, you know, taking algorithms… the fundamental algorithms that were developed by others and applying them in this domain so that you could take imagery from an ordinary camera on board and register that to radar data taken from an orbiter. So that kind of cross registration of different sensor modalities can tell us where the balloon is. And then… so that’s the NASA side. I’ve been working on defense robotics for, you know, more than 20 years and that grad school most of my work was funded by the defense department. So, you know, as a funding strategy, I’ve tried to find things to work on that had relevance to both NASA and other agencies. So, off-road navigation fits, so the focus on terrain understanding. So even while I was at grad school I collaborated with Hans Moravec and Alberto Elfes on… they had a fundamental breakthrough in an algorithm called occupancy grids for world model representation and Alberto and I wrote the first paper on combining data from multiple sensors in occupancy grids, so that was sonar and stereo. And then with funding from DARPA and the Army Research Lab, I was the first person to use near-infrared imagery for terrain classification of vegetation for off-road navigation. The first person to recognize that you could use LIDAR to classify tall vegetation which can be difficult to discriminate, you know, you can drive through it but it can be difficult to discriminate that from a real obstacle that you have to go around. There’s something that we call negative obstacles in this business, so basically, any hole in the ground. They’re very hard to see. They’re still very hard to see. You know, the only good way that’s ever been discovered to find those is from the air. But you can’t always send an aircraft ahead of you. So, along the way, I discovered that there’s just kind of a physical property in the way heat transfer works… that the

interior of a negative obstacle tends to keep itself warm. That has not completely solved the problem but it was an interesting advance that makes it somewhat easier to detect those than it would be otherwise. In looking for things that I could work on that other people weren’t doing… water bodies in off-road navigation that are hazards to navigation. We’ve… myself and collaborators here have put a lot of effort into how do you detect water bodies from a ground robot and, you know, we’ve made a lot of progress towards solving that problem. So that’s… you know, it doesn’t have wide impact but it’s something fairly distinctive that we’ve done. I think that’s a fair selection of highlights.

Q: Has optic flow been a concept that you’ve used in some of your visual odometry?

Larry Matthies: Certainly have used it. So, what we do in visual odometry, tracking features through the image sequence is, you know, there are many different flavors of optical flow, so that’s a flavor of optical flow. The collaboration I had with Takeo Kanade and Rick Szeliski was essentially using a flavor of optical flow. In my work on ground robotics, I focused on stereovision for 3D perception and motion estimation, whereas there have been other people in the community that have tried to use a single camera and so the problem with a single camera when you’re driving forward is, you don’t have depth perception directly in front of you because you have no visual parallax. So, optical flow per se, has not been as important part of my work because of that but in the last few years I’ve started to work on micro air vehicles under funding from the Army. Which have very important applications in military reconnaissance and, you know, applications in the civil arena and inspecting bridges and so forth but is also very relevant to things for NASA. So if you’re trying to explore a comet or an asteroid, or even fly a balloon in the atmosphere of Titan or Venus, you know, there are very different scale of machine. But the vision problems have a lot in common. Or if you wanna have a micro-inspector like a little ball, you know, a foot or two in diameter, on a human spacecraft like the space station or if in Apollo 13 we had a micro-inspector that could’ve gone outside and had a look, that’s a lot – has a lot in common with the microware vehicle for earth applications. So, there, you can’t really fit a stereo system and expect to see very far. So optical flow is quite important in those domains. So we’re using it. We haven’t developed new approaches to optical flow, but we’re using it for perception for those kinds of systems.

Q: So for these various, like, near-infrared and LIDAR and vegetation detection, what were the robots that you put these systems on [inaudible] demonstrations [inaudible]?

Larry Matthies: Some of that work was done under the DARPA Demo II program, which was using Humvees that were turned into robots. Then there was the DARPA Tactical Mobile Robotics program which – essentially that was before iRobot became big and successful. And they developed a little tract vehicle, which initially was called the Urban Robot and later became the Packbot. So that program was the genesis of their Packbot product line and, as you probably

know, those are now used in the middle-east for bomb disposal. So we put sensors, including the well-known SICK LIDAR single axis scanner as a range finder on a Packbot and used that to classify vegetation. The near-infrared was on the Humvee. The thermal infrared, that came out of another program called PerceptOR where we were basically using all-terrain vehicles that were outfitted as robots. You know, the quad all-terrain vehicles. And General Dynamics Robotic Systems has been a big player in army robotics. They built some custom vehicles that are, I think, around 3,000 pounds. They call them experimental unmanned vehicles. So my work has been integrated on those vehicles. You know, sort of Volkswagen-size vehicle. What else? I think those are the highlights of that.

Q: Can you tell us a little bit about the organization of the groups that you’ve worked in and how they’ve changed through time, how different groups interact with each other within NASA, and what it looked like when you got here.

Larry Matthies: Okay. Well, I was originally in a robotic vehicles group run by Brian Wilcox. Somewhere along the way, the organization decided it was time to have a vision group and I became the leader of the vision group. And when that was first created, actually, there were a small number of us working in vision, which they merged with a slightly larger number of people working on star trackers. So star trackers are used in interplanetary crews to keep track of the orientation of the spacecraft. So I really was not interested in star trackers and felt that spending a lotta time on them was gonna be detrimental to the stuff that I was interested in. So I didn’t really take care of it very well and eventually management recognized that and moved that off somewhere else and left me with just the vision part. And that grew from, you know, originally a half dozen people to as many as 18 and then we, you know, had the succession of reorganizations and the group would get smaller and then it would get bigger again. And along the way, we recognized that from a, you know, a strictly, a funding strategy perspective there’s actually a lot more money in visual surveillance than there is in robotics, or at least vision for robotics. So we thought we should participate in that somewhat. So we started to get a toehold in airborne surveillance work. And then there was another reorganization where we formed a separate group that is carrying on that work and I’m focusing, again, on vision for autonomous navigation. I have avoided moving further up in management ‘cause I didn’t wanna be purely managerial. So if the question is, you know, “How has the organization evolved within JPL?,” that’s it. And then just within the last year/year and a half, you know, I felt that it was important to try to get more visibility with top management here and more appreciation by top management of the importance of unmanned systems outside of NASA. And therefore, it should be important to address those applications within JPL. So I’ve taken sort of a day a week level of responsibility to try to provide some strategic coordination of robotics for non-NASA applications across JPL.

Q: You mentioned earlier that with some of these missions, the idea that mobility is important became clear. What has been the status of robotics or of unmanned vehicles within NASA? Has it changed through time?

Larry Matthies: Well, I think NASA has been interested in surface mobility since the ‘60s with the Lunar Surveyor program. So there’s always been the interest there but I think it wasn’t until the Mars Pathfinder Mission that it was really appreciated as essential, and also that it was recognized that it was feasible. So I think there was a watershed there around 1997. And then with the Mars Exploration Rover Mission, we’ve shown that it’s not only feasible for short-range motion, it’s feasible for long-range motion and that it’s crucial to find rock outcrops that are many kilometers away from where you landed. But, you know, as the ebb and flow of research funding goes, it’s been a little bit of a hand to mouth existence because NASA – there’s been swings of the pendulum within NASA where for periods of time there is very little research funding, per se. You know, there’s very focused money or specific things needed for the next mission. But that’s another reason why it’s been essential to be, you know, having other sponsors at the same time, in order to manage the ebbs and flows of research dollars within NASA. So to maintain, you know, a steady cadre of people working on these problems so that when the opportunities came back within NASA we were still here and still equip to address them.

Q: So how does the funding work exactly? How much of it does the group have to create for itself?

Larry Matthies: Most of the funding that we get, we have to raise ourselves. So there is some funding that comes through what’s called Focused Technology Programs that’s, you know, assigned to JPL to solve specific problems for specific missions that will kind of trickle down through management to us. But at the, you know, more basic and applied research level, almost all the funding that we get is competitively ordered. So I liken it to – you know, in universities there’s a concept of a soft money position, which is non-tenure track funded by research contracts. It’s very analogous to that.

Q: So, in a sense, the size of the group also depends on the ability to find that kind of… .

Larry Matthies: To raise money.

Q: Okay.

Larry Matthies: Right. Yes.

Q: And how much does the funding shape what you decide to do research on versus how much is driven by your interest?

Larry Matthies: Well, they go hand-in-hand. And, you know, you have to be pragmatic and realize that if you have to have a charge number, you know, and a contract behind that charge number, you have to work on things that people will fund. But that’s a bit of a chicken and egg thing. If you bring an interesting new idea to a sponsor then sometimes they will apply some funding to it. So you need to sorta be working under the radar on things that you believe in even if, you know, the country is not ready to apply funding to them. And there are ways to get funding for that at, you know, a small level. So JPL has some internal seed funding pots you can apply to. You can collaborate with people elsewhere. So there could be different agencies like the National Science Foundation that fund basic research that maybe NASA or the Defense Department aren’t ready to fund; but if you collaborate with those people, you can kinda get the ball rolling and then move those ideas into other agencies.

Q: And have you had much interaction with companies as well?

Larry Matthies: Yeah. So the Robotics Collaborate Technology Alliance that the army’s been funding, we’ve always been part of a team, led by companies. So General Dynamics. The DARPA Demo II program was led by Martin Marietta. We were – I don’t remember if our money came through then. I think our money, then, came directly from DARPA but we were basically working on a team with Martin Marietta. When we were working with CMU and the NASA Robotics Engineering Consortium we worked a little bit with some companies. So Toro was interested in automation for some of their, you know, lawnmower products. We’ve had some of our vision software licensed by small businesses. So – no small businesses that’ve made it big yet, but we’ve had some of our software licensed. I actually collaborated with people here a little bit who pioneered the development of CMOS Imagers. So the cameras that are in all the cell phones these days, some of the pioneering work on that technology was done here. So I was part of a patent for on-chip block averaging for multi-resolution readout, which was part of the package of patents were licensed. That particular capability hasn’t had much impact in the market yet but you can buy it. Let me think. We’ve collaborated with other companies in other DOD funded programs. So we’re doing work for the navy now where we’re a subcontractor to other companies doing unmanned boats for the navy. There’s probably others that I’m not remembering. So iRobot, you know, we collaborated with iRobot in some programs. We’re collaborating a little bit with iRobot’s competitor, QinetiQ. In the robotics technology lines these days, QinetiQ’s part of that team. We work for Boston Dynamics on some of the DARPA programs for legged squad support system, basically for legged robots. We provide vision systems for legged robots for Boston Dynamics.

Q: So what do you see as the big outstanding problems facing field robotics over the next few years?

Larry Matthies: Well, one of the things that we’ve been working on, which there’s been progress – I don’t think it’s done but for many years, all of robotics research was done in static environments. So the problem was hard enough when nothing else was moving that you didn’t wanna have anything moving in the environment. ‘Cause we just didn’t have the perception capability to understand that yet, or the path planning capability to reason about things moving in the environment. And a lot of that was just driven by limitations in computational power. So we’ve been working on the perception and representational and planning capabilities to operate in dynamic environments. For Mars there is still an issue of understanding the difference between hard soil and soft soil. So the rovers have gotten stuck in soft soil. And there are problems yet to be solved in detecting that before it happens and making sure we get out of it. Back on earth, you know, in the mainstream computer vision community, there’s been a lot of work on object recognition. You know, that’s a dominant theme in computer vision these days. That will play into mobile robotics. It hasn’t been as significant yet but it’s going to become important. As we get the more basic problems of safe mobility solved then you need to do more purpose of mobility and that means you’ve gotta understand things around you as more than just 3D structures. You’ve gotta understand what they are and what they’re for. So even just, you know, going through a door, helps to understand what doors are in a semantic sense not just in a geometric sense. And there’s a growing trend to work on mobile manipulations, so robots with arms. So now you’re combining all of the problems that come with manipulation, including object recognition, with the problems of mobility. So you know, if you have arms on a mobile base, now you have to reason about the degrees of freedom of the arms and the mobile base. Let’s see. I think that’s fairly good.

Q: What do you think is the future of robotics in NASA?

Larry Matthies: In NASA?

Q: Hmm… hmm.

Larry Matthies: Well, the holy grail for Mars is sample return so rovers would be part of that. Right now for Mars we’re limited to terrain that you can go to with a wheeled vehicle. But some of the more interesting science may be in terrain that you cannot access that way. So we’re looking at robot systems that can, say, rappel down cliffs. NASA would like to enable sample return from comets because comets may, you know, be where some of the chemistry arose that gave rise to life. And, you know, an unmanned spacecraft is not the same as robots that you see, you know, roaming around on earth but they have many of the same requirements for autonomy. So free flying spacecraft that can operate in close proximity to other objects, whether those

objects are planetary bodies, like comets, asteroids, or moons, or other spacecraft. So like a micro-inspector. And then long-term for NASA, I believe that there will be value to robots for satellite servicing. So this has been a debate, you know, in the commercial world, “Is that really economically viable?” And just within the last six months, I think, there was announcement that for the first time a big satellite operator has awarded a contract, in this case, to a Canadian company to do satellite servicing. But what’s made that economically viable is that this is for communications and Intelsat has many satellites, essentially in geostationary orbits, on the equator. And so you can go and refuel those with one mission that just kind of drifts along the equator, servicing many in one mission. So that becomes an economically viable business model. But for NASA – so NASA spent, you know, $5 billion or more on a mission called the James Web Space Telescope, which is going to be on the far side of the moon. And it is not designed to be serviceable. And I really know very little about the mission but I have to wonder, if we could do robot servicing, could we do a better job of great observatories in space and perhaps make them more affordable because they don’t have to be so failsafe upfront. You know, servicing missions aren’t cheap but, you know, if you can spend $500 million on a servicing mission as opposed to, you know, throw another one and a half billion in to greater reliability so you don’t have to service, maybe it’s a good idea. And we’ve seen that with the Hubble. Right? We extended the life of the Hubble many times through servicing missions.

Q: There are, of course, autonomous robots and then there’s also a lot of telerobotics at NASA. Have you worked on any of the telerobotics and how do you see those two working together or against each other or whatever you would call it?

Larry Matthies: I haven’t had much involvement in telerobotics. You know, there was a lotta work on that when I first got here but I chose to work on rovers. I don’t see that as competition. It’s a matter of an evolution and so you do telerobotics where it’s feasible and where autonomy is not yet feasible. And you do autonomy where that’s the only possible way to do it and, therefore, you’ve invented a way to do it. So you can do telerobotics in Earth orbit or potentially on the moon because communi – basically what makes telerobotics possible is relatively low communication delays. And where it must be autonomous is where the communication delays are too long, so essentially Mars and anything beyond the moon has to be autonomous. And then, you know, there are analogies on earth. The same thing goes. So, you know, applications that you would do telerobotically on earth – you could think of surgery as a telerobotic kind of application, where you wouldn’t dream of not having a surgeon involved with today’s technology. But, at the same time, you’d much rather send the robot in to Fukushima than have to send a person. And it’s sort of unfortunate that we aren’t ready as a society to have robots that could be sent in there to do that job. But I think, you know, that’s a rare occurrences that disasters like that happen, and therefore the – even though technically I think it’s feasible, the investment hadn’t been made, you know, prior to that incident to have the robots ready to do more. So they’ve certainly taken robots in there but we could do a lot more. The capability exists to do more. It’s just not matured to the point it’s quite deployable yet.

Q: And I noticed that you were also on the editorial board of “Field Robotics,” the journal.

Larry Matthies: The journal of “Field Robotics,” yeah.

Q: And “Autonomous Robots.” So I was wondering how long you’ve been on those boards and how you’ve seen the field change while you were there, or emerge [laughs].

Larry Matthies: I’ve been on the editorial board for “Autonomous Robo” – I can’t remember how long I’ve been on either of those editorial boards. I’ve been on “Autonomous Robots” longer. I haven’t been the most active editorial board member [laughs]. You know, for “Autonomous Robots” I review papers occasionally. For “JFR” I’ve been a guest editor for special issues occasionally, particularly in space robotics. So how has the field evolved? Since I can’t remember exactly when I started those roles, I don’t really have a good answer for that.

Q: It can be more broad than just from the journal itself.

Larry Matthies: Well, we have examples of things that have actually been fielded. So, you know, we’ve been talking a lot about space. You know, prior to 1997 there had never been an autonomous planetary rover mission. And so, since then, we’ve had substantial missions for that. On earth, things like the Packbot and the Talon from QinetiQ are the first deployed military robotic systems. Since I really haven’t been that involved in commercial applications I can’t say as much about that side but I think there are applications in agriculture and mining and material movement in shipping ports and things like that where we’re seeing autonomous vehicles. You know, we’ve been seeing a lotta work in the last decade or more. So the whole robot road following thing, which, you know, got a big start in the Darpa ALD program in the mid to late ‘80s – you know, that’s gone from something that was just barely possible in a very big, very expensive research vehicle to something that actually works pretty well and is, you know, doable in an automobile now. And now we have cars that can park themselves. And so this has been driven by advances in sensing and especially by advances in computing and by reducing the cost of those things. A big driver that I believe we’re gonna see in the future is cell phones. So, you know, small cameras for cell phones, which lead to – now you’ve got images in cell phones, you need to be able to process those images. So now we’re getting very substantial image processing capabilities in cell phones. The next thing that’s gonna happen is now you’ve got two cameras in a cell phone so you can do – have a camera and do, you know, Skype on the cell phone. But there are also things coming out so you can have two cameras facing the same direction to aid with teleconferencing. So that’s a stereo system that you can use for a robot. And because it’s a cell phone it’s not only gonna have a lot of processing power, it’s gotta be very low power dissipation. So I think we’re gonna see processes develop for cell phones eventually go into small – in fact, not eventually, it’s already happening – go into small mobile robots, both on the ground and in microware vehicles. So robots, robotics by itself, is too small a market to drive

substantial innovation in sensors and computers. But commercial electronics is a huge market that is driving those and robotics will basically ride that wave in sensors and computers.

Q: Our final question. Generally, if you could give some advice to young people who are interested in robotics what would it be?

Larry Matthies: Get a solid grounding in mathematics but not just that. You need to be hands- on so you need to get a solid grounding in computing and the ability to program. It, given the multi-disciplinary nature of the field, it might actually be useful to start with an undergraduate degree in an engineering discipline and then possibly do, you know, with some computing options, and then possibly move to a computer science department in grad school. You know, I’ve talked to students recently who are grooming themselves for grad school who are getting a very diverse background in computer science and controls and elements of electrical engineering. And I’m very impressed with, you know, the level of maturity and forward thinking I see in people like that because it’s a very diverse field. And when you get to grad school, you know, you can’t be completely sure in advance which direction your interests are gonna go and which direction your opportunities are gonna go. So if you come in with that breadth of background as an undergrad, not only are you more likely to get accepted to a good place but you are more prepared for whatever opportunities arise. You know, if possible, at the high school level. So, you know, through the first program it’s good to get hands-on experience with projects at the high school level. You know, if you can manage to attach yourself as an undergrad to the faculty doing research, you know, get summer jobs or, you know, part-time winter internships – not internships really but, you know, work with faculty who are doing research, as an undergrad on whatever they need to do, to get that experience and exposure. But also remember that it’s, at least today, it’s still a fairly small field. So while it’s growing, there are more jobs in other fields than there are in robotics. And, yes, it’s exciting but it is still a small field, although I believe, is gonna grow, so.

Q: Okay. Thank you.

Larry Matthies: You’re welcome. #### End of LarryMathies01.mp4 ####