Interview with Terry Fong

An interview conducted by Selma Šabanović, June 10 2011

Q: We can start with your name and where you were born.

Terry Fong: Where I was born? Wow. Okay. Well, myMy name is Terry Fong. I was born in Pittsburgh, Pennsylvania.

Q: Did you go to school there as well?

Terry Fong: School at what age?

Q: Just quickly.

Terry Fong: Oh. Throughout. So well I grew up outside of Chicago. I went to grade school, junior high, and high school in the Chicago area. And then I went got to do my Bachelor’s and Master’s in Boston and my Ph.D. in Pittsburgh.

Q: And which school were you in Boston?

Terry Fong: I was at MIT and in Pittsburg it was Carnegie Mellon.

Q: And what major were you pursuing at MIT?

Terry Fong: MIT, I did a Bachelor’s and Master’s in aAeronautics and astronautics and then at Carnegie Mellon, I did a Ph.D. in robotics.

Q: And how did you get interested in aeronautics?

Terry Fong: Well, wWhen I was growing up, as a kid I always wanted to build and then fly airplanes. That was sort of like my dream. I thought I was actually going to be an aircraft designer until and then I went to college and I realized discovered that designing aircraft can be quite tedious. But, then , “Well, actually there’s a lot more to building aircraft than I imagined,” especially a lot more math. And I started working actually got interested inon robots and found them to be because they did things that were qmuch uite a bit more interesting on many levels., more interactive with people.

Q: So did you see any robots while you were at MIT?

Terry Fong: I did.! I mean tThere were robots all over the place, in the aeronautics department, EE, computer science. Lots of different kinds of robots too –, small, big, ones all shapes and sizesbuilt for contests and that was really kind of fun and interesting because there were so many different types.

Q: Did you participate in any contests at the time?

Terry Fong: No, actually I didn’t! and iIt’s actually kind of funny because now these days here at NASA AMES, I work with have many students and engineers who have participated lots of people that work on various outreach projects and, of course, they work with lotsin robot contests. of kids, lots of students of all ages. And I’ve certainly been a judges for robot contests. But those kinds of things butI, myself, have never ever actually participated in a robot contest.

Q: What was the first robotics project that you were involved in?

Terry Fong: The first robotics project? Boy, let me think about that. The first real projectIt was probably when I was a sophomore in college. I applied for a part-time job in the and at that point in time I was working basically part time in the MIT Space Systems Lab and startedin MIT doing work on a, I think a large underwaterneutral buoyancy robot armics.

Q: And who did you work with on that?

Terry Fong: The Space Systems Lab So that was run by in Dave Akin, who ’s lab. Dave Akin was a professor in the MIT aero/astro department at MIT at the time. HThee had a lab, the Space Systems Lab tha used t did neutral buoyancy robotics as a way of developing and testing space systems. And then he moved from MIT to the University of Maryland pretty much just after I finished up at MIT.

Q: And what kind of work did you do with the arm?

Terry Fong: So I’ve always been a software nut, and from a very early age and so for me robotics was primarily about writing code to a thing that allowed me to use software to make – well, allowed us to make cool things move.

Q: And what was the next thing that you did that was robotics related?

Terry Fong: So I actually worked in the Space Systems Lab at MIT throughout my – pretty much the rest of – when I was in undergrad and then I decided to stay at MIT for grad school. And I did my thesis, my Master’s thesis, in the Space Systems Lab and I built a large underwater robot arm. The arm It was designed to be similar in scale to the Space Station arm, but with different capabilities. iIt was an interesting system because it had a serial component. So a three-joint serial part and configuration that was serial and then a parallel – large parallel end – effector based on that was a Stewart platform. I called it So it was athe “Stewart Platform Augmented Manipulator” or SPAM. The idea was to use it to study and it was a fun project to really try to look at this idea of a very large scale manipulation that had sort of like a coarse large positioning component so a large – very large three joint arm with a very fine precision parallel thing at the end. So a large-scale, coarse/ fine positioning. – mobile arm manipulation.

Q: What was were some of the challenges in designing that system?

Terry Fong: Well, I built the arm with it was built by me and a team of undergrads. So I had freshmen, sophomores, and a couple seniors working on it with me. At the start, this and so we really had no idea how to build anything like this. This was – it was actuallysuch a pretty large, complex thing for us.. And it was interesting bBecause it was an underwater arm, and it was – which meant that we had to decide, “Well, how are we going to actually power the system?” Since Iit was meant to be very large, we decided that we could not just . It was not something that you could actually build large motors and seal it under water because we wanted it to be able to move very, very large things. So we ended up usinglooking at a combination of hydraulics – using water, not oil – and pneumatics. I In this case, it was actually pushing water, not oil or anything, and air so lots of pneumatics. And it was really interesting I think just trying to figureing out how do you actuallyto build the system and size it so that it could move large things underwater, including people.in a way that you could actually move people because the Lab actually did a lot of simulated space suite work as well underwater. And so it was trying to take all of these things. So I remember calling lots of like plumbing supply places and asking for various manifolds and otherof things and they wanted to know what I was going to use them ’m using it for. I said, “Oh, I’m building a robot.” At that time, it wasn’t really common for students to build all that common to see these research underwater robots built like that. Although tThese days, however, you see kids – even from like fourth or fifth grade building underwater robots – and they have all thesemany contests. So, it and I think it’sis a lot more common now.

Q: When did you finish your master’s degree?

Terry Fong: I finished my masters in 1990. And then after that I came out to California to work at NASA Ames. for the first time so I’ve been actually at NASA twice. The first time I was here from 1990 to 1994. at NASA AMES and tThat was a – I think a – for me it was really was a break from school for me bbecause I wasn’t sure if I wanted to do a Ph.D.. I thought it

might be interesting to I decided I wanted to come out and actually do some work in a research environment and so I came out here to NASA AMESAmes.

Q: What were you working on when you came here?

Terry Fong: Oh, boy. So NASA AMESmes has always been an interesting place to work because it’s – the work hereit has beenis very diverse. I do recall that tThe first time I was here – from 1990 to ‘94, – I worked on everything from high performance computing to virtual environments. We actually did some parallel computing on a system called the iWARP, which actually grew out of a project at Carnegie Mellon called the WARP, basically a high powered distributed parallel computing system. The iWARP used something y actually called it “systolic computing”, which is an analogy to kind of like basically like the cardiac system because data would be “pumped” through the system in lock step. So we did some work there with parallel processing.But, Wwhat the robotics group became most e also actually got famous for I think at that time was virtual reality interfaces. It was bBack in the early ‘90s and there was a lot of interest and enormous hype about how , “Oh. We use VR and virtual reality could be environments used for all kinds of things.” The group NASA here at AMESAmes really was one of the a pioneer in pioneers of trying to useing VR interfaces for robotics. At that time, high-end graphics It was a real kind of different thing and at that time computers were large – like silicon graphics, octane computers. These refrigerator size – literally refrigerator size computers, like the Silicon Graphics Onyx, that cost several hundred thousand dollars. But, and at that time that’s what you needed for doing real-time interactive 3D graphics. So wWe used virtual environments for testing robots at NASA Ames. doing a lot of things for controlling robots in our test areas here. We also worked on did robots that went to the Antarctic. One was There was a ROV that we deployed under the ice – the ssea ice at McMurdo Sound and that was remotely operated from with a virtual environment. Later on That led into additional work. We ended up we working ed on the Dante II robot with Carnegie Mellon,. Dante II the robot that walked into the Mount Spurr volcano in 1994. So it’s really interesting trying to look at how you use these virtual immersive kind of environments as a way of representing the remote world and for understanding what the robot’s doing.

Q: What were some of the challenges of designing these environments and interfacing them as an interface for the robot?

Terry Fong: Well, you have to recall that back in the early ‘90s virtual environments reality really werewas sort of cutting edge. People were trying to use looking at theseVR for all sorts of things – from simulation of robots, to financial markets, to social interaction. At the time, and we did not really know everything there wasn’t a whole lot really known about the psychophysics of how people perform should really be in these environments. Most VR developers did not understand how to create Just whole questions of how do you put say sstereo

3D graphicsdisplays and head mount display in such a way tthat people could stay immersedn there for for more than a few minutes at time without becoming nauseousbecause it was really tiring if you don’t have the graphics right. It was’s also really tiring at that time because head mount displays were pretty heavy at that time. Some were essentially I mean they were like 20- twenty pound helmets that you put on with these big actual CRT monitors in them. And then of course the head tracking was done with these magnetic – electromagnetic systems made by companies such as like a Polhemus and Tracker. I remember there was something called the Ascension. These trackers Flock of Birds and these were extremely expensive and very non- linear, sensors and so sometimes you would move your head and there would be a big jump in what you saw seam goes like this and that was a real big challenge. Overall, I think trying to create environments that people would be comfortable in for a long time and be productive in was a real interestingvery difficult research area.

Q: So who was in charge of the robotics research here then and who else was working in that group?

Terry Fong: So wWhen I first came to NASA AMESmes in 1990, there were a couple of people involved with that. One was Mike Sims. He was washere running the robotics group. when I first arrived and in fact he’s still here at NASA AMES, but I think aAbout maybe 10 months later, Mike after arrived he stepped down from thatfrom that position and Butler Hine took over. the group. And Butler was the one who really gotpushed the group into developing virtual environments, virtual reality user interfaces. We also actually worked a lot with Steve Ellis, who is a very ’s a pretty well-knownn researcher in the area of spatial displays and human performance. Actually Steve is still here. I actually used to carpool with Steve to work, so we’d get an extra two hours of work everyday and just tto talking about virtual environments and robotics.

Q: Were there any other people that you worked with who were related to the virtual environments or the robotics group?

Terry Fong: Well, actually – so we did – At that time,I think one of the things that really distinguished the robotics group at the time and awas thatt that time it was called the Intelligent Mechanisms Group. We tried to look at we usedusing virtual environments and to remotely operate operate these robots, but they were really motivated by using robots to performfor scientific explorations. So wWe actually wworked pretty closely with a number of scientists, planetary scientists, who were interested in using VR and robots to trying to see what did the robot actually find and trying to understand what the remote environment was likes. So fFor example, we worked on one project with Carol Stoker to the project that we worked on with remotely operate the an underwater ROV that went to in Antarctica, that was a project that we worked fairly closely with Carol Stoker. Carol is a planetary scientist here at NASA AMESmes.

She does a lot of has worked inon a pretty broad range of things. She’s worked in underwater environments. She’s done a lot of drilling work. Also, sShe’s pretty well known for her work on a lot of Mars related planetary science and because of that it was interesting. To me, the blend of engineering and science was it was really exciting and motivatinga blend of really engineering because we’re trying to build robots and trying to build interfaces, but it was motivated primarily by those robots being used for science. And so that was an interesting thing from NASA. We hadn’t actually seen within NASA this real sort of blend of science and engineering around things like virtual reality interfaces and robots.

Q: And what were some of the motivations and constraints that came from this interaction with science and with scientists?

Terry Fong: Well, part of our motivation it wwas that we were trying to find to ways of represent ing the data in a way that wasould be meaningful to both engineers and to scientists, especially data that came from instruments carried by robots. In theIt was pretty unclear at that time really what’s the right way to represent that. You have to think back to the early ‘90s and as I was saying , the best computer systems graphicals systems at that time were made by sSilicon gGraphic systems, but they were nowhere near what you see on the market today. Back then, I meanhigh performance meant real-time rendering of we’re still talking flat shaded polygonal models of perhaps ofseveral tens of thousands of polygons at most. So tTodayhese days, you pick up anything, even your cell phone and it’s probably gothas 100 times – or more – ofbetter graphical performance of what you saw back then. So trying to represent scientifically meaningful data and engineering data in a meaningful way with such limited resources was a huge challenge. One approach to increase realism that was quite common was to useI do recall that time it was sort of real cutting edge to look at the idea of texture mapping, that is to say mapping you could actually take images and put them on top of flaton polygons. And that was our way of increasing the realism of those scenes but it was very, very limited. Overall, Iit was very hard to really create real-time 3D representations things that were both, I think, meaningful and accurate – at least from a scientific perspective. That was, I think, probably the biggest challenge.

Q: What made you decide to go back to get a Ph.D.?

Terry Fong: Well, I had workedbeen here at NASA for four years and I had worked on a lot of different remote operations projects, a lot of different kinds of software and I decided that I really wanted to spend time and be able to focus on one area. because I think one of the things that’s probably true just about anywhere in a research environment, especially in an applied research environment is that you can work on have projects after project after project without everand you never really get a sense to just focusing on just one thing for a long period of time. just because that’s the nature of I think research in these sort of settings. So, I wanted to go back

Formatted: Font:

Formatted: Font:

and get a Ph.D. so I could because I wanted to actually be able to spend multiple years working on something. Then, I had to make a And so at that time it was a decision of, “Well, do I stay in California?” – because clearly there are a number of good schools here – or, d”Do I go to someplace else?” And I chose to go to Carnegie Mellon because it was (and is)and still is in my mind the best place for learning about robots.

Q: And you mentioned that you had worked with some of the Carnegie Mellon folks on Dante and so did you already have some connections with people there from that project?

Terry Fong: YeahYes.! So I actually knew many a lot of the people at Carnegie Mellon extremely well before I went there for my Ph.D. because we had worked together. Plus, oI had previously visited several times while working at NASA. Each time I had been there for a n visits, the reaction had been there I would go in and of course, “Hey, the guy from NASA is coming so let’s go show him around and meet people.” So, I was extremely familiar with a lot of the research at CMU and I actually knew many of the professors there extremely wellat a professional level before going back as a student. In particular, I had spent quite a bit of time workinged quite a bit with Reid Simmons, and Red Whittaker, and Chuck Thorpe, – I and so knew them all extremely well before going. And, I also because of the work that we had done onwith the iWARP, I kknew a lot of people in the computer science department, such as Thomas Gross, and Jon Webb. Knowing all of these people as researcher made itI can’t remember who else but it was an interesting transition for me because of course having worked as a researcher and really knowing all those people and then going back as a student it was a little kind of odd actually because I went there and I knew all these people – actually knew them really well. So it was an interesting transition.

Q: How did you choose an advisor once you got there?

Terry Fong: Well, actually I went there knowing that I really wanted to work with Chuck and with Red because I had worked with both of them before. We had actually iIn the course of the projects we had done together, we had talked a lot and foundwe had a lot of many similar interests. And, of course, having come from NASA, I wanted to do something space related, or something that was really in the field. and bBoth Chuck and Red really did projects like that. But, I did also consideredtalk to other people too, including . Obviously, I knew that Reid Simmons and did a tremendous amount of work at that time, the Ambler Project was a big project at that point. Eric Krotkov was at CMU at the same time. In fact, I really almost decided to choosego with Eric because when I was talking to him dduring the sort of “marriage process” they have at Carnegie Mellon – where students arrive and you spend time meeting all the different professorseople and then there is it’s sort of a matchmaking between students and the advisors, – one of the things that really struck me about Eric was he said was that he had a his goal in doing research was to be able to do robotics research in all of the five senses. That is to

say, So he wanted a robot that could taste, and one that could smell, and then one that could use its vision, etc. and I thought that was a really great goal and I was really this close to choosing him., b But, then, you know, I think the lure of really working on field robotics really drew me to Chuck and Red.

Q: And what were some of the projects that they had going on at the time?

Terry Fong: So this was bBack in the early ‘90s. , Tthe Navlab Project was still going very strong. At that time, I think there were only maybe two Navlabs. But, Bby the time I left, there I thinkwere maybe half a dozen (or even more) than that Navlabs of various sizes and shapes. The Ambler Project had wrapped up. The Dante Project had wrapped up,. but tThere was some work in building some Lunar Rover prototypes that I recall being done. And I ended up in mMy initial work at CMU focusinged on trying to adding proprioception periphery reception to Navlab, so that the robot could , “really just trying to figure out could you drive by feel”, which is an interesting thing.. Obviously cCars, even back then, often but certainly today have all these traction controlsafety systems that are designed to and they try to – the cars themselves can take some action to safeguard the driver if you start running off the road or skidding around,. aAnti- lock braking is a good example of that. But wWith Navlab, what we we’re trying to do is figure out was, “Well, cCould we feed some information back into the system that when it’s driving off road it would allow it to perform better when driving off-road?” In other words, we wanted to robot Soto realize that,, “Oh. This terrain is really, really bumpy,” or, “Okay. I’m shaking left and right or mainly going right, therefore I should try to correct and go left.” And that was the research that I started working on really primarily with Chuck and I think it was driven by the fact that Navlab at that time was moving more and more towards off road scenarios.

Q: And what were some of the ways that you implemented these solutions with Navlab?

Terry Fong: So what we were looking at was I started think a combination of the sensing because this was really meant to be periphery receptive so I start off by thinking, saying, “Look. I’m not going to use look at anything that looks outside of the robot. I’m only going to uselook at say accelerations, changes in orientation, etc. and trying to pick out patterns.” And fFor example, I was interested in how rapidly we’rethe vehicle was bouncing up and down or maybe how strongly it waswe’re bouncingdoing that. But, It was really interesting, challenging kind of area of research because it wasn’t clear (at least to me), – well, do you need just one sensor or do you need lots of sensors?. You have to look at, for example, a wheel slip as well as inertial measurements. For I think for us as humans, it’s clearer that when as wewe move, we rely on have a lot of proprioception. periphery reception just because of our limbs and our joints and our head motion and everything, but at the same time you close your eyes you think about yourself like on a rollercoaster because whether or not you want to look out a rollercoaster you certainly feel yourself going up and down. You can infer a lot about the outside world, or – at least what

you think you know is going on in the outside world, and you make a lot of judgments based on thatose inferences. Being in I think that the work that we did with Navlab certainly showed that you could tell a bit of what’s going on, but the problem is that just like humans you close your eyes, you move around. You think you know what’s going on and that’s only part of it because you’re not grounded in the world. I think whether you’re on a rollercoaster or actually if you’re in aan airplane in thethe cloudbanks is a good example. You may think that, “Hey, I’m flying level because my body tells me I’m level,” but in reality actually when you come out of the fog clouds and you may find that the plane is out, “Oh. We’re actually in a bank. W steep over,” that’s actually a bit of surprise. And we had the same sort of, I think, challenges with Navlab too.

Q: What was your thesis on?

Terry Fong: So Well, let me first say that I have never followed the usual path through life. I guess I’ve been kind of an unusual person in that respect. I’ve (almost) always chosen to take the not just even “the road not traveled”, but “the road not yet built”. and I think that was reflected in the fact thatBecause I I had worked in this industry or in particular at NASA and then went back to school, . And so I was already a kind ofI was a bit of an unusual grad student – being an older grad student, somebodyone who had done research and had published papers already, and certainly someone who knew all the faculty at more of a pure eer level. On top of this, Then what happened was the – after my first year and a half at CMU, because my wife (who was also doing a Ph.D. at Carnegie Mthere)ellon, she had joined a research group that did all of their research in Switzerland. So, after a year and a half we left Pittsburgh and moved to Switzerland and I ended up doing my Ph.D. research at a Swiss Uuniversity, the Swiss Federal Institute of Technology in Lausanne or “EPFL”. Although and at the same time the research was done at EPFL, but my degree was from Carnegie Mellon.! And during my the first year at EPFL, my CMU advisor at that time was Chuck, but . At that point and time I had decided to just focus on my research with Chuck. Hhe actually wasn’t able to support me because most of his research funding came from DARPA and it was difficult for him to support a researcher working out of the country and completely alone. So, I was in this situation thafound myself thinkingt, “Okay. I’m in Switzerland. I’m trying to do my research for CMU and where am I going to get funding from?” But, I was fortunate because at the same time some people that I had worked with at NASA AMESmes actually had decided to leave NASA and create a startup company and they asked me to join it. So for a period of time I was a CMU grad student doing my research in Switzerland while being paid for by a California startup company.

Q: What was the company?

Terry Fong: It was a company called Fourth Planet that actually did work based on the virtual environment research we had done, created ing these virtual environment visualizations of real-

time data. Things completely unrelated to robots. Things like computer network monitoring, just looking at say bandwidth usage and connections for various places on the Internet. And so lLike I said, it was a pretty kind of different approach. I probably wouldn’t recommend anybody to go through grad school with the goal of doing your research 6,000 miles away from your university, 6,000 miles away from your advisor and getting being paid for by a startup that you created in California, but it was a path.

Q: So it’s at least two jobs in one?

Terry Fong: It was at least two, maybe three jobs, but you know it was fun. And like I said, I’ve always tried to take the road not even yet seen.

Q: Did you work with any of the people at EPFL at all?

Terry Fong: Yeah. So at EPFL had at that time there actually was an Institute of Robotics., but Ththeir focus was primarily on very small-scale systems… . There were a lot of insect and behavior-based robots, as well as a lot of industrial systems. EPFL was known for They did a lot of manipulation for very precision, high speed, pick and place type work. But T the group that I actually joined there was the “Virtual Reality and Active Interface” group, which and that group was interesting from the sdeveloped a broad range of user interfacestandpoint of really looking at interfaces broadly. So there was cComputer-aided surgery. There was just vVisualization of remote environments. And some robotics. and there was a little bit of robotics in that group too. That was within this larger institute that did those other robotic things that I talked about. And Wwhen I first arrived at EPFL I did spendt some time talking with several professors. One was – at the time it was Reymond Clavel, . He was a professor there who was famous for this robot called the Delta robot. The DeltaIt’s is widely used in especially drug manufacturing lines for very high-speed manipulation. I also spoke with worked a bit with Jean-Daniel Nicoud, who’s was – his group actually created several well-known a number of small robots, including . There was one called the Koala and Khepera, which then led to a startup company called the “K-Team for a wh”ile. I actually don’t even know if the K-Team is still in existence, but certainly their robots live on in many research labs. This really nice small little tabletop robots. I actually think I have one even on the table behind you someplace or maybe it’s around here. But, yeah, so In any case, EPFL was certainly a big change for me from Carnegie Mellon. It was a big shift in terms of the way people approached research, in terms of just the facilities, many things. Going from a place thatlike Carnegie Mellon, which had hundreds of researchers working in robotics, to and a place, which had maybe a dozen roboticists was a big change,. bBut, it was also I think really kind of liberating in the sense that there weren’t any expectations of, “Oh. You can only do this research,” because there’s were so few people in robotics at EPFL and everybody just did what they felt was really interesting.

Formatted: Font:

Q: What did your thesis wind up being on?

Terry Fong: Well, when I So my thesis, you know, actually having moved to Switzerland I, of course, wasn’t able to couldn’t ccontinue my work on proprioception peripheral reception with the Navlab because the Navlab I was using at the time was a retrofitted Humvee. It was pretty hard to put to a Humvee in my suitcase and take it to Switzerland with me. So I ended up going off on a in a completely different area, working on and the advisor that I had in Switzerland, his name was Charles Baur and so it was kind of funny that I actually had two advisors named Charles, although one was Charles in Switzerland the other was Chuck in Pittsburg. And we ended up getting this whole area of Human -Robot Interaction, or “HRI”. At that time, the HRI community was really very nascent. I mean tNhere wasn’t really ot really even any sort of recognition of the term, “Human Robot Interaction”. Instead, people were concerned about There were sort of, “Wait. Oh. They’re robot interfaces and humans and ro and bots.” There was a conference at that time. It still runs today called RO-MAN for human-robot communication. , bBut, it wasn’t this notion that there really was human robot interaction, what did that mean, but it was starting. Aa lot of people around the world were starting to look at, starting to define “HRI” that. And I was really interested in the problem that this idea that as you get humans and robots working together on similar tasks, how do they communicate? But when I say “communicate”, I do not mean “Not because I’m interested in semantically, how do they express themselves?” or even necessarily the display, but rather really w”What is that communication useful for?” And, so my thesis ended up addressing working in this area that what I I eventually end up callingcalled “collaborative control.” The central idea in collaborative control is that The notion that if a human is working on something and has a problem, ithe should be able to ask a question to the robot – and vice versa. because it was clear to me – you know, part of this was driven by the fact that I was in Switzerland, which was a very collaborative environmentThe bottom line is that nobody has all the answers. , but And so therefore maybe if if you could ask questions, you couldan benefit from the theother’s knowledge of others. This is, especially true when you consider look at robots and humans, which are situated in the world and have different perspectives because they’re different locations, different sensing modalities, different areas of expertise, different levels of precision or ability for doing repeti, etctive tasks. That you could actually benefit from having a team where theIf humans and robots can aresupport each other by exchanginging information, then they can and are working together from ain a very collaborative mannerpoint of view. That approach was quite different I think from what other HRI research people had been doing at that time.

Q: Were you talking to Reid too because he has that concept of mixed autonomy, which seems somewhat related but I don’t know if it is.

Terry Fong: Yeah. Well, it’s the case too that for a long time within the AI community researchers have looked atthere’s been all of these notions of mixed initiative and, adjustable

Formatted: Font:

Formatted: Font:

Formatted: Font:

Formatted: Font:

autonomy, and sliding autonomy, and I don’t know how many other types of other aautonomy. kind of terms like that at the time. But all of these architectures those were really focused on thise idea that the robot has some level of autonomy and you just need to ’re just trying to “dial in”, you know, the right bit of autonomy for thea particular moment in the time, particular tasks, particular scenario, whatever. and II wasn’t interested in that because that seems to me that is really sort of like tweaking knobs on a control system. I was more interested in thise notion that humans and robots could work as partners, or as peers, just like you and I are we’re talking. In other words, That when I have a question, I should be able to ask it of you and I could benefit from your knowledge. In my research, I was very focused on So to me it was a whole question of, “Okay. How do you use this and when will this be useful?” So I was trying to look at scenarios. These are very task driven scenarios. I was very interested in situations where team members might say to one another, It’s not in terms of like social interaction, but it’s more like, “Well, wWe’ve got a job to get done. Therefore, we have some common ground because we’re trying to do the same thing. A and, therefore I know that if I have a question, I can ask it of you because we’re working towards the same sort of goal.” So, my work focused more on task- oriented information exchange, rather than control of autonomyAnd so I don’t have to worry about trying to really completely model you or try to understand what you’re doing. We’re trying to work towards the same objective and therefore I think sort of the dialogue – the range of questions, the range of knowledge that we need to have in the system can be more limited because it’s very task specific.

Q: So what were some of the tasks that you tried out for this?

Terry Fong: So tThe most basic one was just moving from Point A to Point B, which is just sort of thea classic navigation tasks. More importantly, this task is , which we know is still a challenge for robots of any size, shape, or form, . In particular, I think that humans because of our ability to look at the world and relate our prior knowledge about objects to things we haven’t seen, which might be similar. I see, for example, you have this bottle here of some drink I’ve never seen before, but I recognize it as a bottle because it has these characteristics, which say, “Oh. That’s plastics and it’s got a label on it and therefore it looks like a bottle to me. And so I’m going to say, “Well, it’s a plastic bottle there.” Are really, really hard still these days for robots. And back when I was doing my research, it was clear that if you’re trying to drive a robot especially if the robot is operating in through a natural environment where that the perception of obstacles, the perception of terrain hazards, etc. is still was a huge problem. So I was interested in looking at the quhaving robots ask estion of, Could have the robot ask questions of the humans, sayuch as, “Hey. Is this dangerous? Can I drive through this? Can I – dDo I have to drive around it?” To do this, I ran some And so my research was really focused on sort of navigation issues, trying to see could we a simple robot to actually make progress in the natural world if it has the ability to just ask a question whenever it’s uncertain. So I would do these interesting studies where I would put take, for example, you’ve probably seen these flat paper cats – . tThey look like a cat, but are it’s actually just a picture of a cat, cut out and mounted on cardboard piece of paper. – So I put those in front of the robot and have people

remotely operate the robot. and tThe robot would drive until it detected an obstacle and then would see something and say, “I don’t know what this is? Is it a rock?” because it clearly couldn’t tell if it was a cat or not. “Or is it a cat? It is a real cat?” And it would send a picture back to a human and sayask, “Hey, cCan I drive through this?” Depending on the situation, And, of course, depending on the user the answer might bewould be, “Well, yeah, it’s a flat plastic – paper cat, so yeah just drive over it.” Or maybe, “No, I don’t know if that’ is a cat or not, so drive around.” In short, even with And that was a really sort ofa really basic thasking –, driving basic skill that robots need to have to be able to go from Point A to Point B, robots but still something that couldcan really benefit from this sort of interaction with people.

Q: And was it mostly natural language that in a sense the robot was using so it was asking questions or were there other types of communication that you were working on?

Terry Fong: TIt was – the dialogue was very scripted from the standpoint that I had chosen a very, very specific task . and because of that I only used end up trying to encode basically a number of different primitive pieces of communication. So a pre-defined list of questions. or at least topics to focus on. And Tthe challenge there was then trying to figure, okay, if you have like in your vocabulary – I don’t know, maybe vocabulary’s not the right word – but basically, say, your ability to ask a hundred different questions. How do you choosedeciding which question to ask, because the robot you mmight want to ask about a lot of different things at the same time. Ideally, you want the robot to But you actually want to end up choosing e the question that is the most important for thisa given moment in time. But, And it’ is also the case, too, that you don’t want to be annoying because you don’t want to ask question after question after question… If you do, then that ppretty soon the human says, “Ah, Tthis is just too annoying. I’m gonna [laughs]ing to stop answering questions.” So one thing I tried was to trying to do a bit of user modeling, thinking that if a robot is interacting with was important, trying to say, “Well, okay, if I have an expert in navigation, it should ask I should ask a certain types of questions. But, if the human is a novice (in navigation) If I have a novice” – maybe they don’t normally drive robots but they’re still pretty skilled in some areas ‘cbeause they’re human after all – you should ask other questions. trying to figure out how do you choose to communicate with those different types of people. Thatis ended up being was a big part of my thesis, really, just deciding trying to figure out how do you choose the questions, how do you decide when to ask these questions. And, for different users, doshould you ask different questions?

Q: What were people’s reactions to the notion of robot as peer?

Terry Fong: I think it was something that within the HRI community – and as I said, the human/robot interaction community was really getting started at that point in time – was really a thing that people didn’t quite have a good sense of, “Well, iIs this a good thing or a bad thing?”

And in fact, even today, I would say, today that nobody really knows for sure. We see There are notions that robots are certainly in everyday environments and, well, “Well, iIs it a peer? Is it a partner? Is it a tool?,” is a great topic for place to have long and lengthy discussions and debates about.. But, to me, that’s not even so much the issuereally the key point. I mean, I don’t necessarily care whether, or not, a robot is a peer, or a partner, or a tool if, for a particular task – because I’m very task driven, especially here at NASA – that how we (humans) work with it makes us – give us the ability to actually do achieve something better than we could without it. And, tTo me, that is was really the sort of the fundamental thing. Whether or not the robot is it’s a peer, or needs to be a peer tool or not, I don’t know.

Q: And so far at the EPFL you used small mobile platforms generally for this kind of work?

Terry Fong: Yes, I actually did worked with a bunch of Pioneer robots, which were made . At that time, it was by Active Media, I think. They’ve since changed their name into Mobile Robots and I think recently they were recently purchased by Adept. In any case, when I worked with the Pioneers there were But at that time, Mobile Active Media was selling a line of Pioneers. They had some indoor modelsones and eventually created these kind ofa somewhat pseudo rugged outdoor versionrobots and they did a lot of research in the lab and a bit of research outdoors, in terrains which were actually pretty benign but still at least in a natural environment.

Q: Did any of this research then carry on when you came to NASA – NASA Ames?

Terry Fong: Yeah, so wWhen I came back to NASA –, after having been gone for 10 years – I found that the group came back and the group had changed quite a bit, but was still from intelligent mechanisms to intelligent robotics and certainly had continued doing research in the area of using robots for remote exploration robotics. But when I first came The main reason I came back was to here – I actually came back to work with my friend, Illah Nourbakhsh, who , because Illah had decided to take some time off from being a professor at Carnegie Mellon and was running the group at NASA Ames. He Illah was a professor there for a while and he and I had had known each other for years and years and years, but never had worked together. At that time, he actually came out to NASA Ames and was running the group here, wasn’t sure if he was gonna actually go back to Carnegie Mellon. He eventually did but, when he was first here, I decided, hey, it was a great chance to – ‘cause I’m gonna actually get to work with Illah who I think the world of. And So, when I arrived here, we created thisa project called the “Peer to Peer Human Robot Interaction Project” and that was really sort of an outgrowth of my thesis. , this whole notion that humans and robots could work as peers or as partners, really exchanging information. And there, wWe were trying to go beyond the kinds of things I had done in my thesis, and which were limited to pretty simple navigation tasks, to things that really were able to get into the area offocus on spatialecial dialogue. We wanted to be able to So you could say to a robot, “Hey, sShine this a light here,” and because it would have some knowledge of you, and

what does here mean, and the object and the task at hand, be able tohave it react more appropriately. The And that was a great project because it was a project involvinged the group here at NASA Ames, some researchers from the rRobonaut group at NASA Johnson, . Involved Carnegie Mellon with because Reid Simmons, was part of this and Alan Schultz from the Naval Research Lab. So it was a really, I think, great project for about a year and a half, and looking at this whole notion of can you really push toward sort of more peer interaction – at least focused on the area within a specific task. Just not in general. I mean, we never had the goal of trying to create robots that could just freely interact but really interact within the context of a specific area of work.

Q: What were some of the results and things that you learned from this project?

Terry Fong: Well, thatis project was also, I think, much broader than my thesis because. wWe were trying to used natural language . So there was speech recognition . involved. WWe were talso rying to used spatial reasoning, so there was some computational cognitive modeling involved as well. WAnd, ewe employed were trying to really do some perspective taking, that is to saytrying to figure out, “Okay, iIf I’m here and the robot knows that I’m here but it’s over there and I say, ‘Move this there,’ what am I talking about? Is that reference to me and my body frame? Is it a reference to the object and the task I’m working on?” So iIt was a bit of a challenge trying to figure out whatich area we really wanted to explore first, which area youwe wanted to make the most progress on. and But, overall, I think that, for us, the primary result big contribution was that we were able to show NASA that robots and humans could try to work together in a more autonomous manner. It wasn’t like the traditional NASA approach, which is basically that said, “Hey, we’ve got a robot. We’re going to gonna ssend it off. Then, Wwe’re going to command it and we’re going to nna monitor it,.” Instead, we showed the ideabut really this notion that humans and robots could work more closely as independent peers. And that was a huge, interesting I think, notion for NASA at that time. I doI remember doing lots of demonstrations to people.… We have managers from the NASA headquarters saying come by and then they, “Oh, wWow. That’s great. That’s just like science fiction.” And I thought, “Wow, okay, great. So now we’re making science fiction into, hopefully, some a bit of sciencesmall bit of fact.” That was a lot of fun.

Q: How did you get them interested in this because we have heard a lot about the kind of command and control approach and also the need to be very conservative when you’re working on actual missions? So this is obviously a more research-oriented project, I guess I would say. So could you tell us a little bit about how you got it passed through and how you got people at NASA to accept this different way of looking at things?

Terry Fong: Yeah, well, I don’t wanna take a tremendous amount of credit for that because I think it was a lot was due to of good timing. Right when You know, when I was had finished my

thesis and getting ready to I came cometo work here with Illah, at that time, NASA had had decided to started a very large, new, technology development program. This program was designed to And as part of that they were really looking at a very, very broad range of research areas. Some of those areasthings were near-term, but many a lotta them arewere very far-term, really as sort of open-ended,kind of “Hey, wWhat might be possible in our 10/15/20 years?” And because of that, we had the freedom to try really do something that which was very different from, like you were saying, this sort of mission-focused, “Okay, well, lLet’s reduce the risk as much as possible because we’re sending a $2.3 billion rover to Mars,.” like we will in a few months. And because of thaIt was good timing to t, we had the freedom to really do more, I guesstry , really fundamental, very, very different – very risky actually, I would say, – research.

Q: You said it took about a year and a half with how the project ran. What happened then and what did you do after?

Terry Fong: Well, the usual sort of thing, I think.! That brand new program that I mentioned gotwas cancelled [laughs] and, because of that, you know, NASA ended up shuffling around the various projectsprograms it was doing and ended up creating a whole different set of research projects that which were much more, I think, focused on driven towards supporting near-term missions. So we wrapped up the Peer to Peer Human-Robot Interaction project and moved on to other things.

Q: Can you tell us about some of the things that you did then?

Terry Fong: Yeah, so oOver the past, I would guess I would say maybe five years, we’ve actually moved from Peer to Peer Human Interaction – at least here at NASA Ames – to what I’m now terming “Robots for Human Exploration”. The idea is And this is the notion that robots, unlike the ones that we use on Mars right now, really can be used to improve the way that humans do exploratione. It’s a little bit of a subtle difference here because iIf you think about the way that we use robots today, such as Spirit and Opportunity and eventually Curiosity, thosewe have them are robots that have to do the exploration end to end. They’ve gotta do everything because there are no humans with themthere. Therefore, we are extremely conservative in how we operate them. We don’t wannat to lose them because they’re the only things we can use in a place that we probably are not going to nna be able to get to again, even with another robot, for a very long time. And then, contrast with that – with the approach that we’ve taken with towards human exploration. And it’s interesting if you think about this. The last time we really had a humanoidn explorer – and I’m not talking about just, like, low-Earth orbit in space around the Earth but on another planetary surface. That was almost 40 years ago, in. 1972 was when Apollo 17 was on the moon. Jack Schmitt, who was actually the only scientist to actually go into space, –was the onlythe geologist on Apollo 17. , I mean, trained as a scientist. That was in December 1972 and iIt’s has been a long, long time since then., Bubut

we’ve certainly learned to do a lot more in terms of how do we use things like robots, whether they are rovers or landers or spacecraft, orbiters, satellites, that kinda thing. And so there’s been a big change, I think, in what you can do from a technology standpoint since the last time we had humans on another planet. And oOne of the things that we’re interested in now is this whole question of “cCan you use robots to improve the way that humans carry out exploration?”do operations. Can you improve the things that humans do by, for example, having robots work before humans – doing things likesuch as scouting, setting up equipment, setting up communications relays? Perhaps even do initial survey work. And then, can you have those same robots, perhaps, support humans, while they’re there, in a different manner as automated transport or maybe safeguarded transport. So you might have a robot, which was working independently before, but now humans can jump on it and, because it has sensors, it can avoid hitting obstacles when they’re driving. So they don’t have to worry about that. And then, after the humans leave and go home, if you might use take those same robots to do follow-up workand after humans leave, do follow-up work.. By “follow-up”, I mean The idea that you can completeing tasks that were started by humans or doperforming tasks that are complimentary, or supplementary, to what humans are doing. The upshot of all of this is that And, for us, that’s a real big change because here, now the we do not have to rely uniquely on humans, or robots, for everything aren’t focused on doing the exploration end-to-end. Instead, we can use Here they’re trying to do those things which robots for tasks, are good at but which would be very unproductive for humans to do. Systematic survey is a good example of that. A lot of the work that we need to do to understand an area involves making thousands and thousands of repetitive measurements in a very structured way. Well, sSending a human to another planet, which is extremely costly, which is and extremely risky, simply to go make thousands of measurements in sort of these a lawnmower patterns does not really seem rational. is a huge, huge waste of time and Many people would say, “Why would you do that?” Well, you do it if that’s the only way you can understand the environment. But, if instead you had the option to employ robots for survey, perhaps interacting with local human explorers when needed, well that is a game changer. Overall, thewould say, “Hey, this sort of repetitive thousands and thousands of measurements in a lawnmower pattern, that’s perfect for a robot” – well, so you should use a robot for that. So as I was saying, looking at the idea of “robots for human exploration” is something that I think is very powerful for NASA and we’ve spent a lot of time in the past few years trying to understand how do you build robots to work before, in support, and then after humans.

Q: Is there a view to implementing this any time soon in one of the missions?

Terry Fong: Well, aAs I said, we’re driving towards supporting human exploration. So, of course, the basic question is “wWhen are humans gonnaing to get off the planet and actually step foot on some other planet or an asteroid, for example?” And uUp until last year, the answer was, “Oh, we’re heading towards the moon. We’re – think we will ’re gonna be there in the 2020 timeframe.” Well, Tthat’s changed again, both . I think bebecause of political realities and but aalso because of just the economy. There’s are some things that we’re just not able to afford

right now and so I don’t know when we’re gonninga to see humans back on the moon, or on the surface of Mars, or on an asteroid. I do firmly believe it’s gonnaing to be sometime during my lifetime, but weI don’t know the specific date. At the same time, however, that means that NASA continues to still is spending a lot of effort to actually try to ffigure out how do you to create robots that can support humans in futurethose sort of exploration scenarios. So we are continuing to do a lot of work, primarily testing here on Earth in planetary analog environments – so places on Earth that have some characteristics that are similar to the moon or Mars in terms of terrain, in terms of geology, for example. And because of that, tThat’s really where our focus is these days.

Q: You also became the director… [off topic conversation]

Q: …of the robotics group. How did that happen and what was the group like when you took it over and how have you been developing since?

Terry Fong: Yeah, sure, as I said, As I said before, when I came back to NASA Ames, I came here because Illah was here running the group and I’d always wanted to work with him. Like I said, wWe had known each other for a long time but never really worked closely together, other than, well, – actually it was interesting. Jjust before coming here, we actually had put togetherwritten a survey paper of human/-robot interaction,. That paper which is incredibly dated now and I’m actually horrified that people still scite it that because it’s so old now. But…

Q: But it’s the first one [laughs].

Terry Fong: But it was the first – it was certainly the first one out there and because we enjoyed writing it, of that we really thought, “Hey, this is great. We should work together.” So I came here, back to California, came back to NASA Ames and worked with . Was working for Illah. But, then and six months later, Illah after I got here, he actually found out that he got tenure at CMU and decidedhe made the decision that, well, “bye-bye California, back to Pittsburgh.” So he left.! So I had move [laughs] back here to work with him, worked with him six months. Actually, it was a great six months together and we continued working together while he was in Pittsburgh on the Peer to Peer Human Robot Interaction Project, even when he was back in Pittsburgh. But, when he left I basically inherited the group from him. The group, aAt that time, had I think there were about 15 people in it. and wWe’ve really grown since then, over the past – like I said, it was about 6 years ago – past 6 years, to where we now have 32 people and a group that does pretty we do broad research that’s pretty broad. Some of the work we do now

people would say, “Oh, why is that robotics?” And iIt’s has been a really interesting journey, I think, over the past six years.

Q: What are some of the priorities for the robotics group?

Terry Fong: So wWe are still focused on being able to createing technology thatto really improves the way that you can exploreation of remote environments. And that’s sort of like our core mission, trying to find ways to improve remote exploration. But tThat doesn’t mean that we have to do that exploration only with robots. that can drive on a surface, that have wheels and instruments and have to be really sort of in Sid-shield. But wWe’re also interested in using orbiting looking at how can you really approach exploration more globally and part of that is that NASA and other international agencies have all of these spacecraft orbiting around other planets, other planetary bodies, providing lots and lots and lots of detailedto capture detailed mapping information. And tThe challenge there is really how do you take that information, which iscan be very, very large these days – perhapsI mean, petabytes of data – and visualize it in a way that anybody, whether they are a scientist, an educator, a student or youra grandmother, can understandactually look at that data. Because this is wonderful, rich data of places that we just can’t go as humans but which we care to know a lot about. So oOver the past few years, we’ve takening some of the software that we originally developed for our robots for navigation – basically looking at, say,using stereo images andto creatinge these 3D maps that are useful for navigation – and turning it into them to software you can use forto building planetary-scale maps. So aA large part of my group now works on automated planetary mapping. We take lots of data from Mars orbiters. We’ve been working with the HiRISE Imager on the Mars Reconnaissance Orbiter. We’ve also worked with a number of other datasets overfrom Mars and over the moon. For example, There is the Lunar Reconnaissance Orbiter right now, which has a camera called the Lunar Reconnaissance Orbiter Camera, LROC. We’ve been processing data from LROC that as well as historical datasets. Some of the richest data, some of the best data, believe it or not, is actually from Apollo. Apollo 15, 16 and 17 had thesefilm cameras called the Apollo Metric Cameras, mapping camera. And dDuring the past several years, there’s been a project led by Mark Robinson at Arizona State University has been to scanning the original film with a photogrammetric film scanner. So tThis is a scanner one tthat can actually scan things down to film grain level. So you end up with these enormous, really enormous, images of film, the film strips that were perhaps a meter long and maybe 20 or 30 centimeters wide. My group then So we takes that information and . We applyies computer vision methods that we originally used for our robots to create these large-scale area mosaics – so image-based maps. And then, for areas where we have, say, stereo overlap, we can create 3D trainerrain models. But, And then the the thing that’s been so exciting for the past few years is that our next door neighbor – I mean, literally across the fence, – our next door is Google. We’ve actually worked with them to create versions of Google Earth for the moon and Mars. So today, for example, if you go download Google Earth, there’s a little icon on the toolbar that looks like Saturn, which apparently is the universal planet, at least in Google’s eyes. If yYou click on that, and you can switch from looking at the Earth to actually looking at the moon or looking at Mars. And I’m

incredibly proud that just about everything you see there was developed by my group: the base maps, the images, the tours, which allow people of all – from all domains, all areas of interest, all areas of learning, to actually interactively explore look at some of the data collected by NASA. To me, that’s a phenomenally wonderfulinteresting thing.

Q: If I remember correctly, Illah had been working with the Google Earth folks at some point too. Were you also connected with that?

Terry Fong: Yeah, so jJust before Illah left, he and Randy Sargent started a project called Global Connection. This , which was really focused on the idea of how could we use tools like Google Earth or – systems, at the time, there was only a prototype, a gleam in, I think, Randy Sargent’s eyes that was just an – that would’ve eventually be come to known as GigaPan. And that was something that we really talked with Google about, the idea that you could interactively exploreing images in really new a ways. that you hadn’t been able to do before, that you could use tools like Google Earth to explore environments in a way that you couldn’t do before. And that led into a very long, I think, series of research projects and development of different pieces of robot hardware and software to sort of realize that vision. And now these days, I’ve gotta say, thatOne thing that Illah and Randy have created this was a truly amazing, wonderful thing called GigaPan. , which just really allows you to really browse and study images, in remote environments actually, in ways you couldn’t before. It’s also interesting for tThe inspiration for GigaPan project that the real inspiration of that, believe it or not, came from Mars. There’s a camera called the Pancam on both the Spirit and Opportunity rovers and some of the original pictures that came down from that camera system were things that Randy, who was working here at NASA Ames at the time in the robotics group, really wanted to view in a better waybe able to just visualize. Because he had little bits and pieces and people had to stitch them together in, say, Photoshop and he had – with this panorama but it was a static panorama. I mean, you could zoom in but it was hard. So he actually started creating what became the first version of the GigaPan browser, being able to actually work with panoramic data that came from Mars, and that was the inspiration for what became GigaPan.

Q: You mentioned that one of the things that you’re working and are really excited about is having these visualizations and this different kind of data and exploration open to more people. There are a lotta lot of people who are now looking at how you could use crowdsourcing to actually explore things. Is that part of the vision at all or [laughs]…

Terry Fong: YeahYes, it definitely!, it certainly, is. So itIt’s nice the way a lotta of things come full circle, that you may start off on one direction and create something new and then eventually it wraps back around. and comes back. So GigaPan’s a good example of that. It started with Mars data as a way of trying to visualize and allow scientists to browse these panoramic datasets. Eventually it became a robot camera, which by the way and actually a spin-

Formatted: Highlight

off company is as well now selling these things commercially. And then last year, what we did is we actually took some commercial systems. We went out and used GigaPan to basically help do scouting ahead of a simulated human exploration mission. We captured and then we used that set of data, these panoramas, . We put them on-line, and let the general public vote on where the humans should explorelook at these and took their input to help plan the simulated human mission. that we were gonna do in an area. This happened to be in the Arizona deserts. In other words, So here we started off with a robot system that was inspired by built on exploration from data from a robot on Mars, which eventually led to the creation of a commercial platform whichthat was used for us to then plan a simulated human mission for that we’re trying to test outing new technology and techniques for exploration, including but then involveedment of the general public. And we had the public involved in doing analysis of these panoramas. We also had them involved in just sort of voting on, hey, what’s the best place to go scout that? So we had this large voting project online and, based on what the general public did – I mean what they decided as, “Hey, this is the right place to go,” we use that, incorporated it into part of our traverse planning for that mission.

Q: How did that work out?

Terry Fong: It worked out really well on two levels. First, One was that I think it really raised the awareness of some of the future mission work that NASA’ is doing right now –, trying to plan out for thate day in the future when humans do go back to other planets. Second, And it was really rewarding to me seeing some of the comments from the public, ‘cause we had an online voting system but, of course, was a place to also add comments. And some of the comments we got were just phenomenal, especially from young students saying, “Oh, hey, this is why I want tona grow up and be a rocket scientist.” So, to me, that was really nice to see that connection to students. We aAlso had a number of some geology students at the Arizona State University used these panoramic look at these images and actually try to do field geology. from the comfort of their classroom. So tThey used these panoramas to do some for analysis that we then fed back into the planning for the mission. So you mentioned crowdsourcing before and, of course, I think crowdsourcing can be on different levels depending on the level of expertise, depending on what you’re trying to do. So iIn one case we we’’re working with looking at a the very broad public. Just asking them, “Hey, vVote on what you think is most interesting,” and could be interesting by a number of different criteria, bBut then we also crowdsourced try to get more detailed scientific information by working with students, who had . In this case people, undergrad students for example, have some limited geology knowledgebut still very limited field experience – as a way of sort of triaging these very large data sets and helping us plan better where we should go do exploration.

Q: What was Chuck Thorpe like to work with as an advisor?

Terry Fong: Chuck was great. He was – I was so phenomenally lucky to have him as an advisor. and tThen, when I got to EPFL to also have actually, Charles Baur as a co-advisorin Switzerland. I wasam absolutely convinced that they wereare twins separated at birth. They would [laughs] probably be horrified to hear that ‘because they look so different, and they obviously have such different backgrounds, but they have a very common approach towards research. They werewere always very, very open to new ideas. They were very willing to give you me as much rope to as you want to hang youmyrself on, but then helped to actually support youme when things didn’t go right. So they were phenomenal. They were really the kinds of people that – I don’t think that I could’ have actually done a thesis where I was in Switzerland, doing aresearch for a CMU Ph.D. whilethesis funded by a startup company if I hadn’t had those two. SI still feel o that was phenomenally lucky.

Q: What do you see as the major challenges facing robotics over the next five to ten years?

Terry Fong: Well, I think that these days you see robots in many a lotta different places. They’ve certainly become much closer to the average person. A lott ofa people have robot vacuum cleaners. The Roomba is hugely successful. The military, of course, is using robotics more than ever before… and that’s both expected and somewhere disturbing. , I think, too, especiaI think lly because whenever you have these sort of new technologies that kind of rradically change things there’s always a question of,of “Well, iIs that technology going to nna be used appropriately?” That’s not to say that you should n’ot use robots for military operations. It’s a question of, “Well, how do you do that? aAnd what is the implication of doing that?” The same thing, I think, could be said about just almost any sort of new technology. For space, the it’s a real question ofis “cCan we use robots to improve the way humans do exploration?” because that’s really, I think, the thing that drives most people to come work at NASA. It’s true that we’ve learned a tremendous amount from using robotic explorers, whether they are rovers or landers or spacecraft. But, at the end of the day, there’s this fundamental urge, I think, for humans to get onto exploration explore and it’s clear that, if we’re going to nna have humans spending more and more time off the surface of the Earth, we need to find ways to makehow do you do that in a way that becomes more productive, that’s less risky and perhaps is more cost- effective?. And clearly robots offer a possibility of actually addressing all those things.

Q: For young people who might be interested in a career in robotics, what kind of advice do you have for them?

Terry Fong: Well, I’m a software guy, so my advice is to learn how to write code early. Learn every single language, every platform, out there because so much of robotics these days is based ion software. This is not to say that creating new mechanisms isn’t important. I’m continually surprised by the way robots evolve mechanically and electronically. But, to me, the thing that really makes robots different from, say, just remotely controlled vehicles is that they can actually

think and they can be independent and that’s all software. So for people who want tona get into robotics, I’d say “it’s – lLearn everything you can about software engineering, everything you can about perhaps A.I. Learn everything you can about creating user interfaces.” And that’s all based on software.

Q: You mentioned that you were there at the beginnings of the HRI community. Could you tell us a little about who else was – you communicated with about that, other than Illah, at the time, what it was like, how it developed in the last, I don’t know, 10 years or [laughs]…

Terry Fong: Yeah, it’s, to me, I’m sometimes askedit’s kinda interesting because a lotta people contact me and they say, “Oh, boy. You were there at the start of HRI, so and, well, how come you’re not there now?,” Well, and it’s not because I have a lack of interest! in that. I mean, that’s really been, I think for my thesis work, one of the things that I’m ve been most interested in, thies whole question of how do humans and robots interact. OIt’s just that over the past few years here at NASA, however, we’I’ve focused on, like I said, on this notion of robots for human exploration. This approach And so that does requires some interaction between humans and robots, but it is not just doesn’t have to be proximal. And, Iit doesn’t have to be real time. It’s more on the human/robot coordination level. But, in terms of human/-robot interaction as a community, as a domain, as a specialty of robotics, I do feel like I was there certainly at the startin the early days of HRI. There was the creation of the Human/-Robot Interaction Conference, which has become as sort of, like, the core conference that focuses onfor issues of human/-robot interaction and human/-robot teaming and human/-robot coordination and all that sort of thing.. And wWhen that was first getting started, the people that I think who were most really involved were people like Illah, like Mike Goodrich at Brigham Young, Alan Schultz at Naval Research Lab. There were other people who were, I think, still in school at that time, including even, like Holly Yankle – sorry Holly Yanco. And, Tthere were certainly people that you are still see involved today. Maja Matarić, for example. was certainly a key person there. . And then there were other people in government labs – like Jean Scholtz at that time, who was just coming towards the end of her government career – being interested in trying to figure out, “Well, if we’re going nnato studyhave human/-robot interaction, how do we measure thatit? How do we assess at some level, in some way, the way that humans and robots interact?” And that, to me, was something that I think really led to the creation of the human/robot interaction conference, which has really, I think, spurred this sort of growth of human/robot interaction as a subdomain of robotics.

Q: Does HRI feed into the rest of robotics or is there still a division there? What are some of the challenges to having those [laughs] communities?

Terry Fong: Well, it’s the case that rRobotics, by definition, is so interdisciplinary, multidisciplinary. I’ve even heard the word “transdisciplinary” recently, though . I’m not quite

sure what that really means. And it’s bBasically – it’s just something that, to me, is inherently broad. So, one And so it’s a question of really is whether there is a “boundary” around robotics? And, in some sense, this is the same question that the people asked years ago about artificial intelligence. I’ve heard “Oh, ‘cause clearly, oh, eEverything you achieve in robotics, that’s artificial intelligence.” Well, not really true! I think eeverything you do is robotics and artificial intelligence is this little tiny piece. So it’s hard for me to really try to wrap my arms around it that and say, “Yeah, tThis is within robotics and thisat is not.”

Q: They were talking about HRI but I don’t know if you wanted to say anything else about…

Terry Fong: Yeah, I guess the only thing I would say about HRI is that because it is so multidisciplinary, I think it has the effect of pulling others into the robotics domain, or at least increasing awareness about robotics. People from design, for example, who may not have even thought about robots before. People interested in ethics, for example. Just looking at how do these things that we’ve now created that are more or less autonomous or semi-autonomous – how do they interact with us? And, to me, perhaps the biggest contribution of HRI is the fact that it is pulling more and more people and more and more domains into robotics in general, and that’s a good thing, because that means that robotics touches other fieldthingss and can learn from other thingfields and can give back to other areas. And, to me, wWhether, or not, HRI exists within – as this bubble called robotics, or if it’ is something that bumps into it from various places, that’s actually irrelevant. The fact that it’s there, the fact that people care about it and are actually learning about things together… that is is perhaps the most important thing.

Q: Great, thank you. Anything else? Is there anything you’d like to add or something you think we missed?

Terry Fong: No, I don’t think so.

Q: Great robot stories [laughs].

Terry Fong: Yeah, great robot stories.. Yeah, I don’t know. I think I’ve just been, I think, really fortunate into havinge worked with a lotta of great people in robotics, people who really care about trying to create these things which you can go out and actually doing do things meaningful things. And certainly, if you haven’t yet talked to them, – and I’m sure that you will – people like Chuck Thorpe and Reid and Red and others. They are the people that really carry the torch and really created this great thing to happen called robotics.

Q: Thank you very much.

Terry Fong: Sure, my pleasure!.

Formatted: Space Before: 24 pt, Line spacing: Exactly 15 pt, Tab stops: 2.75″, Centered