Interview Transcript
Nancy Amato: I grew up, I was born in Portland, Oregon. We lived there until I was in like sixth grade, then we moved up to the Seattle area. I ended up going to a high school, which we didn’t have advanced mathematics there. I always liked math, but I had this one math teacher that was really, she made a big impression on me and she pushed me to go actually take calculus at the University of Washington, because we didn’t have calculus at my high school. I think that probably was a distinguishing factor that helped me get into my undergraduate program which is at Stanford. Really pushed me to study. I was an applied math major and an economics major in undergrad. I think that I would really attribute that to her. I went to a Catholic high school. Her name was Sister Maureen Rose, and I still think fondly of her and every year when I give to my high school, I give a donation in her honor.
Q1: Did you have any mentors during your undergraduate studies?
Nancy Amato: I call myself an accidental computer scientist. I’m a faculty member here in computer science, actually the department head now. I didn’t study, I just told you I studied applied math and economics. When I was graduating from my undergrad, I applied for basically the jobs that would fit my major in the career center and I was offered two. One was to go work on Wall Street and the other was to go basically this one year on campus program through Bell Communications Research, and they wanted me to go get a master’s in computer science. I had not studied computer science. My family was pushing me to join the real world to get a job. I still remember they flew down from Seattle to California and we drove around with my little brother in the backseat and they were trying to convince me that I should not be afraid to join the real world. But I didn’t join the real world. I did go get a… I took that job.
Then I went to Berkeley for my master’s in computer science. I had to go late because I hadn’t applied to graduate school. I hadn’t prepared. And so I was kind of a math person, so I took theoretical computer science courses. I took an algorithms class. I loved it! And the professor of that class, I went to him and I was just asking him, “Is it possible to do a master’s in theoretical computer science?” Because I’d been told that it wasn’t. And he thought I was asking him to be my advisor. And he immediately said yes. And he was the best advisor I could have had at that point. He was just very accepting, very supportive, wonderful man, Manuel Blama, a Turing Award winner, if you’ve heard of that. That’s like the Nobel Prize of computing and just a really awesome person. So he was a great mentor. My other mentor at Berkeley was Dick Karp, who’s another Turing Award winner. And he was more pragmatic, I guess. But they were really great mentors to me for my master’s. And immediately after my first semester, I knew I wanted a PhD. So they were really formative for me, I would say.
Q1: Tell us about your work experience.
2
Nancy Amato: I mentioned I got my master’s at UC Berkeley. Well, I actually came here to University of Illinois, Urbana-Champaign to get my master’s in computer – I mean my PhD in computer science. And I came here together actually with my husband because we wanted to find someplace where we, he wasn’t my husband then, but where we could do our PhDs together. So I actually came here for my PhD in computer science. And I was a theoretical computer scientist. When I was graduating with my PhD, I had worked on parallel algorithms for computational geometry problems. So those are real complicated problems that we need a lot of computational power to solve. And so when I was finishing and I was starting my position as a professor, I thought, I’d like to work on something that’s more applicable, more practical. And I was thinking, what kind of problems would actually need those skills? So I thought, well, maybe robotics. So I went to, in fact, the library here, because I was just a graduating student, I went and I found the ICRA proceedings. And that’s when they were still publishing those proceedings in hard copy. And so there was like, there was four volumes, each one like this thick. So I had like this much proceedings to look through. I’d never been, and I just got super lucky. I found a paper, which was a very first paper on probabilistic roadmap methods by Lydia Kavrocki. She’s a professor at Rice now. And I read that paper and I just loved it. I thought it was so beautiful. And I basically, I was hooked. I’ve been working in that area ever since. So I just got super lucky. I randomly thought robotics. I picked up that proceedings from that year. I didn’t remember what year it was. And I found that one paper and it’s like a needle in the haystack. And it really set me on this path. And it’s been super exciting til today.
Q1: What inspired you to pursue the filed of robotics?
Nancy Amato: Before I actually became a robotics researcher, I was always interested in like science fiction, Star Wars movies, Star Trek, those kinds of things. And here, there’s like 2001 Space Odyssey where we were in Urbana, Illinois, where Hal was. So yeah, I was very interested in those kinds of things. So I guess I was interested in science fiction and things like that.
Q1: What was the first robot you heard about?
Nancy Amato: Again, I didn’t really know much about robotics. And so when I found those ICRA proceedings, I was looking through them and seeing these, at those times, very primitive robotics. But they were, I guess, these very large industrial manipulator arms. That’s what I first saw. And that’s what we, the only robot we had at Texas A&M where I started as a faculty member, I was a faculty member there for 24 years actually, before coming back here in 2019 as department head.
Q1: How did early robots influence your views on robotics?
3
Nancy Amato: Really, we were using them for very specific tasks in manufacturing. They were not part of our daily lives. That was really still science fiction for the longest time. I would say really up until the last, the recent past where we really, I’m absolutely convinced that within my students’ careers, early careers, we’re gonna have robots that are really actually assisting us with useful things in our daily lives. That was not even, yeah, we couldn’t even really conceive of that when I was first starting.
Q1: What was the first robotics projects you worked on?
Nancy Amato: I told you about that paper that I had found. So I read that paper. It was a beautiful, it’s called, we call it today, sampling-based robotics algorithms. So I read that paper, but I had an idea about how to improve it. And in fact, that was the very first problem that I worked on. That method was called obstacle-based probabilistic roadmap methods or OBPRM. My very first student was a master’s student. That was his master’s thesis. And it was, I would say, kind of a pretty simple modification of that basic method, which allowed the method to be useful in a more, I would say more realistic settings, problems that are hard, that have this thing we call the narrow passage problem in robotics.
Q1: Describe your notable robotics projects that followed.
Nancy Amato: I came from theoretical computer science and mathematics, and I’ve really been working on the algorithmic side of robotics. So I work on coming up with new motion planning and task planning algorithms. And they are all actually still, I would say, in this kind of general area of the sampling-based planning. So I’m kind of on the algorithmic side. And I would say almost all the work of my group has been, in some sense or another, figuring out how to bias these algorithmic methods so that they’ll work in a particular scenario. So the very first one, oh, here’s why I want to show you my alpha-puzzle. So you’ve probably seen, I don’t know if you can see this, but it’s a puzzle that, once you know the trick, it’s easy to do, but it’s actually, it’s like, how many different ways are there to do this? If I tried to come up with an algorithm to discover that on its own, it would go forever.
So what we’ve tried to do in my group really was to figure out strategies where we can still apply that very simple kind of sampling-based strategy and solve hard problems like that. Another problem that was hard to apply this, the basic sampling-based strategies is if you have, to have contact. So if you want to, for example, have, you know, touch something and trace the surface of that. That’s a strategy, whereas if we’re just doing random sampling, the chance that, you know, my finger is going to touch the surface is zero. I’m never going to find that accidentally, right? If I have two arms that need to collaborate and like work together, if I plan for each one of them separately, or I need to maintain this closed chain constraint, it’s really going to be, it’s impossible to do without having some way to kind of guide the planning process. So basically what we do is we’ve looked for, we found problems that we couldn’t solve, and 4
we’ve tried to think about why aren’t these existing strategies able to solve those problems? What can we do to kind of make that work? And, you know, it’s really pretty astounding to me, even today, the different problems that we can solve with the same basic underlying algorithms.
So for example, one time we had a faculty candidate come to the, at Texas A&M, where I was a faculty member, and he was giving his job talk, and he had a robot and it was folding a box. And so I asked him, did you use a motion planner to fold that box? And he literally laughed at me, he’s like, oh, motion planner couldn’t figure out how to fold that box. So I thought our planner could fold that box. And I was sitting there with my student, and I said, you know, go on, I think we could probably fold that box. And so by the time the faculty candidate came to my office for our one-on-one meeting, I had a little animation of this box folding. So that started kind of, I’m still friends with that guy by the way, today, and he’s a great roboticist, but you know. So that like kind of pushed us to work on that. Well, then I came to another conference and I showed, you know, the animation. We started working on origami, you know, paper folding. And I showed those animations to Jean-Claude Latome, who’s like the father of, one of the fathers of motion planning. And when I showed him that, he said, “Nancy, did you ever think about folding proteins?” You know, I had never thought about folding a protein.
So I went back home to Texas A&M and I reached out to a biochemistry professor and I, who was, you know, I looked up, you know, who was doing protein folding. He invited me to his lab meeting. Then I started working on that. And it turns out the exact same algorithm we can apply that we use to plan for like a robot arm, we can use to model a protein in its molecular motions. Cause a protein is really just a string of amino acids with a bunch of angles. And so it’s really incredible, pretty much anything that, you know, in space that moves, we can apply these algorithms to.
Q1: Which research questions sparked your interest, and how have they eveolved?
Nancy Amato: Going back to this planning problem, the narrow passage problem. So that’s kind of refers to a scenario here, like, you know, if we have an open area, it’s, you know, easier to kind of plan for the motion. But if you have like a very crowded scenario or you have a narrow corridor, and I need to plan for my robot to move safely through that without, you know, colliding with the wall or with other robots, that narrow region is harder to plan in. So we have, you know, the challenge for these, you know, sampling based algorithms, which are the algorithms really that are used today on anything that we’re really planning with, you know, autonomous vehicles, everything. Those are the cases where they’re not working well. So we needed to come up with strategies for, you know, biasing the sampling in those regions without giving up the completeness guarantees that we need from our algorithms. And in fact, these algorithms are not complete, but they have this property that’s called probabilistic completeness, which essentially means if we sample enough, if we run them long enough, they will eventually converge and, you know, find a solution if one exists. But one of the challenges is we don’t always know if a solution exists. And for those, for that problem, you know, it’s still something 5
we don’t have, you know, good solutions for, but fortunately, you know, most of the cases that we actually do, you know, practically need to work in, the solutions do exist.
Q1: What challenges have you faced in your position?
Nancy Amato: One of the frustrating things I’ve had, frankly, is I have too many, I can never choose between things. I find everything interesting and I want to work on everything all the time. But there have been a few ideas I had over the years that I wanted to pursue, but I couldn’t like convince a student that it was like a cool thing to work on. And so I had a few of those things that really took me, you know, sometimes maybe even 10 or 15 years before I could get a student to work on them. So that I think was, you know, but that’s the way working with students is. Everyone has their own passion and we have to find the kind of the right way to get them excited about the things we want to work on.
Q1: How did you home in on your particular focus in robotics?
Nancy Amato: One thing about robotics is it’s truly multidisciplinary. It’s not honed in, you know, I’m in a computer science department, you know, but if you go to a robotics conference, people are in computer science, mechanical engineering, electrical engineering, cognitive science, you know, mathematics, statistics, everything. So in fact, that was one of the challenges actually at, I would say institutional challenges at my previous institution is that, and I think many institutions, that it’s kind of hard to work across boundaries. And when I was actually an assistant professor and I wanted to start working in robotics, I was advised by the senior faculty in my department, including my department head, who was a roboticist, that I shouldn’t do it. I should wait until I got tenure, you know, because it was a high risk thing that changed areas and the robotics conference, you know, conference and community is very multidisciplinary. I wouldn’t necessarily have this immediate home.
But in fact, I found the reverse. I found that robotics is one of the most open communities. And I think it is because it is so broad, you know, where, you know, I don’t even know like half my colleagues that I see at conferences. I don’t even know what their home department is. I would have no idea. And so I think that’s actually really exciting. And that’s one of the reasons I want to come back here to Illinois. And it’s one of the reasons I came here as a grad student is we do not seem to have the same kind of disciplinary silos that many institutions have here. We have a lot of these multidisciplinary institutes, like this thing called the Coordinated Science Lab. That’s multidisciplinary engineering. And actually as a grad student, I had my office there. And it’s a natural home for things like robotics. So that’s, I think, more institutions, where everything is, the boundaries between all disciplines are blending and fading. And I think we’re gonna see more and more of that. But, you know, up until the past like 20 years, that’s been a challenge, I think, for many people.
6
Q1: What tools have impacted the trajectory of your research?
Nancy Amato: Number one, like just the improvement in computational power that’s available to us has really, there were things that at the early time in my career, you know, we couldn’t even think of doing. So one example is I was, or one of, what’s it, it must’ve been at least 20 years ago. We were interested in kind of blending virtual reality and doing haptic interaction, where you could have, you know, a haptic device, a virtual reality device, where you can actually touch something. And there were potential to have like augmented reality, where you could kind of blend the physical and the virtual. And one of the challenges there is our computers were just too slow as we couldn’t actually do like collision detection fast enough to give real-time feedback. And so we were able to do a lot of approximations and make some starts, but it really was not, you know, gonna be in anyone’s home anytime soon.
And today, that’s completely different. And so there are, you know, really that is actually a current, actually it’s become, again, a research topic in my group, that we are, you know, trying to blend the physical and the virtual and have, you know, truly interactive planning where we can bring the human guidance into the loop and as another input. You know, it’s not completely offline like it was. So that’s something that’s really different. Large language models, we’re all crazy about them today. And they are having an impact in many different areas, not as much right now in the physical world of robotics, but I think that’s just around the corner. We need to kind of figure out the right way to integrate them. Right now we can do, you know, very small specific tasks. You know, we need to, the training is one of the problems. We don’t have enough data. Well, I think that problem we’re going to have, you know, we’ll have surmounted that, I would say, within five years.
Q1: Please describe your experience relating to research funding.
Nancy Amato: I’ve been really fortunate. Since my work is more algorithmic in nature, I haven’t needed like huge robotic test beds. We could come up with algorithms and then team with others who would maybe use them. And the National Science Foundation has been, you know, the US National Science Foundation has been a big supporter of computer science research. In fact, they fund like, I think it’s something north of 80% of all fundamental computer science research. And that has, the NSF funding is very flexible, let’s say, and allows you to really kind of do fundamental research. If you write your proposal, it’s what you think you’re going to do. It’s your best intention, but, you know, a year in, if you, you know, some new thing pops up and you want to, it kind of pushes you in that direction. You can do that and they’re supportive of that. So I’ve been able to really kind of pursue, you know, if we get any crazy new idea, we can always kind of tack and support that and investigate that. And that’s one of the reasons I love being frankly, a faculty member too, because it does give us that freedom to do that. You have to convince the students to work on what you want. So it’s not quite like a company where you can say, we’re all going to do this, but, you know, that kind of is then on me to be more convincing to them. So I can tell them, yeah, this is something we really should work on. 7
Q1: Do you partner with companies?
Nancy Amato: Sure. In fact, you know, there’s, there are a number of industry, university kind of partnerships that we have centers on campus. So that’s an easy way to do that. So that kind of provides umbrella funding where we can propose projects to that center and they will be funded. And we often have, you know, partners in that company to work onto. They give us and our students access to their test beds and their, you know, computational resources. The students can go for an internship and learn about proprietary problems that, you know, we do have to kind of check with them before we publish, but through those things I’ve worked with IBM, Toyota, those are kind of the two main collaborators, C3AI, some others, 3M, GE, I guess there’s quite a few of them.
Q1: Are there interesting projects you haven’t yet pursued?
Nancy Amato: Oh yeah, a ton of things. So I just told you, I get a lot of ideas. And so I think, you know, at any given time, there’s probably, you know, 10 or 15 things I’d like to work on. So there’s, you know, at the moment there’s, you know, we get to investigate maybe three, four or five of those. So a lot of them actually, I would say. I tell my students, they should go to seminars all the time because at a university we have a lot of seminars and it’s probably even more important for them to go to the seminars that aren’t in their area rather than the ones that they are an expert in because, there, they’re gonna be exposed to some different problems, some new idea. They might come across something that we can actually apply our methods to, or they get an idea from some other method that we can then apply to ours. So, you know, that’s, you know, I would say two or three new ideas a week.
Q1: is there a specific project you’d like t work on?
Nancy Amato: Well, I guess maybe this human planner interaction, this virtual reality is maybe an example of something. So, you know, it was about, you know, say 20 some odd years ago, we started working on it. We had some interesting results, then we stopped working on it. And right now I’m in the phase where, you know, I’m excited by it. One of my colleagues is. I’m trying, I’m in the process of getting the students excited to it. So it’s, I wouldn’t say that we’re definitely taking that path but I really hope so. Hopefully by the end of the summer, I’ll have a convert. (laughs)
Q1: What is your most significant contribution to robotics so far?
8
Nancy Amato: Probably the most important contribution is my students. So, you know, as a faculty member, we are, you know, training students, we’re, you know, exciting them about research. We’re sending them off to, you know, work in industry and research labs, startups, you know, as professors and they are the, our biggest product. But in terms of research ideas, I think this whole kind of sampling base planning, the idea of making those methods customized and biased to particular types of scenarios is something that really, I think my group has had a big impact in. Some of our ideas are used, you know, as is, are directly used. Others, you know, they inspire to other people. And I think that’s kind of the place that we have had the biggest impact.
Q1: How has robotics changed and how do you foresee its future?
Nancy Amato: Yeah, so when I started really working on this and I would go to the conferences, you know, we’d see this huge, you know, industrial arms. We were, you know, robots were in the factory. They would, you know, have this huge setup time. You know, they build the factory for like a year and then they could use it. They’d plan to use it for years or something. And the idea of like us having, you know, Rosie the robot in our home that would vacuum for us was like, you know, science fiction. Today, you know, robotics are doing a lot in manufacturing. They’re super, you know, you know, they’re really there. We wouldn’t be able to do what we do today. But they’re also in really making useful progress in a more instruction setting. Still, you know, not like in the wild, they are they’re not here in the university with us, but we’re starting to see them deployed. And I think that’s what we are going to, they will be assistance for us. I don’t think, you know, people shouldn’t be worrying that they’re gonna take over. But there are a lot of settings where they are already doing really important things like places where it’s dangerous for people to be, or remote settings. So we have released remote operations. They go off to Mars, right? And they run around and they get data for us. They are doing experiments, but they’re still not, you know, with us in our kitchen, you know, standing alongside me. We don’t trust them in some ways, their capabilities for working and adjusting are not where they need to be.
That’s where I think the huge advances we are making in computation and, you know, basically like cloud robotics, we have access to data from so many different places that this is going to really push us quickly to get to that point. And I think we will see in the near future, you know, I have this dream that, you know, there will be a robot that will kind of be with you throughout your life. It, you know, imagine like when you’re a child and you need to practice things over and over again, you know, that’s something that a robot would be great for, right? It won’t get tired. It won’t get bored. You know, they, when you, you know, customize tutoring, you know, the physical presence to kind of sit there and help you with your studies would be, you know, more impactful.
Even though they maybe don’t need to do anything physical, but we know like, you know, in assistive settings, just having a physical presence can have a different kind of impact. When you, we need physical therapy or something, or you need a trainer or someone to push you and say, yeah, you really should do your exercises today. You know, I could imagine having this, you know, partner with me that’s going to help me, remind me when I need to do those things. Now, 9
when you get to the other end, when you start forgetting and things like that, I could imagine like kind of a companion with your whole life. And, you know, it’s, you know, today we have our phones, but I could imagine having this more physical presence that will, you know, it’s not going to replace a person, but it can, you know, be there, a physical presence and do certain things better for us than just a virtual presence.
Q1: Has your research been applied to real-world scenarios?
Nancy Amato: So I did mention that our algorithms are used, I think, you know, pretty much any, you know, deployed, you know, algorithm is using some of the ideas from our sampling based methods. I think the application that most surprised me, it’s not robotics. So I did mention the protein folding, right? So these are different ways to apply to different things. Another really surprising application is orthodontics. You know, teeth actually move, right? So if you have crooked teeth, what are we doing? We’re trying to move them to straighten them out. And it turns out that one of my colleagues had this, you know, saw that connection and I have worked as a consultant with 3M and was designing planning algorithms to plan teeth motions to straighten them out.
Q1: What’s your advice to students interested in pursuing robotics?
Nancy Amato: Oh, go for it. It’s a great field. You know, it’s the funnest community I’ve been with. There’s space for everyone, whatever your passion is, we need it in robotics. We need people to design mechanical systems. We need people to design algorithms. We need people to understand, you know, psychology so that we can have robots that we want to work with and that can help us. Yeah, so there’s a place for everyone. And I do think that we are really gonna change the world. For example, one of our problems like with elder care, right, you know, people wanna stay home. They want to be comfortable. They wanna be taken care of. Well, we’re not gonna be able to do that without some kind of robotic assistance. You know, it could really have a huge impact on our lives. And I think it will. So I think that’s, if I had to pick any field that I think has the potential to really, you know, improve lives for many, many people, I think robotics is absolutely up there.
Q1: Is there anything else you’d like to add?
Nancy Amato: One thing we didn’t touch on, which I know, I think many people are concerned about and we need to find ways to reassure them, is their, you know, fears about, you know, being taken over by robots or our work, our jobs being taken over, you know. And I think that’s just not gonna be a problem. They are going to be assisting us. We, things will change, right? There will be certain tasks that we don’t need to do because they can be automated, but that’s gonna open up new possibilities for other types of jobs. So I think, and that’s always been the 10
case, but we should be mindful as roboticists that people are concerned. And so I think one of the things that we do need to worry about is trying to communicate this properly and reassure people. We shouldn’t be like promising that things are going to happen overnight and tomorrow because they’re not going to. And we also need to be aware of potential abuses just because, you know, we are not, you know, ethicists or, you know, that’s not our field. It doesn’t mean we shouldn’t be, you know, aware of these so that we can, you know, educate our students and reassure our colleagues that, you know, and try to design in ways to make protections if we can. 11