Interview with Giulio Sandini

Giulio Sandini: You mean early before the university?

Q1: Yes.

Giulio Sandini: Yeah, okay, good, yes. Hi, I’m Giulio Sandini. I was born in the middle of Italy, more or less, in a very small village called Correggio, close to Reggio Emilia. I spent my early life in other cities. I lived in Milan for a few years. I lived in Como for a few years. And then I moved to Genova when I started university and I’ve been staying more or less in Genova since then, apart from periods I was outside in the States.

Q1: And can you tell me a little bit about how you got into college or how did you get into kind of an interest in robotics or related domain? You were first, I think, neuro, for neuroscience, right?

Giulio Sandini: Well, yeah, I mean, I always liked the engineering and building things, you know. That’s probably because my father was a bricolage person, so he was a pharmacist, so nothing, not building anything, but he was working at home, building different kinds of things. And when I was a kid, I used to help him to do these things. So I sort of have this idea of using the hands to build things for, you know, since I was a kid, basically. So building things, then it gives you the idea that you have to follow the technical stream of education. You know, so I did my, Liceo Scientifico is a sort of, this is high school on the scientific aspects. And then I joined the university in Genova in the faculty of engineering. In those time, there was, in Genova, there was no computer science, you know, computer science was basically exploding. I started programming in Fortran, by the way, so it gives you the idea of, you know, what kind of environment I was working on in those times. So I did the first two years at the university, basically, you know, studying and giving exams. So it was a very, let’s say, very theoretical approach, you know, for engineering, you have to study mathematics, you know, geometry and all these sort of theoretical bases of engineering.

But then, you know, the third year, I wanted to put my hands on things. So I started going and asking the professors there if there was a possibility to enter into a lab. And I must say I had my first stroke of luck because I met with a professor, his name was Vincenzo Tagliasco, who was building a group of academics working on bioengineering in Genova. And he offered me the possibility of starting to work in the lab, you know, and then working in the lab at that time, you know, the group was interested in studying the models of the eye-head coordination system in humans. So which meant, you know, recording from people behaving, basically moving their head in different ways and recording them on a computer, which was one of the very first 2

mini-computers. And then after you record it, you have to process. And, you know, there was at that time, the eye-head coordination system was a very well-studied part of the human, you know, because it’s basically a control system. So the stream of research started in the United States by a person called Larry Stark, who was an MD. But he had this idea of using control theory to explain how eye-head, you know, coordination can be controlled.

​ So there was an idea of this model, you know, different boxes. And so, I mean, that’s the way I entered in the field somehow, you know, because, you know, I saw that it was possible to describe with mathematical models, physical things on one side. And then I discovered that I was very much interested in biological systems. So I like this idea, you know, of joining the study of models from mathematical point of view, specifically on biological systems, you know, humans in that case. So that was, let’s say, the entrance to robotics. Still not to robots, but, you know, to the field. And so I graduated there with a thesis on eye-head coordination. And at that time, I had the chance of meeting some people working on the visual aspects of eye movements, okay? And the group was a group in Pisa at the CNR lab of neuroscience.

​ And I had my second stroke of luck, because there I met a neuroscientist. His name is Lamberto Maffei. And at that time, the topic of interest was to study the response of single neurons, individual cortex of animals. Those times, you know, the lab animals were cats. So I was, I spent basically three years, you know, doing experiments to measure and describe how these vision neurons are replied to stimuli at different spatial frequencies, using bars, moving, I mean, how simplified things, but basically that was the principle. I want to mention this because the idea was to describe the visual system, the peripheral visual system as a spatial frequency analyzer. So claiming that vision starts by separating the images into different spatial frequency bands, which was a theory that was sort of in competition with another theory that was coming from a group at Harvard University led by Hubel and Wiesel that were recording from the same areas in monkeys. And they were describing the replies of the answer of the neurons as if they were replying to local features. Okay, so edges of different orientation. Of course, I mean, the two theory were describing the same thing, you know, you can describe something in space or can describe something in frequencies. But then, you know, that was a sort of, so it started resonating in my mind, you know, that, you know, now we know how, we know, we pretend to know how the visual system works, at least the periphery, you know, and then the engineering side of my brain popped out and said, “okay, why don’t we try to do that?” You know, and then at that time also, there was a very active field called computer vision. So I started working on computer vision algorithms, you know, edge detection, so frequency filtering, you know, all these, trying to describe mathematically and computationally what we knew at that time about the visual system.

​ So the other thing that was interesting to me at that time, you know, being in that environment, sometimes you hear conferences or lectures on aspects which you don’t know, you know, that’s the reason why you wrote the conferences and lectures. And I remember there was a lecture about the structure of the retina. You know, how the human retina, but also other mammals’ retina, is not a constant array, you know, like a camera, but, you know, it has a, the central part has a much higher resolution than the periphery. And then I started, you know, trying to see what was the effect of these space variant, a sensor into the processing itself, you know, what is the effect and why is it built like that? I mean, why doing, you know, at that time the 3

cameras and at this time too, they have constant resolution, you know, so why doing something like this? So the first thing, so I built a simulator, simulating what kind of transformation does an image goes through, going from the retina to the primary visual cortex, so which is the first, you know, one of the first points of processing. And, you know, I found that, you know, that the main motivation is the fact that you reduce the amount of information that you acquire, that the system acquires, exactly from the beginning. So you don’t start with a huge matrix. And then you sample it out, you just acquire information, visual information, which is lower in resolution, you know, and what does this means is that you need a much smaller number of neurons to process the image, okay? So when I say much smaller, I mean, you know, 100 times less information than if you had a constant resolution image with the same field of view and with the same highest resolution. So on one side, it was a compromise because you can do precision things like manipulating small things, but you also have a wide field of view. So if you go around, you don’t bump into obstacles and so on and so forth.

​ So this was the functional utility, you know, the energetic, you know, the metabolic advantage was that you had, I mean, if you make a calculation of how many neurons you need to process a constant resolution image of that size, then you make, you know, well, this kind of calculation, you end up with a brain which size is, weight is 30 tons. So it’s really a huge amount of information, you know. So that was the reason, that was the why our visual system is like that. And then, you know, I stopped asking, yes, but you know, on one side you have less information, but on the other you have to move the eyes, okay? And then, you know, came back to the model of the eye-head coordination again, and said, okay, why don’t we try this time, instead of building a software model, why don’t we build a hardware model? And just to test what kind of strategies you can put into actions if you want to look around the world, you know. And again, this was very much in synchrony with a stream of research that was exploding more or less at that time, which is conductive vision, you know, Regina Bashi, Edna Ballard. And the two things went together, because I mean, there was, on one side, the need for a biological system to explore the world. So you need to move the eyes. But on the other, it was also because of this space variant resolution, the fact that you have a low resolution image in the periphery that forces the system to look around in order to acquire the information you need.

​ And so that is the reason why I built my first robot, it was a robot head, in fact, two cameras. It was a huge device because in those times, I mean, cameras were about a kilogram in weight and so the motors were very big and so on and so forth. But, and that was, you know, well, that was the first robot I built. With the idea of studying exploration strategies, you know, how you explore the world and you build sort of a unified representation from different snapshots, okay.

Q1: And was this still at Pisa, or?

Giulio Sandini: Well, this was, at that time I was working three days in Pisa and three days in Genova, okay. I was assistant professor at the Scuola Normale in Pisa. So my official place was still Pisa, but I had the contact with the group of bioengineers in Genova who are also interested in this, and we had joint projects funded by the Italian CNR. Okay, so that was, so it was, I mean, I’ve always been, in all my life, I’ve always been jumping from one side to the other, you know, 4

trying to do this thing. So I built this robot and then, you know, you start, because I mean, then you meet a robot, you build strategies and you study the strategies for, you know, looking around, but then, you know, you really, then you want to understand, you not only want to understand the environment, you also want to interact with the environment. And, you know, at that time I was, I did, well, first of all, I moved from Pisa to Genova. I started a lab called LIRA-Lab. LIRA stands for Laboratory for Integrated Advanced Robotics, because I wanted to invest more time, not only in exploring the world, but trying to see how you can, you know, affect and modify the environment.

​ So I bought a second hand Puma arm 260, a small one, and attached it to the head with the idea of, you know, seeing not only the world, but also trying to modify the world. But then there was another important period for me. I was a somatic, I had a somatic field at MIT and working with the group of Rod Brooks, Rodin Brooks. And I had this idea of trying to see what we can learn, how to build a system by looking at how a human body develops, human body and within brackets, “the human mind” develops. So I started the reading articles and books and at MIT there is a period which is around February, it’s called Independent Activity Program, where there are courses offered by professors there. And I attended a course by Adele Diamond. She’s a developmental psychologist. So I was really taken by seeing, you know, how during development, you really built intelligence. I mean, a human baby, when he’s born, is not able to do anything, it’s a very small number of reactions. But then, you know, following steps of development, first, and it is interesting from the engineering point of view because instant of time, the system is a self-standing and complete system, but the system is evolving and becoming more and more intelligent, more and more able to act in the world. So I thought, why don’t we try to do that with machines, you know, with robots?

​ You know, and so when I came back to Genova in my lab, we decided to start working on what is now called developmental robotics, you know, which is exactly the idea of, you know, looking at human development of humans and try to see if it is possible to follow the same path and, you know, first of all, you learn how to move the neck, for example, then you learn how to control the coordination with the eyes, and then you start exploring with the hand. So there is, I mean, it is from the engineering point of view, it’s really very interesting how development, you know, in babies happens. So I had a head and then I was saying, I bought this second-hand Puma arm, and we put them together and we called it Baby Bot. That was the idea there. And we started working on, you know, coordination, how you learn how to reach for object, how you learn how to coordinate your eyes with the hand, the first shaping of the hand when you approach an object, you know, if it is, depending on the size and, so I was working in these aspects. And then, I’m going too long?

Q1: No, no, no, it’s great. I was just curious about one thing. When you were on team, it was a time where Cog was also—?

Giulio Sandini: Exactly, yes, yes.

Q1: Did you work on that project at all? Or did you work on your own? 5

Giulio Sandini: No, no, no, it was basically, for me it was a static period, but the environment there was a fantastic environment. It was called by Rod, it was called the zoo because there were people with different backgrounds. There was a a theologist, a psychologist, of course, engineers, and we met often. And the people working on Cog were, we were working on Cog there. So I had the chance of talking to them and see what were the problems, and discuss about, you know, implementing things like reaching and so on and so forth. So I did not do any, you know, experiment myself there, but there was this, you know, interaction. We had a meeting very often and discussing about what to do and how to do it. So when I came back to Genova, I had an idea, a more precise idea, of what I would like to do, you know, the sequence, I would like to pursue. So that was the, the transport of that.

Q1: And when you were working on the Baby Bot, and then, were you collaborating with other folks from other disciplines? You mentioned you were inspired by—

Giulio Sandini: Yes, yes.

Q1: You were inspired by works from a developmental psychologist and how did you, did you have your own zoo?

Giulio Sandini: Well, you know, the other discovery that actually changed the, this happened a little bit earlier, but I mean, it is very much related to the Baby Bot project because the other discovery that, for me, was very important, you know, for what was important was the mirror neural stuff. You know, so I still had this goal of merging, trying to merge the interest of engineers and neuroscientists and in fact, I organized a workshop in Genova where I invited neuroscientists, developmental psychologists, you know, just a few names, a media Emery Bizza, motor control, Klaus INAUDIBLE was a developmental psychologist, and Chano Fadiga was one of the group that actually discovered INAUDIBLE, Concetta Morrone, she was working on visual aspects and we were doing the PhD together, so it was a good friendship. There was a group from Canada, David Milner it’s called, I mean, I mean, he wrote a book with Goodale. They wrote a book on – The Visual Brain in Action, it was called and the idea that, you know, the processing of visual information goes through two different pathways, you know, what they call the “what” and what they call the “where.” The “what” is more temporal and it is, I mean, simplifying very much, but its main goal is to recognize, understand objects, recognize objects and why there was another path going more, from the occipital to the frontal part which they called the “where” pathway. I mean, it’s basically responsible of the use of visual information to control movement.

​ So this was on one side, so if you want to control movement, you know, it’s a peculiar thing. The other thing is mirror neurons, you know, mirror neurons to me is the quintessential existence of embodiment, you know. Why? Because it was a very, it is a very important 6

demonstration that, if you want to understand what others are doing, you have to rely on your own experience. So on your own ability to move, you know, this. So if you want to understand the world, to interact, you know, while understanding what you are doing, then you need your own, to put your own body in the equation. So all these things, you know, development, mirror neurons, the “what” and the “where” pathways, they claim that if you want to understand how our brain works, you have to do it with a robot, okay? And if you want to try to study that and to model these in humans, then you have to use a humanoid because that’s an important part of the equation, you know, the fact that the shape is important in that respect. So this was the, let’s say the Baby Bot Project.

​ And then, you know, those years the European community funded a rather well, a stream of research which was related to this, which is motor cognition, basically, how you reason about movement. So we had in the lab, we had a stream of funded projects, always with, or more or less with the same group of people, you know, that I was referring to, Fadiga and INAUDIBLE another person who was very key to me was David Vernon, another cognitive scientist, with whom, by the way, we had the very first European funded project together in 1994. Basically, when this IEEE robotics and automation started and the project name was Image and Movement Understanding, the title. So it was a continuity there.

​ So at the end of that period, I also met, Giorgio Metta was one of my students in Genova. And, you know, he, well, of course he’s a very bright person. He’s also an excellent engineer. You know, you can ask him to write a program, to do something and he was able to do it, whatever question you ask. So it was very instrumental to the development of Baby Bot and then we built another humanoid in the middle. But the group altogether was, you know, very much instrumental to these RobotCub project, which was, let’s say, the last part of this cognitive science period. And this RobotCub project is the one that built the iCub humanoid, but the goal there was, we had two goals, you know. One was to build a platform, humanoid platform that could be shared with labs. Not with labs where they don’t have, they didn’t have the expertise to build their own human. So building a platform, which was a sort of a reference, to try to engage as many groups as possible with the same platform, so that you can sort of increase the amount. So this was one, was a technological thing, you know, building a humanoid, that could be shared with others. But the important scientific question was cognition. So understanding interaction, basically, the human machine interaction through gestures and this kind of communication too. So putting together mirror neurons and the “what” and the “where” pathways and development. Okay, so starting with a simple, and the acquisition of simple information. So this was basically, you know, we are now more or less, you know, 20 years ago, okay.

Q1: So can you talk a little bit about the development of the iCub, particularly this shareable platform, because, of course, putting something together for your own lab is one thing, and then having something that’s —, to give to others is quite another feat.

Giulio Sandini: Oh, yes, yes. No, this was, you know, on one side, it was a strategy that we used to convince the European Union that we were doing something useful for the community, you know. So the idea has been to build an open platform. In fact, I think it was the first project 7

worldwide where we had, as a goal, an open platform. When they say open, I mean, mechanical things, electronics and software, and so on and so forth. So to have it open, an open platform. And in fact, we also got funding to build, not only a prototype, but to build eight prototypes and to give these eight prototypes, I mean, one prototype plus eight copies to other labs in Europe, in order to start this convergence of software production and collaboration. And so this, of course, poses problems which are, let’s say, not academic altogether, because you need to have something that can be reproduced, you know, without, and you know that, you know, if you build a robot in an academic environment, it works from time to time. But then if you have to remake it, it’s different and so on and so forth. So we had two small companies involved in the group, which by the way, you know, it was a group of European labs. And the people working in these labs, you know, they are all now here at the conference for the 40 years. You know, there was INAUDIBLE Billard, Darwin Caldwell, Rolf Pfeiffer, like Amim Asfoor, Ludovik Rigetti, you know, it was really the size of, you know, Luciano Fadiga and David Vernon and Klaas van Nofte. So it was really a mix of minds, you know, which to me, it was, I mean, I think it’s still unique. So when you hear people saying, who is the father of iCub, you know, I say that the father of iCub is this group. So there is not a single one who can say that he’s the father. Someone work on the mechanics, somebody else work in other things, but basically that’s, I claim that I am the mother of iCub, but just because I create the environment for the iCub, you know, to grow and be born somehow, but that’s another issue.

​ So we built these robots and we gave eight of them to labs and some of them were labs were involved in the project itself, but we also put up a sort of a call for applications. We wanted people to address cognition problems using a humanoid. And so we received, I think, I remember something like 30 proposals from different labs and we selected four. And then the project was, you know, more or less at that stage. When it happened, another important event in my life, you know, and that important event was that the Italian government decided that it was needed, it was necessary to start an Institute of Technology in Italy. Okay, so this was, I mean, the law established this, it was around 2004, which was exactly the year that the RobotCub project started. So I was involved as one of the three persons writing the scientific program of IIT. And that was a dream come true for me because it was possible to propose, you know, to have engineers and neuroscientists in the same place, plus material scientists because I thought it was.

​ So the, let’s say, the main theme was anthropomorphic robotics, so the study of neuroscience and in fact, which Luciano Fadiga was involved, Giorgio Mehta, of course, was involved. And so the group of RobotCub was there, ready to start to work on a project which fitted very well. The, then of course, there were other aspects of neuroscience and particularly other aspects of material science that were new to the idea of a RobotCub. But we thought that, you know, these three ingredients, you know, they have synergies that can be exploited, that can be exploited in an institute which starts without barriers. You know, academia, in academia you have the School of Engineering, the School of Medicine, you know, they are, then you can collaborate, but you know, somehow they are, they have different interests.

​ There, you can start from scratch and say, okay, now let’s put this together and see what is coming out. So this, and this was a lucky, a very lucky situation because, you know, the RobotCub project was a five year, but the first lab of IIT was, you know, from 2007. So it was already three years into the project. And, you know, after five years, the project would have, you 8

know, sort of disappeared somehow, you know, the robots would still be there and everything. But then, you know, with the staff of IIT, there was a place where this project could be supported more, you know, for a longer time. And in fact, you know, it’s now still a very lively project. And not only that, but there was also the origin of other projects related to robotics, and related to neuroscience, which are now, you know, the mainstream of the Italian Institute of Technology. So that’s more or less, you know, then, you know, since IIT started, you know, we continue to develop both the RobotCub, but I had the possibility of collaborating with young researchers, both working on neuroscience aspects and working on robotics. And now, I mean, everything is in their hands, you know. I still work with them and collaborate, you know, Alessandra Schutte and Monica Gori and George, of course, George Mettern still there, even higher than, now the scientific director of IIT. So I’m glad because I think the idea worked out and I see that this will go on in the future.

Q1: You mentioned kind of different kinds of funding you had initially from the European Union. In terms of then continuing beyond the five years, I think within IIT, how is, can you say a little bit more about how, you know, funding works at that point? Yeah, I understand the issue of, you know, these kind of short-term projects with an end, and then you don’t know how to get going. And so this seems like a very unique example of what continued.

Giulio Sandini: Oh, yes, yes. No, it was unique and very lucky because I mean, there was a period, it was, you know, from, I would say, 1998, 1990 to 2004 where the funding of European Union, it’s targeted, broadly speaking, it’s targeted to different areas. And one of these was in fact, cognitive science. You know, at that time, there was one of the commissioners, also one of the people that, you know, listened what the researchers are interested in and then tried to condense and propose as a stream of research. And there was this stream on cognition, basically, starting from cognitive vision and then going and developing. But it was a period of more or less of five, six years. And how this projects works, work is that, of course, you have to propose an idea about the project, you have to find a consortium, so you have to find different laboratories collaborating with you.

​ And we started with vision, basically, and we had projects which were more or less interweaving with each other continuously for these five and six years. RobotCub, I would say, is the top, because in that case, it was what was called an IP project, which meant a very big project. It was about 8.5 million euros funding. So it was not a small, even for the European Union. It was a big one. And then, you know, if there wasn’t IIT, we would have continued to ask for funding to continue the project there. But, you know, the objectives goes up and then go down. So cognitive science was steady for a given period. And then, sort of, I wouldn’t say that it disappeared, but it decreased very much. So if it wasn’t for the funding from IIT, the iCub project, theRobotCub project, would have gone down slowly. Because it takes a lot of effort to maintain a platform to maintain an environment which is still open, you know, still people can go there and get the software and processing or put their own software and share the software there. So it’s still ongoing, and it has been ongoing for 20 years now.

9

Q1: So if you developed both your research and your platform, were there particular tools or innovations or technological innovations that you felt really helped transform your work, make it easier, enable you to do the things you were dreaming of that might have been, you know, tougher when you had the baby robot, for example, right? So it’s just more like if there are, you know, particular transformative technologies that have helped you along the way.

Giulio Sandini: Well, yes and no. And I would also, on one side, I mean, in terms of individual technologies, I would say the scheme that was developed for iCub is something which was very unique and still very unique. So that was the sensor, in fact, that was not available. All other sensors, you know, the vestibular, the cameras, the microphones, these were, you know, standard kind of things. So the innovation there was to be able to build a system of that complexity, you know, cable driven, which was possible to program with, you know, the development software and the software architecture already available and so on and so forth. So we had the chance to use the iCub, in fact, in two ways, okay. The first one was sort of a classical, you know, a model, okay, a model of behavior, you know, that to see how you can learn how to grasp consistently an object, you know, how you can use two hands, you know, so more a model of the system. The other that came up as, in my view, even more interesting is the use of the iCub as a partner, as a stimulator, I call it, you know. And that is, you know, because you can control the movement in a very accurate way. So it was possible to study more precisely what are the messages that we exchange when we communicate, you know, not only hand gestures, but also more subtle things, you know, like if you are nervous, you start moving in different ways.

​ Our own movement, well, scientifically, is defined as biological motion, you know, and which means that we start by accelerating, we go at maximum velocity, and then we slow down. And we were able to control this kind of movement very precisely to see which are the parameters of the movement that communicate a message, you know, is the velocity, is the trajectory, you know, all this kind of question. We could do experiments on trying to see what is the effect of the motion of the iCub on the movement of the person interacting with the iCub. So, if I move in a nervous way after a while, the other person also does that. So there is this mode of contagion, they call it, you know, this synchrony. So it became, it has become an experimental platform to study social robotics, human robot interaction in different ways. If I grasp an object and I move an object in another point and you are looking at me, you basically implicitly understand what is the weight of the object, okay? So it’s like seeing forces, you know, which is something strange, you know, from the visual point of view. And so this is still now the main research stream that is carried out in the lab I work with, which INAUDIBLE now. And so it’s this kind of thing, you know, to try to see what are the parameters of the movement, or the parameters of the interaction. What is needed, you know. Eye contact, for example, the way we look around. All these gestural messages that are known in psychology, but strictly speaking, nobody knows what are the parameters of these messages that we exchange, which are really relevant. And, you know, in fact, it is important if we want to build robots interacting with humans in a sort of a friendly way.

10

Q1: So you also think of, you’ve been talking about the kind of scientific and engineering-oriented contributions. When you think about the kind of societal implications or significance of this work, what do you think are the most important components in terms of societal—?

Giulio Sandini: Societal outcome. Yes, that’s the other point. I mean, we also have these, we always have these, you know, loop of science, technology, and society. But directly speaking, we don’t have a personal robot. We don’t have a robot companion. So in terms of the robot itself, no outcome, no social outcome. We have, you know, partial outcomes, because, for example, the same electronics and the same actuation that was used to build the eye camera, we also used it to build what we call the rehabilitation robot for the wrist. Okay. So this is one thing. Now, the other thing is, you know, for example, by studying how we learn to understand the movement of other people, you have to think about how you represent your, let’s say, the shape, the kinematics, and the movement of your own body. Okay. So how you develop that, you know, how, when you are born, you don’t know about your own body. I mean, for you, your hand is something strange, you know. And when you grow, your body shape changes. So, I mean, there must be a mechanism there which relates vision, the vision of your own hand with the movement of your own hand.

​ Okay. So that, why I’m telling you this, because this means that during development, along your own life, you basically have to store somewhere the correspondence between what you see and what you do. Okay. A relationship between your own body and the movement of your own body. So we started asking, you know, if you are born blind, what does this do? You know, people who are born blind, motorically, they are not very good because they don’t see. So they tend to stand very much, you know, and they don’t see their own body. So they don’t have, they cannot build the correspondence between a position in space of your hand. They don’t have this representation. They have an motorical representation, but not a visual representation. So there is no relationship between these two. So in the group, you know, working on visual perception, people was interested in understanding development of the visual or multimodal or sensor information.

​ So we came up with the idea of trying to supplement the lack of visual information in other ways. And one of the projects that we developed is now, let’s say, a product. It’s not a commercial one, but it is provided to centers who are interested. It was a very simple device, you know, a small watch which contained inertial measures and a beeper. So, I mean, if you move your hand, it makes a noise. And if you move it fast, it is higher, so you can modulate the frequencies. And we had a collaboration with a center working with blind children. And in fact, we showed that it was possible to build the representation of the body instead of using a vision, using audition. And so that was an outcome. Then, you know, linked to that, there are others. But what is important, I think, is certainly the device, you know, the outcome. But what is important is to have people thinking together about these aspects so that, you know, you hear someone talking about body representation and then people working on visual perception think, okay, why, if you don’t have vision, what does it do? What happens? You know, what are the relevant information that you have to build to represent, to build up the representation of the body of 11

someone else? So, it is, from these discussions, that ideas and projects come out, you know. So, it’s not only the device itself or the individual outcome.

​ So, we had a few of these. But, you know, that’s what I was saying. I mean, it depends on, it very much depends on having the people in the same place, you know, interested in similar things, you know, discuss among themselves. You know, we had a history of very interesting experiments done by psychologists to highlight aspects of development which we have not known, you know. How early you develop the affordance, for example, the ability to discover affordances. Or how early you start anticipating your movement if you have to grasp an object which is moving. So, these were interesting for robotics. But developmental psychology never had the idea or whatever, you know, to do these kind of experiments. So, you know, Plasmon-Nofsten published at least two or three papers on science and nature coming out from the robot camera project. So, it’s not only the device, it’s knowledge of the problem that falls forward.

Q1: Wonderful. Thank you. So, I think we’re almost out of time.

Giulio Sandini: Sorry, also —

Q1: I was going to say, is there something we haven’t covered that you weren’t asked?

Giulio Sandini: No, the other thing I wanted to say is that, you know, IIT started with the iCub. Now we have plenty of different kind of robots, you know, like the robots that are often used to help humans to fatigue things, you know, during work. Even robots that, you know, will fly someday. So, that’s the other outcome. So, it generated the technology that was used and is used in iCub as being used and further developing the building other kind of robots. So, that’s where, you know, it’s easier to have technologies coming out. We have actuators coming out, new actuators, new touch sensor coming out from other projects. But, you know, that sprouted from the iCub one. So, it’s extending. Yeah.

Q1: Is there anything else you would want to add before we conclude?

Giulio Sandini: Well, you know, maybe five minutes on the dark side. You know, that’s… You know, the… Not just the dark side. That’s… You know, if I had to tell you now that I am satisfied with the outcome that we had on cognition with respect to the objective that we had at the beginning, I would say that, yes, I’m happy, but not extra happy. And that is related to the fact that we use this platform to implement different kind of behaviors. So, how to reach, how to grasp, you know, how to communicate nervousness, how to, you know, establish a dialogue with the person you are interacting with and so on and so forth. But each one is its own… It’s its own experiment. I mean, they share the platform, but they don’t share, you know, the ingredients, you 12

know, to do something if I’m using these three ingredients, to do something else, and using three other ingredients.

​ So, in that sense, in that sense, it is something which is big, because we thought, you know, you have a platform so that you can put your software in and then you can integrate things. It didn’t work. You know, we have individual solutions. Okay. That is while, you know, so the robot is able to execute a lot of actions. Okay. What we discovered we miss is the fact that the robot can execute but cannot imagine movements, you know, like, you know, I can close my eyes and imagine to be playing tennis, you know, I can see myself playing tennis, you know. I can see myself, you know, doing a somersault, even if I’m not able to do it. So I can imagine all these things, you know, and to be able to do this, you know, which is a neuroscientist has demonstrated that we actually simulate movements before doing them, you know. So it’s a part of the system, you know, robots don’t have this, and the fact that they don’t have this, well, it poses a big limitation on the kind of performance you can do. You cannot beat the experience. If you interact with somebody, when you turn it off, it’s off, you know. So this ability to imagine, it has to be part of the system. And if you think about that, it highlights the kind of processing somehow that you may need to do that, you know, you need a memory where you can store, but you can also retrieve very easily and you can associate things. In machines, we are using new computation while our brain associate things, you know.

​ And so I think in order to advance, we should start thinking about what I was calling, you know, the imagination machine. What do you need to be able to imagine action, and to acquire the experience of what you do? And if you think about what you need, then on that structure, on that model, you could think of adding the ingredients, but, you know, sharing the same structure, you know, the, which again is trivial. If you think about development, it is true that a child learns one thing at a time, you know, very slowly and so on and so forth. But the structure is already there. So the brain does not change much during, during development. Of course, you know, some neurons are dying, you know. So the interaction shapes the structure, tunes a little bit the structure, but the structure is there. So if you learn how to play tennis, implicitly, you learn how to throw a stone, because it’s the same kind of movement. Why? If you have a robot, you know, you learn one thing and then you have to relearn it if you have to do something, something like that.

​ So I think that we, I mean, the, let’s say the big obstacle, but also the roadmap, you know, if you want to be able to really, you know, intelligent or cognitive systems, it is to think about what you need, you know, for a robot to be able not to act, but to imagine actions. To be able to replay, in its mind, the actions to do. They call it cognitive architecture. You know, that’s, that’s basically, stream of research which has been going on for many years. But now I think, you know, if we want to be able to build the personal robots we are claiming, you know, helping elderly or disabled people, you need the robots that are able to understand what the other person is doing to establish a relationship. You know, you need a robot to be neural somehow, you know, think explicitly. But in order to do that, you need to have an idea of what is the structure behind. Okay.

Q1: Anything else that you want to add? That we missed? 13

Giulio Sandini: I don’t think so. I mean, I like this, this feed very much. And yeah. Yeah. No, no, no, no.

Q1: You have so much experience, and a rich story, from the kind of beginnings to where it’s going.

Giulio Sandini: Yeah. Initially I thought it was much easier. You always think about that. That’s always the case. But, you know, that’s that is — so when I hear about, you know, artificial intelligence now. I’m not even sure if the technology that we are using is enough, is good enough. You know, these computational, certainly not, not GPUs. GPUs may be a part of the architecture, but it’s not the solution. You know, it’s the association, which I think is relevant, you know. It’s not memorizing things. It’s retrieving things from memory. If you, if you, if you smell something, you know, which reminds the cake that your grandmother was cooking for you, I mean, how does it happen? You know, we don’t have any physical system, you know, the, what they call the machinery of the mind is not, I don’t think is based on computation. You know, that that’s the example I was making, you know, is that you have to learn how to do math, but you don’t have to learn how to associate this, the smell of an apple to the shape of an apple or to the taste of an apple. You do it naturally because it’s there. The system is done, is built to do that. And we don’t have no associative memories. It’s a it’s a very important stream of research in my view, but it’s still not there. INAUDIBLE

Q1: Thank you!

Giulio Sandini: Good. Okay.

Q1: Thank you, Giulio!

Giulio Sandini: Okay, well, I hope it’s going to work. 14