Interview Transcript
Q1: So let’s start with you telling us your full name, where you were born, and just a little bit about your early life, perhaps.
Alois Knoll: My name is Alois Knoll. I was born in Stuttgart, Germany—the famous city where the car was invented. I was educated at several universities: the University of Stuttgart, the Technical University of Berlin, and then I went to the University of Bielefeld, which is also in Germany, where I became a full professor very early in my life at the age of 31 or 32.
Later I moved to Munich, which is arguably Germany’s best university. I have been in this field of robotics and AI—which I have always considered to be the same thing—for more than 40 years. So, four decades of AI. I have seen many summers and winters of AI in the meantime, and it’s been a very interesting journey.
Q2: Can you tell me a little bit about the early times of your journey? For example, when did you first become involved with AI and robotics, or both? Was it both at the same time or separately? Just a little bit of how that came to be an interest.
Alois Knoll: That was in late 1985 when I became a research assistant at TU Berlin before the wall came down. It was a totally different world. The topics that were en vogue at that time were things like collision avoidance, path planning, blocks world, and robotics programming languages—how do you program a robot?
The equipment we had was not overwhelming. It was a PUMA 560, a standard six-axis robot. The first computer vision system I worked with used a CCD camera with a resolution of around 570 by 400—very primitive compared to today. But we developed our ideas and published our first papers on that. My very first publication was on an AI topic: scripts as a representation for scenarios in robotics.
Then I specialized in sensors—ultrasound, cameras, and sensor fusion. At that time, sensor fusion was something nobody really understood; people thought if you had one sensor, that was enough. So conceptually, it was a totally different world. There were robots and there was AI, like expert systems, which was the fashionable fad and hype at the time. AI has always been a fascinating topic for researchers since its inception in 1956.
That was the early days, from 1985 to maybe 1988. Then I went more into multi-sensor fusion and human-robot interaction because it was clear that you wanted to talk to robots rather than use a teach pendant to program every single movement and motion pattern. We were thinking about defining action primitives and how to trigger them through language statements.
In the early-to-mid ’90s, I was involved in a collaborative research unit sponsored by the German National Science Foundation (DFG). In that unit, we built the very first robot system that made it possible to assemble a wooden toy airplane based on language input. We combined language, text, speech, vision, and action. It worked quite well, but it was a very expensive and sometimes exhausting process that took us about five years to build. In the end, the system ran on about 12 Sun workstations working together. It had a mean time between failures of five minutes, but it worked! That was really the first demo of an online dialog management system combined with a robot cell of three robots capable of assembling a toy airplane. That was a real breakthrough, and even today it is quite impressive to see how well we did before high-performance, highly sensitive, torque-controlled robots even hit the market. We had to build it all by ourselves.
Q3: You mentioned it was a collaborative research unit. Who was the collaboration between?
Alois Knoll: A collaborative research unit—or Sonderforschungsbereich—funded by the German National Science Foundation is a large-scale project scheme involving 15 to 20 research groups that receive funding for four-year periods, which can be extended up to 12 years total. However, it only gets granted when you demonstrate that the sum is greater than the parts and that the groups work closely together.
In the case of this Bielefeld collaborative research unit, which was called “Situated Artificial Communicators,” it was a collaboration between the Department of Linguistics and the Department of Technology. That was way ahead of its time in the early ’90s.
Q4: When did that collaborative unit run, and what was its focus?
Alois Knoll: It started in 1993 and ended in 2005. I was part of it during the second phase, from 1996 to 2000. During that time, we built up the entire robot demonstration system and conducted research on sensors and multimodal human-robot interaction. It was a pioneering project, and I was not aware of anyone else doing that at the time.
Q5: You mentioned realizing that we can’t just keep programming robots manually and need to communicate in natural language. Where did the need for that really arise?
Alois Knoll: Roboticists have always tried to broaden the range of applications for robots. If you can talk to a robot the way you talk to a colleague, you can use it wherever you want. Programming every single step in a motion pattern using a programming language or a teach pendant is cumbersome, boring, and won’t take you far.
A robot has to be aware of its environment. There are two sides to it: perception of the environment and perception of the human instructor. A crucial issue is establishing a common understanding—if I talk about a “lab,” the robot also needs to have a concept of what a lab is. Otherwise, it cannot understand you.
It was a long-term endeavor where we learned a lot about human language and the human mind. It was very intense trying to make yourself understood by a technological object. Having linguistics from the humanities working alongside technology and robotics was a very progressive and forward-thinking approach that helped guide the future of robotics.
Q6: What were some of the most interesting insights or philosophical takeaways that came out of this interdisciplinary collaboration focused on human-robot understanding?
Alois Knoll: On a philosophical level, demonstrating that it was possible to talk to a robot system over a narrow domain was a major step toward intelligent robotics, even though we didn’t call it AI at the time.
Secondly, we observed how difficult this is because human language is highly ambiguous and context-dependent. It depends on the experience, background, and mood of the dialogue partner, yet you expect the robot to understand what you mean when even another human might not in that specific context.
When it finally worked, people’s reaction was often, “So what? I can do that better.” Of course a human can do it better, but modeling that capability into a machine is extremely difficult. It is satisfying to see how that concept is gaining traction in all fields today, but those were really the beginnings: managing a live dialogue between a human and an agent that can actively change its shared physical environment.
It was important that this was a common workspace where the human and robot worked together in real time, so response times had to be acceptable. If your GPS doesn’t understand you, you repeat yourself; but if a robot misunderstands you in a shared workspace, it could do something wrong or even harm you. Meeting those nonfunctional properties made it much more difficult, but it worked—at least for five minutes until we had a crash!
It paved the way for many interesting follow-up research projects and helped advance AI, even if people at the time criticized it and asked why anyone would spend so much money assembling a toy aircraft with robots when human labor was cheaper. Now people recognize how far-looking that work was.
Q7: Did you get critical feedback at the time, and how did you motivate the project?
Alois Knoll: There was gossip where people said, “This is just a toy collaborative research unit; we don’t have to take this seriously, we do the real stuff.” There was no connection between what we were doing and what other people thought was relevant, which was frustrating at times. On the other hand, we had our funding, we could do what we wanted, and in the end, we were very successful.
Q8: What were some of the offshoot projects that continued from that work?
Alois Knoll: The next thing we built in the mid-’90s and early 2000s was a mobile manipulator—a mobile robot used in biotechnology pilot plants for sampling products from fermenters, in collaboration with Bayer and UC Berkeley/San Francisco.
In biotechnology, when developing processes for mass production, you have large fermenters and cell cultures producing agents or medications. You need to draw samples into vials at precise time intervals for automated analysis. This is a slow, meticulous process that must be performed in a sterile cleanroom environment.
We developed a mobile manipulator controlled by a camera—featuring hand-eye visual coordination—to handle these vials according to strict FDA regulations. Achieving the required precision and handling delicate equipment with needles in a lab environment was extremely challenging, but we successfully built an operational prototype.
However, it ultimately turned out to be too expensive commercially because cheaper manual labor beat the robotic solution at that time. Nevertheless, it was a very interesting lesson in understanding the pitfalls and strengths of mobile manipulation.
Interestingly, ten years later, we implemented a similar system for blood analysis in a hospital setting. The original project took us five years from start to prototype, but the hospital project only took us six months, demonstrating the massive increase in productivity for developing robotic solutions over that decade.
Q9: What key tools contributed to that boost in productivity for building robotics applications?
Alois Knoll: A crucial tool for boosting productivity was ROS (Robot Operating System). It started as a small research project at Stanford and Willow Garage and was never originally intended to be a worldwide standard, but with community support, it became the de facto standard. ROS made it possible for roboticists to focus on higher-level project content rather than reinventing hardware device drivers, firmware, and basic infrastructure every single time.
Q10: What were the technical challenges regarding mobility and precision with that mobile manipulator?
Alois Knoll: For that mobile manipulator, the central challenge was combining sensing and visual servoing (visual control) to allow the robot to drive through large rooms while achieving a end-effector precision of around two millimeters at the gripper. Very few solutions at the time were capable of that.
We tried to motivate the automotive industry to adopt similar mobile solutions, but it was too early. Even today, speed remains a major issue in non-industrial robotics. Complex tasks that humans perform quickly are still done very slowly by robots, such as humanoids folding a shirt. Speed and flexibility remain important lines of ongoing research.
Q11: How did you form relationships and connect with industry partners like Bayer for these application domain projects?
Alois Knoll: It requires living and working in an environment with dense opportunities—universities, research institutions, and industry nearby. Clustering is key. Silicon Valley is a prime example of a network, and in Germany, we had a dense carpet of research universities, institutions, and automotive companies working closely together. Progress typically happens where people are in close spatial proximity.
Beyond proximity, you have to actively network, attend trade fairs, and go to academic conferences. When I started around 1987 or 1988, conferences didn’t have industry exhibitions. Today, exhibitions at conferences are vital for networking and human dialogue, which is where science happens across disciplines—from linguistics and control theory to electronics, sensors, computer science, and psychology.
Q12: What were some of the other major initiative projects you led after that?
Alois Knoll: One major project was ECHORD (European Coordination Hub for Open Robotics Development), a large-scale project funded by the European Union that ran for ten years (2009 to 2018) with a budget of roughly 50 million euros.
Having a long-term horizon is essential for complex, interdisciplinary work. ECHORD was designed to bridge industry, research organizations, and academia. European projects often suffered from a reputation for being bureaucratic and difficult to access, so we structured ECHORD around “experiments”—small-scale projects with short proposals, limited budgets, and 12-to-18-month timelines.
This encouraged teams to rapidly develop products or use the experience to submit larger follow-up research proposals. We evaluated 200 to 300 proposals. Even teams that weren’t funded found the process valuable because it prompted new collaborations. ECHORD produced several startups and established Robotics Innovation Facilities, which was a novel approach at the time.
Q13: How did that lead into your involvement with the Human Brain Project?
Alois Knoll: ECHORD was preceded by a European Network of Excellence called NeuroIT, which explored combining IT and neuroscience. That experience naturally fed into the Human Brain Project (HBP).
From 2007 to 2009, I served on the Future and Emerging Technologies advisory group in Brussels, where we defined the “European Flagship Projects” to put Europe on the global map for large-scale science. We convinced the European Commission to allocate 1 billion euros over ten years for flagship initiatives.
After a competitive process involving 23 initial proposals and six full proposals, two flagships were awarded: the Human Brain Project (coordinated by EPFL and Henry Markram) and Graphene.
The Human Brain Project was highly ambitious and controversial, involving over 150 research groups across neuroscience, robotics, physics, and computer science. Managing a 10-year, 1-billion-euro project across diverse academic cultures was extremely complex. Despite the ups and downs, it was a massive success that produced thousands of published papers, valuable open research data, and a lasting European research infrastructure (EBRAINS) that continues to support follow-up research today.
Q14: You’ve highlighted many successful projects. What about initiatives or paths that didn’t work out as planned?
Alois Knoll: Nobody likes to talk about their failures! However, that is a fair critique of the scientific publishing system—you generally cannot publish negative results or failed approaches, even though knowing what doesn’t work is often more useful to the community than knowing what does, as it prevents others from going down blind alleys.