Lessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems

1 Introduction

The Amazon Picking Challenge 2015 (APC) tested the ability of robotic systems to fulfill a fictitious order by autonomously picking the ordered items from a warehouse shelf (Fig. 1). The system presented here outperformed the 25 other entries, winning by a significant margin. In this paper, we provide a technical description and experimental evaluation of our system. Our system-building experience led to the following insights: Robotic systems can be characterized along four key aspects. To develop a shared understanding of system building we should explore these spectra. For manipulation in unstructured environments, we believe that certain regions of each spectrum match the problem characteristics most adequately and should be examined by roboticists with increased emphasis. Those four key aspects are:

∗Authors contributed equally to this work and are ordered alphabetically. We gratefully acknowledge the funding provided by the Alexander von Humboldt foundation and the Federal Ministry of Education and Research (BMBF), the European Commission (SOMA project, H2020-ICT-645599), the German Research Foundation (DFG) (Exploration Challenge, BR 2248/3-1), and the travel grant provided by Amazon Robotics. We would like to thank Barrett Technology for their support and our team members Raphael Deimel, Roman Kolbert, Gabriel Le Roux, and Wolf Schaarschmidt, who helped creating the winning system.

A. Modularity vs. Integration: In robotics, success is determined by the behavior of the entire system, not by individual modules [Brooks, 1990]. Still, a high degree of modularity allows breaking down problems into simpler subproblems. Wrong modularization, however, can make solving problems unnecessarily difficult. Until we fully understand which modularization is most adequate for manipulation in unstructured environments, we suggest to build tightly integrated systems and constantly revise their modularization.

B. Computation vs. Embodiment: Robot behavior results from the interplay of computation (software) and embodiment (hardware). Computation is a powerful and versatile tool but adapting the embodiment sometimes leads to simple and robust solutions. We suggest that in manipulation, one should consider alternative embodiments as part of the solution process.

C. Planning vs. Feedback: Planning performs search in a world model, leading to verifiable solutions. Feedback from physical interactions, on the other hand, reduces uncertainty and allows to find local solutions without expensive computation. We suggest to use planning only when necessary and explore the use of feedback as an alternative when the manipulation task does not require global search.

D. Generality vs. Assumptions: For robotics research, finding general solutions is highly desirable. However, solving the most general problem might be unnecessary or even unfeasible. We suggest to search for reasonable and useful assumptions that aid solving manipulation problems in unstructured environments. By extracting, sharing, and revising assumptions that prove useful for an increasingly broader variation of a problem, we will naturally progress towards a general solution.

These aspects are not novel and certainly will not surprise the robotic practitioner. However, what should come as a surprise is the sparsity with which the corresponding spectra have been explored by our community and how rarely these aspects are used explicitly to characterize robotic systems. Case in point: Our solution to the APC explores different regions on these spectra than most other challenge entries [Correll et al., 2016]. We believe that these differences were crucial for our success.

We propose that by making the four key aspects of robotic systems (and possibly additional ones that we did not identify yet) explicit, our community will begin to understand the mapping of problem characteristics to the appropriate regions on these spectra. Our paper is, of course, only a single data point in this endeavor. But if our community starts characterizing robotic systems along the proposed axes, thereby making design choices transparent and comparable, we might move towards a scientific theory of system building [Atkeson et al., 2015; Hawes et al., 2010; Katz and Brock, 2011].

2 The Amazon Picking Challenge 2015

The APC consists of autonomously picking twelve out of 25 objects from a warehouse shelf and placing them into a storage container (Fig. 1) within 20 minutes. The robot knows which objects are contained in each of the shelf's twelve bins, but not of their exact arrangement. For each successfully picked target object, the robot receives 10, 15, or 20 points, depending on how many additional objects were in the same bin. The 25 objects varied widely in size and appearance.

3 Technical SystemDescription

We now describe the hardware and algorithmic components of our solution. We will mention connections between our design choices and the four key aspects.

Hardware Components We use a 7-DoF Barrett WAM mounted on a Nomadic XR4000 mobile base. The inclusion of holonomic mobility-a choice of embodiment that set our solution apart from most other entries in the APC-greatly facilitated the generation of motion. The ability to reposition the base enabled the arm to easily reach inside all of the bins. Our end-effector consists of a modified crevice nozzle with a suction cup mounted at its tip. An off-the-shelf vacuum cleaner generates sufficient air flow to lift up to 1.5kg. It can reliably pick up all challenge objects except for the pencil cup. Grasping success is rather insensitive to the exact contact location with the object, leading to reduced requirements for perception. At the same time, the end-effector's thin shape reduces the need for complex collision avoidance, as it easily fits in between objects, pushing them aside if necessary. This simple choice for the end-effector illustrates that an appropriate embodiment simplifies different aspects of the overall solution, including grasp planning and perception.

Motion Generation Objects are picked using two predefined grasp strategies: a top-down grasp, and a grasp approaching the object from the side. Both primitives deliberately move the end-effector into the object and push it against the floor, walls, or other objects. This is an example of exploiting the environment to guide manipulation [Eppner et al., 2015] using haptic feedback. The execution of picking motions is realized with continuous feedback controllers. We transition between these controllers based on discrete sensor events. This behavior can be described by hybrid automata [Egerstedt, 2000]. The automaton also contains countermeasures for common failures. The resulting hybrid automaton consists of 26 states and 50 transitions, of which 34 deal with error handling, e.g. if the robot detects an undesired contact with the shelf, it retracts from the bin and reattempts to pick the object later.

Object Recognition Based on data from an RGB-D camera three steps are performed: feature extraction, object segmentation, and bounding box fitting. The first step extracts a number of task-specific features for every pixel of the RGB-D image. These features include information about color, distance to the tracked shelf model and height within the bin. Instead of searching for features that could solve the general object recognition problem, these task-specific features rely on strong assumptions (e.g. that objects are placed in a known shelf). Statistics about these pixel-features for each object are derived from manually segmented training images and enable the second step to find the image segment that has the highest probability of belonging to the target object. This step exploits additional assumptions to facilitate segmentation, e.g. that only a small subset of all objects is present in every bin and that the robot knows which objects these are. The third step takes the point cloud for this segment and fits a bounding box. This allows the robot to decide where and from which direction it should perform the pick. Our object recognition pipeline is described in [Jonschkowski et al., 2016].

4 Evaluation

The APC provides an in-depth evaluation of our system, comparing its performance with 25 teams from around the world. We complement these results with nine additional experiments, using object configurations from the competition.

Quantitative Evaluation During the competition we scored 148 out of 190 points (2nd : 88, 3rd : 35). We attempted to pick all twelve objects and were successful for ten. An average picking motion took 87 seconds. This allowed us to maximally attempt 14 picks within the 20 minutes of challenge duration.

We reenacted all five shelf configurations that were used in the challenge for further testing. We performed two trials per setup, using the robot system from the challenge without modifications. During 200 minutes the robot picked 95 objects, of which 85 were target objects. On average we collected 117.6 points (σ = 29.2) which is 62.5% of all available ones. This shows that the competition run was on the upper end of the system's capabilities. Still only one (72 points) out of ten trials would have lead to our team placing second.

SystemLimitations From the 120 attempted picks, the system picked ten wrong objects and failed to pick 25 objects. We think that all failure cases (Table 1) can be addressed by shifting along the spectra of the proposed aspects.

Object Recognition Failures: We attribute 13 failed picking attempts to the object recognition pipeline. These failures occur when our local features cannot discriminate between the objects present in the target bin, resulting in wrong object boundaries or mistaking another object for the target object.We believe that object recognition can be improved most effectively by shifting along spectrum A towards tighter integration, and along spectrum C towards more feedback. We could reject poses that result in physically implausible configurations by tighter integration of segmentation and geometric pose reasoning. Moreover, we could decrease the likelihood of picking wrong objects by weighing objects or visually inspecting them after the pick.

Bulky Objects Stuck at Removal: Eight scenarios contained a large box which could only be removed from the bin by tilting it. Our system failed on all attempts. The long bottle brush also got stuck once on the shelf lip and dropped. To address these failures, we need to shift along spectrum C towards (motion) planning. Planning would allow us to reason how to reorient objects to remove them from the bin.

Small Objects: Out of ten attempts, the robot failed nine times attempting to pick up the small spark plugs. In the competition run, the robot even picked up a non-target object instead. These failures result from the fact that the reaching movement is executed open-loop, accumulating a significant error. This can be addressed by shifting along spectrum C towards more feedback, e.g. by using visual servoing.

Displacing Objects: In five out of ten attempts, the robot toppled over the glue bottle. The bottle then required a reattempt from the top. In two cases the robot did not have enough time for a reattempt and lost points. As before, this failure case can be alleviated by additional feedback; tumbling could be detected earlier and lead to a different strategy.

Pencil Cup: The meshed metal pencil cup does not have enough solid surface to pick it with suction. This failure mode shows a limitation of our chosen embodiment (Sec. 5.2). It suggests possible extensions to our end-effector, e.g. adding a mechanical or magnetic gripper.

5 Key Aspects of Building Robotic Systems

We will now generalize our experience from the APC to building robotic systems in general. For each aspect we present arguments and examples for both ends of the spectrum and position our system on it.

5.1 Modularity vs. Integration

There is a continuum between tightly integrated and modular solutions which has been investigated in systems engineering, computer science and product management.

Modularity Building systems of arbitrary complexity without structuring them into modules is difficult. Modularity decomposes complexity by breaking down a problem into smaller sub-problems that can be solved and tested individually. Because of the power of compositionality, building modular systems is the prevalent paradigm in robotics. This is reflected in the separation of robotics into the classical fields of perception, planning, control, etc. as well as in the produced software. For example, high modularity is one of the core concepts of ROS [Quigley et al., 2009], a popular framework for implementing robotic systems. Similarly, libraries like OpenCV, PCL, and MoveIt! represent commonly employed modules for computer vision and planning.

Integration Robotic systems generate behavior as a result of integrating many software and hardware components [Brooks, 1990; Cohen, 1996]. The usefulness of a robotic system is determined by the performance of the integrated system, rather than by the performance of individual components. To ensure that the performance of the entire system is maximized, and to avoid making wrong commitments or addressing sub-problems that are unnecessarily difficult, all components of the system should be chosen to maximally exploit potential synergies between components. To identify these synergies in the absence of established system-building guidelines requires early integration [Johnson et al., 2015; Katz and Brock, 2011]. Important advances were achieved by overcoming existing "modularizations", e.g. by combining interaction and perception [Bohg et al., 2016; Mart´ın-Mart´ın and Brock, 2014].

Our Design Choice on the Spectrum Our system used ROS [Quigley et al., 2009] and relied on various standard modules, e.g. for visual processing and navigation. However, we embraced tight integration at various levels. We integrated planning and control using hybrid automata, adapted our picking strategies to the embodiment and the requirements for object recognition to the picking strategies. Furthermore, we adapted many ideas from agile development [Schwaber, 2004]: rapid prototyping, early and continuous integration, adversarial testing, and shared knowledge.

5.2 Computation vs. Embodiment

The idea that mechanisms and materials expose behavior that is normally attributed to computation is known as morphological computation [Pfeifer and G´omez, 2009].

Computation Since computation is more flexible and can be altered easily, compared to the embodiment (hardware), it allows building highly complex systems with diverse behaviors. Purely computational approaches to robotics [Coleman, 2015; LaValle, 2006] also have the advantage of potentially being hardware-agnostic. Many examples of computationfocused approaches exist in robotics, e.g. posing grasping as a contact point planning problem [Miller and Allen, 2004]. It is appealing as it abstracts the problem away from the hand and the environmental context. Similarly, perception has been traditionally seen as a passive, purely computational problem [Marr, 1982].

Embodiment Tailoring the hardware to a particular problem can reduce the required computation. Hardware solutions are often simple and robust, especially when uncertainty is present. Grasp planning can benefit substantially from embodiment, as exemplified by simple under-actuated robotic hands [Deimel and Brock, 2016; Dollar and Howe, 2010]. Appropriate embodiment also facilitates perception, e.g. adaptive hands reduce the need for accuracy in object pose estimation. Moreover, placing a vision sensor on the robot arm increases the sensor's field of view and reduces the effect of occlusions [Aloimonos et al., 1988].

OurDesignChoice on the Spectrum We reduced the need for computation by using a suction cup. The reduced number of degrees of freedom simplified grasp planning and object pose estimation. We also reduced the need for computation by increasing the number of degrees of freedom by mounting the robot arm on a mobile base. This allowed us to generate motion mostly through feedback control, rather than resorting to motion planning. However, in the APC we failed to pick the pencil cup due to our chosen embodiment.

5.3 Planning vs. Feedback

Classical robotics and AI employed the sense-plan-act paradigm, assuming the robot can build a perfect model of the world. The difficulty of obtaining such models initiated a shift towards feedback-driven approaches [Brooks, 1990].

Planning Planning finds global solutions, where controllers based on local feedback would fail. The most common application in robotics is motion planning [LaValle, 2006]. Practitioners need to provide models of the environment, calibrate the robot, and localize it in the environment [Thrun et al., 2005]. Under these prerequisites, motion planners serve as general and versatile black-box solvers.

Feedback If global search is not required or not possible, feedback control based on task-relevant features is often sufficient to generate successful robot motion. Feedback can be exploited in the visual [Espiau et al., 1992] or contact domain [Lozano-Perez et al., 1984; Eppner et al., 2015] to simplify manipulation tasks. Feedback approaches are particularly useful in the presence of uncertainty, high dimensionality, long time horizons, and inaccurate models. In these cases, planning would be computationally demanding and often intractable [Papadimitriou and Tsitsiklis, 1987].

Our DesignChoice on the Spectrum Our system relies on very simple planning. We use on-line grasp approach planning and execute the motions using pre-defined, feedbackguided motion primitives, avoiding configuration-space motion planning altogether. This positions our solution far to the feedback-side of the spectrum, in contrast to the majority of other challenge entries (80% of the teams used motion planning, 44% used MoveIt! [Sucan and Chitta, 2016], [Correll et al., 2016]). Feedback control is so successful in the APC setting because the task only requires a limited range of motions and the shelf provides plenty of contact surfaces. However, some shortcomings of our system, such as the lack of in-bin reorientation of objects, should be addressed by some form of planning.

5.4 Generality vs. Assumptions

This spectrum is reflected in the no free lunch theorem [Wolpert, 1996], the bias-variance trade-off [Hastie et al., 2005], and the balance between generality and specificity in system design [van Gigch, 1991].

Generality Finding general solutions is an important goal in robotics. A general solution applies to a wide range of problems and reflects a deep understanding of the problem at hand. In contrast, solutions strongly tailored to a specific problem instance (e.g. a robot demo) might not lead to insights or contribute to a broader understanding of system building. A number of general approaches were successfully applied in robotics, e.g. task-generic planning algorithms such as A∗ are widely used for mobile robot navigation. Recursive Bayesian estimation is a very generic framework that helped solving many different problems in robotics.

Assumptions In machine learning, search, and optimization, the no free lunch theorems prove that no problem can be solved without making appropriate assumptions [Wolpert, 1996; Wolpert and Macready, 1997]: averaged over all possible problems, there is no method that outperforms random guessing. The only way to improve on random guessing is by making assumptions about the problem. We believe that problems in robotics are characterized by a significant amount of reoccurring underlying structure. E.g. in motion planning, adding information about workspace connectivity can reduce the computational complexity by up to three orders of magnitude [Rickert et al., 2014]. In reinforcement learning, adding explicit knowledge about physics makes the learning problem tractable by reducing its dimensionality [Jonschkowski and Brock, 2015; Scholz et al., 2014].

Our Design Choice on the Spectrum Our APC system used available (general) solutions whenever they proved sufficient to solve the problem. Since we could not find existing approaches for reliably locating the target object, we used various assumptions to simplify the problem, e.g. that the objects are placed in a known shelf or that the contents of each shelf bin are known and therefore only a small number of objects need to be considered. Our general solutions included a particle filter for localizing the robot, and standard joint and operational space controllers for motion generation.

6 Conclusion

We presented and evaluated our winning system for the 2015 APC. To describe the system, we proposed four key aspects of system building. A systematic description of robotic systems according to these aspects (and additional ones proposed by others in the future) will facilitate accumulating general knowledge for building robotic systems.

Our lessons on building robotic systems are consistent with those others derived from their experience in similar robotics challenges. In the area of autonomous driving [Buehler et al., 2007] and humanoid robotics [Atkeson et al., 2015], the resulting insights have led to significant advances. We hope that the APC and our lessons learned will be equally useful for manipulation.