By the use of powerful simulation tools for the economical and time-efficient virtual learning of AI robot manipulation skills in handling and assembly the project Sim4Dexterity strives to lower two major obstacles in robot utilization for end-users and systems integrators to boost automation capabilities in production:

  1. Automatic and economic robot learning and self-teaching of new robot skills 
    → effects cost savings, increased flexibility, and independence of robot programming experts
  2. Virtual feasibility studies and virtual commissioning 
    → effects planning certainty and security of investments, cost savings, and minimized interruption of production

Thus, the project results represent the first step towards the implementation of processes that could not previously be automated economically and the flexibilization of robot manipulation capabilities.

Major exploitation goals of the project partners are the use of these tools for developing better AI manipulation technologies, for conducting virtual validation and commissioning, and for creating scientific challenges and impartial industrial benchmarks for common handling tasks.

Sim4Dexterity will create a comprehensive workflow for simulation-based learning of handling skills by addressing the following topics and tasks.

© Fraunhofer IPA

Sim4Dexterity will create a comprehensive workflow for simulation-based learning of handling skills by addressing the following topics and tasks.

© Fraunhofer IPA

Bin Picking

In bin picking, a robot separates unsorted components delivered in boxes. If bin picking is used to pre-sort large ranges of components in an industrial warehouse, it is referred to as kitting. AI methods that can be trained on synthetic data include object pose estimation, grasp planning, (tactile) grasp execution, or manipulation strategies. Due to the infinite arrangement possibilities of the parts, essential quality criteria such as the complete emptying of the bins as well as rare error cases can only be determined through extensive data analysis.

© Fraunhofer IPA

Shelf Picking

An innovative future topic for the retail industry are mobile robots for picking customer orders from store shelves, e.g. for online retailing or nighttime service during closing hours. Relevant AI methods here are, among others, object detection for large and rapidly changing product ranges, active vision, grasp planning, (tactile) gripping in confined spaces, picking strategies, and sensor-guided orderly packing (bin packing). In this use case, the problem complexity and the rate of change of the requirements are so high that the generation of comprehensive training data including rare error cases as well as extensive testing and proof of functionality cannot be implemented on the basis of real data.

© Fraunhofer IPA


Due to their complexity, assembly processes are rarely automated today. By learning new assembly skills independently, significant costs are reduced when automating assembly. Assembly usually involves the force-based joining of individual components. Human sensitivity can be implemented on the gripper using tactile sensor technology. In the switch and automotive assembly applications considered in Sim4Dexterity, examples include assembly steps such as plugging switch housings together, plugging fuses, and gluing model marks to doors. Applicable AI methods include object localization and tracking, and manipulation strategies with tactile feedback. The greatest challenges lie in the realistic simulation of the physical processes and the resulting tactile sensor signals on the robot gripper.

AI Handling and Assembly Technologies


Product Flyer AI Bin Picking


Product Flyer Handling Technologies for Warehouse Automation