The project ‘Self-learning and self-explanatory machine’ (SLEM) is a research project the SABO Mobile IT and partners worked on during 2021. The idea behind SLEM is to develop an intelligent assistant which adapts to different levels of user expertise during the operation of machines, as well as guiding the user through complex production processes. For this purpose, the machine and interaction data are recorded with the help of various sensors. This data is then used for the training of the artificial intelligence (AI). In this way, the AI can observe the user and learn how the machine should be operated.
Together with the Fraunhofer Institute for Manufacturing, Engineering and Automation (IPA) and Knowtion GmbH, SABO Mobile IT developed the ‘Self-learning and self-explanatory machine’ in an agile project environment. Fraunhofer IPA was responsible for gathering and processing machine interaction data. Thus, data about how the user interacts with the machine and its surroundings. Knowtion GmbH gathered and processed machine data to provide additional information about machine states. The fusion of both of the data packages from FraunhoferIPA and Knowtion was part of the work package of the SABO Mobile IT. In addition, we worked on how to guide a junior operator based on the data gathered from a senior operator and derive a manual or user guideline.
The main goal of SLEM is to develop a system which observes a trained user and learns independently how the machine should be operated. Thereby, SLEM is able to recognise the level of information needed to guide users of different levels of expertise through complex processes.
The SLEM-Service consists out of three different services. First, Fraunhofer gathers the human data, which in the context of this project consists out of human activities extracted from the video data. The video data is captures by a camera which is observing the operator while he or she works with the machine. At the same time Knowtion gathers IoT data from the machine and analyses it with the help of AI. The two data packages gathered previously are then merged and processed further by Sabo Mobile IT. Depending on the seniority level of the operator the SLEM-service is either working in the learning mode or the teaching mode. In the learning mode (senior operator) SLEM extracts a guideline based on the gathered data and trains multiple machine learning models to abstract the operator behaviour into a guideline. SLEM is also generating a manual with the help of a text generation algorithm. Contrary to the learning mode there is also a teaching mode which is selected when a junior operator is working with the SLEM enhanced machine. The learning mode provides the manual to the user and in addition it uses the previously trained machine learning models to detect and predict possible mistakes done by the junior operator.That way SLEM is guiding an untrained operator through the process.
Unfortunately, the development of SLEM was also influenced by the COVID-19 pandemic. The restrictions related to the pandemic made gathering data for the various machine learning tasks extremely difficult. During the development we overcame the following challenges:
· Gathering and processing machine data with the help of AI
· Gathering and processing human-machine-interactions with the help of AI
· Data fusion
· Training machine learning models to learn from a senior operator
· Generating a text-based manual from the gathered information
· Guiding a junior operator with the help of the trained models
In addition, due to the previously mentioned COVID-19 restrictions, we had to develop algorithms which can work with a limited amount of data and still provide reasonable results.
The SLEM project delivered a proof of concept (PoC) for an intelligent assistant which learns by observing the operator and processing the relevant machine data. SLEMis capable of generating a text-based manual out of the activities detected by image processing. In addition to that it can display the manual to an untrained user.Furthermore, it can detect discrepancies between what the user is doing and what SLEM expects according to the generated guideline. The self-learning aspect was achieved by the implementation of a training mechanism that can be further extended and optimized by gathering more data. This improves the overall process over time. In this way, for example, the maintenance of special machines, the operation of machines and production processes of complex products can be taught to operators in a simplified way.
In the future SLEM can increase productivity by reducing the training effort of untrained operators. The plan of the project partners for the future is, to bring SLEM to the next level and refine it further. For example, by the integration of a natural user interface (NUI). The user could control the workflow by using gestures or speech. Also, there could be other concepts to look into in another project. This could make the interaction with SLEM easier and even more self-explanatory.
