© J. Posada et al., Published by EDP Sciences, 2021

1 Introduction

Industry 4.0 is a worldwide trend intrinsically related with the incorporation of the latest IT technologies coming from other digital domains into the industrial production and advanced manufacturing. First and foremost, in the shopfloor (with the entrance of the Industrial Internet of Things), but also in other aspects related with the relationship with providers and customers in the whole value chain, including product-service schemas. Clearly there is a focus shift from the purely physical dimension of production towards a digital dimension of transformation, in which the potential of technologies such as Industrial Big Data, Human-Robot Collaboration, IIOT & 5G, Cybersecurity, Cyber-Physical Systems, Semantic Technologies, Visual Computing and other related technologies have a strong impact in the new ways the smart manufacturing is reconfiguring the production.

In this context, Visual Computing is a key enabling technology for Industry 4.0. Augmented Reality, Virtual Reality, advanced Computer Vision, Human Computer Interaction and related technologies are central to many disruptive applications in a Smart Factory perspective [1,2]. Applications already present in the literature include the use of digitized geospatial data in the context of Extended Reality (XR) as a key technology to enhance worker cognition in industrial scenarios [3], the inclusion of digital twins for assisted training courses [4] and maintenance processes [5]. The use of 3D reconstruction methods via optical systems also enables the development of non-destructive metrology and dimension assessment methods, facilitating real-time visual inspection and aiding some fast decision making directly in the shop floor [6,7]. Other applications of optical systems include the implementation within augmented reality applications, which can be used to visually assist operators in assembling small electronic components [8,9].

Aside from Computer Vision and Virtual Reality techniques, another relevant aspect in the Industry 4.0 framework relates to the use of physical-based simulation tools. These tools are important for the assessment and optimization of many manufacturing processes and, although they can be used offline, there's a tendency to implement such tools within interactive virtual environments. As an example, the implementation of heat transfer simulation techniques in the context of sheet metal cutting, enables interactive assessment, optimization and path re-definition within a digital twin of the real CNC machining tool [10–14]. Furthermore, structural analysis is quite important in the context of additive manufacturing when aiming to new printing techniques and/or materials [15]. However, special care must be taken when trying to apply numerical methods (e.g. Finite Element methods) in virtual environments as their high computational complexity may render them incompatible to many interactive approaches. In a more general perspective, real-time interactive visualization and simulation is an enabling technology in Industry 4.0 but also in many fields, especially in those where the humans adjust iteratively their designs or their response [16].

In this article, we present a theoretical framework for the incorporation of Visual Computing and Physically-based Simulation technologies to Industry 4.0 and Operator 4.0 scenarios. The presented framework has been already implemented and supports different smart manufacturing scenarios that are already running in many SME companies in the fields of automotive and aeronautics. In particular, three practical case scenarios are presented and discussed.

2 Methods

The methodology of the work is based in an applied research conceptualization of the role of Visual Computing in the paradigms of Industry 4.0 and the Operator 4.0, followed by a set of 3 practical examples that are already running in several companies, most of them SME dedicated to the industry & manufacturing sector in fields such as automotive and aeronautics. It is worth to mention that this work is based in our own research line with highly referenced works (see for instance [1,3,6]) in the scientific community.

Visual computing technologies are an are an important enabling technology that must be considered in any comprehensive analysis regarding Industry 4.0. In Figure 1, we adapt the proposed framework of [1] to give the proper relevance of Visual Computing in that context, and provide a set of specific examples related with industrial real applications developed for real industries from a R&D perspective. As we explained in that work, several of the more classical Industry 4.0 technologies such as big data, advanced HMI, 3D models and simulations, cloud computing, CPS, Internet of Things and Services, machine to machine, and smartization technologies can be applied in an isolated matter to some industrial challenges, but in many cases Visual Computing acts as a “glue factor” that enables a comprehensive solution involving the user in a central role (in the roles of interpretation, analysis and interaction).

From a methodological point of view, we have followed an applied research approach consisting in: (1) detection of real world needs demanding an Industry 4.0 solution, (2) identification of the relevant technologies that can be applied from a dual perspective: potential application impact and scientific relevance, (3) development of research prototypes in Technology Readiness Levels (TRLs) 4–7 in real world scenarios for industrial final users, but always in collaboration with IT integrators and SW companies, (4) validation of the solutions and deployment in industrial settings, making a technology transfer strategy to the companies (both industrial users and IT providers) in TRLs 8-9.

Taking this methodology in consideration, we identified two main technology lines of work directly related with Visual Computing:

• Virtual Engineering and Simulation: Where we aim at performing process simulation and training through visualization and interaction. This includes:

  • Virtual Process: Simulation of Robot, machine tool, automation 3D planning, design and simulation, CAD/CAM/CAE, FEM/FEA.

  • The Human Factor: Augmented and virtual reality for assistive or training systems applied on maintenance or assembly.

  • Cyber-Physical Equivalence: Model algorithms in simulation tools for testing in virtual worlds and deploy in real machines, and synchronize both worlds.

• Cognitive Vision: Where we aim at the Improvement of process and product quality control by using non-destructive techniques. This includes:

  • Cognitive Robotics: Robot environment knowledge through optical techniques.

  • Dimensional Engineering: 3D Reconstruction, volumetric imaging and Geometric dimensioning and tolerance.

  • On-Line inspection and monitoring: Optical Techniques for process control or quality control.

On the other hand, Visual Computing is an integrative concept of several technologies, and its relevance for the particular field of Industry 4.0 can be considered following a matrix structure, relating the priorities detected in production and industrial settings with specific visual computing technologies (Visual Analytics, Human-Machine Interaction, Virtual Engineering, Virtual Reality and Virtual Environments, Augmented Reality, 3D Reconstruction, Computer Vision, 3D Geometric Modelling, Simulation / Visualization, IoT in 3D/Web3D, GIS/Visualization, Multimedia). Table 1 presents this matrix structure (adapted from [1]), relating the 3 selected project results with the respective visual computing priorities.

Fig. 1 Visual Computing as Key Enabling Technology for Industry 4.0 [1]. Examples in (a) In-Line dimensional Inspection with 3D reconstruction [6], (b) Augmented Reality for Operator 4.0 [8], (c) Laser cutting process simulation in GPU (towards Digital Twin) [10]. Figure adapted from various papers of our previous research.

3 Results

In Table 1 we show the three selected examples, related to the conceptual framework for specific technologies of Visual Computing that are relevant in Industry 4.0 (adapted from [1]). These are: (a) In-Line dimensional Inspection with 3D reconstruction [6]: This project example has been done with one of the leading industries in the automotive sector worldwide (GKN), and has allowed the in-line dimensional inspection of warm-die forged revolution workpieces using 3D mesh reconstruction. (b) Augmented Reality for Operator 4.0 [8]: This project example has been done in collaboration with a relevant SME in the electronic assembly sector (IKOR), with plants in Spain, Mexico and China. A Real-time automatic optical system to assist operators in the assembling of electronic components was developed. (c) Laser cutting process simulation in GPU (towards Digital Twin) [10]: This project example has focused on the use of GPU architectures for fast simulation of laser heating processes on thin metal plates with FFT using CPU/GPU Hardware.

3.1 In-Line dimensional inspection with 3D reconstruction

In the context of dimensional assessment of hot-forged workpieces, the use of Visual Computing has proven to be crucial for early detection of forge-line manufacturing problems. Although highly precise, classic dimensional inspection methods are highly time-consuming (20 min per part) due to prior air-cooling requirements [6]. On the other hand, visual computing methods are less precise, but much faster due to their contactless nature. In a factory where 1200 workpieces can be forged every hour, the contactless methods are crucial for early detection and correction of problems in the forge line.

The Visual Computing method for dimensional assessment presented in [6] automatically digitizes every workpiece straight from the forge line. A 3D mesh reconstruction of the hot workpiece is performed using blue-spectrum laser scanners. The reconstructed mesh is aligned against a CAD reference using mesh registration techniques, resulting in a colormap of deviations on the CAD surface (Fig. 2) which enables further visual inspection (as illustrated in Fig. 2). Finally, the desired dimensional measurements are computed and reported automatically allowing the current operator to take preventive actions if required.

Since the forge line performs a repetitive task, an identified defective workpiece results in the scrapping of all recently forged parts up to the last identified correct one. In this sense, the method presented in [6] results in an estimated reduction of scrap part production of 95%, in which the cadence of each measurement is improved from 1 piece every 20 minutes (classic method) to 1 piece every minute (visual computing method). Furthermore, this high cadence allows the acquisition of data for potential data analytics [6].

Fig. 2 Visual Dimensional Inspection of a forged workpiece using 3D reconstruction and mesh registration [6].

Fig. 3 Information presented to the operator via screen. Green rectangles denote mounted components, red rectangles denote missing component, and orange rectangle denote current mounting instruction [8].

3.2 Augmented reality for Operator 4.0

In the context of industrial production lines, it is well known that manual operations performed by human operator during long periods of time present undesirable ratio of errors mainly due to the human factor. The concept of the Operator 4.0 [17] is to reduce the problematic of the human factor, empowering operators with new capability and increasing its adaptability to the industrial environment by means of technologies such as Augmented Reality.

The work described in [8] presents an Operator 4.0 system for manual assembly of electronics components on boards. The presented system is composed of a Computer Vision (camera) and an Augmented Reality System (display, projector), both systems holding from the ceiling at a sufficient height such as the system is non-invasive with respect to the operator workspace. The Computer Vision system recognize the correct mounting of a component on the board, preventing components absence or incorrect mounting, which is a major problem of the production line. The Augmented Reality system interact with the user informing about the current state of the board and displaying the mounting instructions in a screen and projecting them on the board [9].

The system was proven to offer several benefits, not only related to mounting efficiency, reduction of time and errors, but also in the experience itself, as operators claimed to a leverage of cognitive load, and enhancement of ergonomy and comfort [8]. Figure 3 shows the information presented to the worker via projection as well as the real interaction of a worker with the system.

3.3 Laser cutting process simulation in GPU

Computer Numeric Control (CNC) Machines are relevant for the machining of metal workpieces, which includes processes such as milling, drilling and laser cutting. In this context, heat transfer analysis is crucial for the optimization of the CNC parameters (speed, trajectory, energy output, type of laser gas, etc.). Numerical methods such as Finite Element Analysis are standard tools for the simulation of these type of problems. However, due to many different variables (such as complex laser trajectories and metal ablation), the solution of these problems requires massive computing resources, even for very small domains and simple laser trajectories [11].

In this sense, the method presented in [12] develops an analytic solution to the laser cutting problem for metal plates, which (under certain simplification assumptions) allows almost-interactive simulation rates at the cost of some accuracy. This method has been integrated into an interactive visual simulator which integrates a physical module and a geometric module that enables fast visual assessment of (a) the surface temperature at any given time and (b) the produced cuts and current geometry of the metal plate [13]. This interactive coupling allows the engineer to simulate any given CNC program to manually fine-tune the parameters before production, resulting in improved quality, less material scrap and less energy usage. GPU-hardware can be used to significantly improve the interactivity rate of the simulator allowing more complex problems and results with higher resolutions [10].

The aforementioned method has been also extended to allow simulation of multi-laser machining. The method presented in [14] allows the cutting of up to 100 laser heads while keeping interactive simulation time rates (as illustrated in Fig. 4).

Fig. 4 Interactive simulation for multi-laser cutting process (3 laser heads) [14].

4 Conclusions

This work presents a theoretical framework for the incorporation of Visual Computing and Physically-based simulation technologies to Industry 4.0 and Operator 4.0 scenarios. On the one hand, two key research lines are identified as crucial in the context of the developed framework: 1) Virtual Engineering and Simulation, where the objective is to train models and to simulate processes through visualization and interaction, including Virtual Processes, Augmented and Virtual Reality, and Cyber-Physical modelling (e.g. Digital Twin); and 2) Cognitive Vision, where the objective is to improve the manufacturing processes via non-destructive product quality control techniques, including Cognitive Robotics, Dimensional Engineering and On-line Inspection and Monitoring. On the other hand, a workflow that considers all the stages of the product/process design has been developed, taking into account the cross-correlation of priorities detected in the industrial settings and the different Visual Computing Technologies.

To validate the presented framework, three success study cases are presented: (1) a dimensional inspection tool based on a computer vision system that enables in-line fast detection of manufacturing defects in the forge line, (2) an augmented reality system to aid visually the operator in the assembly of electronic components, and (3) an interactive heat transfer simulation tool, integrated within a digital twin of a CNC laser cutting machine for interactive path planning and tool-path optimization. These study cases have been successfully deployed in several manufacturing companies in the sectors of automotive and aeronautics.

Acknowledgments

The research projects presented in this article have been partially funded by the Basque Government, in the research programs Basque Industry 4.0, HAZITEK and ETORTEK, an by the companies Lantek, Ikor and GKN.

References

All Tables

All Figures

	Fig. 1 Visual Computing as Key Enabling Technology for Industry 4.0 [1]. Examples in (a) In-Line dimensional Inspection with 3D reconstruction [6], (b) Augmented Reality for Operator 4.0 [8], (c) Laser cutting process simulation in GPU (towards Digital Twin) [10]. Figure adapted from various papers of our previous research.
In the text

	Fig. 2 Visual Dimensional Inspection of a forged workpiece using 3D reconstruction and mesh registration [6].
In the text

	Fig. 3 Information presented to the operator via screen. Green rectangles denote mounted components, red rectangles denote missing component, and orange rectangle denote current mounting instruction [8].
In the text

	Fig. 4 Interactive simulation for multi-laser cutting process (3 laser heads) [14].
In the text