© H. Cui, Published by EDP Sciences, 2024

Research highlights

Using hand-drawn images, typography, and graphics, “Hand-drawn art food packaging design” (HAFPD) is a fresh approach to the design of food packaging.

• Computer-aided design (CAD) software has the potential to be an effective resource for HAFPD development.

• This research suggests a CAD system for HAFPD that is powered by a deep neural network (DNN).

• Designers can use the proposed method to easily create unique designs for food packaging without losing the original look of hand-drawn art.

1 Background

Hand drawing is often regarded as being among the most intuitive and effective information recording methods available to humans. To transmit information as earlier as ancient Times, Egyptians devised various writing characters and scratched them on limestone cliffs using tools to record. These symbols were used to record information. There is no question that a diagram that has been hand-drawn is an effective means of expression that may contribute to the representation of people's thoughts. They can freely grow their styles and incorporate symbols that can explain themselves. Research on diagram identification has a long and illustrious history during this time [1].

Usually speaking, hand-drawn drawings are always associated with the domains of creativity, design, instruction, and recreation. People's inherent designs for ideas are established via hand-drawn sketches, and they are capable of producing their natural hand-drawn designs. According to the industrial sector, the design procedure is the most well-liked and initial step. The fundamental concepts are first freely sketched with the use of hand drawings, and only then are they transformed into 2D and 3D representations. Yet in the industrial industry, getting acknowledgment for it is the hardest part. Architects, designers, and engineers often use pencil and paper to draw their ideas. The computer processes every hand-drawn picture before transforming information into a format that is more useful for technical purposes, such as a diagram, 3D image, flow diagram, etc. Despite the popularity and widespread of 3D graphics images, conventional 2D sketching continues to be a significant creative way of communicating information and is still crucial in the planning stages of motion graphics production [2]. The main benefit of 2D animation sketching is its natural, entirely unrestricted environment. Each frame must be carefully crafted to produce a professional painting or animation, and certain chores may easily get monotonous [3].

Food items must be protected from environmental, microbial, and transportation hazards by means of packaging that encapsulates and preserves them. Being as inert as feasible with little food-material interaction is an important safety feature for food packaging materials in contact with food. Moreover, the packaging is linked to branding, environmental marketing, logistics, and distribution processes [4]. More effort is needed to adopt ecologically friendly packaging. Initially, while creating friendly environment packaging, it is important to take into account the roles that packaging serves. Protection, storage, loading, and transport, as well as sale, advertising, service, and warranty, are the major purposes of packing. Given that food waste is more harmful to the environment than packing, it is important to highlight the fact that packaging lowers food waste. Sales, promotion, and customer service all play important but sometimes overlooked roles. Consumers’ final purchase decisions dictate packaging's commercial success [5].

Recently, packaging has played a dynamic role in food preservation, confinement, protection, and marketing due to innovation and ingenuity. Innovative packaging technologies are referred to by a variety of names, including interactive, active, smart, intelligent, and other features that are user-friendly and aid in maintaining food safety and quality. In times of intense competition, package design is also helpful for brand marketing, awareness, and development. Beautiful and eye-catching packaging may catch customers’ attention and have an impact on their choice to buy. It also conveys information about a product's reliability and competitive advantages, and it can encourage prospective customers to make impulsive purchases when the time is right [6]. Even if you never enter a supermarket, you will almost certainly come into contact with product packaging multiple times every day. Product packaging is an integral element of Western culture. It would be difficult to eat, drink, or take a bath without being exposed to as well as handling the packing that a great deal of everyday things arrive in [7]. People may feel more at ease and naturally intimate while seeing hand-drawn illustrations. It may enhance the creative package design, draw attention to the qualities of the goods, increase the added value of the items to increase sales and infuse cultural connotation while also communicating the corporate philosophy and the emotional appeal of the product to the customer.

The original variety of food packaging has been given new life by packaging in the shape of hand-painted illustrations [8]. Food packaging design using DNNs has the potential to revolutionize the packaging industry by enabling designers to create more effective, efficient, and sustainable packaging solutions. However, it is important to note that this approach is still in the early stages of development, and many challenges and limitations must be addressed before it can be widely adopted.

2 Related works

Table 1 shows the description of related works.

3 Materials and methods

In this section, we discuss in detail about the computer-aided design for food packaging using a deep neural network model. The viewing duration of the product package may be extended by using CAD visuals in an appealing manner, which supports the efficient dissemination of product information. Prior to creating a hand drawn design for a product package, we must thoroughly research the item, including its features, functions, recommended uses, and safety measures.

3.1 Dataset

To investigate the process of human sketching and human sketch recognition, we gathered hand-drawn sketches and named them as the TU-Berlin sketch dataset. It is nowadays, the biggest and most widely used human sketch dataset. There are 250 categories in all, with 80 drawings in each. 1,350 users submitted it via Amazon Mechanical Turk, resulting in a variety of categories and drawing techniques within each. In terms of the total number of object categories, it is comprehensive. In addition, since they gather an equal number of sketches for each class—and because one class's worth of sketches is sufficient for a large-scale retrieval benchmark—the bias problem is avoided. The goal of the study is to develop a sketch-based 3D model retrieval benchmark. The datasets used in the study, called SHREC12STB, are based on the Princeton Shape Benchmark (PSB) datasets, from which 1258 appropriate models have been identified for 90 of the 250 classes in total [26].

3.2 Computer aided design (CAD)

Computer-aided design (CAD) can be used to design and visualize food packaging before it is manufactured. CAD software allows designers to create 3D models of the packaging and simulate different scenarios, such as how it will look on a store shelf or how it will protect the food inside. Some CAD software specifically designed for food packaging design includes ArtiosCAD, Solid Works, and Adobe Illustrator. These programs allow designers to create packaging from scratch or modify existing designs, adding logos, text, and other graphics. Additionally, CAD software can assist in creating accurate die lines and optimizing material usage, which can save time and reduce waste. Figure 1 depicts the CAD of the food package structure.

One key advantage of using CAD for food packaging design is the ability to create virtual prototypes, which can be viewed and tested without the need for physical samples. This can save time and money during the design process, as changes can be made quickly and easily without having to produce multiple physical prototypes. Overall, CAD can be a powerful tool for food packaging design, helping designers create functional and visually appealing packaging that meets the needs of both consumers and food manufacturers.

Fig. 1 CAD of the food package structure (Source: author).

3.3 Examples of CAD models

To examine the various elements like boundary conditions, loads, and materials in the CAD model, we have selected 5 examples. The following are the 5 examples selected for the CAD model.

• The initial module was served by a Beam. The beam was tested using standard steel with Young's modulus of 2,06,000 MPa and a Poisson's ratio of 0.3, subjecting one edge to continuous line load and restraint, and using engineering mechanics for stress and displacement results.

• A holding shrub served as the next example. Pressure was applied to outside surfaces and an axial surface load was applied. Radial bearings and axial bearings were installed for further limitations. The steel used was the same as in the first example, illustrating how different software interprets surface loads.

• A crankshaft served as the third example. Every side of the crankshaft has a radial bearing, and one side has an axial bearing. The crankshaft's material has the same characteristics as the first example. This example's primary goal was to validate the outcomes of the previous one.

• A commercial diesel engine's piston served as another example of a CAD model. The piston head, bowl surfaces, and borehole surfaces were subjected to pressure, with the thrust bearing being used for the piston pin. Steel was chosen due to its ability to withstand pressure loads.

• The final example involved simulating a fan with loads and boundary conditions, using a thrust bearing and weighted rotor blades. The plastic used had a Poisson's ratio of 0.335 and a Young's modulus of 17,000 MPa, making it the most complex and the second to use pressure as a load.

3.4 A DNN model's interpretation

This section provides an overview of the challenge of comprehending a deep neural network (DNN) taught idea. The neurons in a DNN are organized in many layers, with each layer receiving the action possibilities of the neurons below it as input and using them to do an easy calculation. A complex nonlinear translation from inputs to outputs is achieved by the network's neurons collectively. By modifying the parameters of each neuron using a method known as the error training algorithm, this mapping is learned from the data.

A neuron on the top layer often represents the idea that has to be understood. Top-layer neurons are impersonal, but the DNN's input domain is often comprehensible. In the sections that follow, we’ll go through how to create an interpretable prototype of the learned abstract notion in the input domain. Under the scope of activating maximization, creating the prototype may be defined. Figure 2 shows the DNN architecture.

The input layer of the deep neural network (DNN) would use features taken from the hand-drawn sketches by using the TU-Berlin sketch dataset. These characteristics might include forms, creative details, and stylistic components that are present in the sketches and offer a comprehensive depiction of human sketching styles. The 250 categories with 80 drawings each in the dataset would provide a wealth of patterns and correlations between different sketching styles that the hidden layers would learn through iterative training on the TU-Berlin sketch dataset. Using the unique characteristics of hand-drawn sketches as a basis, the output layer would create design recommendations that might be used for food packaging. Because 1,350 people contributed to the dataset, it is comprehensive enough for the DNN to perform well across a range of sketching styles and categories. This will ultimately help in the development of a reliable computer-aided design tool for hand-drawn art-inspired food packaging.

Fig. 2 Architecture of DNN (Source: author).

3.4.1 Activation maximization (AM)

An analytical method called activation maximization looks for an input sequence that generates the largest model response for a certain quantity of interest.

Imagine a DNN classifier that associates a collection of classes (ω_v)v with data points. The class probabilities that were modeled are encoded by the output neurons b(y|ω_v). Optimizing: yields a prototype y^* equivalent of the class ω_v.

$${\displaystyle \underset{y}{\max }}logB\left({\omega }_v|y\right)-\lambda {||y||}^2$$(1)

The DNN models class probabilities as gradient-based functions. This makes it possible to use gradient ascent to optimize the goal. A −norm regularizer that applies a tendency for inputs that are near the origin is the objective's rightmost word. Prototypes thus resemble primarily gray pictures when used for image classification, except for a few edges and various colors at key areas. While they generate a strong reaction from the class, these prototypes might seem artificial.

3.4.2 Improving AM with an expert

The f₂-regularize may be swapped out with the more complex one named "expert" to produce more meaningful prototypes. For instance, the expert may be a model b(y) of the data. This brings up the most recent optimization issue:

$${\displaystyle \underset{y}{\max }}logB\left({\omega }_v|y\right)+logb\left(y\right)$$(2)

The prototype y^* that is produced as a result of addressing this optimization challenge will exhibit both significant class response and data resemblance. The Bayes’ rule may be used to identify the newly specified goal as the category data density logB(ω_v|y), up to model uncertainties and a constant term. Hence, the learned prototype will match the most probable inputs y for class ω_ν.

The Gaussian RBM is an option that the expert may choose. It has gradients in the input vector and may describe complicated distributions. Here is how its log-probability function is expressed:

$$logb\left(y\right)={\displaystyle {\displaystyle \sum }_i}{l}_i\left(y\right)-\lambda {||y||}^2+cst.,$$(3)

where the terms $l_j\left(y\right)=\text{log}(1+\exp \left(u_i^{D}y+p_i\right)$are an overlay to the initial f₂-norm regularizer and are learned from the data. Convolutional RBM/DBMs or pixel-RNNs are more complicated density models that may be used to analyze notions like natural picture classes. In reality, the final prototype's look is significantly influenced by the decision of the expert b(y).

The optimization issue is reduced, on the one hand, to the maximizing of the instead of function b(y|ω_ν) by a coarse expert (a). The optimization issue is effectively reduced to the maximum of the expert b(y) itself in the case of an overfitted expert (d), which is the opposite extreme.

An overfitted expert (d) should be specifically avoided when using AM to validate a DNN model since it might conceal the model's intriguing failure modes. So, a little under-fitted expert (b), such as one who merely prefers photos with natural hues, may be adequate. On the other side, the goal should be to avoid underfitting ω_ν while using AM to learn about a notion that the DNN accurately predicted. The prototype y^* would not be a true representation of aaa since an under-fitted expert (b) might reveal (y|ω_ν). Optima may be far from the data. So, it is crucial in such a scenario to train a density model that is as similar to the actual data distribution as feasible (c).

3.4.3 Performing AM in code space

Data density models b(y) may be challenging to acquire up to high accuracy or very complicated, making it challenging to maximize them in certain applications. Generative models are different kinds of unsupervised models. The next two processes are often used to sample from the density function, which they do not directly give.

A representative example of a straightforward distribution o (h) ∼ M (0, 1) is defined in an arbitrary code space H. Apply the decode function H→χ, which remaps the samples to the original data domain, to it.

The deep convolution network is but one model. It acquires a decoder function g that makes it as difficult as possible to distinguish the created data distribution from the real data distributions. A differentiator between the produced and genuine distributions is learned in competition with the decoding function g. Multilayer neural networks are often utilized for the classifiers and the decoding function.

By including such a prediction model in the activating maximization framework, they suggested creating a b(y) prototype. The optimization issue is reformulated as follows:

$${\displaystyle \underset{h\in H}{\max }}logb\left(u_v\text{|}s\left(h\right)\right)-\lambda {||h||}^2$$(4)

where the second term is an f₂-norm regularizer in the coding space, and the initial term is a combination of the newly added decoding and the original classifier. The prototype for ω_ν may then be retrieved by decoding the answer, which is represented by the code x^*=s(h^*) in the optimization problem.

The f₂ penalty λh² favors coding with high probability whenever the code distribution q(h) is selected to be approximately normal since it is equal logq(h). The code space maximizing discussed in this section will, however, only roughly maximize the desired number logB(ω_ν|y) since high likelihood codes do not always translate to high-density areas in the input data.

We explore the issue of understanding classes when they are represented by a three-layer DNN to highlight the significant difference between the techniques of the above three sections. For this challenge, we take into account a straightforward f₂ −norm regularizer $\lambda y-{{\displaystyle \overline{y}}}^2$, where xx indicates the information mean for ω_ν, second a Gaussian RBM expert, b(y), third a generative network with a two-layer decoded function, and the f₂ −norm regularizer $\lambda h-{{\displaystyle \overline{h}}}^2$, where h represents the code mean for. The DNN assigns each prototype a classification with absolute confidence. Unfortunately, the prototypes only seem crisp and lifelike with just an expert or decoding function.

3.4.4 From global to local analysis

Probability functions b(y|ω_ν) and b(y) may be multimodal or significantly extended when taking into account sophisticated machine learning challenges, making it impossible for a single prototype to properly reflect the modeled idea ω_ν. The problem of multimodality is brought up by those who show the value of understanding a class ω_ν utilizing many local prototypes rather than a single global one in the context of picture classification.

It's not always essential to provide a detailed explanation of the modeled notion ω_ν, however. Instead, one may concentrate on a specific area of the input data. Biomedical data, for instance, is best assessed about a certain topic or organ, depending on the stage of development of a medical problem.

A localization term∥y − y_o ∥ ², where y_o is a reference position, might be added to the AM goal as a quick approach to include the locality in the analysis. The degree of localization is controlled by the parameter. Algorithm 1 shows the pseudo code for DNN.

The minimization strategy minimizes the cross-entropy loss between the expected and actual outputs by iteratively modifying the weights and biases of the deep neural network (DNN) in the training algorithm as mentioned in Algorithm 1. The biases and weights are initially initialized at random. The DNN produces predictions for the input data by doing forward pass calculations with the sigmoid and softmax activation functions. The difference between these forecasts and the actual outputs is then measured by the cross-entropy loss function. Adjustments can be made in the direction that minimizes the loss by using backpropagation to compute the gradients of the loss with respect to the weights and biases.

The learning rate is used to scale these modifications to control the update size. Through a series of epochs, the training process iterates, gradually improving the weights and biases to enhance the DNN's predictive accuracy of the intended outputs. The utilization of an iterative optimization technique guarantees that the DNN converges towards producing more precise predictions, therefore augmenting its efficacy in the computer-aided design of hand-drawn art food packaging.

4 Results

First Fixation duration (FFD) can provide insights into the effectiveness of the packaging design and how it captures the attention of potential customers. FFD can be influenced by various factors, including the color, shape, size, and branding of the package. Eye-tracking studies can be used to collect data on FFD and other eye movement parameters to inform packaging design and marketing strategies. Figures 3a and 3b depict the FFD. Tables 2a and 2b depict the result of FFD.

View statistics can provide insights into user behavior and preferences, which can help improve the design and user experience of the content or platform. Figures 4a and 4b demonstrate the view statistics. Tables 3a and 3b demonstrate the result of CAD view statistics.

Researchers often use fixation count in combination with other eye-tracking measures, such as fixation duration and saccade amplitude, to gain a more comprehensive understanding of visual attention. Figures 5a and 5b show the fixation count. Tables 4a and 4b show the result of the fixation count.

Computer-aided design (CAD) watch duration statistics are often used in the design and development of watches, allowing designers to create and refine watch designs in a digital environment. One way to analyze the effectiveness of the design process is to gather duration statistics on how long it takes designers to complete various tasks in the software (Tab. 3b). To track duration statistics for watch design using CAD software, designers can use a time-tracking tool or plugin (Tab. 4a). These tools can track the time spent on different tasks, such as sketching, modeling, rendering, and refining the design. Figure 6 and Table 4b depicts the CAD watch duration statistics.

The DNN model was trained on the dataset and during training, a few of the models diverged. Figure 7 illustrates a sample case. We selected the “early-stopped” model as the optimal model, which explains how the prediction model was selected to minimize the test loss rather than the training loss.

Fig. 3 (a) First fixation duration (Source: author). (b) First fixation duration (Source: author).

Fig. 4 (a) CAD view statistics (Source: author). (b) CAD view statistics (Source: author).

Fig. 5 (a) Fixation count (Source: author). (b) Fixation count (Source: author).

Fig. 6 CAD watch duration statistics (Source: author).

Fig. 7 A developed model for mean squared error as a function of epoch count.

5 Conclusions

The elegance of antiquity may be reflected by combining the traditional pattern with the design theme and applying it to a contemporary design arrangement. There are many different types of package designs for creative and cultural goods, but hand-drawn images stand out for their powerful integration of culture, art, emotion, and commerce. Giving creative and cultural brands deep feelings and vibrant individuality as well as spreading culture and passing down history via artistic language is crucial for study. We presented a Deep Neural Network (DNN) model for this study. It may be helpful to maintain the goal of advancing social equality in nations via package design. It can inform the public on fostering acceptance and understanding of various ethnic cultures in countries. For instance, it might enhance its cultural worth and creatively include the finest packaging with cultural connotations in the future.

Funding

1. [2018 Shandong Art Science Special Project, the title of the project is “Characteristic Teaching Practice of Traditional Handicraft Applied in Pattern Mixed Reform Course”, approval number: YJ201811141].

2. [2019 Shandong Art Education Special Project, the title of the project is “Teaching Practice Research of Qilu Culture Incorporating into Pinning Enamel Painting”, approval number: YJ201911133].

Conflict of Interest

The author declares no conflicts of interest.

Data availability statement

The data used to support the findings of this study are available from the corresponding author upon request.

Authors contributions

The research was devised by Hui Cui, and she also wrote the paper. Hui Cui coordinated the effort, and had a hand in both the design and analysis of the research. In-depth examination, study, and assessment of Hui Cui. Hui Cui collected the information. The final text was reviewed and approved by her own.

References

All Tables

All Figures

	Fig. 1 CAD of the food package structure (Source: author).
In the text

	Fig. 2 Architecture of DNN (Source: author).
In the text

	Fig. 3 (a) First fixation duration (Source: author). (b) First fixation duration (Source: author).
In the text

	Fig. 4 (a) CAD view statistics (Source: author). (b) CAD view statistics (Source: author).
In the text

	Fig. 5 (a) Fixation count (Source: author). (b) Fixation count (Source: author).
In the text

	Fig. 6 CAD watch duration statistics (Source: author).
In the text

	Fig. 7 A developed model for mean squared error as a function of epoch count.
In the text