Two Messages out of One 2D Matrix Bar Code

Abstract

With the proposed principle of two-dimensional matrix bar code design based on masks, the whole surface of a 2D bar code is used for creating graphic patterns. Masks are a method of overlaying certain information with complete preservation of encoded information. In order to ensure suitable mask performance, it is essential to create a set of masks (mask folder) which are similar to each other. This ultimately allows additional error correction on the whole code level which is proven mathematically through an academic example of a QR code with a matrix of size $9{\times}9$. In order to create a mask folder, this article will investigate parameters based on Weber's law. With the parameters founded in the research, this principle shows how QR codes, or any other 2D bar code, can be designed to display two different messages. This ultimately enables a better description of a 2D bar code, which will improve users' visual recognition of 2D bar code purpose, and therefore users' greater enjoyment and involvement.

1. Introduction

Bar codes are defined as information that is recorded in graphic form, which is then scanned by a computer, interpreted as binary information, and transformed into a computer protocol for a specific action [1]. Today there are about 400 bar code symbologies, each of which has a specific purpose and area where it is applicable [2]. Through their development, bar codes have increased the ability to store information, so now they are generally divided into one-dimensional (1D) and two-dimensional (2D) bar codes. Among 2D bar codes, QR codes have become the most popular ones [3], because of their commercial and practical-oriented application. Today, content has become the main reason why users scan QR codes [4]. But in order to get to the content users need to be attracted to the bar codes. The fact that people scan QR codes 25% to 30% more often when they are designed [5] proves that bar code personalization is a good way of getting users to scan them. How this is done is also important. Although stated otherwise [6-7], the possibility for the user to see what is the content of the QR code does not exist [8], except by scanning through a computer application or a bar code reader. In order to personalize a QR code one should understand its technical limitations.

 

2. Related Work

First and foremost, it is essential to ensure the technical qualities – i.e., that the QR code can be reliably, accurately and correctly encoded and decoded, yielding all the information contained without errors. Therefore, one has to take into account that QR codes are partially resistant to noise [9], distortion of perspective [10], inconsistent lightning conditions or partial covering of a part of the code. The distance between the bar code reader and the bar code surface, lack of smoothness of the surface and underexposure can also influence readability [11]. From a hardware standpoint the successful reading also depends on camera resolution and on the pixels per module of the QR code [12-13] [9]. After a successful scan of a QR code, its image needs to be corrected before decoding the bit streams. The methods for doing that rely upon image denoising [14] or on camera shaking [15-16]. But based on the number of different decoding algorithms to read QR codes [15] [17-19] [13] [20-22], it is noticeable that no single standardization method of reading a QR code exists. Only after these technical conditions are met, one can approach their design.

Considering that, ideally, only the pixels in the center of the QR code are relevant for correct decoding [23], modules can be shrunk 3 times in a QR code and still remain readable as proved by H.-K. Chu et al. [24]. More modules in the same space also means the creation of higher resolution images. Similar can be seen by Visualead [7] and by Garateguy et al. [23]. It is also possible to embed an image without using QR codes’ Reed Solomon codes for correcting errors with a method called image blending [8]. In order to facilitate reading of a QR code with an embedded picture, the brightness and color of modules in such a method are changed. As encoded information does not always take a maximum number of characters per mode, there is a vacant area which is filled with redundant and pad codewords. This fact is used for creating methods which grants space for a visual image [25-27]. The restriction of these methods comes from the fact that the size of the implemented image depends on the length of the codeword and their number [27]. Besides that, the user is limited by the surface on which he can place an image. In methods presented in innovations from the authors [7-8] [24-26] only one message is displayed. Methods that imply position changing of an embedded image [28] or its transformation [29] also display only one message for the user. Other authors [30-32] use colors as means to increase the storage capacity of a QR code. So for instance, the method which uses RGB red, green, blue and white for encoding messages can store twice as much information in comparison to monochrome QR codes [30]. Method which uses cyan, magenta, and yellow printing colorant channels for data encoding, enables three times greater storage capacity [31]. And in the method where three standard QR codes are placed in one colored one, the capacity is increased four times [32]. The use of colors is implemented in the field of document security as well. By adding three new colors on the same bar code, a new message can be read. The black and rest-as-white modules contain one message, while the rest-as N-ary modules contain progressive information [33].

When it comes to identifying content, from these innovations one can understand that the methods do not enable users to see what information is embedded in the QR code except through bar code readers. Also, one can observe that the whole surface is not used for design, which is possible to conclude from the visible function patterns. So in design methods known until now there isn’t a solution where the whole surface of a 2D bar code can be designed and carry two messages for the user.

 

3. The Approach

Some parts of a bar code (function patterns) must remain visible in order for a 2D bar code to be readable. The existing function patterns therefore reduce the space for designing graphic patterns, which the user desires. Because of that there are also limitations of how one can design the bar code. In order to achieve a greater interaction between users and 2D bar codes, and to increase their intelligibility for the users, a new principle is proposed which enables the creation of two different messages intelligible to the users. By introducing a new principle users are going to have the possibility to visually see which content they are going to receive before they use their bar code reader. By knowing the technical limitations of a QR code (section 2) this paper investigates the possibility of placing two messages in one 2D bar code symbol without embedding images. This includes the use of graphic patterns. The reason for using a black and white graphic pattern is its higher reliability for scanning. In order to bridge the design limitations of a 2D bar code caused by function patterns, they are transferred to digital space and so remain intact. This means it is necessary to add new elements which will enable that transfer. By changing the visual layout of the function patterns, error correction algorithms and the coding region (by switching 0 to 1 and vice versa), it is possible to create symbols, images or words out of squares intelligible to man, similar to pixel art. This is done by using masks. Masks represent a method of overlaying certain information, while completely preserving the encoded message. Masks do not interrupt the security and decoding of the bar code’s message because they do not represent pictures but information. As masks are in fact information, every module has to be correctly read in order to get to the final encoded message. This is the reason why masks (created images) are made like pixel art. This idea opens the possibility of having two messages in one bar code symbol. There are two ways of doing so. In the first case one message can be intended for the user, and the other one for the computer. In the second case, one message can be read with one bar code reader, and the other message in the same symbol with another bar code reader. Both bar code readers could be installed on a single smartphone. To prove that this concept is possible to carry out in practice, two important steps are investigated (see Fig. 1). In the first step linear block codes are used in order to explain and prove the principle mathematically. In the second step an eye tracking device is used with which a minimal mask change based on Weber’s law [34] is investigated. In order to see if the principle described previously is even possible it is necessary to research step 1 and step 2. The first step is explained in the next chapter.

Fig. 1.Visual scheme of creating a principle for displaying two different messages

 

4. (Masked Matrix) MM Code Design Procedure – an Academic Vision

The aim of this step is to mathematically prove that it is possible to encode a message in a 2D matrix bar code and use its whole surface for embedding an image. This is presented in the encoding phase. In order to prove that the encoded message can be obtained from such a bar code it is necessary to present that process in the decoding phase.

4.1 Encoding Phase

Step 1: Defining the code For principle explanation purposes, the first 128 characters (from 0 to 127) from the ASCII string are taken. Due to the simplicity of presenting the design procedure, an academic example of dimensions 9 × 9 is used which may be seen as equivalent to the QR code. Although this academic example is not applicable in practice, for explanatory purposes it will be called (Masked matrix) MM code. The name describes what this principle uses, from a technical standpoint, in order to achieve its visual appearance. In addition, a binary linear code with parameters [9,7,3] can be taken to encode the 128 characters into codewords of length 9 [35]. This code ensures the correction of one possible error in every codeword.

Step 2: Setting the information in the graphic structure For a message that is encoded in this example, the following 9 optional codewords are taken: 100011101 [encoded letter G], 110110000 [l], 010111000 [.], 111001111 [s], 110110000 [l], 110111101 [o], 111011010 [v], 110111101 [o] and 100000101 [A]. A part where the 2D bar code or any other information is embedded is called the inner matrix, in our case it is the square of order 9. Information contained in the inner matrix is called the message. The part of the structure in which the surrounding bits are arranged around the inner matrix, is called the rim. In this example the remaining 4 bits first four bits represent the dimensions of the inner matrix. For that reason the remaining 4 bits of the message need to be stored elsewhere (see Fig. 2, a).

Fig. 2.a) Empty rim and filled inner matrix to be read from the left lower angle: 4 bits for message dimension followed by 9 codewords of length 9; the last 4 bits of the 9th codeword to be saved elsewhere b) Example of a mask (under ordinal number 0101), representing the letter A from an old Croatian script

Step 3: Creating masks Unlike designed QR codes, masks that represent characters or images that one can recognize do not reduce their reliability. In addition to the listed codewords, the dimension of the inner matrix needs to be stored as well (in Fig. 2 represented by 1001 in the lower left corner). Therefore, the last 4 bits of the last codeword (in our example) to be stored elsewhere determine the ordinal number of the mask from the mask folder (not greater than 24) which will be chosen to represent the MM code. Each mask in the mask folder has to be as similar as possible to one another, but also to differ on as many positions as possible. The author of a mask folder needs to find a successful compromise between these two contradictory conditions. The masks in our example, taken to be all visually identified as variations of the letter A (see Fig. 2. b), differ from each other in at least 13 positions (see Fig. 3). The mask folder in our example contains 16 masks.

Fig. 3.Mask folder showing the glagolitic letter A

Masks allow for correcting errors on the level of the whole inner matrix in addition to correcting errors on the level of individual codewords.

Step 4: Display matrix In order to visually change the message, using a mask, it is necessary to introduce the display matrix. The display matrix is created by the binary addition of the message and the corresponding mask, which is then placed in the rim. Having done so, the encoding procedure is completed and the final visual form of a MM code is created (see Fig. 4, a).

Fig. 4.MM code a) without errors b) with 6 errors differing from the mask labeled 0101 and more than 6 with the others

4.2 Decoding Phase

Step 5: Correcting errors in the inner matrix The correction procedure is the following. The received mask bits which contain errors (see Fig. 4, b), are compared to each mask in the mask folder representing the same graphic pattern (see Fig. 3) and analysed with regard to the number of positions in which the values differ. That defective mask is replaced with the correct mask with which it differs in the least number of positions. In our example it is the mask under number 6 (see Fig. 3), with a binary label 0101. In this way 6 errors were corrected in this example.

Step 6: Correcting errors in the outer rim In order to retrieve the message, binary subtraction from the mask and display matrix is performed. This operation retrieves the encoded message containing errors. To correct the remaining 4 errors in this example, the dual code is used [35]. In this process, errors are detected word by word and the correct codewords found with respect to the minimal Hamming distance principle.

Step 7: Displaying the message contained in the MM code Once all the codewords are known, the ASCII character table is checked to find which codeword has been assigned to which sign. This concludes the decoding process and the message embedded in the MM code, which is in this example “Gl.slovoA”, is shown.

4.3 Mask author’s influence

The author of a mask has a direct influence on the final appearance of the MM code (Fig. 5), which enables him to better communicate the encoded messages to the user. This level of human influence on the bar code wasn’t possible before.

Fig. 5.Some examples of the simulated MM code

By designing masks, it is possible to create one mask in the form of a QR code, which contains Data matrix, Aztec code or some other 2D bar code inside the inner matrix (see Fig. 6). This designed solution allows for displaying two different messages by containing two different bar codes in one MM code. It is also possible to create a large number of identical masks which display one single QR code, but each bar code contains a different coded message. In this chapter it has been proven that the hypothesis set in this article is valid.

Fig. 6.Scheme of decoding two messages in the new structure

 

5. Experiment

In order to visually change the message, using a mask, a mask folder has to be designed. Masks in a mask folder have to be as similar as possible to one another, but also to differ on as many positions as possible. The goal of the experiment is to prove whether there is a statistically significant variable by which the MM code users, i.e., the participants in the experiment, recognized the designed form modifications for each graphic pattern (sample) in a mask folder. Therefore the following two research question were answered in the experiment:

1. Can participants consciously recognize the minimal changes in given graphic patterns? The research was conducted by a survey.

2. Do participants visually perceive the minimal changes in given graphic patterns? The research was performed using a Tobii X60 eye tracking device.

If so, if examinees can indeed correctly recognise and perceive the changes in graphic patterns the idea of the principle could not be supported. Otherwise the principle described in this paper can become the basis for creating two messages.

5.1 Experimental setup

The modifications in graphic patterns, expressed in percentages, are going to be modeled by Weber's law of just noticeable difference (jnd) [34]. Jnd is a difference that an examinee notices in 75% of attempts [36]. In order to express it, there should always be a constant comparison between the stimulus and every previous form variation [37]. The experiment was conducted at the Faculty of Graphic Arts (GF), University of Zagreb and the Faculty of Electrical Engineering and Computing (FER), University of Zagreb. In total, 85 students (47 examinees from FER, and 38 from GF) have participated. All of the examinees had 20/20 eyesight and did not wear prescription glasses. The monitor was a 22 inch Lenovo ThinkVision L2251x with a resolution of 1680 × 1050 pixels (16:10 display aspect ratio). The research was done in normal room lighting conditions. The approximate distance of the examinee from the monitor was 30 to 40 cm in order to simulate a situation in which the user is holding a product with a bar code in a store. The examinee, had a supervisor who gave instructions regarding their task (how to start the test, what to do, etc.). The examinee had to detect the minimal difference between given graphic patterns on MM codes sized 2 × 2 cm. The examinee had to compare the reference sample of each graphic pattern to other samples and determine which, if any, changes occurred. By placing every MM code on a different product’s packaging, except for the reference sample, real-life situations in which MM codes appear are faithfully simulated. In order for the examinees to concentrate on the graphic pattern the background was made solid white (see Fig. 7).

Fig. 7.Sample No 18 and its area of interes (AI) for the research

Graphic patterns, created in such a way that they resemble geometric, typographic and complex shapes, were presented in MM codes (see Fig. 8).

Fig. 8.Graphic patterns (reference samples) that examinees had to remember in order to compare them to other similar variations (samples)

For a geometric shape a composition of vertical and horizontal lines which crossbreed was taken. For a typographic shape the letter B was used and for the complex one a shape in the form of a man riding a horse was used. All patterns were of the same dimensions (11 × 11 modules), which is equivalent to the smallest version of Micro QR code M1. Having this grid, graphic patterns as well as the laws of minimal change will be easier to notice.

The variable changes in the above samples are shown to the examinees at random in order for them not to know which expected change has occurred. After every slide of a presented graphic pattern the examinee had to answer the question: “Which changes have occurred compared to the original picture?” in a survey questionnaire with the following answers: 1 just the change in thickness in the same parts of the form, 2 just the change in height, 3 there wasn't any change, 4 the change in the size of the form, 5 the change in the form. All the answers are unambiguous and every examinee had to circle just one. These answers in the experiment represent the dependent variables (see Table 1). It is then determined which of them contribute the least and which of them the most to minimal modification of a graphic pattern.

Table 1.List of variables with defined number of manipulation levels

The independent variables (see Table 1) with which the dependent variable is being researched are as follows: First fixation duration – duration of first fixation on the AI (area of interest) (measured in seconds); Fixation count – a number which shows how much the users have fixated on the AI; Visit count – duration of each individual visit inside the AI measured in seconds. Each of these independent variables is implemented in an area where the change had happened (AI) and in an area where the change had not happened (not the area of interest).

5.2 Results and Analysis

Regression analysis is used in order to find the analytical connection between the variables. Linear regression is used because it examines the relation between one dependent variable and one (or more) independent variable. It shows how much an independent variable effects the dependent variable. In this work it was of interest to find what, and in what measure, can influence correct answers. The ANOVA test has also been used, which serves for finding factors that had the greatest influence on linear regression. In this article only a combined statistical analysis of Faculty of Graphic Arts and Faculty of Electrical Engineering and Computing is shown because it represents a statistically significant influence on the dependent variables. The following table shows the results obtained by using descriptive statistics for each individual sample (see Table 2). The reason for using descriptive statistics is to describe overall results. This descriptive statistics for overall results has 6 columns. Minimum (Minimum) and maximum (Maximum) values are a span of represented values in our analysis for each individual variable.

Table 2.Descriptive statistics of survey answers. Survey question: “Which changes have occurred compared to the original picture?”

By grouping an examinee’s results for each shape group seperately the following results have been obtained.

Table 3.Descriptive statistics for the first 18 samples (abstract group)

From this table one can understand that, on average, the examinees have most frequently answered the question with the answer under number 3 (there wasn’t any change). From this the conclusion can be drawn that the examinees have not noticed the changes in given graphic patterns.

Table 4.Descriptive statistics for the samples from No 19-22 (typographic group)

In the typographic group examinees have mostly given answers under number 4 (the change in the size of the form). The correct answer for this group is number 5 (the change in the form). Therefore a conclusion can be made that examinees haven’t noticed which change had happened in this group.

Table 5.Descriptive statistics for the samples from No 23-26 (complex group)

The examinees have also given answers under number 4, while the correct answer is under number 5. The same conclusion can be made as for the typographic group.

Because of results shortening, only examples of frequency tables where the correct answer is of the lowest frequency are shown (2nd,4th and 18th sample). The correct answer is marked with a filled color.

Table 6.Results for sample No 2

In the table for the second sample the correct answer (4) has been noticed only by 3 examinees (only 3,5%). Most of them have answered number 3 (there wasn’t any change) which means that the change in this graphic pattern wasn't noticed.

Table 7.Results for sample No 4

In question number four one can understand that the change in the size of the graphic pattern (4) wasn’t noticed (only 7/85 - 8,2% examinees have noticed the change).

As for the 18th sample only 9,4% examinees have answered correctly (2) (see Fig. 7).

Table 8.Results for sample No 18

The following tables represent eye tracking results from linear regression for all variables.

R-square (determination coefficient), is empirically significant (p=0,008 what indicates statistical significance) so this model describes this data well (see Table 9).

Table 9.a. Predictors: (Constant), Visit fixation thickness, First fixation form, Visit fixation form, Count fixation height, First fixation width, Count fixation form, Visit count height, First fixation height, Count fixation width, Visit count width, First fixation thickness, Count fixation thickness

Table 10.a Predictors: (Constant), Visit fixation thickness, First fixation form, Visit fixation form, Count fixation height, First fixation width, Count fixation form, Visit count height, First fixation height, Count fixation width, Visit count width, First fixation thickness, Count fixation thickness

Because in this case the p-value is significant, at least one of the coefficients in the model, which stands by an independent variable, is different from zero. This means there is a difference between dependent variables (see Table 11).

Table 11.Coefficients of the linear regression model

Model:

Number_correct_answers=0,359-0,130*First_fixation_height+0,004*Count_fixation_height-0,012*Visit_count_hight+0,028*First_fixation_width+0,023*Count_fixation_width+0,003*Visit_count_width+0,094*First_fixation_form-0,031*Count_fixation_form-0,058*Visit_fixa tion_form-0,044*First_fixation_thickness+0,028*Count_fixation_thickness+0,032*Visit_fix ation_thickness

Free member (0,359) is interpreted in a way that if one assumes that all the other variables are zero, 0,359 is the expected number of correct answers. Other coefficients show that if the number of correct answers is increased by 1, the number First_fixation_height will be decreased by 0,130, the number Count_fixation_height will be increased by 0,004 etc.. Based on conducted regression analysis nothing can be concluded with a certain statistical significance.

To summarize this section, participants of the survey have answered 20/26 questions correctly. Based on eye tracking research by studying three independent variables, a statistically significant variable by which the examinees recognized form modifications (from 4 to 53%) doesn’t exist. It is therefore possible to conclude that despite recognizing form modifications as seen from the survey (see Table 2), the examinees haven’t noticed which dependent variable has changed. This research shows that the percentage is not crucial for changing individual variations of a graphic pattern. It is important to notice that the change in variation depends greatly on the placement of the modules (see Table 11). By analyzing the results one can understand that the pattern changes, in observed percentages, are still recognizable as the reference graphic pattern to the users. Afterwards, the question of which bar code they find visually attractive was also asked. Possible answers were standard QR codes or the new 2D bar codes with the applied principle (MM code). From 85 examinees, 12 of them (14,1%) found QR codes visually attractive, while 43 (50,6%) found MM codes visually attractive. The rest of them, 30 examinees (35,3%) said that they do not scan bar codes and therefore cannot tell which bar code is more visually attractive. So for the examinees who use bar codes the majority found MM codes with the applied principle more visually attractive than standard QR codes.

 

6. Conclusion

Future plans, based on the research performed for this article, are to investigate the minimal recognition changes depending on the position of the change. Research of how many changes are possible to place on one mask (beyond the investigated changes in this paper) before a user recognizes the form modification is also planned. In order to explore the limitations on printed media, it is necessary to create a mobile application specifically for reading the MM code. With that tool, the parameters of reproduction techniques can then be fully investigated (resolution of the new printed structure, the efficiency of the finder pattern, determining the minimal size of squares and triangles in the new structure, incorporation of different grey tone values in the new structure). In this research, where Weber's law was investigated by simultaneous examination of two variables, results have shown that the examinees were unable to recognize form modification of up to 53%. One can also see that it isn’t just variable change that is important, but also the position of the change in comparison to the other variables. By knowing this fact it can be concluded that QR codes or any other 2D code can be used for displaying two different messages through the explained principle. It is mathematically proven that with the help of one message type it is possible to display another one. Thereby this article has shown how it is possible to read both type of messages. Ultimately this principle will contribute to a better connection between users and bar codes. It will achieve a new level of visual and cognitive communication between them, which can greatly increase the use of bar codes in practice.