Jigsaw Puzzle Solver (JPS) Tayyab Bin Tariq 1

Ahsun Chohan

Ehtasham ul Haq

[email protected]

[email protected]

[email protected]

Department of Computer Science, FAST NUCES, Islamabad, Pakistan

Abstract In this work we present Jigsaw Puzzle Solver (JPS) that automates assembly of straight cut jigsaw puzzles using texture matching. Existing techniques use both texture and shape information for this process. However, JPS solves this problem using only texture information. Using an adaptive threshold JPS is able to reconstruct ~90% of test images correctly. We also discuss the use of JPS for covert transfer of images.

Introduction Given the pieces of an image in arbitrary orientation and without knowledge of the original image, the image needs to be put together such that all the pieces are in their original place. These pieces are not constrained to be the same size; however the scale is supposed to be true. (Radack and Badler 1982) formally stated the problem as: “given a set of simply connected planar regions (silhouetted puzzle pieces), rotate and translate each peace so that the pieces fit together into one region, with no significant area gaps or overlapping pieces.’’ This definition however does not take into account texture and color information, which is vital to the reassembly process of jigsaw puzzles. (Yao and Shao 2003) give a more formal definition: “the

jigsaw puzzle problem can be defined as follows: For SP = {P1, P2, P3, … PN-1} , where Pi represents the ith jigsaw piece (i  {1, 2, 3, N1}), and has the attributes of a closed boundary and a solid texture; and for Pi, there exists Pj, (i ≠ j, i, j  {1, 2, 3, … N-1} ) so that a boundary part of Pi is completely identical with a boundary part of Pj, and the textures and color near these two boundary parts are most similar, that is, Pi and Pj are neighbors (or can be connected), then find the neighbor for Pk (k  {1, 2, 3, … N-1}) and connect all pieces in SP into one large piece.” We propose a texture matching approach that uses an adaptive threshold for straight cut jigsaw puzzles. This system is called Jigsaw Puzzle Solver (JPS). JPS is tested on a variety of images and achieves an accuracy of over ~90%. We also describe how JPS can be used for covert transmission of images.

Related work Jigsaw puzzles can be solved by using shape information, texture information or a combination of both. In this section we cite some examples and compare them using a table originally presented by (Yao and Shao 2003). Lastly JPS is categorized in relation to these examples to establish its purpose, capabilities and limitations.

1 Corresponding Author. Email: [email protected] . Phone +92 345 590 5527

1

(Freeman and Garder 1964) first worked on the jigsaw puzzle problem. Using the limited resolution and computing power at the time they addressed the problem using shape information alone. (Radack and Badler 1982) also proposed a technique based on shape information from the pieces. As stated in the previous section their definition of the problem is limited to shape information and does not take into account texture information. (Webster, LaFollette and Stafford 1991) isthmus critical points for solving jigsaw puzzles. This technique efficiently solved canonical off the shelf jigsaw puzzles. This technique only used shape information. (Kosiba, et al. 1994) derived a set of features based on shape and color information to solve canonical off the shelf jigsaw puzzles. (F. Yao, et al. 1997) proposed using the fine segmentation of boundary curves of off-the-shelf puzzle pieces. The two works mentioned above utilize a combination of texture and shape information to efficiently solve jigsaw puzzles. (Yao and Shao 2003) published an improvement on their formal work. In this paper they classified the pieces into 16 categories based on different shape features and defined which categories could be combined together. This improved the efficiency by reducing the search space. Table 1 was originally published by (Yao and Shao 2003) (the said paper has been added). It can be seen that all discussed techniques utilize shape information. Three techniques use both texture and shape information, however there are no techniques that use only texture information. JPS is a technique for solving straight cut jigsaw puzzle problems using only color information. Although, the same theory can be extended to solve canonical jigsaw puzzles, JPS only handle straight cut puzzles.

Method proposed by

Canonical Puzzles?

Texture Informatio n

No

Shape Infor matio n Yes

(Freeman and Garder 1964) (Radack and Badler 1982) (Webster, LaFollette and Stafford 1991) (Kosiba, et al. 1994) (F. Yao, et al. 1997) (Chung, Fleck and Forsyth 1998) (Yao and Shao 2003)

No

Yes

No

Yes

Yes

No

Yes

Yes

Yes

Yes

Yes

No

Yes

Yes

Yes

Yes

Yes

Yes

No

Table 1: Categorization of methods to solve Jigsaw Puzzle Problem (Yao and Shao 2003)

Design A technique based on the similarity condition given in the introduction section is proposed. Although this technique holds good for most jigsaw puzzles, intended to be put together based on color and texture information, problems are faced when a strong image edge coincides with a puzzle piece edge. Such as the case of a chess board image cut on the edges of the squares. However, it must be noted that if squares cut in the above fashion are given there is no possible way to know whether the original image was like that shown in Figure 1 (a) or (b) or some combination of the squares. The problem is formally stated using terminology used by (Yao and Shao 2003). Let SP = {P1, P2, P3, … PN-1}, where Pi represents the ith jigsaw piece (i Є {1, 2, 3, N-1}), and has the attributes of four edges and a solid texture; and for Pi, there exists Pj, (i ≠ j, I, j Є {1, 2, 3, … N-1} ) so that an edge of Pi is completely identical with and edge of Pj, and the textures and color near these two edges are most similar. Given a piece Pi find Pj satisfying the above 2

condition. The four edges of a piece Pi are represented by vectors EPi1 through to EPi4 containing gray values of the edge.

The above two conditions are enough to define the similarity condition given in the definition put forth by Yao, Gui-Feng Shao (2003). Using this condition however, gives a number of false positives. As a result if a single piece was matched incorrectly it could substantially hurt the matching process. This is due to the fact that all pieces correctly matched with an incorrectly matched piece are incorrectly placed.

Figure 1: (a) and (b) different arrangements of chess board pieces According to the proposed technique two pieces Pi and Pj satisfy the above condition if at least one of the edges pairs EPis and EPit (s, t ϵ {1, 2, 3, 4}) satisfy the conditions stated below. 1. Length of the edges is the same. 2. The edges have similar information.

image

Condition 1 is formally stated as:

Size( EPis )  Size( EPjt ) To state condition two formally a difference vector D and a set S of satisfying values from D are defined.

D  | EPis  EPjt | S { vD  v } Where φ is an experimentally determined threshold discussed in the results section. Condition 2 is satisfied iff

|S| t |D|

Figure 2: Incorrectly matched edges on a flat background. To reduce this effect the value of φ is linked with the image information present on the edge, thus the modified definition of S becomes.

S  { v  D  v

 10C 2

C

C

x 1

x 1

min{ STD( EPisx ),  STD( EPjtx )}}

In the above equation C is the number of color channels in the image and STD is the standard deviation. The effect of this new equation is that the condition for membership of S becomes more and more stringent as the histograms of the edges become flatter. This modification is introduced keeping in view the assumption that an informed decision cannot be made based on edges that do not contain enough image information. Using this new definition of S the results are improved by ~30%.

Where |S| is the number of elements in S, |D| is the number of elements in D and t is an experimentally determined threshold. This threshold is discussed in the results section. 3

Implementation The described technique is implemented using Matlab. As a first step an image is read and divided creating the puzzle pieces. The pieces are randomized and rotated arbitrarily. This forms the input for the technique. A randomly chosen piece is taken to be the initial completed image. The implemented algorithm selects unmatched pieces one by one and each one of its edges is compared with the edges of the completed image using the criterion described in the previous section. If a match is found the piece is combined with the image in the appropriate orientation and is removed from the list of unmatched pieces. The algorithm does not maintain a list of suitable neighbors and treats the first match as appropriate. This confidence in the matching is due to the carefully selected threshold values (discussed in the results section) and the modification in the value of S. The most important implication of this decision is that only one edge match is deemed necessary for two pieces to be placed together. Meaning that only one edge match per piece is required (N in all) instead of two for each corner piece, three for non-corner side pieces and four for internal pieces. This approach gives substantial increases in performance.

Results To evaluate the performance of JPS two sets of tests were performed. The first set of tests was performed to find the optimum values of the two thresholds described in the design section. For this a dataset consisting of 23 images was used. These images contained images of humans, everyday objects, indoor and outdoor scenes. The images in the dataset were classified based on the background and foreground color properties. Then the optimum values φ and t were found for all categories. The categories and the corresponding values of φ and t are summarized in Table 2.

Background Flat texture

Foreground

with Colorful

φ

t

35

0.7

None

Mixed Colors

60

0.8

Flat

Dull

30

0.7

with Single color

30

0.8

Flat texture None

Colorful

35

0.7

Flat

Mixed Colors

60

0.8

35

0.8

Flat texture

with Dull

None

Single color

15

0.8

Flat

Colorful

35

0.8

35

0.7

Flat texture

with Mixed Colors

None

Dull

30

0.8

Flat

Single color

15

0.8

Table 2: Image categories and thresholds Using the above values a second set of tests was performed on a larger dataset containing 50 images similar to the dataset described before. JPS was tested with both definitions of set S given in the design section. The results are summarized in Table 3. Correct Matches 68%

False Positives 32%

False Negative 0%

Without Adaptive Threshold 96% 0% 4% With Adaptive Threshold Table 3: Puzzle Assembly Results

Some of the matched images and processing screen shots are shown below. 4

categories. The values of the thresholds can also be automated by categorizing the images based on color histogram analysis.

Figure 3: Pieces of the image to be assembled

Figure 4: Image assembling process in progress.

The technique provided in this paper can be applied for the secure transfer of images from source to destination in security applications. The images can be cut into pieces at source and each piece can be transmitted separately, possibly through separate medium and put together at the destination. The two main advantages of this approach are; (1) each piece contains very little image information so only a small portion of the image is revealed to transmitting entity (2) since the information on each piece is small, in the event of compromise very little information would be lost. This technique can be extended to other non-image forms of data.

References Chung, M.G., M.M. Fleck, and D. Forsyth. "Jigsaw puzzle solver using shape and color." 1998. Freeman, H., and L. Garder. "Apictorial jigsaw puzzles: The computer solution of a problem in pattern recognition." IEEE Transactions on Electronics and Computers. 1964. Kosiba, D.A., P.M. Devaux, S. Balasubramanian, T.L. Gandhi, and R. Kasturi. "An automatic jigsaw puzzle solver." Proc. 12th IAPRInternat. Conf. on Pattern Recognition. 1994.

Figure 5: Assembled image.

Conclusion & Future work In this paper a technique for assembling jigsaw puzzles is proposed. The matching technique is based solely on image information. The technique was successfully tested on a variety of images with over 90% accuracy. These results however depend on the accurate approximation of the thresholds. The researchers have been able to experimentally find the appropriate values of these thresholds for different image

Radack, G.M., and N.I. Badler. "Jigsaw puzzle matching using a boundary-centered polar encoding." Computer Graphics Image Processing. 1982. Webster, R.W., P.S. LaFollette, and R.L. Stafford. "Isthmus critical points for solving jigsaw puzzles in computer vision." IEEE Trans. Systems Man Cybernet. 1991. 1271–1278. Yao, F.H., G.F. Shao, A. Tamaki, and K. Kato. "Recovery of connection relationships among twodimensional objects." IPSJ J. 1997. Yao, Feng-Hui, and Gui-Feng Shao. "A shape and image merging technique to solve jigsaw puzzles." Pattern Recognition Letters, 2003. 5

6