Employing structural representation for symbol detection, symbol spotting and indexation in line drawing document images Muhammad Muzzamil Luqman [email protected] PhD student (II year)

Thesis supervisors Jean-Yves Ramel

Université François Rabelais de Tours, France

Thierry Brouard

Université François Rabelais de Tours, France

Josep Lladós

Universitat Autònoma de Barcelona, Spain

Wednesday, 02 June 2010

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Plan

Part 1 Representation and recognition of graphics content in line drawing document images

Part 2 Unsupervised indexation and content based (focused) retrieval for line drawing document image repositories

-2

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Plan

Part 1 Representation and recognition of graphics content in line drawing document images

-3

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Representation phase

Representation phase Representation of structure of graphics content by an Attributed Relational Graph. Description phase Learning and Classification phase

-4

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

-5

Description phase

Representation phase Description phase Extraction of signature from ARG.

Learning and Classification phase Number of primitives in symbol

Number of nodes

Number of connections

L-Junctions T-Junctions Intersections (X)

Parallel connections (P) Successive connections (S)

Density of Connections at nodes

Distribution of relative angle of connections

Number of nodes with Low density of connections

Number of Small-Length primitives

Number of Small-Angle connections

Number of nodes with Medium density of connections

Number of Medum-Length primitives

Number of Medum-Angle connections

Number of nodes with High density of connections

Number of FullLength primitives

Number of Full-Angle connections

Arrangement of connections (between primitives)

Distribution of relative length of primitives

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Number of primitives in symbol

Number of nodes

Number of connections

L-Junctions T-Junctions Intersections (X)

Parallel connections (P) Successive connections (S)

Description phase

Density of Connections at nodes

Distribution of relative angle of connections

Number of nodes with Low density of connections

Number of Small-Length primitives

Number of Small-Angle connections

Number of nodes with Medium density of connections

Number of Medum-Length primitives

Number of Medum-Angle connections

Number of nodes with High density of connections

Number of FullLength primitives

Number of Full-Angle connections

Arrangement of connections (between primitives)

Distribution of relative length of primitives

-6

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Number of primitives in symbol

Number of nodes

Number of connections

L-Junctions T-Junctions Intersections (X)

Parallel connections (P) Successive connections (S)

Density of Connections at nodes

Distribution of relative angle of connections

Number of nodes with Low density of connections

Number of Small-Length primitives

Number of Small-Angle connections

Number of nodes with Medium density of connections

Number of Medum-Length primitives

Number of Medum-Angle connections

Number of nodes with High density of connections

Number of FullLength primitives

Number of Full-Angle connections

Arrangement of connections (between primitives)

-7

Description phase

A value laying here fully contributes (i.e. membership weight 1) to the interval “Small”

Distribution of relative length of primitives

A value laying here contributes in part to the interval “Medium” and in part to the interval “Full”

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Number of primitives in symbol

Number of nodes

Number of connections

L-Junctions T-Junctions Intersections (X)

Parallel connections (P) Successive connections (S)

Description phase

Density of Connections at nodes

Distribution of relative angle of connections

Number of nodes with Low density of connections

Number of Small-Length primitives

Number of Small-Angle connections

Number of nodes with Medium density of connections

Number of Medum-Length primitives

Number of Medum-Angle connections

Number of nodes with High density of connections

Number of FullLength primitives

Number of Full-Angle connections

Arrangement of connections (between primitives)

-8

Distribution of relative length of primitives

Two iterations over set of ARGs: First iteration 1. Compute ‘connection density counts’ for all ARGs 2. Distribute these ‘connection density counts’ in an optimal number of bins 3. Arrange the bins in a fuzzy fashion to form overlapping intervals for ‘Low’, ‘Medium’ & ‘High’ connection densities. Second iteration Compute signature for graphic symbols (ARGs)

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Learning phase (Structure & Parameters of BN)

-9

Representation phase Description phase Learning and Classification phase Encoding of Joint Probability Distribution of signatures by a Bayesian Network. P(Nodes)

P(Class|Nodes)

P(DenH|DenM)

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Classification phase (Graphics Recognition)

- 10

Representation phase Description phase Learning and Classification phase Encoding of Joint Probability Distribution of signatures by a Bayesian Network.

Bayesian probabilistic inference for recognition.

Bayes rule:

Posterior probabilit y

P (ci | e)

P (e, ci ) P (e)

Likelihood * Prior p robability Marginal l ikelihood

P (e | ci ) P (ci ) P (e)

where e

f 1, f 2,..., f 21 k

P(e)

P(e, ci )

P(e | ci ) P(ci ) i 1

Query is recognized as class which gets highest posterior probability!

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Example images

- 11

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Noise and deformations

- 12

2D linear model symbols from GREC databases Learning on clean symbols and testing against noisy and deformed symbols

Results presented in CIFED2010 – With Fuzzy Intervals

Results presented in ICDAR2009 – Without Fuzzy intervals

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Noise and deformations

2D linear model symbols from GREC databases Learning on clean symbols and testing against noisy and deformed symbols

Comparing results with (Qureshi et al., 2007) and (Luqman et al., 2009)

- 13

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Context noise

2D linear model symbols from GREC databases (SESYD dataset) Learning on clean symbols and testing against context-noise

Results presented in CIFED2010 – With Fuzzy Intervals

- 14

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Some remarks

Based on vectorization and hence is sensitive to noise and deformation (which produce irregularities in signature). The proposed signature is more vulnerable to symbols that are composed of circles/arcs.

However, lightweight signature and use of an efficient classifier makes it suitable to be used as a pre-processing step to reduce search space or as a quick discrimination method for sufficiently large number of graphic symbols … an application to symbol spotting!

- 15

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Generalizing fuzzy signature - Explicit Graph Embedding

Vector for explicit embedding of attributed graphs

Fuzzy zones for “features for node degrees” (for example)

A value laying here contributes in part to the interval “Fi2” and in part to the interval “Fi3”

- 16

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

ICPR2010 contest on Explicit Graph Embedding (GEPR)

ICPR2010 contest Graph Embedding for Pattern Recognition (GEPR)

Results on sample contest data ALOI COIL ODBK

(Performance Index: 0.379) (Performance Index: 0.376) (Performance Index: 0.353)

ALOI - Amsterdam Library of Object Images COIL - Columbia Object Image Library ODBK - Object Databank

Performance Index measures the quality of clustering (that could be obtained for the embedded vectors). The closer it gets to zero the better the embedding results are!

- 17

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Plan

Part 2 Unsupervised indexation and content based (focused) retrieval for line drawing document image repositories

- 18

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

A Symbol Spotting & Focused Retrieval System

Localization results

QBE

Utilisateur

Spotting system

Document base

- 19

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

A Symbol Spotting & Focused Retrieval System

Unsupervised indexation of line drawing document images 

Represent document images by attributed relational graphs



Spot Regions Of Interest (ROIs) in the ARG of document image



Learn parameters for fuzzy structural signature from the set of ROIs



Describe each ROI by a fuzzy structural signature



Cluster signatures of ROIs



Prepare an index (clusterID vs ROIs vs documentImage) and



Learn a BN

- 20

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

A Symbol Spotting & Focused Retrieval System

Content based focused retrieval for line drawing document images 

Represent query ROI by attributed relational graph



Spot Regions Of Interest (ROIs)



Describe each query ROI by a fuzzy structural signature



Classify query ROIs using BN and



Retrieve documents using repository index

- 21

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

A Symbol Spotting & Focused Retrieval System

- 22

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Experimentation

Dataset SESYD (Systems Evaluation SYnthetic Documents)

During learning phase our system detected a total of 10285 ROIs in electronic diagrams and 4586 ROIs in floorplans, which approximately corresponds to 108% of the symbols in each of the datasets.

- 23

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

- 24

Experimentation

Document Retrieval Results

Each point in the graph represents the precision and recall values for a query image.

Results presented in ICPR2010

Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion

Conclusion and Future work

The Overall framework allows to prepare an index for the document repository in an unsupervised fashion, which is a very important contribution.

However the underlying method for ROI localization is based on a set of heuristics and does not return a single symbol in most of the cases and needs to be improved.

Future lines of work include the designing of a method to replace the manually selected heuristics by automatic learned heuristics for spotting a ROI.

- 25

References

 Delalandre et al., “Building synthetic graphical documents for performance evaluation,” in GREC, vol. 5046 of LNCS, pp. 288–298, Springer, 2007.  Delaplace et al., Two evolutionary methods for learning bayesian network structures, in LNAI 2007.  Luqman et al., A Content Spotting System For Line Drawing Graphic Document Images, International Conference on Pattern Recognition, 2010, to appear.

 Luqman et al., Vers une approche floue d’encapsulation de graphes: application à la reconnaissance de symboles, Colloque International Francophone sur l'Ecrit et le Document, 2010, 169-184.  Luqman et al., Graphic Symbol Recognition using Graph Based Signature and Bayesian Network Classifier, Tenth International Conference on Document Analysis and Recognition (ICDAR), IEEE Computer Society, 2009, volume 10, 1325-1329.  Luqman et al., Employing fuzzy intervals and loop-based methodology for designing structural signature: an application to symbol recognition, Eighth IAPR International Workshop on Graphics RECognition (GREC), 2009, volume 8, 22-31.  Qureshi et al., Combination of symbolic and statistical features for symbols recognition, in IEEE ICSCN’2007.  Qureshi et al., “Spotting symbols in line drawing images using graph representations,” in GREC, pp. 91–103, 2007.

- 26