Employing structural representation for symbol detection, symbol spotting and indexation in line drawing document images Muhammad Muzzamil Luqman
[email protected] PhD student (II year)
Thesis supervisors Jean-Yves Ramel
Université François Rabelais de Tours, France
Thierry Brouard
Université François Rabelais de Tours, France
Josep Lladós
Universitat Autònoma de Barcelona, Spain
Wednesday, 02 June 2010
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Plan
Part 1 Representation and recognition of graphics content in line drawing document images
Part 2 Unsupervised indexation and content based (focused) retrieval for line drawing document image repositories
-2
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Plan
Part 1 Representation and recognition of graphics content in line drawing document images
-3
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Representation phase
Representation phase Representation of structure of graphics content by an Attributed Relational Graph. Description phase Learning and Classification phase
-4
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
-5
Description phase
Representation phase Description phase Extraction of signature from ARG.
Learning and Classification phase Number of primitives in symbol
Number of nodes
Number of connections
L-Junctions T-Junctions Intersections (X)
Parallel connections (P) Successive connections (S)
Density of Connections at nodes
Distribution of relative angle of connections
Number of nodes with Low density of connections
Number of Small-Length primitives
Number of Small-Angle connections
Number of nodes with Medium density of connections
Number of Medum-Length primitives
Number of Medum-Angle connections
Number of nodes with High density of connections
Number of FullLength primitives
Number of Full-Angle connections
Arrangement of connections (between primitives)
Distribution of relative length of primitives
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Number of primitives in symbol
Number of nodes
Number of connections
L-Junctions T-Junctions Intersections (X)
Parallel connections (P) Successive connections (S)
Description phase
Density of Connections at nodes
Distribution of relative angle of connections
Number of nodes with Low density of connections
Number of Small-Length primitives
Number of Small-Angle connections
Number of nodes with Medium density of connections
Number of Medum-Length primitives
Number of Medum-Angle connections
Number of nodes with High density of connections
Number of FullLength primitives
Number of Full-Angle connections
Arrangement of connections (between primitives)
Distribution of relative length of primitives
-6
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Number of primitives in symbol
Number of nodes
Number of connections
L-Junctions T-Junctions Intersections (X)
Parallel connections (P) Successive connections (S)
Density of Connections at nodes
Distribution of relative angle of connections
Number of nodes with Low density of connections
Number of Small-Length primitives
Number of Small-Angle connections
Number of nodes with Medium density of connections
Number of Medum-Length primitives
Number of Medum-Angle connections
Number of nodes with High density of connections
Number of FullLength primitives
Number of Full-Angle connections
Arrangement of connections (between primitives)
-7
Description phase
A value laying here fully contributes (i.e. membership weight 1) to the interval “Small”
Distribution of relative length of primitives
A value laying here contributes in part to the interval “Medium” and in part to the interval “Full”
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Number of primitives in symbol
Number of nodes
Number of connections
L-Junctions T-Junctions Intersections (X)
Parallel connections (P) Successive connections (S)
Description phase
Density of Connections at nodes
Distribution of relative angle of connections
Number of nodes with Low density of connections
Number of Small-Length primitives
Number of Small-Angle connections
Number of nodes with Medium density of connections
Number of Medum-Length primitives
Number of Medum-Angle connections
Number of nodes with High density of connections
Number of FullLength primitives
Number of Full-Angle connections
Arrangement of connections (between primitives)
-8
Distribution of relative length of primitives
Two iterations over set of ARGs: First iteration 1. Compute ‘connection density counts’ for all ARGs 2. Distribute these ‘connection density counts’ in an optimal number of bins 3. Arrange the bins in a fuzzy fashion to form overlapping intervals for ‘Low’, ‘Medium’ & ‘High’ connection densities. Second iteration Compute signature for graphic symbols (ARGs)
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Learning phase (Structure & Parameters of BN)
-9
Representation phase Description phase Learning and Classification phase Encoding of Joint Probability Distribution of signatures by a Bayesian Network. P(Nodes)
P(Class|Nodes)
P(DenH|DenM)
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Classification phase (Graphics Recognition)
- 10
Representation phase Description phase Learning and Classification phase Encoding of Joint Probability Distribution of signatures by a Bayesian Network.
Bayesian probabilistic inference for recognition.
Bayes rule:
Posterior probabilit y
P (ci | e)
P (e, ci ) P (e)
Likelihood * Prior p robability Marginal l ikelihood
P (e | ci ) P (ci ) P (e)
where e
f 1, f 2,..., f 21 k
P(e)
P(e, ci )
P(e | ci ) P(ci ) i 1
Query is recognized as class which gets highest posterior probability!
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Example images
- 11
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Noise and deformations
- 12
2D linear model symbols from GREC databases Learning on clean symbols and testing against noisy and deformed symbols
Results presented in CIFED2010 – With Fuzzy Intervals
Results presented in ICDAR2009 – Without Fuzzy intervals
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Noise and deformations
2D linear model symbols from GREC databases Learning on clean symbols and testing against noisy and deformed symbols
Comparing results with (Qureshi et al., 2007) and (Luqman et al., 2009)
- 13
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Context noise
2D linear model symbols from GREC databases (SESYD dataset) Learning on clean symbols and testing against context-noise
Results presented in CIFED2010 – With Fuzzy Intervals
- 14
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Some remarks
Based on vectorization and hence is sensitive to noise and deformation (which produce irregularities in signature). The proposed signature is more vulnerable to symbols that are composed of circles/arcs.
However, lightweight signature and use of an efficient classifier makes it suitable to be used as a pre-processing step to reduce search space or as a quick discrimination method for sufficiently large number of graphic symbols … an application to symbol spotting!
- 15
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Generalizing fuzzy signature - Explicit Graph Embedding
Vector for explicit embedding of attributed graphs
Fuzzy zones for “features for node degrees” (for example)
A value laying here contributes in part to the interval “Fi2” and in part to the interval “Fi3”
- 16
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
ICPR2010 contest on Explicit Graph Embedding (GEPR)
ICPR2010 contest Graph Embedding for Pattern Recognition (GEPR)
Results on sample contest data ALOI COIL ODBK
(Performance Index: 0.379) (Performance Index: 0.376) (Performance Index: 0.353)
ALOI - Amsterdam Library of Object Images COIL - Columbia Object Image Library ODBK - Object Databank
Performance Index measures the quality of clustering (that could be obtained for the embedded vectors). The closer it gets to zero the better the embedding results are!
- 17
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Plan
Part 2 Unsupervised indexation and content based (focused) retrieval for line drawing document image repositories
- 18
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
A Symbol Spotting & Focused Retrieval System
Localization results
QBE
Utilisateur
Spotting system
Document base
- 19
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
A Symbol Spotting & Focused Retrieval System
Unsupervised indexation of line drawing document images
Represent document images by attributed relational graphs
Spot Regions Of Interest (ROIs) in the ARG of document image
Learn parameters for fuzzy structural signature from the set of ROIs
Describe each ROI by a fuzzy structural signature
Cluster signatures of ROIs
Prepare an index (clusterID vs ROIs vs documentImage) and
Learn a BN
- 20
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
A Symbol Spotting & Focused Retrieval System
Content based focused retrieval for line drawing document images
Represent query ROI by attributed relational graph
Spot Regions Of Interest (ROIs)
Describe each query ROI by a fuzzy structural signature
Classify query ROIs using BN and
Retrieve documents using repository index
- 21
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
A Symbol Spotting & Focused Retrieval System
- 22
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Experimentation
Dataset SESYD (Systems Evaluation SYnthetic Documents)
During learning phase our system detected a total of 10285 ROIs in electronic diagrams and 4586 ROIs in floorplans, which approximately corresponds to 108% of the symbols in each of the datasets.
- 23
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
- 24
Experimentation
Document Retrieval Results
Each point in the graph represents the precision and recall values for a query image.
Results presented in ICPR2010
Part1: Recognition of graphics content Experimentation Some remarks Part2: Content based (focused) retrieval Experimentation Conclusion
Conclusion and Future work
The Overall framework allows to prepare an index for the document repository in an unsupervised fashion, which is a very important contribution.
However the underlying method for ROI localization is based on a set of heuristics and does not return a single symbol in most of the cases and needs to be improved.
Future lines of work include the designing of a method to replace the manually selected heuristics by automatic learned heuristics for spotting a ROI.
- 25
References
Delalandre et al., “Building synthetic graphical documents for performance evaluation,” in GREC, vol. 5046 of LNCS, pp. 288–298, Springer, 2007. Delaplace et al., Two evolutionary methods for learning bayesian network structures, in LNAI 2007. Luqman et al., A Content Spotting System For Line Drawing Graphic Document Images, International Conference on Pattern Recognition, 2010, to appear.
Luqman et al., Vers une approche floue d’encapsulation de graphes: application à la reconnaissance de symboles, Colloque International Francophone sur l'Ecrit et le Document, 2010, 169-184. Luqman et al., Graphic Symbol Recognition using Graph Based Signature and Bayesian Network Classifier, Tenth International Conference on Document Analysis and Recognition (ICDAR), IEEE Computer Society, 2009, volume 10, 1325-1329. Luqman et al., Employing fuzzy intervals and loop-based methodology for designing structural signature: an application to symbol recognition, Eighth IAPR International Workshop on Graphics RECognition (GREC), 2009, volume 8, 22-31. Qureshi et al., Combination of symbolic and statistical features for symbols recognition, in IEEE ICSCN’2007. Qureshi et al., “Spotting symbols in line drawing images using graph representations,” in GREC, pp. 91–103, 2007.
- 26
