Algorithm Validation

Posted on by

Brief description of this line of research

The validation of ARIA (automatic retinal image analysis) algorithms is a very important task. Validation indicates the process by which the output of an algorithm can be compared with a reference standard which is usually represented by a ground truth defined by expert performance. Our aim is to contribute towards the objective validation of these algorithms by proposing new measures of evaluation and providing publicly available datasets.

 

RIM-ONE DL

RIM-ONE DL is the new revision of the original versions of RIM-ONE specifically designed for its use in Deep Learning applications [1].

Please visit our GitHub repository for more information and download options: https://github.com/miag-ull/rim-one-dl

 

[1]
F. J. Fumero Batista, T. Diaz-Aleman, J. Sigut, S. Alayon, R. Arnay, and D. Angel-Pereira, “RIM-ONE DL: A Unified Retinal Image Database for Assessing Glaucoma Using Deep Learning,” Image Analysis & Stereology, vol. 39, no. 3, pp. 161–167, Oct. 2020, doi: 10.5566/ias.2346. [Online]. Available: https://www.ias-iss.org/ojs/IAS/article/view/2346. [Accessed: 30-Jul-2021]

Abstract: The first version of the Retinal IMage database for Optic Nerve Evaluation (RIM-ONE) was published in 2011. This was followed by two more, turning it into one of the most cited public retinography databases for evaluating glaucoma. Although it was initially intended to be a database with reference images for segmenting the optic disc, in recent years we have observed that its use has been more oriented toward training and testing deep learning models. The recent REFUGE challenge laid out some criteria that a set of images of these characteristics must satisfy to be used as a standard reference for validating deep learning methods that rely on the use of these data. This, combined with the certain confusion and even improper use observed in some cases of the three versions published, led us to consider revising and combining them into a new, publicly available version called RIM-ONE DL (RIM-ONE for Deep Learning). This paper describes this set of images, consisting of 313 retinographies from normal subjects and 172 retinographies from patients with glaucoma. All of these images have been assessed by two experts and include a manual segmentation of the disc and cup. It also describes an evaluation benchmark with different models of well-known convolutional neural networks.

 

RIM-ONE

RIM-ONE is an open retinal fundus image database with accurate gold standards of the optic nerve head (ONH) provided by different experts. It includes images from healthy eyes as well as images from eyes with glaucoma at different stages. A variability measurement by zones of the optic disc is also proposed for the purpose of validation. Three hospitals have contributed to the development of this database: Hospital Universitario de Canarias, Hospital Clínico San Carlos and Hospital Universitario Miguel Servet.

 

DCSeg – An interactive tool for optic disc and cup segmentation of stereo and monocular retinal fundus images

We have developed a new tool to ease the manual segmentation of the optic disc and cup of retinal fundus images, which can be used on monocular and stereo images. By using this tool we have developed the release 3 of RIM-ONE.

Version 0.2b includes a Self-assessment mode, very useful for teaching and learning. This is a beta release that we will continue updating and improving.

The DCSeg tool and instructions can be downloaded from: DCSeg 0.2b

Demo video of optic cup segmentation in stereo mode:

We are glad to hear any comments about it!

 

RIM-ONE Release 3

The third version of the RIM-ONE database is finally here. This dataset consists of 159 stereo retinal fundus images. The optic disc and optic cup of each image has been segmented by 2 experts in ophthalmology to create the ground truth. The average segmentation is also available to use as the reference segmentation or gold standard.

RIM-ONE third release: RIM-ONE r3

 

RIM-ONE Release 2

We have published a new version of the database, with more images and better precision on the manual segmentations.

RIM-ONE second relase:  RIM-ONE r2

MATLAB function to convert gold standard files to mask images:  convert_gs_to_image

In this second release, the format of the gold standard files is a little bit different than the first version, it uses more radii to manually segment the optic nerve head and the classification of the images is also different. There is a MATLAB function to convert the gold standard files to mask images. The usage of this function is as follows:
- First unzip the file “RIMONE db r2.zip”
- Once unzipped, it will contain two folders: “Normal” and “Glaucoma and glaucoma suspicious”
- To convert the gold standard files to images:
convert_gs_to_image("path/to/Normal", "path/to/Output Normal GS images")

convert_gs_to_image(“path/to/Glaucoma and glaucoma suspicious”, “path/to/Output Glaucoma GS images”)

 

RIM-ONE Release 1

The first release of the RIM-ONE database can be downloaded from here:  RIM-ONE r1

The MATLAB code for the variability measurement procedure:  variability_measurement_procedure

RIM-ONE Variability among experts

RIM-ONE Variability among experts

Related publications:

[1]
F. J. Fumero Batista, T. Diaz-Aleman, J. Sigut, S. Alayon, R. Arnay, and D. Angel-Pereira, “RIM-ONE DL: A Unified Retinal Image Database for Assessing Glaucoma Using Deep Learning,” Image Analysis & Stereology, vol. 39, no. 3, pp. 161–167, Oct. 2020, doi: 10.5566/ias.2346. [Online]. Available: https://www.ias-iss.org/ojs/IAS/article/view/2346. [Accessed: 30-Jul-2021]
[1]
F. Fumero, J. Sigut, S. Alayon, M. González-Hernández, and M. González de la Rosa, “Interactive Tool and Database for Optic Disc and Cup Segmentation of Stereo and Monocular Retinal Fundus Images,” in Short Papers Proceedings - WSCG 2015, Pilsen, Czech Republic, 2015, pp. 91–97 [Online]. Available: http://wscg.zcu.cz/DL/wscg_DL.htm
[1]
S. Alayon, M. Gonzalez de la Rosa, F. J. Fumero, J. F. Sigut Saavedra, and J. L. Sanchez, “Variability between experts in defining the edge and area of the optic nerve head,” Archivos de la Sociedad Española de Oftalmología (English Edition), vol. 88, no. 5, pp. 168–173, May 2013, doi: 10.1016/j.oftale.2012.07.005. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S2173579413000765
[1]
F. Fumero, S. Alayon, J. L. Sanchez, J. Sigut, and M. Gonzalez-Hernandez, “RIM-ONE: An open retinal image database for optic nerve evaluation,” in 2011 24th International Symposium on Computer-Based Medical Systems (CBMS), 2011, pp. 1–6, doi: 10.1109/CBMS.2011.5999143 [Online]. Available: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5999143&tag=1

 

Useful links:

[1]
E. Trucco, A. Ruggeri, T. Karnowski, L. Giancardo, E. Chaum, J. P. Hubschman, B. Al-Diri, C. Y. Cheung, D. Wong, M. Abràmoff, G. Lim, D. Kumar, P. Burlina, N. M. Bressler, H. F. Jelinek, F. Meriaudeau, G. Quellec, T. Macgillivray, and B. Dhillon, “Validating retinal fundus image analysis algorithms: issues and a proposal,” Invest. Ophthalmol. Vis. Sci., vol. 54, no. 5, pp. 3546–3559, May 2013 [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/23794433

 

Leave a Reply