Webinar: Breaking the interpretation bottleneck
Examining the utility of an automated genomic interpretation algorithm in a clinical genetic lab
The utilization of exome and genome sequencing in clinical practice has become widespread in recent years. However, the genotype-phenotype interpretation remains challenging and time-consuming. By automating the variant prioritization and classification processes, machine learning technologies can unblock the genomic interpretation bottleneck, and improve the power and efficiency of the analysis.
Dr. Meng will present the results of a joint Baylor Genetics-Emedgene study demonstrating the utility of machine learning for interpretation in a 180-case cohort.
Dr. Tzur will follow with a presentation of the machine learning capabilities in action, displaying their value in a case review.
We’ve provided a transcript for this webinar below. The transcript was created with automation and may contain inaccuracies. We recommend accessing the recording for an accurate representation of the session content.
As a lab director, I’ve always seen that at the dry lab of Baylor Genetics, we often face the bottleneck, that we are unable to keep up, we’re seeing increased case numbers. So, in today’s talk I am going to share with you our experience of working with Emedgene’s platform to do whole exome analysis in our lab.
The Interpretation Bottleneck
So, what is the bottleneck we’re facing? We all know that the cost of genome sequencing has fallen to levels, which it is no longer the bottleneck for creating clinical reports. However, the cost of the dry lab is huge compared to the wet lab. This is due to the information overload as we’ve lots of variants from exome and genome sequencing that need to be interpreted. Each type of variant requires highly trained individuals to assess evidence and do the judgments.
ACMG provided interpretation guidelines for variant interpretation, however, it hasn’t been integrated. We are now using the AI platform. And our last challenge is always the need to reinterpret exome and genome data to keep abreast with new published information from the literature.
Why Automate Genomic Interpretation?
Our lab has been working with exome sequencing for a very long time. In the publication (from 2013), we published in the New England Journal of Medicine, here is the kind of workflow at that time. As you can see, for each of the exome cases after filtering, there are about 400-700 variants left that we need to manually assess (each of them) to do the decision making, including the variants’ pathogenicity, the phenotype matching, and the inheritance, in order to decide whether we want to put the variant on report or not. This is a very manual process and takes a lot of time for exome review.
Automated Genomic Interpretation
In order to increase our work efficiency about a year ago, we started to work with Emedgene to look for the options to do more automated genomic interpretation. So, here is a high-level kind of a summary of the Emedgene platform. So, they have their knowledge database which they’ve created from the scientific literature and scientific database. They use machine learning that evaluate each of the variant using the patient’s variant, patient’s phenotype to cross-check with the knowledge database and to form each of the hypotheses. This process is automated for each of the variants from the exome.
Accuracy Study of Emedgene Algorithm
Before we started implementing Emedgene platform in our lab, we would like to know whether the platform is doing the job in the correct way. So, we performed accuracy study using 180 non-selective positive WES cases from our lab. All those cases have been previously manually reviewed by our lab scientists and lab directors. Using Emedgene’s platform, we wanted to measure the ranking of the diagnosis variants and the percentage of the cases that were captured by the top 10 rank variants. And, also later we will talk about our clinical production cohort including the validation, implementation, and our experience with 334 clinical WES cases. We also looked at the automated reanalysis (possibly in the last slide).
Accuracy Cohort vs Production Cohort
So, our accuracy cohort is quite similar to our production cohort, so it is correct representation of our real clinical work. The only difference between accuracy cohort and production cohort is that the accuracy cohort has all positive cases whereas the production cohort is the mixture of positive and the negative cases. Both cohorts have a very similar phenotype presentation as well. The top phenotypes are the neurologic, and the neuromuscular presentations.
Rank of Reported Variants
So, we first evaluated the diagnosis from the 123 trio cases. So, in the graph you see here, the X-axis shows the ranking of the diagnosis variant and the Y-axis shows the number of cases. Out of 123 trios, we found 109 cases have the diagnosis variants listed as top 1 variant and over 98% of the cases have the variants listed in the top 10. Our diagnosis includes all possible inheritance patterns including autosomal dominant-de novo and inherited situations, autosomal recessive compound heterozygous and homozygous situation, and X-linked-de nova, and inherited situations. Next, we looked at singleton cases as you can see the performance is little bit lower compared to trio cases and this is quite expected as in singleton cases we don’t have the parents’ data available for the inheritance model analysis and the cases also include the similar inheritance patterns.
What are the limitations of the algorithm?
Next, we looked at those cases that has relatively lower ranking scores, trying to understand the limitations of the AI algorithm. So, we found out that the first category of cases is those that have incomplete genetic data. So, showing here 3 cases. In the first case, the patient has autosomal recessive Ehlers-Danlos syndrome. WES identified a single heterozygous VUS in PLOD1 gene which is ranked at 38th. CMA later found an exonic deletion in trans. The second case is similar which has a apparently homozygous pathogenic variant in the RBM8A gene. This variant is little bit tricky because by itself even in homozygous status, it is not penetrant. It is only penetrant in chances of deletion which is causing the tau syndrome. CMA later found the deletion and solved the case. The last case is a case that is relatively complex which earlier break into the two …… and they may contribute to the relatively low ranking of this case.
The next category of cases is those which has atypical phenotype information, however in all cases we consider them as a diagnosis due to the relatively strong genetic evidence.
For example, the first one is a 7-week female who has compound heterozygous likely pathogenic variant VUS in the COASY gene which is causing the autosomal recessive neurodegeneration disease. As you can see, the phenotype match score given by the Emedgene platform is ‘0’ in this situation, however, we cannot rule out that the patient is too young to develop the phenotype. Similar situation in the second case who has a pathogenic variant in the FOXP1 gene. The patient is a newborn who has not developed a disease yet. Again, the phenotypic matching score in this case is 0. Some cases are similar, and the last case is a likely pathogenic variant in gene of unknown significance. There are only two publications indicating for autism spectrum disorder at that time and therefore, we think that may contribute to the ranking.
We also looked at the possibilities of the platform to do the dual diagnosis. In our cohort, there are 3 cases that has definitive dual diagnosis and in each of the cases, the AI platform is able to rank both diagnosis variants at high level.
As the summary of the accuracy study, we found that the automatic analysis is able to rank the diagnosis variants at the top 10 for 98% of the trio cases, 91% of the proband cases, and overall, 96% in our entire cohort.
There are a couple of factors that can affect autoanalysis that include incomplete genetic data, such as singleton analysis, single heterozygous variant in the autosomal recessive genes and complex alleles. Another situation that can affect autoanalysis is the weak phenotype evidence that include either atypical phenotype or non-OMIM gene of unknown significance. We found that autoanalysis can pick up dual diagnosis.
Validation for Clinical Use
After seeing the performance of Emedgene autoanalysis, we started doing a more comprehensive work of trying to implement this platform in our clinical lab. We did extensive validation using another 60 case and based on them, we developed our review SOP. By working with the Emedgene’s team, we did extensive workflow customization and also the interface customization in order to fit our lab’s needs. Finally, we did side by side run on 50 cases with our old analysis and the new Emedgene’s analysis to compare the results …. consistence.
The sequencing and bioinformatics calling were done at Baylor genetics and AI analysis was done at Emedgene which gives us first tier of 10 variants called “Most likely” and the second tier of 108 variants called “Candidates”. In addition to that, we performed independent Baylor Genetics Preset filtering to complement those autoanalysis and all those variants still under manual review of the scientists as well as the lab directors. The reporting was done at Baylor genetics.
Up to October this year, we’ve analysed approximately 334 cases and we were able to achieve 29% of diagnosis rate in both singleton cases and trio cases. And this diagnosis rate is consistent with the previously published WES cohorts from other US reference labs.
Large Scale Reanalysis
Lastly, we looked at the possibilities to use AI system to do the large-scale case reanalysis. As we all know that at the reference lab, we accumulate more and more cases throughout the year. It is very labor-intensive to manually assess those historic cases. In order to speed up the process, Emedgene developed a new model to assign a solved/unsolved probability score to each of the cases. That score will provide the probability of whether the case will be solved by the WES analysis. We applied the model to the 334 production cases and in the figure, you can see that on the X axis is the number of cases, ranking by the probability score from the lowest to the highest. So, the lowest means the case is more likely to be unsolved while highest means the case is more likely to be solved. If we look at the bottom quartile of the cases, from 0 to 0.25 of the score, there are only 5% of the cases that are solved and 95 % of cases unsolved based on the manual review. And if we look at the top quartile of the cases, 77% of the cases are solved based on the manual review.
We’re currently working with Emedgene to see how we can best utilize the model to do reanalysis and how we can implement that in our clinical workflow.
To summarize our clinical production experience, Emedgene autoanalysis has been validated, customized, and integrated in our clinical exome review at Baylor genetics. We still currently do two-step manual review in order to ensure our exome quality. The diagnosis rate in 334 cases is about 29% that is in range of the previous manual review cohorts. And currently, we are looking to explore options to do more automated analysis for case reanalysis.
[hubspot type=form portal=6458790 id=9bec1dbc-9486-4aec-bcda-88c46596ccad]