Protein Folding With AlphaFold2: Chapter Three
In the last chapter, we have introduced AlphaFold2 [1] architecture with its preparation of data for the neural network module. Now, the next question comes: what is the Neural network architecture that it follows? What is the network supposed to do? The inspiration behind it.
The AlphaFold2 network comprises two main stages, (i)Trunk or Evoformer block inspired from the transformer architecture, (ii)Structure module introducing invariant point attention for 3D structure prediction of protein. In this blog, we are going to take a slightly deep dive into the Evoformer block structure.
What is Evoformer?
Evoformer is a type of transformer architecture, commonly used in natural language processing tasks. In AlphaFold2, the evoformer is used to integrate evolutionary information from homologous protein sequences into the prediction model. The Evoformer consists of 48 consecutive blocks and each of the building blocks uses the attention mechanism similar to the transformer architecture to get the information about the importance of residues and their interactions. The inputs, here, are the embeddings from the sequence processing module for both MSA features and pairwise features of the residues. This module strengthens the feature information after applying multiple attentions to get the proximity information of residues in 3D space.
What is inside the building block of Evoformer?
A building block of Evoformer consists of a series of update operations, which can be categorized into three main parts, namely, (i) updating MSA features, (ii) combining the MSA information with the pairwise features, (iii) updating pairwise features.
In the first part, it applies the self-attention in a row-wise fashion. It identifies pairs of interacting or correlated amino acids in the aligned proteins, and focuses the attention using information from the pair representation. Next, the column-wise self-attention identifies pairs of strongly correlated amino acid sequences and heuristically, the model learns which aligned sequences are informative for the target structure.
In the second part, the MSA representation updates the pair representation through an element-wise outer product that is summed over the MSA sequence dimension.
In the third part, it works with the pairwise representation. The pair representation is intended to represent a set of distances between residues of the protein, but distances in Euclidean space respect the triangle inequality, so we need to enforce this rule. To capture the proximity of the residues in 3D space, it applies the necessary constraints like triangle inequality on distances. Here, combinations of two types of updates ‘with’ and ‘without’ attention are used involving three nodes or residues to improve the accuracy. The triangle multiplicative update is cheaper than the symmetric one.
Finally, both these representations of MSA and pairwise features are fed into the next block of Evoformer. At the end of the 48th block, these will be used in the next structure prediction module of Alphafold2, which will be discussed in the next blog of the series.
At Molecule AI, we’re creating cutting-edge approaches that harness the power of deep learning in the realm of protein design. To learn more, feel free to contact us at info@moleculeai.com.
References:
[1] Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, T., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S., Ballard, A., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Stengger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., & Hassabis, D. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873), 583–589.
