Google launches ALBERT, a Lite version of Google BERT, a tool for self-supervised learning of language representations.

In a post on Google 's Artificial Intelligence , the company's researchers, Radu Soricut and Zhenshong Lan, announce the release of a lightweight, open-source version of BERT, called ALBERT.

The BERT technology According to the researchers, ALBERT Stanford Question Answer Dataset (SQuAD v2.0) and the RACE for SAT- style reading comprehension .

ALBERT was released as an open-source implementation through TensorFlow and includes several pre-trained, ready-to-use language representation models for ALBERT.

What is Google BERT?

Official launch of ALBERT

In the post about the launch of ALBERT, we can read in detail:

Since the advent of BERT a year ago, natural language research has adopted a new paradigm, leveraging large amounts of existing text to pre-train a model's parameters using self-supervision, without the need for data .

Therefore, instead of needing to train a machine learning model for natural language processing (NLP) from scratch, one can start with a model already equipped with knowledge of a language. But, to improve this new approach to NLP, it is necessary to develop an understanding of what exactly is contributing to the language understanding performance – the height of the network (i.e., number of layers), its width (size of the hidden layer of representations), the learning criteria for self-supervision, or something entirely different?

It becomes easier to understand the role of Natural Language Processing in understanding, through Semantics , how a language works, contributing to Google's understanding of searches (with BERT) and now being usable in individual projects with ALBERT.

Google's new search algorithm: BERT

The Importance of Semantics for SEO

The reason I place so much importance on the release of tools like BERT or ALBERT, like Amazon's Silver , is that, in order to make the necessary leap in quality in understanding the content created online , we need to make computers understand (in the various languages ​​and programming languages ​​we use) what we want to say.

Semantic Optimization: A Case Study

This is the final step towards the definitive creation of SEO , an optimization of digital projects that can use all the power of the machines we have today, and those that are being created (have you heard of quantum computers? ) to make the tools that deliver results for the questions we ask deliver increasingly better answers.

Natural Language Processing in SEO

Identifying the dominant factor in NLP performance is complex – some configurations are more important than others, and, as the Google study reveals, a simple individual exploration of these configurations would not produce the correct answers, hence the importance of developing innovations such as ALBERT and Google BERT.

The Power of the Knowledge Graph

According to Google researchers, the key to optimizing performance, implemented in the ALBERT design, was to allocate the model's capacity more efficiently.

Through incorporations at the input level (words, sub-tokens, etc.), which needed to learn context-independent representations, such as a representation for the word "bank," for example.

Seeking meaning and context with ALBERT

On the other hand, hidden layer embeddings need to refine the models into context-dependent representations, for example, one representation for "bank" in the context of financial transactions and a different representation for "bank" in the context of sports or real estate.

If you are interested in the technical aspects behind ALBERT, read the original excerpt from the post that originated this post, in English:

The key to optimizing performance, captured in the design of ALBERT, is to allocate the model's capacity more efficiently. Input-level embeddings (words, sub-tokens, etc.) need to learn context- independent representations, a representation for the word “bank”, for example.

In contrast, hidden-layer embeddings need to refine that into context- dependent representations, eg, a representation for “bank” in the context of financial transactions, and a different representation for “bank” in the context of river-flow management.

his is achieved by factorization of the embedding parameterization — the embedding matrix is ​​split between input-level embeddings with a relatively-low dimension (eg, 128), while the hidden-layer embeddings use higher dimensionalities (768 as in the BERT case, or more). With this step alone, ALBERT achieves an 80% reduction in the parameters of the projection block, at the expense of only a minor drop in performance — 80.3 SQuAD2.0 score, down from 80.4; or 67.9 on RACE , down from 68.2 — with all other conditions the same as for BERT.

Albert's Success

The success of ALBERT demonstrated the importance of identifying the aspects of a model that give rise to powerful contextual representations.

Research that focused improvement efforts on aspects of the model's architecture has shown that it is possible to significantly improve the model's efficiency and performance across a wide variety of NLP tasks.

If you are interested in this field of study, Google is offering open-source ALBERT to the research community .

Is Google Bert outdated?

In this post from Search Engine Journal, I read that Google published an article with information about a research study that discusses a new algorithm called SMITH. According to the information, it outperforms BERT when it comes to understanding long queries and documents.

According to reports, Smith surpasses BERT in its ability to understand passages within lengthy documents. It's not yet confirmed whether Smith is being actively used by the search engine, but we are already seeing search that extract entire sections from texts and highlight them in SERPs.

What is the Smith algorithm?

According to Google's research, SMITH is a new model that seeks to understand entire documents. It's clear that the intention is to comprehend the document , elevating the study and application of Semantic SEO to a new level of importance.

In contrast, BERT was trained to understand words within the context of sentences. Semantics within the document is limited because the relationship between sentences is not considered.

While algorithms like BERT are trained on datasets to predict words hidden randomly from context within sentences, the SMITH algorithm is trained to predict what the next blocks of sentences will be.

Roger Montti – Search Engine Journal

According to the article, this training allows the algorithm to understand longer documents with higher quality than its predecessors.

But is Google using the Smith algorithm?

We all know that Google doesn't disclose everything about which algorithms it uses or doesn't use, but the company's researchers claim that the new algorithm surpasses BERT, and all SEO professionals should keep an eye on the tools and searches.

Reference:

ALBERT: A Lite BERT for Self-Supervised Learning of Language Representations
December 20, 2019
By Radu Soricut and Zhenzhong Lan – Google Search Researchers