paper: https://arxiv.org/pdf/1706.03762
Abstract
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer,based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
Recurrent model concept
Input: A - B - C Process: h1 = H(h0, A) h2 = H(h1, B) h3 = H(h2, C) (h = hidden state)
1.Introduction
Problem for recurrent model is that it inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples. In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.
recurrent model: The output of the current position depends on the output for the previous positions.
Problem for old models
the number of operations required to relate signals from two arbitrary input or output positions grows in the distance between positions.
2. Model Architecture
Here, the encoder maps an input sequence of symbol representations (x1, …, xn) to a sequence of continuous representations z = (z1, …, zn). Given z, the decoder then generates an output sequence (y1, …, ym) of symbols one element at a time. At each step the model is auto-regressive (only the decoder), consuming the previously generated symbols as additional input when generating the next. (auto regressive ≠ recurrence)
Left part is encoder, right part is decoder.
2.1 Encoder and Decoder Stack
Encoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, positionwise fully connected feed-forward network. We employ a residual connection around each of the two sub-layers, followed by layer normalization. That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension dmodel = 512.
Decoder: The decoder is also composed of a stack of N = 6 identical layers. In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack. Similar to the encoder, we employ residual connections around each of the sub-layers, followed by layer normalization. We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i.
2.2 Attention
An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.
Concept Understanding (not accurate, but help understanding):
Query: search: NTU Key: NTU, National Taiwan University, Nanyang Technological University… Value: the content in the websites
output = W1V1 + W2V2 + W3V3 Wn = compute_attention_score_function
2.2.1 Scaled Dot-Product Attention
Concept Understanding (not accurate, but help understanding):
When dot product value is big, softmax will make the gradient really small, therefore the sqrt(dk) at the denomiator as a counteract.
2.2.2 Multi-Head Attention
Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.
2.3 Positional Encoding
Because the attention mechanism contains no recurrence and no convolution, in order to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. To this end, we add “positional encodings” to the input embeddings at the bottoms of the encoder and decoder stacks.
pos in sin and cos can create an unique set of numbers for latent variables. i is for each dimension of the position (Although the sin and cos might get the same value for different parameters, but the (sin, cos) set guarantees the uniqueness.)
3. Conclusion
In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention. For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers.
In other word, it is good.