scalation/scalation/scalation.modeling/scalation.modeling.forecasting/scalation.modeling.forecasting.neuralforecasting/Attention

Attention

scalation.modeling.forecasting.neuralforecasting.Attention

See theAttention companion object

trait Attention(n_var: Int, n_mod: Int, heads: Int, n_v: Int)

The Attention trait provides methods for computing context vectors, single-head attention matrices and multi-head attention matrices.

Value parameters

heads: the number of attention heads
n_mod: the size of the output (dimensionality of the model, d_model)
n_v: the size of the value vectors
n_var: the size of the input vector x_t (number of variables)

Attributes

Companion: object
Graph
Supertypes: class Object

trait Matchable

class Any
Known subtypes: class TrEncoderLayer

Members list

Value members

Concrete methods

Compute a Self-Attention Weight Matrix from the given query (Q), key (K) and value (V).

Value parameters

k: the key matrix K
q: the query matrix Q (q_t over all time)
v: the value matrix V

Attributes

Compute a Multi-Head, Self-Attention Weight Matrix by taking attention for each head and concatenating them; finally multiplying by the overall weight matrix w_o. The operator ++^ concatenates matrices column-wise.

Value parameters

k: the key matrix K
q: the query matrix Q (q_t over all time)
v: the value matrix V
w_o: the overall weight matrix to be applied to concatenated attention
w_q: the weight tensor for query Q (w_q(i) matrix for i-th head)
w_v: the weight tensor for value V (w_v(i) matrix for i-th head)

Attributes

Compute a Context Vector from the given query at time t (q_t), key (K) and value (V).

Value parameters

k: the key matrix K
q_t: the query vector at time t (based on input vector x_t)
v: the value matrix V

Attributes

Compute the Query, Key, Value matrices from the given input and weight matrices.

Value parameters

w_q: the weight matrix for query Q
w_v: the weight matrix for value V
x: the input matrix

Attributes

Concrete fields

In this article

Generated with