ApplyRotaryPositionEncoding (aka.

Attention builds an attention layer, optionally using cache to store a limited amount of context.

AttentionTypeString retrieves an enum value from the enum constants string name.

AttentionTypeStrings returns a slice of all String values of the enum.

AttentionTypeValues returns all values of the enum.

Block implements one transformer block for the Gemma model.

DecodeTokens use the same table as EmbedTokens to convert embedding back to the tokens -- or to token logits.

EmbedTokens using weights in Config.

GatedFeedForward layer for Gemma: - hiddenDim: one intermediary layer.

GemmaTypeString retrieves an enum value from the enum constants string name.

GemmaTypeStrings returns a slice of all String values of the enum.

GemmaTypeValues returns all values of the enum.

GemmaWithCache creates a forward path on a Gemma model for one decoding step, using the weights in Config to initialize the variables.

HuggingFaceGatedFeedForward layer for Gemma, the HuggingFace version, with transposed weights: - hiddenDim: one intermediary layer.

KernelEinsum multiplies the input by a kernel of the given shape, using the given graph.EinSum equation.

Must panics if the error is not nil.

Must1 panics in case of error, otherwise returns the one return value.

NewConfigFromContext creates a transformers config model, based on the structure of the variables in the given context -- the scope has to be set directly to the model variables.

QueryPreAttentionNormalisationTypeString retrieves an enum value from the enum constants string name.

QueryPreAttentionNormalisationTypeStrings returns a slice of all String values of the enum.

QueryPreAttentionNormalisationTypeValues returns all values of the enum.

RMSNorm normalizes by its root-mean-square x = x / √(mean(sqrt(x), axis=-1) + epsilon) and applies a learned scale.

SoftCap using Tanh, so values won't go beyond +/- cap.

QueryNormTypeByEmbedDimDivNumHeads indicates whether to scale the query by `embed_dim // num_heads`.

QueryNormTypeByOneOverSqrtEmbedDimDivNumHeads indicates whether to scale the query by `1/sqrt(embed_dim // num_heads)`.

QueryNormTypeByOneOverSqrtHeadDim indicates whether to scale the query by 1/sqrt(head_dim).

RoPEDefaultMaxWaveLength is a default value to use for rotary positional encoding.

Cache is a state cache of a (batch of) sequence being encoded/decoded.

Config Gemma transformer model.

QueryPreAttentionNormalisationType defines how to normalize query before attention.