RMSNormalization
class RMSNormalization<T : DType, V>(normalizedShape: IntArray, eps: Double = 1.0E-5, val name: String = "RMSNormalization", initWeight: Tensor<T, V>? = null) : Module<T, V> , ModuleParameters<T, V> (source)
RMS (Root Mean Square) Normalization layer. Unlike LayerNormalization, RMSNorm has no bias and normalizes using only the root mean square of the input, making it simpler and faster.
Used extensively in LLaMA-family models.
Parameters
normalizedShape
The shape of the normalization (typically the last dimension)
eps
Small value added to the denominator for numerical stability
name
Name of the module
initWeight
Initial weight (scale) parameter