GitHub

MLP components

Layers

Dense layer can be used to construct MLP. Its parameters are handled explicitly, meaning after instantiation d = Dense(1=>2), parameters are obtained with θ = NerfUtils.init(d, Backend). Where you pass one of the supported Backends to initialize the model directly on the device.

You can then pass inputs to the layer y = d(x, θ).

Chain can be used to stack multiple dense layers. Its parameters are handled in the same manner:

c = Chain(Dense(1=>2), Dense(2=>1))
θ = NerfUtils.init(c, Backend)

x = ...
y = c(x, θ)
NerfUtils.Dense — Type
function Dense(
    mapping::Pair{Int64, Int64}, activation::F = identity,
) where F

Regular dense layer with activation.

source
NerfUtils.init — Method
init(d::Dense, kab)

Create parameters for the Dense layer on the given backend kab.

source
NerfUtils.init — Method
init(c::Chain, kab)

Create parameters for the Chain on the given backend kab.

source

Optimizer

There's only one optimizer available - Adam. It handles parameters that are either tuples (or named tuples) or plain arrays.

c = Chain(Dense(1 => 2), Dense(2 => 1))
θ = NerfUtils.init(c, Backend)
opt = Adam(Backend, θ; lr=1f-3)

After computing gradients ∇, applying the update rule can be done with NerfUtils.step! function.

NerfUtils.step!(opt, θ, ∇; dispose=true) # Free immediately gradient memory afterwards.
NerfUtils.step!(opt, θ, ∇; dispose=false) # Do not free.
NerfUtils.Adam — Type
Adam(kab, θ; kwargs...)

Adam optimizer.

θ must be either plain array or a tuple (or named tuple).

source
NerfUtils.step! — Method
step!(opt::Adam, θ, ∇; dispose::Bool)

Apply update rule to parameters θ with gradients ∇.

Arguments:

  • dispose::Bool: Free memory taken by gradients ∇ after update.
source