Block Update Order

The default order the blocks are updated is cyclically through each factor of the decomposition D::AbstractDecomposition, in the order of factors(D). For AbstractTucker decompositions like Tucker, Tucker-1, and CP, this means starting with the core, followed by the matrix factor for the first dimension, second dimension, and so on.

As an example, this would be the default order of updates for nonnegative CP decomposition on an order 3 tensor.

BlockedUpdate(
    MomentumUpdate(1, lipschitz)
    GradientStep(1, gradient, LipschitzStep)
    Projection(1, Entrywise(ReLU, isnonnegative))
    MomentumUpdate(2, lipschitz)
    GradientStep(2, gradient, LipschitzStep)
    Projection(2, Entrywise(ReLU, isnonnegative))
    MomentumUpdate(3, lipschitz)
    GradientStep(3, gradient, LipschitzStep)
    Projection(3, Entrywise(ReLU, isnonnegative))
)

Randomizing Block Updates Order

The order of updates can be randomized with the random_order keyword.

X, stats, kwargs = factorize(Y; random_order=true)

By default, this will keep momentum steps, gradient steps, and constraint steps for each factor together as a block, in this order.

A possible order of updates could be the following. Note that the updates for each factor are grouped together, but each factor is updated in a random order.

BlockedUpdate(
    BlockedUpdate(
        MomentumUpdate(2, lipschitz)
        GradientStep(2, gradient, LipschitzStep)
        Projection(2, Entrywise(ReLU, isnonnegative))
    )
    BlockedUpdate(
        MomentumUpdate(1, lipschitz)
        GradientStep(1, gradient, LipschitzStep)
        Projection(1, Entrywise(ReLU, isnonnegative))
    )
    BlockedUpdate(
        MomentumUpdate(3, lipschitz)
        GradientStep(3, gradient, LipschitzStep)
        Projection(3, Entrywise(ReLU, isnonnegative))
    )
)

For more randomization, use the recursive_random_order keyword which will also randomize the order in which the momentum steps, gradient steps, and constraint steps are performed.

X, stats, kwargs = factorize(Y; recursive_random_order=true)

A possible order of updates could now be the following. The updates for each factor are still grouped together, but the updates within each block appear in a random order.

BlockedUpdate(
    BlockedUpdate(
        Projection(2, Entrywise(ReLU, isnonnegative))
        MomentumUpdate(2, lipschitz)
        GradientStep(2, gradient, LipschitzStep)
    )
    BlockedUpdate(
        MomentumUpdate(1, lipschitz)
        Projection(1, Entrywise(ReLU, isnonnegative))
        GradientStep(1, gradient, LipschitzStep)
    )
    BlockedUpdate(
        GradientStep(3, gradient, LipschitzStep)
        Projection(3, Entrywise(ReLU, isnonnegative))
        MomentumUpdate(3, lipschitz)
    )
)

The opposite of this would be to keep the outer order of blocks as given, but randomize the order which the updates for each factor gets applied, use the following code.

X, stats, kwargs = factorize(Y; recursive_random_order=true, random_order=false, group_by_factor=true)

A possible order of updates could now be the following. Note the order of factors is preserved (1, 2, 3) but the inner BlockedUpdates have a random order.

BlockedUpdate(
    BlockedUpdate(
        Projection(1, Entrywise(ReLU, isnonnegative))
        MomentumUpdate(1, lipschitz)
        GradientStep(1, gradient, LipschitzStep)
    )
    BlockedUpdate(
        MomentumUpdate(2, lipschitz)
        Projection(2, Entrywise(ReLU, isnonnegative))
        GradientStep(2, gradient, LipschitzStep)
    )
    BlockedUpdate(
        GradientStep(3, gradient, LipschitzStep)
        MomentumUpdate(3, lipschitz)
        Projection(3, Entrywise(ReLU, isnonnegative))
    )
)

Note all the previously mentioned options still keeps the various updates for each factor together. For full randomization, use the following code.

X, stats, kwargs = factorize(Y;
    recursive_random_order=true, group_by_factor=false)

A possible order of updates could now be the following. Note that every update can appear anywhere in the order.

BlockedUpdate(
    Projection(3, Entrywise(ReLU, isnonnegative))
    MomentumUpdate(2, lipschitz)
    GradientStep(2, gradient, LipschitzStep)
    MomentumUpdate(1, lipschitz)
    GradientStep(1, gradient, LipschitzStep)
    Projection(2, Entrywise(ReLU, isnonnegative))
    MomentumUpdate(3, lipschitz)
    MomentumUpdate(2, lipschitz)
    Projection(1, Entrywise(ReLU, isnonnegative))
    GradientStep(3, gradient, LipschitzStep)
)

The complete behaviour is summarized in the table below.

`group_by_factor`	`random_order`	`recursive_random_order`	Description
`false`	`false`	`false`	In the order given
`false`	`false`	`true`	In order given, but randomize how existing blocks are ordered (recursively)
`false`	`true`	`false`	Randomize updates, but keep existing blocks in order
`false`	`true`	`true`	Fully random
`true`	`false`	`false`	In the order given
`true`	`false`	`true`	In order of factors, but updates for each factor a random order
`true`	`true`	`false`	Random order of factors, preserve order of updates within each factor
`true`	`true`	`true`	Almost fully random, but updates for each factor are done together