shadow.utils module

class shadow.utils.ConstantCW(last_weight=1)[source]

Bases: shadow.utils._CWScheduler

Constant valued consistency weight scheduler.

Scheduler function to control a weight, often used to weigh a consistency cost relative to a supervised learning cost (e.g. Cross Entropy). This is intended to be stepped after each epoch during training to increase or decrease the weight accordingly. This provides a constant weighting function that does not change.

Parameters

last_weight (float, optional) – Final consistency weight. Defaults to 1.

Example

>>> alpha = ConstantCW(last_weight)
>>> for epoch in epochs:
>>>     train(...)
>>>     loss = criterion + alpha() * consistency
>>>     validate(...)
>>>     alpha.step()
class shadow.utils.IgnoreUnlabeledWrapper(criterion, ignore_index=numpy.NINF)[source]

Bases: torch.nn.Module

Wraps a loss function to filter out mising values for a Semi-Supervised learning task.

Parameters

  • criterion (callable) – Used to compute the supervised loss.

  • ignore_index (bool, int, float, complex, optional) – Specifies a target value that is ignored and does not contribute to the input gradient. Defaults to negative infinity.

Example

>>> ssml_loss = IgnoreUnlabeledWrapper(criterion=torch.nn.MSELoss())
>>> y_true = torch.rand(3, 1)
>>> y_hat = y_true.clone()
>>> y_hat
tensor([[0.1543],
        [0.1572],
        [0.0404]])
>>> ssml_loss(y_hat, y_true)
tensor(0.)
>>> y_true[1] = np.NINF
>>> y_true
tensor([[0.1543],
        [  -inf],
        [0.0404]])
>>> ssml_loss(y_hat, y_true)
tensor(0.)

Example

>>> ssml_loss = IgnoreUnlabeledWrapper(criterion=torch.nn.BCELoss())
>>> y_hat = torch.Tensor([[0], [1], [1], [0]])
>>> y_true = torch.Tensor([[ignore_index], [1], [ignore_index], [1]])
>>> ssml_loss(y_hat, y_true)
tensor(50.)
forward(y_hat, y_true)[source]
class shadow.utils.QuadraticCW(last_epoch, last_weight=1, first_weight=0, epochs_before=0)[source]

Bases: shadow.utils._CWScheduler

Quadratic consistency weight scheduler.

Scheduler function to control a weight, often used to weigh a consistency cost relative to a supervised learning cost (e.g. Cross Entropy). This is intended to be stepped after each epoch during training to increase or decrease the weight accordingly. This provides a quadratic weighting function.

Parameters

  • last_epoch (int) – Number of epochs until scheduler reaches last_weight.

  • last_weight (float, optional) – Final consistency weight. Defaults to 1.

  • first_weight (float, optional) – Consistency weight at beginning of ramp. Defaults to 0.

  • epochs_before (int, optional) – Number of epochs to hold weight at first_weight before beginning ramp. Defaults to 0.

Example

>>> alpha = QuadraticCW(last_epoch, last_weight, first_weight, epochs_before)
>>> for epoch in epochs:
>>>     train(...)
>>>     loss = criterion + alpha() * consistency
>>>     validate(...)
>>>     alpha.step()
class shadow.utils.RampCW(last_epoch, last_weight=1, first_weight=0, epochs_before=0)[source]

Bases: shadow.utils._CWScheduler

Linear ramp consistency weight scheduler.

Scheduler function to control a weight, often used to weigh a consistency cost relative to a supervised learning cost (e.g. Cross Entropy). This is intended to be stepped after each epoch during training to increase or decrease the weight accordingly. This provides a linear ramp weighting function.

Parameters

  • last_epoch (int) – Number of epochs until scheduler reaches last_weight.

  • last_weight (float, optional) – Final consistency weight. Defaults to 1.

  • first_weight (float, optional) – Consistency weight at beginning of ramp. Defaults to 0.

  • epochs_before (int, optional) – Number of epochs to hold weight at first_weight before beginning ramp. Defaults to 0.

Example

>>> alpha = RampCW(last_epoch, last_weight, first_weight, epochs_before)
>>> for epoch in epochs:
>>>     train(...)
>>>     loss = criterion + alpha() * consistency
>>>     validate(...)
>>>     alpha.step()
class shadow.utils.SigmoidCW(last_epoch, last_weight=1, first_weight=0, epochs_before=0)[source]

Bases: shadow.utils._CWScheduler

Sigmoidal consistency weight scheduler.

Scheduler function to control a weight, often used to weigh a consistency cost relative to a supervised learning cost (e.g. Cross Entropy). This is intended to be stepped after each epoch during training to increase or decrease the weight accordingly. This provides a sigmoidal weighting function.

Parameters

  • last_epoch (int) – Number of epochs until scheduler reaches last_weight.

  • last_weight (float, optional) – Final consistency weight. Defaults to 1.

  • first_weight (float, optional) – Consistency weight at beginning of ramp. Defaults to 0.

  • epochs_before (int, optional) – Number of epochs to hold weight at first_weight before beginning ramp. Defaults to 0.

Example

>>> alpha = SigmoidCW(last_epoch, last_weight, first_weight, epochs_before)
>>> for epoch in epochs:
>>>     train(...)
>>>     loss = criterion + alpha() * consistency
>>>     validate(...)
>>>     alpha.step()
class shadow.utils.SkewedSigmoidCW(last_epoch, last_weight=1, first_weight=0, epochs_before=0, beta=1, zeta=1)[source]

Bases: shadow.utils._CWScheduler

Skewed sigmoidal consistency weight scheduler with variable ramp up speed.

Scheduler function to control a weight, often used to weigh a consistency cost relative to a supervised learning cost (e.g. Cross Entropy). This is intended to be stepped after each epoch during training to increase or decrease the weight accordingly. This provides a skewed sigmoid weighting function with variable ramp up timing speed.

Parameters

  • last_epoch (int) – Number of epochs until scheduler reaches last_weight.

  • last_weight (float, optional) – Final consistency weight. Defaults to 1.

  • first_weight (float, optional) – Consistency weight at beginning of ramp. Defaults to 0.

  • epochs_before (int, optional) – Number of epochs to hold weight at first_weight before beginning ramp. Defaults to 0.

  • beta (float, optional) – Controls how sharp the rise from first_weight to last_weight is. beta = 1 corresponds to a standard sigmoid. Increasing beta increases sharpness. Negative values can actually invert the sigmoid for a decreasing ramp. Defaults to 1.

  • zeta (float, optional) – Skews when the rise from first_weight to last_weight occurs. zeta = 1 corresponds to a rise centered about the middle epoch. zeta = 0 corresponds to a flat weight at last_weight. zeta < 1 shifts rise to earlier epochs. zeta > 1 shifts to later epochs. Defaults to 1.

Example

>>> alpha = SkewedSigmoidCW(last_epoch, last_weight, first_weight, epochs_before, beta, zeta)
>>> for epoch in epochs:
>>>     train(...)
>>>     loss = criterion + alpha() * consistency
>>>     validate(...)
>>>     alpha.step()
class shadow.utils.StepCW(last_epoch, last_weight=1, first_weight=0)[source]

Bases: shadow.utils._CWScheduler

Step function consistency weight scheduler.

Scheduler function to control a weight, often used to weigh a consistency cost relative to a supervised learning cost (e.g. Cross Entropy). This is intended to be stepped after each epoch during training to increase or decrease the weight accordingly. This provides a step weighting function.

Parameters

  • last_epoch (int) – Number of epochs until scheduler reaches last_weight.

  • last_weight (float, optional) – Final consistency weight. Defaults to 1.

  • first_weight (float, optional) – Consistency weight at beginning of ramp. Defaults to 0.

Example

>>> alpha = StepCW(last_epoch, last_weight, first_weight)
>>> for epoch in epochs:
>>>     train(...)
>>>     loss = criterion + alpha() * consistency
>>>     validate(...)
>>>     alpha.step()
shadow.utils.flatten_to_two_dim(input_tensor)[source]

Flatten tensor along the first axis ([2, 3, 4] -> [2, 12])

Parameters

input_tensor (torch.Tensor) – input tensor

Returns

input_tensor flattened along first axis

Return type

torch.Tensor

shadow.utils.init_model_weights(model, value)[source]

Set all weights in model to a given value.

Parameters

  • model (torch.nn.Module) – The model to update. Weight update is performed in place.

  • value (float) – The weight value.

shadow.utils.set_seed(seed, cudnn_deterministic=False)[source]

Sets the seeds for max reproducibility.

Sets seeds for random, numpy, and torch to seed, and can also enable deterministic mode for the CuDNN backend. This does not guarantee full reproducibility as some underlying options (e.g. atomicAdd) still have sources of non-determinism that cannot be disabled.

Parameters

  • seed (int) – Seed used for random, numpy, and torch.

  • cudnn_deterministic (bool, optional) – Sets the CuDNN backend into deterministic mode. This can negatively impact performance. Defaults to False.

Note

PyTorch provides only minimal guarantees on reproducibility. See <https://pytorch.org/docs/stable/notes/randomness.html> for more information.