Changelog¶

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog.

This project adheres to Semantic Versioning, with the exception that v0.X updates include backwards-incompatible API changes. From v1.0.0 and on, the project will adherence strictly to Semantic Versioning.

Unpublished¶

[1.2.4] - 2025-03-15¶

Fixed¶

Multilayer transformers when num_layers>=3.

Added¶

Tests up to python 3.12.

[1.2.3] - 2023-06-16¶

Fixed¶

Ensure state_index remains on the same device after clean_state.

[1.2.2] - 2023-05-24¶

Fixed¶

Option to use strings to specify transformer activation.

[1.2.1] - 2023-03-24¶

### Added

Onnx as dev requirement.

Changed¶

Allow torch>=2.0.

[1.2.0] - 2023-03-16¶

### Added

Skip module.
“leading” mode in Residual.

[1.1.3] - 2023-03-15¶

### Added

Description of state handling to README.

Fixed¶

Documentation formatting for co.Identity() examples.
Horovod check for newer pytorch lightning versions.

[1.1.2] - 2023-01-13¶

### Added

query_index argument to SingleOutputTransformerEncoderLayer.

Fixed¶

Residual centred residual and Delay auto_delay forward_step.

[1.1.1] - 2023-01-10¶

### Added

Support for GroupNorm and InstanceNorm

[1.1.0] - 2022-12-19¶

### Added

append function to co.Sequential.
Production-ready docstrings for public functions.
reduce_max to Reduce.

### Changed

Rename Unity to Identity to follow torch.nn.
Major overhaul of README, improving descriptions and adding benchmark.
Major overhaul of docs, improving descriptions and adding benchmark.
MHA warnings to only log once.

Removed¶

Unused parameters batch_first and bidirectional for RNN, GRU, and LSTM.

[1.0.4] - 2022-12-07¶

Fixed¶

co.Conditional onnx support for single-option config.

[1.0.3] - 2022-12-07¶

Fixed¶

co.Conditional onnx support.

[1.0.2] - 2022-12-06¶

Fixed¶

co.Conv onnx export for kernel_size=1.

[1.0.1] - 2022-12-02¶

Added¶

Ability to access onnx from root, i.e. co.onnx.

[1.0.0] - 2022-12-02¶

Added¶

ONNX compatibility to all library modules!
co.onnx.export function mirroring torch.onnx.export.
purely functional _forward_step function to all modules.
_state_shape and _dynamic_state_inds properties to modules.
Add about info to package root.

Changed¶

Change call_mode internally from enum to tensor.
Change internal state_indexes to tensors.
Change stride to tuple.
Change padding to tuple.

### Fixed

Fix assertion bug in co.Lambda.

### Removed

TensorPlaceholder in favour of None.

[0.17.1] - 2022-06-02¶

Added¶

Missing part on Continual Transformers in README.

Removed¶

Conv cpp impl.

[0.17.0] - 2022-05-12¶

Added¶

Citations for Continual Inference lib paper.
Docs.
Automatic conversion for RNN modules.
Continual Transformer modules, including:
- RecyclingPositionalEncoding
- RetroactiveMultiheadAttention
- SingleOutputMultiheadAttention
- SingleOutputTransformerEncoderLayer
- RetroactiveTransformerEncoderLayer
- TransformerEncoderLayerFactory
- TransformerEncoder

[0.16.0] - 2022-04-04¶

### Added

“lagging” option for shrink in co.Delay and co.Residual.
co.RNN.
co.LSTM.
co.GRU.

Changed¶

phantom_padding renamed to residual_shrink.

[0.15.6] - 2022-03-18¶

Fixed¶

Missing cpp file in package.

[0.15.5] - 2022-03-05¶

Added¶

CoConv step impl in C++.

[0.15.4] - 2022-01-28¶

Fixed¶

FLOPs module registration compatibility with ptflops >=v0.6.8.

[0.15.3] - 2021-12-13¶

Added¶

Call-mode specific functions in co.Lambda

[0.15.2] - 2021-12-11¶

Added¶

Support for functor in co.Lambda

Removed¶

nn.LayerNorm from automatically convertible modules

[0.15.1] - 2021-12-10¶

Added¶

nn.LayerNorm to automatically convertible modules

[0.15.0] - 2021-10-29¶

Added¶

ParallelDispatch module.
Conditional predicate print in __repr__.

Fixed¶

Sequential padding computation.
Lambda __repr__ function prints.

Removed¶

CI testing for python v3.6.

[0.14.0] - 2021-09-20¶

Added¶

Added phantom_padding to Residual.
Added receptive_field property.
Added Reshape module.

Changed¶

Rename forward_shrink argument to auto_shrink in Delay.
Torch requirement to v1.9.
Replace Lambda unsqueeze_step with takes_time and new default to False.

Fixed¶

padding property in sequence.
delay property in sequence.
strict mode in load_state_dict.

Removed¶

Assertion error in BroadcastReduce for modules with different delays.

[0.13.0] - 2021-09-14¶

Added¶

Add forward_shrink option to Delay and Residual.

[0.12.0] - 2021-09-14¶

Added¶

Add Constant.
Add Zero.
Add One.

[0.11.4] - 2021-09-08¶

Fixed¶

co.ConvXd cuda compatibility.

[0.11.3] - 2021-09-08¶

Added¶

Add flatten_state_dict state variable.

Removed¶

Debug message for Convolutions with padding.

[0.11.2] - 2021-09-08¶

Fixed¶

call_mode for Linear.

[0.11.1] - 2021-09-06¶

Added¶

Add call_mode.
Add warm_up.

Changed¶

Container implementations to use __call__ with alternating call_modes. This change was necessary to properly trigger the torch hooks needed in ptflops.

Fixed¶

ptflops compatibility.

[0.11.0] - 2021-08-31¶

Added¶

co.Linear module.
Improved repr in co.Lambda.
Option to skip unsqueeze in co.Lambda.forward_step.

[0.10.0] - 2021-08-27¶

Changed¶

Renamed co.Parallel to co.BroadcastReduce.

Added¶

co.Broadcast module.
new co.Parallel module.
co.Reduce module.
Automatic inference of co.Broadcast.num_streams in co.Sequential.

[0.9.0] - 2021-08-26¶

Added¶

co.Lambda module.
co.Add module.
co.Multiply module.
co.Unity module.
co.Conditional module.

[0.8.1] - 2021-08-26¶

Fixed¶

Bug in forward_stepping.
Bug in clean_state.

[0.8.0] - 2021-08-24¶

Fixed¶

Bugs in forward_step(s) with update_state=False.

Changed¶

forward_steps interface to always include pad_end argument.
Name of “interface.py” to “module.py”.
Implementations of forward_step(s) to be consolidated in CoModule.

Removed¶

Padded interface.

[0.7.0] - 2021-08-24¶

Added¶

Independent state_dict and load_state_dict functions.
Added nonempty check for aggregation functions in Parallel.
update_state argument to all forward_step(s) methods.
Additional tests for edge-cases

Changed¶

Changed default pad_end value to False.

Fixed¶

Continual interface and conversion to support both class and module.
Replicate padding in co._ConvNd

[0.6.1] - 2021-08-23¶

Changed¶

co.Residual modules to be unnamed. This allows the module state dicts to be flattened.

[0.6.0] - 2021-08-23¶

Added¶

Flattened state dict export and loading via a flatten argument. This feature improves interoperability complex modules, that were not originally constructed with the co.Sequential and co.Parallel building blocks.
Context manager for triggering flattened state_dict export and loading.

[0.5.0] - 2021-08-20¶

Added¶

Support for zero-delay in co.Delay
Support for broadcasting in co.Parallel
Mul (hadamard product) aggregation in co.Parallel
Example of Squeeze and Excitation block

Changed¶

co._PoolNd attribute naming: “temporal_*” removed as prefix for kernel_size, stride, dilation, and padding.

[0.4.0] - 2021-08-19¶

Added¶

co.Delay handling for padding.
Handling of initialization and strides in containers

Changed¶

co.Conv build_from behavior to not change dilation and stride. Argument overload supported instead.
pad_start and pad_end args to convolution and pooling modules forward_steps.
Behavior of modules while they initialize. Now, a TensorPlaceholder is passed for initializing steps.

Removed¶

Automatic unsqueeze in pooling.

[0.3.1] - 2021-08-18¶

Added¶

Support for dropout.

[0.3.0] - 2021-08-18¶

Added¶

Support for dilation and stride in pooling.

Changed¶

Pooling API to match torch.nn better.
_ConvCoNd.forward_steps doesn’t invoke clean_state anymore.

[0.2.2] - 2021-08-17¶

Added¶

Automatic conversion of batch normalization and activation functions.

Fixed¶

Separate dilation and stride in pool.

Changed¶

Conv forward to use temporal padding like (like nn.Conv).

Removed¶

co.BatchNorm2d

[0.2.1] - 2021-08-17¶

Changed¶

Renamed unsqueezed to forward_stepping.

Removed¶

Unused utility Zeros

[0.2.0] - 2021-08-16¶

Changed¶

Naming to match torch.nn. This lets the continual modules be used as drop-in replacements for torch.nn modules.
Renamed forward_regular_unrolled to forward, forward_regular to forward_steps, and forward for forward_step.
Renamed from_regular to build_from.
Renamed continual to unsqueezed.

Added¶

Sequential wrapper for sequential application of modules
Parallel wrapper for parallel application and aggregation of inputs
Residual wrapper for adding a unity residual to a module
continual conversion function
register function for 3rd party modules to register their conversion
Additional tests

[0.1.2] - 2021-08-1¶

Added¶

Pooling modules: MaxPool1d, AvgPool3d, MaxPool3d, AdaptiveAvgPool3d, AdaptiveMaxPool3d.

[0.1.1] - 2021-08-10¶

Added¶

Updated README.

[0.1.0] - 2021-08-10¶

Added¶

Initial publicly available implementation of the library.