Since it's a builder, we should really consume self and return Self instead of using &mut self. This makes it easier to convert into the final matrix at the end of building.