triangular matrix noun Mathematics. a square matrix in which either all the entries above the principal diagonal, or all the entries below the principal diagonal, are zero.