Piecewise-Linear Functions: Part I

We are going to divide this document in two parts. The first part, this document, we are going to discuss piecewise linear functions (PLFs) and how to implement one in Python. The second part, Part II, we will discuss how to fit the coefficients, detect the break-points locations, and even optimize the number of the knots using python.

Table of Contents

Piecewise-Linear Functions (PLF)

What are they?

As the name suggests, piecewise-Linear function consists of couple of lines. Essentially you will have a line, i.e. $y=a + bx$ , fitting your data at different intervals. For example, consider the $y = \left | x \right |$ or absolute value of $x$ . If you plot this function it looks like:

Instead of writing this function using absolute value, i.e. $\left | \cdot \right |$ , you can show it as a piecewise-linear function:

$y=\begin{cases} -x & x < 0 \\ x & x \ge 0\\ \end{cases}$

In the above example, we are representing $y = \left | x \right |$ with two line segments: One for the interval where $x < 0$ and another for the interval where $x \ge 0$ .

You might ask: But why should we do that? Using $\left | \cdot \right |$ looks much easier. Well, in this case you are absolutely right. Using Absolute value is a much more concise way of showing this specific function. But you are not always this lucky. There are functions that you are left with no option but to only show them as multiple different segments.

Let’s have a look at this function, which looks like a letter “W”:

This one could be represented as a piecewise-linear function:

$y = \begin{cases} -x -0.5 & x < -0.5 \\ x + 0.5 & -0.5 \le x < 0 \\ -x + 0.5 & 0 \le x < 0.5 \\ x - 0.5 & 0.5 \le x \end{cases}$

This time you have 4 segments and 4 line equations. Although you can say that using absolute value, you can simplify the representation a bit as follows:

$y = \begin{cases} \left | -x -0.5 \right | & x < 0 \\ \left | x - 0.5 \right | & x \ge 0 \end{cases}$

But we already know that the absolute value function is representing two straight line.

Notice that although We are saying Piecewise-Linear Function, notice that “piecewise” and “linear” are mixed into one word. We are emphasizing that each piece is a line function or better to say an affine function. This should not be misunderstood with Linear Functions at all. These are two different concept. For one, a linear function must satisfy $f\left ( \lambda x \right ) = \lambda f \left ( x \right )$ . For example, none of the piecewise-linear function mentioned above are Linear Function. So, have this in mind that these two concepts are different

Piecewise-Linear Function and Linear Function are different concepts;

How do you implement them in Python?

Now as of the question of how do we implement them in Python? Well, it depends. For example, the first function could be implemented as:

import numpy as np

def my_piecewise_function(x):
  return np.abs(x)

But we already see that things are not always that easy. So, another way of implementing this function is:

import numpy as np

def my_piecewise_function(x):
  if x < 0:
    return -x
  else:
    return x

But there is even a better way to do it, and that’s using numpy.piecewise as follows:

import numpy as np

def my_piecewise_function(x):
    return np.piecewise(
        x,  # The value at which the piecewise function should be evaluated
        [   # A list of conditions
            x < 0,
            x >=0
        ],
        [   # One function corresponding to ech of the condition above.
            lambda x: -x,
            lambda x: x
        ]
    )
    

x = np.linspace(-1, 1, 51)
np.all(np.abs(x) == my_piecewise_function_1(x))  # will print True

This is the minimum that you need to implement a piecewise function:

$x$ values for which the piecewise function needs to be evaluated,
a list of conditions that defines how many segments you have, and
A list of functions for each segment.

If the first condition is true, the first function from the list is returned; If the second condition is true, the second function is returned.

That’s it.

Let’s implement the second function:

import numpy as np

def my_piecewise_function_2(x):
    return np.piecewise(
        x,  # The value at which the piecewise function should be evaluated
        [   # A list of conditions
            x < -0.5,
            (-0.5 <= x) & (x < 0),
            (0 <= x) & (x < 0.5),
            (0.5 <= x) 
        ],
        [   # One function corresponding to ech of the condition above.
            lambda x: -x - 0.5,
            lambda x: x + 0.5,
            lambda x: -x + 0.5,
            lambda x: x - 0.5
        ]
    )

Can I drop the last condition?

Yes, you can. If you have one condition less than the number of the function, the last function in the list will be used when all the conditions evaluate as “False”. Try this:

import numpy as np

def my_piecewise_function_2(x):
    return np.piecewise(
        x,  # The value at which the piecewise function should be evaluated
        [   # A list of conditions
            x < -0.5,
            (-0.5 <= x) & (x < 0),
            (0 <= x) & (x < 0.5),
            # (0.5 <= x) 
        ],
        [   # One function corresponding to ech of the condition above.
            lambda x: -x - 0.5,
            lambda x: x + 0.5,
            lambda x: -x + 0.5,
            lambda x: x - 0.5
        ]
    )

x1 = np.linspace(0, 1.5, 151)
x2 = np.linspace(-1.5, 0, 151)

np.all(np.abs(x1-0.5) == my_piecewise_function_2(x1))   # will be True
np.all(np.abs(-x2-0.5) == my_piecewise_function_2(x2))  # will be True

You will get the same results as before.

What if multiple conditions evaluate as True?

It is important to recognize that the list of conditions are not evaluated like a switch-case or if-elif-else. In another word, the following:

np.piecewise(
    x,
    [
        cond1,
        cond2,
        cond3,
    ],
    [
        func1,
        func2,
        func3,
        func4
    ]
)

IS NOT EQUIVALENT TO

if cond1:
    func1
elif cond2:
    func2:
elif cond3:
    func3
else:
    func4

If multiple condition in the numpy piecewise function evaluates as true, the value corresponding to the function that matches that last condition that evaluates to true is returned.

Confusing? Look at the following:

def my_wrong_piecewise_function(x):
    return np.piecewise(
        x,  # The value at which the piecewise function should be evaluated
        [   # A list of conditions
            0.5 <= x,
            0 <= x,
            -0.5 <= x,
            x < -0.5,
        ],
        [   # One function corresponding to ech of the condition above.
            lambda x: 1,
            lambda x: 2,
            lambda x: 3,                
            lambda x: 4,
        ]
    )

my_piecewise_function_3(2)  # will return 3

You might thing that the above function should return 1 for $x=2$ as input, because $0.5 \ge x$ is the first condition that evaluates to “True”; however, 3 is returned because the last condition that evaluates to “True” is $-0.5 \ge x$ and the function associated to that condition returns 3.

So, those conditions, are like switch-case implementation in C/C++ when you forget “break” statement.

So, be careful how you implement the conditions.

One last note on numpy’s piecewise?

Although this document is about piecewise-linear, so we are assuming each piece is an affine function, as you have noticed the numpy’s piecewise, does not impose any restriction on what the function for each piece should be. You can use that to implement any function; they can be none-linear. Or they can be even not a function, let’s say you do need to perform a different action depending on which piece or segment you are.

Are Piecewise-Linear Function continuous?

They don’t have to. They can be non-continuous. But in this writing we will focus on the continuous one.

Rewriting PLF using a [pseudo]-single equation

this new form is helpful when fitting a PLF. If the PLF is non-continuous, all you have to do is to treat it as bunch of separate line fitting. No big deal. The issue is when the PLF is continuous. You need to guarantee the continuousness of the function. So, let’s see how we can do that. We are going to first rewrite the equation in a slight different format. Let’s say we have the following PLF:

$y = \begin{cases} \alpha_{0,0} + \alpha_{0,1} x & x < \beta_0 \\ \alpha_{1,0} + \alpha_{1,1} x & \beta_0 \le x < \beta_1 \\ \vdots & \vdots \\ \alpha_{n,0} + \alpha_{n,1} x & \beta_{n-1} \le x \\ \end{cases}$

This PLF has $n+1$ segments or pieces: counting from zero: segment 0 to segment $n$ . The intercept for i-th segment ( $i \in [0,n]$ ) is defined by $\alpha_{i,0}$ and the slope for it is defined by $\alpha_{i,1}$ . We are going to rewrite the above equation as:

$\begin{array}{ccl} y\left ( x \right ) &=& \lambda_0 + \lambda_1 x + \lambda_2 (x - \beta_0)\delta_0 + \lambda_3 (x - \beta_1)\delta_1\\ & & + \cdots + \lambda_{n+1} (x-\beta_{n-1})\delta_{n-1} \end{array}$

where:

$\delta_i = \begin{cases} 0 & x < \beta_i \\ 1 & x \le \beta_i \\ \end{cases}$

This might look like we were able to rewrite the PLF as a single equation! well, it depends how you interpret it. The definition for $\delta_i$ is still using multiple condition.

The more important question is: How is this guaranteeing the pieces to be continuous? Notice that $\beta_i$ are called break-points, tie-points, knot-points, or simply knots. To check for continuity, just try the functions at the knots, i.e. evaluate that the limit of that function when approach a knot, $\beta_i$ from left and right is the same, i.e.:

$\lim_{x \to \beta_i^-} y\left ( x \right ) = \lim_{x \to \beta_i^+} y\left ( x \right )$

Ok! let me make it easy. Just evaluate $y\left ( x \right )$ at $\beta_i$ for both equations, i.e. the equation specified for condition $\beta_{i-1} \le x < \beta_i$ and the one that is for $\beta_i \le x < \beta_{i+1}$ . Ignore that for the first one it is said $x < \beta_i$ .

Let me do it for $\beta_1$ . in this case all the $\delta_i$ where $i > 1$ are zero. $\delta_0$ is 1. When approaching $\beta_1$ from left, $\delta_1=0$ and you have:

$\begin{matrix} \lim_{x \to \beta_i^-} y\left ( x \right ) &=& \lim_{x \to \beta_i^-} \left ( \lambda_0 + \lambda_1 x + \lambda_2 (x - \beta_0)\delta_0 + \lambda_3 (x - \beta_1)\delta_1 \right )\\ \\ &=& \lambda_0 + \lambda_1 \beta_1 + \lambda_2 (\beta_1 - \beta_0) \times 1 + \cancel{\lambda_3 (\beta_1 - \beta_1) \times 0}\\ \\ &=& \lambda_0 + \lambda_1 \beta_1 + \lambda_2 (\beta_1 - \beta_0)$

When you are approaching $\beta_1$ from the right side you have: $\delta_1=1$ and you get:

$\begin{matrix} \lim_{x \to \beta_i^+} y\left ( x \right ) &=& \lim_{x \to \beta_i^+} \left ( \lambda_0 + \lambda_1 x + \lambda_2 (x - \beta_0)\delta_0 + \lambda_3 (x - \beta_1)\delta_1 \right )\\ \\ &=& \lambda_0 + \lambda_1 \beta_1 + \lambda_2 (\beta_1 - \beta_0) \times 1 + \cancel{\lambda_3 (\beta_1 - \beta_1) \times 1}\\ \\ &=& \lambda_0 + \lambda_1 \beta_1 + \lambda_2 (\beta_1 - \beta_0)$

You can see that both equation evaluate to the same value; hence, you have your continuity guaranteed.

How are the equations in two forms related?

So, how are $\alpha_{i,0}$ and $\alpha_{i,1}$ related to $\lambda_i$ ? Essentially, if you want to know the slopes and intercepts of each segment, how can we get those?

Remember that there were $n+1$ segments and we were numbering them from 0 to $n$ , so when we say i-th segment, remember that $0 \le i \le n$ . That’s is we are counting from 0. Don’t get confused like that episode from Emily in Paris with the elevator and floor numbering, at least not in this document. Now that we have that cleared, the slope for i-th segment is:

$\alpha_{i,1} = \sum_{j=0}^{i}{\lambda_{j+1}}$

In another word, the slope for the first segment the slope is: $\alpha_{0,1} = \lambda_1$ , for the second segment the slope is: $\alpha_{1,1} = \lambda_1 + \lambda_2$ , and so on.

And how about the intercepts? You can recompute the intercepts using the following equation:

$\alpha_{i,0} = \begin{cases} \lambda_0 & i=0 \\ \lambda_0 - \sum_{j=1}^{i}{\left ( \beta_{j-1}\lambda_{j+1}} \right ) & 1\le i \le n \end{cases}$

so, we have:

$\alpha_{0,0} = \lambda_0$ ,
$\alpha_{1,0}=\lambda_0-\beta_0\lambda_2$ ,
$\alpha_{2,0}=\lambda_0-\beta_0\lambda_2-\beta_1\lambda_3$ , and
…

What’s Next?

As mentioned in the beginning, we are dividing this document into two parts. This document, Part I, focused on what Piecewise-Linear Functions (PLFs) are and we tried to ease you in with the mathematics.

In the next document, Part II, we will focus on how to estimate the coefficients, i.e. $\lambda_i$ when you have a data set to fit. We are going to divide the problem into three different types. In the first type, we assume we already know the break points (how many there are and where they are), i.e. all the $\beta_i$ are known. In the second type, we are going to assume we just know how many break points there are; but we don’t know where they are. In the third type, pretty much we just know that we want to fit a PLF. We don’t know how many segments there are and where they are.

References

Piecewise Linear Function – Wikipedia – (here) – last accessed: Feb. 1st, 2024
Numpy’s API Reference (here) – last accessed: Feb. 1st, 2024