Pattern Search

Intro

The Pattern search algorith is an Optimization algorithm.

$X_{n + 1} =$ exploratory move $(X_{n} + (X_{n} - X_{n - 1}))$
if $f (X_{n + 1}) < f (X_{n})$
accept $X_{n + 1}$
else
$X_{n + 1} =$ exploratory move $(X_{n})$

It is said to be a derivative free algorithm because it optimizes with f(X). Though from my analysis it is not derivative free but numerically approximates them through the algorithms structure.

There have been variations and improvements on the pattern search algorithm, so the pattern search algorithm description in one reference can be differ from another. For example variable h_i, exploratory move after pattern move, removal of order bias in the original algorithm. The pattern search algorithm was explosive.

Exploratory Move

See exploratory defined.

For the pattern search algorithm the exploratory move is defined with the magnitude_inc defined to do nothing - and so the algorithm may get stuck but is cheaper to run. If the algorithms stability is important then define the magnitude_inc function.

Generalized Pattern Move

Sequential X_k are derivatives where the change in the numerator is a vector and the denominator an integer 1.

This is supported as each independent dimension with the exploratory move has a difference and an integer denominator change.

To support this view lets expand the Taylor series and derive the first order approximator.

Taylors series about a point with the difference operator Δ.

$Δ X_{n} = X_{n} - X_{n - 1}$
$X_{n + 1} =$
$X_{n} + h Δ X_{n - 1} + \frac{1}{2} h^{2} Δ^{2} X_{n - 1} + ...$

Truncate to first order.

Let
$h = 1$
$X_{n + 1} = X_{n} + (X_{n} - X_{n - 1})$

Which is the first order approximator.

This strategy generalized to any order approximator. Deriving the second order approximator.

$Y_{n + 1} =$ $Y_{n} + Δ Y_{n} + \frac{1}{2} Δ^{2} Y_{n}$

First difference

$Δ Y_{n} = Y_{n} - Y_{n - 1}$

Second difference

$Δ^{2} Y_{n + 1} = Δ Y_{n} - Δ Y_{n - 1}$
$= (Y_{n} - Y_{n - 1}) - (Y_{n - 1} - Y_{n - 2})$
$= Y_{n} - 2 Y_{n - 1} + Y_{n - 2}$

Putting it together.

$Y_{n + 1} = Y_{n} + (Y_{n} - Y_{n - 1}) + \frac{1}{2} (Y_{n} - 2 Y_{n - 1} + Y_{n - 2})$

$Y_{n + 1} =$
$\frac{5}{2} Y_{n} - 2 Y_{n - 1} + \frac{1}{2} Y_{n - 2}$

Generalized Pattern Search Step

You will notice that the pattern search can overshoot. Indeed without the exploratory search the pattern move made no gain in the problem I had. If we see the pattern move as a taylor series then it is clearer.

$X_{n + 1} = X_{n} + hStep * (X_{n} - X_{n - 1})$

Here is some data for the test problem. The exact solution is (1,2,5).

1.00073 2 4.99689
g.fn.counter=211
hstep=1

0.999943 2 5.00013
g.fn.counter=221
hstep=0.75

0.999946 2 4.99997
g.fn.counter=215
hstep=0.68

1.00001 2 4.99991
g.fn.counter=217
hstep=0.55

0.999548 2 5.00121
g.fn.counter=225
hstep=0.4

0.995003 2 5.00475
g.fn.counter=201
hstep=0.2

When the steplength was near the golden ratio number the calculation was best.

Experiments

The higher order approximations of the pattern move did not do much so the idea of the generalized pattern move seems to have failed. Generally it did worse than the exploratory search without the pattern move.

Perhaps this was surprising but the default exploratory move performed the best on my parabola test problem.

To give you an idea of how good the supositly worst case move of the pattern seach is here is comparing it with a higher order approximator.

To compare the algorithms a counter on the number of function evaluations is used.

13th order pattern move aproximator
./main prog=9 dim=13 counter=400

g.fn.counter=405
hstep=1
pat.exp.fmin=8.29927e-13

explore h approximator
g.fn.counter=403
g.fmin=2.94134e-18

Pattern Search
g.fn.counter=400
pat.exp.fmin=1.86047e-10

As you can see the simplest algorith beats the others hands down. As the counter increased this difference became large. I had not expected this result, I would have guessed that the exploratory algoritmm would perform the worst, but it performed the best and is the simplest.

Algorithm

Initialize algorithm

Generate two successive approximations X₀ and X₁ with the exploratory algorithm.

Use the exploratory algorithm that iterates over 0..N-1 dimensions, see First Order Approximator.

loop
{
exploratory (
$X^{*} = X_{k} + (X_{k} - X_{k - 1})$ )

if (result==false)
exploratory $(X^{*} = X_{k} \pm h_{i} e_{i})$
}

Discussion

The pattern search algorithm has global convergence as its default sub algorithm is the exploratory algorithm which has global convergence.

Lets define local global convergence to be the rate of global convergence and that algorithms for practical purposes do not get stuck.

The original algorithm had fixed h_i which where reduced only after all N dimensions were searched with the exploratory algorithm failing.

How this effects local global convergence compared with varying the dimensions h_i I do not know.

Similarly doing an exploratory search after the pattern search and its effect on the local global convergence I have no idea.

TODO - compare the pattern search with other algorithms on some problems.

The pattern search may also be used for non-continuous functions and derivatives. Or in some hybrid.

The algorithm can be used in constrained optimization. eg resetting a value if it is outside the constraint domain. [1]

In the original algorithm after a successful search in a dimension rather than the N dimensions the pattern move was done. Then the dimension counter was reset. This favoured minimizing the 0'th, 1st, 2nd, .. dimensions.

References

W. Fabrycky and J. Mize (1973). Introduction to Optimization Theory. ISBN 0-13-491472-4. Page 117.