As soon as color started to appear, it's now difficult to relax and not worry when to make the decision to harvest and process.
First of all, the color of the skin is not a reliable indicator of the ripeness of the flesh inside the olive. Fruit maturity depends on many factors such as olive variety, temperature, sunlight, and irrigation. A hot fall can cause fruit to ripen quickly, resulting in a narrow window for optimum picking. A cool fall may result in green fruit hanging longer.
Secondly, not all fruit within the orchard (or a tree for that matter) will ripen at the same time. This can make our decision to pick potentially more complicated.
Fortunately, there are methods to ease that panic!
One of the ways we do this is randomly walk through the orchard to monitor and examine fruit. We like it to feel "right" when squeezed which is neither rock hard nor too soft. And when cut with a sharp knife, there should also be a thin layer of purple under the skin (depending on variety), but the rest of flesh around the pit should be creamy.
Some use a maturity index where essentially you take random samples from trees high, low, and from all sides within harvest area and sort them (skin & flesh) by a color scale of 8. Then by simple multiplication & addition you get your mean maturity average.
Ideally, most shoot to harvesting in the middle of the season so that the olives are not too under-ripe (green, difficult to mill) or over-ripe (sweet, lower shelf life). Hopefully, we do it right and produce the best quality oil possible!