It took 3 years to build our monitoring solution after some dead ends (Nagios). Over the years we have learned what works and what doesn't when it comes to best-practices. The main lesson was that simplicity has its own utility and reduces overhead and TCO.