Hi Colby,
I know what you mean. I have worked with people who are naturally good teachers and I have worked with people who have taken many teaching-related courses, and are not good teachers. I think of the programs as a benchmarks for improvement. Are people better teachers after taking a teaching improvement program - yes, I think so - but a different question than are they competent.
I was part of an initiative in Scouts Canada where we tried to ensure volunteer Scout leaders could meet detailed competencies. The idea was to have someone with the appropriate expertise to do a prior learning assessment to see where the leader was already competent and then work with the leader to determine opportunities to acquire the rest of the competencies. Some of the methods of gaining the competencies involved working with experts, taking training courses, coaching and self study. Dates were set for meeting the requirements, and the leader needed to meet with the expert to have the checklist of competencies reassessed.
While the approach worked in many ways, the difficulties encountered were the time it took for the one-on-one approach, the differences in perception of requirements for a specified competence, and indifference to getting it done. Another downside was that leaders did not meet other leaders (as they would in training course) and have the opportunity to form community with that group.
The advantages were the individualized approach where people were not taking courses when they already had many of the competences and more recognition of individual's abilities (which fostered good will). Another upside was that the expert often became a mentor for the leader and that relationship led to learning beyond the required competencies and retention of the leader.
From an organizational perspective, the availability of qualified people to run the competency-based program and the need for regional flexibility presented issues which were not adequately resolved. At the same time, the ethical need for trained leaders was critical to ensuring retention and organizational viability and the struggle to find the best ways to ensure this competency remains an issue.
I find this experience with a volunteer organization parallels the situation in academia quite well. We face the same issues.