It makes logical sense to find a way to restructure the criteria so they reflect a developing level of understanding and application of a skill or understanding? But how to assess the relative weight of that skill/understanding across a program?
I used to faciltate DACUM analysis workshops for groups trying to define the important skills and knowledge within job-defined topics - that was a herculean task but worthwhile as it contributed to a much deeper understanding and appreciation of the range of skills and knowledge required to perform effectively in a job AND the need to assess emerging, developing levels of ability as each person developed experience and expertise.
Could you involve instructors in your program is outlining your "rubric" and then teasing out what they would expect to see as students come to them from earlier courses in the program?
We used a series of booklets by Ruth Stiehl and Les Lewchuk while I was at the College - they had one on Assessments that many of us found worthwhile - time-consuming of course.
Good luck!
Sylvia