Graduates ' Perceptions and Evaluation of Study Programmes : Analytical Framework and Methodological Considerations

This paper aims to draw scientific attention to the significance and usefulness of sociological approach, namely higher education (HE) graduates’ perceptions survey as a one of the mechanisms to collect and use graduate feedback facilitating the study programmes’ evaluation, and in turn, enlarging our knowledge on the enhancement of HE quality, the improvement of graduate employability, and effective ways of smoothing the transition from education to work. The emphasis is being placed on critical description of the ana­ lytical framework set out by authors of the international research project DEHEMS on HE graduates feedback. The DEHEMS project uses secondary processed data based on primary raw data as generated by two Europe-wide and large-scale research projects: REFLEX and HEGESCO.


In tro d u cto ry Remarks: D escription and Background o f th e Issues
The primary purpose of the paper is to highlight the significance of sociological approach, namely higher education (HE) graduates' perceptions survey as a one of the mechanisms to collect and use graduate feedback facilitating the study programmes' evaluation, in order to enlarge our knowledge on the enhancement of HE quality, the improvement of graduate employability, and effective way of smoothing the transition from education to work.When considering this issue, it is worth looking closely at the relevance and completeness of the analytical framework (perceived as a guide for data generation methods and its analysis) as applied in the latest substantial study on HE graduate feedback (the DEHEMS Project, 2009-2011, a European setting)2.
In recent years, the role of graduate research in HE programme evaluation has been receiving increased attention in academic literature and debate of policy makers and HE practitioners on quality assurance in HE.Much has been written and spoken about programme and teaching effectiveness and quality evaluation.However, fewer efforts have been made to develop our knowledge on findings stemmed from empirical research, and/or good practice examples in this respect.Simultaneously, according to the British authors of a guide to good practice in the realm of collecting and using student feedback, in the UK higher education setting, for example, one can observe "a considerable growth in formal institutional arrangements for quality assurance in recent years, in which student feedback surveys play an important role"3.Furthermore, interestingly enough, the special UK Task Group, chaired by Professor Sir Ron Cooke, striving to conceptualise and implement this essential experience, has proposed to make public the qualitative information provided by " feedback from recent graduates, disaggregated by institution, collected through a national survey" (Final Report o f the Task Group, HEFCE)4.Moreover, "The Task Group has discussed the approach to commissioning advice on the design and implementation of a national survey... "5.Although there are different approaches to carry out an evaluation of programme/ teaching effectiveness, in most cases an appraisal of programme outcomes employs a well known indicator of programme quality, i.e. evidence of programme quality derived from surveys and/or interviews of graduates and other internal/external stakeholders, like students, employers, community members or agencies.This ele ment is common to all academic programme evaluation options.It focuses largely on feedback from recent graduates, students and other parties interested in the programme.Generally, the purpose of this process is to evaluate the quality of the whole programme.
Speaking of the importance, rationale and benefits of feedback from graduates, one should emphasise that graduates' opinions and expectations can be viewed as a subjective measure of their employment success.It is also important to keep in mind, that graduates' assessment (more or less formative and summative in character) of the relevance and usefulness of study programmes for professional employment (work preparedness level) is a critical prerequisite in identyfing more formalised performance indicators, and developing a detailed methodology of collecting and analysing graduates' perception data.
As far as application considerations are concerned, graduates' perception and evaluation data is also of interest to authors of HE rankings.A particularly useful conceptual and practical perspective in this respect is embedded in the approach developed by the CHE (Centrum fur Hochschulentwicklung, Hochschulranking, 1998 -to date).The CHE nation-wide ranking (Germany) relies on statistical indicators (broken down into fields of study) developed using data collected, inter alia from graduates on their perception of the quality of the HE programmes.The data sourced from graduates as well as other collected factual data enable classification of all HEI programmes evaluated in three categories: "superior", "average", "inferior"6.
In summary, institutional, nationwide and international graduate surveys play at least a dual role: firstly, being an extremly useful tool and source of information for HEIs' career offices and administration and secondly, serving as a solid procedure, useful for the university units in charge of evaluation and development of study programmes, accreditation/ranking agencies and other relevant organisations.
According to American authors of seminal work on three major applications of graduate surveys, evaluation of curriculum content/structures, and teaching approaches can be perceived as by-products of well-developed graduate surveys7.It is worth mentioning that the most established, "classic" graduate surveys have been used for a quite narrow set of main purposes like: monitoring graduate labour mar ket outcomes, examining graduate engagement and competences, and scrutinising graduate donations to university.
Apart from the main purposes of graduate tracer studies, a possible (wide) range of the purposes and foci of programme evaluation might be implemented covering for example: curriculum structures and content (perception of curriculum relevance/ usefulness), modes of the curriculum delivery, educational challenges presented to the students and the appriopriateness of the method of assessment.
Taking these items into consideration, one can come to the conclusion that all of them are important determinants of the educational and career paths of graduates.They also enable achieving the detailed purpose of typical graduate surveys, that is, to identify the relationship between the graduates' study programmes and the subsequently obtained employment.
Concluding this paragraph, the diverse reasons that underlie fostering a growing popularity and interest in study programme evaluation by graduates can be grouped into quite numerous categories of the rationale in question.These are as follows: -the general trends in public policy, governance and management (NPM), and the growing international competition of HE institutions and systems (considerations for accountability, transparency and evidence-based decision making); -the universal changes occurred in HE learning and teaching in recent decades, inter alia a shift from teacher-driven to student-centred learning patterns; -graduates are seen as primary stakeholders in the European systems of quality assurance (ESG); -graduate surveys function as a measure in institutional strategie development and management; -graduate surveys function as a measure in institutional (internal and external) or national quality assurance (QA); -graduate surveys function as a measure in HEIs' rankings and benchmarking performance; -graduates potentially create an objective perspective for assessing how well is a programmes quality and effectiveness due to their distance from involvement with the programme and student experience, and their work and life experience since graduation; -graduate feedback's primary concern with curricular matters (academic features of study programme) results from their intrinsic nature to the quality of HE teaching and learning, and intended learning outcomes; -graduate perceptions of the study programme serve as independent variables for the further analysis of career success; -graduate employability and transition to the labour market becomes more and more decisive criterion for evaluating a delivery of higher education programmes and learning provision; -given employers' relative little knowledge about graduates' competencies and job performance (specific information asymmetry) one should expect that employers may not rely only on individual attributes of the candidate, but also take characteristics of the education programme (via graduate group perception data) into account; -when making a decision on educational choice by prospective HE consumers, graduate feedback being an important source of evidence, enables informed choice of study programme within and between HEIs; -when considering higher education as long-term investment in human capital, it is obvious and appropriate to use graduate surveys as a way of evaluation of the educational investment; -when experiencing scarce resources and state budget deficit, evaluation by graduates provides reliable and valuable information to enable programme decision-makers to make tough choices between cancelling some programmes/their components and funding others -the existing and newly created ones.
In European countries, including Poland, one deals with a small number of broadbased governmental initiatives as well as research projects addressing the accountability enhancement of universities by collecting comparable data among graduates and employers for purposes of benchmarking, comparing different national HE systems, different HEIs and different disciplines.More specifically, analysis and research in the field of programme evaluation are still overlooked and being a relatively new research area, especially in Poland where systematic comprehensive country-wide graduate surveys have not been carried out on a regular or even ad hoc basis so far.The only exception was the HEGESCO (international) project dated 20 098.
In the 1990's and 2000's, the large international and country-wide surveys on graduate employment comprising, to a greater or lesser extent, section of questions devoted to the description and/or evaluation of study programmes were as follows (see box 1).

Box 1. The large international and country-wide surveys on graduate employment
comprising items related to the description and/or evaluation of study programmes (90's and 2000's)

CHEERS/TSER (H ig h e r E d u c a tio n a n d G rad u a te E m p lo y m e n t in E u ro p e /C a re e rs a fte r H ig h e r E d u c a tio n -A E u ro p e a n R e se a rch S u rve y) -selected European countries; 2. REFLEX (R e se a rc h in to E m p lo y m e n t a n d P ro fe s s io n a l F le x ib ility ) -selected European countries + Japan; 3. HEGESCO (H ig h e r E d u c a tio n as a G e n e ra to r o f S tra te g ic C o m p e te n c e s ) -selected European countries; 4. DEHEMS (N e tw o rk fo r th e D e v e lo p m e n t o f H ig h e r E du c a tio n M a n a g e m e n t S y s te m s ) -selected European countries; project based on REFLEX/HEGESCO primary data; 5. DLHE (D e s tin a tio n s o f Leavers fro m H ig h e r E d u c a tio n , former FDS -F i r s t D e s tin a tio n s S u p p le m e n t); DLHE-LS (D e s tin a tio n s o f Leavers fro m H ig h e r E du c a tio n -L o n g itu d in a l S u rve y) -the United Kingdom; 6. B&B (B a c ca la u re a te a n d B e y o n d L o n g itu d in a l S tu d y ) -the United States; 7. NGS (N a tio n a l G rad u a te S u rve y) -Canada; 8. AGS (A u stra lia n G rad u a te s S u rve y); The AGS comprises GDS (G ra du a te D e s tin a tio n S u rve y) and either CEQ (C o u rse E x p e rie n c e Q u e s tio n n a ire ) or PREQ (P o stg ra d u a te R e se a rch E x p e rie n c e Q u e stio n n a ire ) -Australia; 9. GPS (G raduate P ath w a ys S u rve y) -Australia; 10. AlmaLaurea (comprises: G rad u a te P ro file S u rve y; S u rv e y o f G rad u a te s E m p lo y m e n t C o n d itio n ) -Italy.
Source: own study, see also P. Bielecki

, N a tio n w id e a n d In te r n a tio n a l H ig h e r E d u c a tio n G ra d u a te Tracer S u rveys. S elected M e th o d o lo g ic a l
A s p e cts, material replicated (in Polish), ORSE SGH, 2011.

A nalytical Fram ew ork and M eth odolo gical A pproach
The main purpose of the DEHEMS9 study comprises, inter alia gathering information about study programmes and results of their evaluation by graduates regarded as determinants of graduates' career success in selected six professional domains10.A statistical analysis of the quantitative data as derived from two earlier European graduate surveys was the only strand of methodology when examinining technical issues of the study programmesevaluation.
The text of the paper primarily draws on the research report (Final Report 2012)11 prepared within the frame of the DEHEMS project (2009-2011) which had been building its data sets and findings on two earlier international projects, namely REFLEX (completed in 2007) and HEGESCO (completed in 2009)12.The latter one was an extension of the former project using the same questionnaire addressed to graduates who completed studies in selected HE systems from Eastern Europe (5 countries) contrary to the REFLEX sample which included selected HE system from Western Europe (15 countries + Japan).It is worth mentioning that both the REFLEX and the HEGESCO projects followed on an earlier pioneering study -CHEERS survey conducted in the 1990's13.The REFLEX survey was carried out in 2005, and the twin -the HEGESCO survey in 2008.Due to technical reasons the extent of the final DEHEMS dataset was diminished by eliminating 2 countries: Sweden and Switzerland.The ultimate, combined REFLEX/HEGESCO dataset (labelled as the DEHEMS dataset), includes nationally representative samples of the graduate cohorts in the participating countries (18) and selected broad domains of study (6) included 30177 graduates overall, and 8598 graduates in the business/economics domain of which 289 were graduates from Polish HEIs.The coverage of the DEHEMS survey embraces graduates (ISCED 5A/1997 -bachelor/master or equivalent) who completed their degree study 5 years prior to the moment of the survey.The following national HE systems were involved in the DEHEMS study (country codes are given): AT, BE, CZ, EE, FI, FR, DE, IT, NL, NO, PT, ES, UK, SI, TR, LT, PL, HU.
As mentioned previously, an assessment of the teaching and learning approaches and modes in the business/economics programmes is one of the reported results in the DEHEMS research project (see box 2).HE programme evaluation features are structured according to the components of programme logic framework: i) programme activities and components -i.e process variables -left side of the box 2, and ii) programme impact (consequences) -i.e.outcomes variables -right side of the box 2.

o v e ra ll e va lu a tio n o f th e p re p a ra tio n f o r e m p lo y m e n t)
Source: own study based on the DEHEMS project.
In the DEHEMS project the popular Likert-type scale was used as a format for the questionnaire.The main measure of the data presentation was percentage of highest scores, accompanied by a mean point score based on a 1 to 5 rating scale (calculated for the average score of assessment of programme characteristics and programme outcomes).
In this paper, the choice of business and economics domain has been determined mainly by massive nature of these two fields of study in Polish HE system, and authors interest.The basic feature of the adopted metodology was the comparison of Poland's graduates' rating shares against the overall ratings (arythmetic mean score calculated for all countries as surveyed by REFLEX and HEGESCO projects), and the ratings of two "extreme" countries (with the highest and the lowest scores of evaluation).Finally, one should add, that there is a sole source of the statistics used in the paper: the DEHEMS elaboration based on the REFLEX/HEGESCO primary data.
As mentioned earlier, the countries (national HE systems) surveyed are spread among four categories in order to elicit information on Poland's standing with respect to graduates perceptions against average survey results attributed to overall countries cohort (18 -"Europe"), and two national HE systems getting in top and bottom position.
Table 1 aggregating the data on the objects of evaluation, presents findings of a series of description statistics on specific programme characteristics and programme outcomes perceived by business/economics graduates.Due to limited size of the paper the subset data on the "extreme" country assessment scores is omitted.Therefore, the data is breakdown only by two categories: Poland and all 18 countries surveyed.Moreover, it should be noted that large differences between fields of study in terms of a variety of variables (educational versus non-educational -labour market and personal factors) present a great methodological challenge and potential limitation of research for project providers of large-scale, national/international graduate surveys.According to the authors of the partial (detailed) DEHEMS study14 investigating determinants of graduates' job satisfaction conceived as a key indicator of graduates' career success, in the domains of social Sciences, and business and economics the most relevant determinants are education-related variables, inter alia programme characteristics.On the contrary, in the domains of engineering, medicine and sci ence the greatest impact on job satisfaction have graduates' personal traits and the work environment.These authors further recommend to use more objective source of information concerning programme characteristics pointing to data taken from universities' registers.
This finding of Polish researchers demonstrates that it is justified to claim that making evaluation of study programmes in business and economics domain matters.

Key Elem ents o f Analytical Fram ew ork Presented in Schem atic Form
When discussing issues of conceptual framework, it is noteworthy that figures and boxes as employed in this context address the role of graduate surveys in pro gramme evaluation in general, and graduate evaluation of business and economics/ other programmes in particular (the DEHEMS project's conceptual background).It is worth noting that only one schematic representation, i.e. box 3 can be attributed to the designers and contractors of the DEHEMS project.
As there is no common understanding of what graduate tracer study's purposes and applications are, Figure 1 offers comprehensive classification of purposes based on essential consideration of main three approaches adopted in work by A.F. Cabrera, D.J. Weerts and B.J. Zulick15.Individual items of Figure 1 show the different approaches and intended uses that graduate feedback might have at different levels, inter alia programme level.Graduate perception data on the study programme characteristics and evaluations can be perceived as one of by-products of typical graduate tracer studies (see three items in shaded cells with blue colour -lighter and darker ones).
Basic models (measures) of programme evaluation, both direct and indirect, used in HE management (for the purpose of decision making and policy development) are presented in Figure 2. The special object of our interest is the importance and necessity of perceptual studies among graduates conducted mainly by means of standardised questionnaires (item marked with blue colour).
As can be seen from Figure 3, there is legitimacy of conceptual variation in grad uate perception/evaluation data collection with respect to object of programme evaluation and its intended purpose.Figure 4 depicts two basic distinctions in concept of evaluation, commonly regarded as issues of particular interest: division between formative and summative evaluation, and internal and external evaluation.It is important to note that the boundary between them can sometimes be blurred.
As shown in box 3, conceptual model of graduate career success adopted within the DEHEMS project emphasises the importance of the perceptions of HE programme characteristics as independent variables for the further analysis of graduate career success.
Well known HE programme logic components (outputs and outcomes) that have been implicitly adopted within the DEHEMS project are presented in schematic form in Figure 5.

Direct measures
Indirect measures (curriculum/course embedded assessment; (perceptions of programme and modes examinations and tests) of teaching and learning)

D. Vidoni, A d d r e s s in g G r a d u a te s ' C a re e r Success f r o m th e H ig h e r E d u c a tio n P e rsp e c tiv e -T h e o retica l a n d C o n c e p tu a l F r a m e w o r k , in: E m p lo y a b ility o f G r a d u a te s a n d H ig h e r E d u c a tio n M a n a g e m e n t S y ste m s . F in a l R e p o r t o f D E H E M S
P ro je c t, eds.M. Melink, S. Pavlin, University of Ljubljana, Faculty of Social Sciences, 2012, p. 37.
Put most simply, box 4 illustrates the significance of the German (CHE) and American (B&B) research initiatives in order to enrich our knowledge on the role of perceptual studies among graduates with regard to characteristics/outcomes of study programme to be evaluated.Usually, judgments resulting from programme evaluation concern different features of study programmes that are not completely separate.For this reason, there is a need to broaden concept of programme evaluation and evaluative experience as adopted in the REFLEX and HEGESCO projects.Therefore, extending of traditional approach to evaluation is needed in order to take into consideration such programme features as programme importance or graduate satisfaction as addition to various forms of programme effectiveness (i.e.accomplishment of particular objectives).and comparing different HEIs and disciplines and last but not least identifying some of the programme features as areas for improvement.
In case of a lack of open and direct access to the REFLEX/HEGESCO databases across study domains (groups of field of study), results of the DEHEMS project can meet the information needs experessed by: government agencies, sectoral organisations, university management leaders, and scholars interested in institutional research on evaluation of programme quality and effectiveness.
When considering the importance of the DEHEMS data, author has also been trying to make an effort to carry out the analysis of graduates' judgements impact on programmes in relation to the EUA scores on formal academic autonomy experienced by HE programmes in different countries (The Autonomy Scorecard project16) in order to identify a determinant of potential discrepancies between perceptions and judgements as presented by gradutes from different countries (national HE systems).
The analysis of the results of the DEHEMS -European-wide comparative survey disaggregated by country and group of fields of study (business and economics was of author's interest as the most popular study subject in Poland) revealed some interesting findings.
Initialy, it is interesting to note a general observation that analysis of graduates' perspectives of the study programmes across all programme descriptors (11 items) reveals significant differences between national systems with respect to almost all categories under scrutiny17.The utmost diversification of the curricular and pedagogy arrangements has been identified especially between North-western Europe with highest scores (Portugal was the exception) and Eastern/Southern Europes HE systems with the lowest scores (accompanied by Poland's position as middle of the road).The latter HE systems display, most frequently, a lower level of academic autonomy.It should also be noted that the DEHEMS data allows one to quantify similar considerable variation in graduates' evaluations related to particular pro gramme outcomes (7 items).
Secondly, the position of Polish graduates' ratings against other graduate judge ments for other Europan countries as surveyed in the REFLEX and HEGESCO projects, shows that the percentages of positive answers for Polish graduates are similar to the average indications for Europe as a whole ("high" or "very high extent" rating categories).
Thirdly, countries which represented the highest graduates' rating of the impact of business/economics programmes (good basis for different forms of further pro fessional, personal and educational development) were mostly included into the EUA academic autonomy cluster: top ("high") cluster (HEIs considered as highly autonomous) and "medium high" cluster.In most cases, the lowest ratings of capacity of programmes with respect to their impact on graduate careers, as reported in the DEHEMS survey, were observed in countries with HE systems of low academic autonomy (cluster: "medium low").
Similarly, when analysing and comparing the average scores of programme descriptors (teaching and learning modes in the business/economics programmes) calculated in points graded on a scale of 1 -"not at all" to 5 -"to a very high extent", there are no noticeable differences among Polish and other graduates' percentage of opinions.Poland's graduates tend to perceive individual programme characterstics just like all other graduate populations (Polish mean scores are close to overall mean) or their ratings tend to moderately exceed scores calculated for surveyed graduates at large (labelled as "Europe").In other words, Poland's HE system took an intermediate middle position.This does not mean, of course, that there is an indication of relatively high degree of standardisation in programme design and provision patterns across national HE systems in different regions in Europe.Contrary to this, such uniformity does not exist.
To sum up, the analysis of the data on evaluation of study programmes in busi ness and economics indicates that in the scrutinised Poland's HE national system and the HE sector there are still areas where supplementary effort is needed in order to enhance study programmes run in these two popular professional domains.The question however remains to what degree national and institutional HE policies demonstrate the readiness to shrink the gap between Polish HEIs and their competitors from Western Europe.
The analytical framework, methodological approach and findings of the DEHEMS research report as well as published dataset on programmes' evaluation18 should be viewed against their several limitations.First, there is a relatively small number of participants in the survey of Polish business/economics graduates (the achieved sample amount) as compared to other countries' graduate samples surveyed.The next limitation of quantitative analysis is due to the fact that the adopted checklist of programme descriptors is dominated by atributes referring to basic modes of teaching and learning.Therefore, a much more comprehensive and subject profiled range of variables depicting effectiveness of programmes should be taken into consideration.When envisaging more relevant criteria of programme evaluation, especially pertaining business and economics (unregulated professional domain with vocational orientation), one should bare in mind such ommited descriptors like: reconciliation between academic rigour and programme relevance, emphasis on e-learning and IT applications in the study programme, emphasis on internationalisation of the study content, emphasis on civic engagement/ethics, emphasis on the variety of course offerings, emphasis on interdisciplinarity and integration of sub-disciplines and emphasis on integration of research results.It should be also underlined that some of the above descriptors are adopted by authors of the CHE reputable ranking.It appears that an essential limitation of the DEHEMS methodological approach is its rather restricted scope with respect to attributes to be evaluated at the programme level.
Additionally, also of value would be an investigation of graduates' satisfaction with certain aspects of the programmes they graduated from and the programmes' importance to their lives now.Nota bene, it is im portant to remember that the DEHEMS data creating a peculiar "information file" on study programmes is only reported by graduates (the extent to which phenomena exist) without any judgements on individual programme descriptors.The experience of the American B&B study conducted by the NCES within the U.S. Department of Education would be of great help here.

Box 2 .
Higher education programme evaluation features according to the DEHEMS project Perception of the study programme by business/ economics graduates (12 attributes) Business/economics graduates' evaluation of the study programme (7 attributes) -Rating of study programme as demanding -Familiarity of employers with the content of study programme -Freedom in composing own study programme -Broad focus of study programme -Vocational orientation of the study programme -Academic prestige of the study programme -Emphasis on group assignments -Emphasis on written assignments -Emphasis on multiple choice exams -Emphasis on participating in research projects -Emphasis on project and problem-oriented learning -Emphasis on lectures -Programme as a good basis for personal development -Programme as a good basis for development of entrepreneurial skills -Programme as a good basis for further learning on the job -Programme as a good basis for starting work -Programme as a good basis for performing current work tasks -Programme as a good basis for future career -Utilised knowledge and skills in current work (s u m m a ry p ro g ra m m e eva lua tio n, i.e.

Figure 1 .
Figure 1.Conceptual approaches to alumni survey according to A.F. Cabrera, D.J. Weerts and B.J. Zulick (2003) Assumptions Approaches to alumni survey Institutional quality and effectiveness can be appraised on what alumni have accomplished [a n n o ta tio n b y P .B .: and what was a level of alumni satisfaction with the quality of the programme]

Figure 3 .
Figure 3. Basic distinction in graduate perception/evaluation data collection with respect to object of programme evaluation and its intended purpose Perceptions of programme characteristics and outcomes Evaluation of programme characteristics and outcomes Purpose of data collection: Diagnostic

Figure 4 .
Figure 4. Combination of programme evaluation roles according to B.R. Worthen, J.R. Sanders, and J.L. Fitzpatrick, including graduates as evaluators

Figure 5 .
Figure 5.The DEHEMS focus with respect to objects of programme evaluation External and Contextual Factors

18
See for example DEHEMS Project.Framework Tool fo r Quality Assurance in Higher Education Systems: Domain Overview of Statistical Data Related to Graduates Employability -Economy [Business and Economics -P.B.].W ork Package 5. Country: Slovenia, DEHEMS, 2012.

Employers familiar with the content of the programme 2.7 2.9 To what extent has the study programme been a good basis for*:
Note: 1.The table shows the means of possible answers ranged from 1 = "Not at all'' to 5 = "To a very high extent''; 2. * The data for business/econom ics alum ni is available only for three categories of program m e outcomes; 3. The DEHEMS project com bined data set (2012) is based on the REFLEX/HEGESCO data.Source: E m p lo y a b ility o f G ra d u a te s a n d H ig h e r E d u c a tio n M a n a g e m e n t S y ste m s.F in a l R e p o r t o f D E H E M SP ro ject, eds.M. Melink, S. Pavlin, University of Ljubljana, Faculty of Social Sciences, 2012, p. 99.

Worthen, J.R. Sanders, and J.L. Fitzpatrick, including graduates as evaluators
Source: B.R. Worthen, J.R. Sanders, J.L. Fitzpatrick, P rogram E va lu a tio n : A lte r n a tiv e A p p ro a ch es a n d Pra ctica l G uidelines, Second Edition, Addison Wesley Longman, New York 1997, p. 20 (the diagram little bit revised).

Conceptual model of graduate career success adopted within the DEHEMS project: the importance of the programme characteristics
16 T. Esterm ann, T. Nokkala, M. Steinel, University Autonomy in Europe II.The Scorecard, European University Association, Brussels 2011: see also P. Bielecki, J.M. Salazar Zegers, Comparative Analysis of Governmental Interference in Higher Education Programmes.Poster presented at the 26th EAIR A nnual Forum: "Knowledge Society Crossroads", Technical University of Catalonia (UPC), 5-8 September 2004, Barcelona.
17W ith the exclusion of the item -program m e attribute: "emphasis on lectures" (data n o t available -?).