1990 SACS Self Study Report

Chapter III: Institutional Effectiveness

Table of Contents

Planning and evaluation at the University of North Carolina at Wilmington take place at all levels within the University, involve a broad base of faculty and administrators, focus on each component of the institution's mission, are defined by established procedures, and are used continuously to improve the effectiveness of programs. That is certainly not to say that there are no weaknesses to be remedied - there are - but in totality, the present planning and evaluation processes are adequate.

With specific respect to the educational program, detailed procedures for program planning and evaluation are in place. UNCW defines its expected educational results - and identifies how the achievement of those results will be ascertained - principally at the departmental level and in the Basic Studies Program. Departments and schools, and the Basic Studies Program, have established clearly defined purposes and educational goals linked to the institutional mission statement. Each department and school has means for determining the extent to which its goals have been achieved and uses these assessment results to modify policies, procedures, or programs to improve institutional effectiveness.

Evaluation of academic research and public service is accomplished mainly by the annual review of individual achievement in those areas. Additionally, several offices, centers, and institutes within the University have the primary role of planning, conducting, and evaluating academic research or public service.

The planning and evaluation functions of the University are supported by an excellent institutional research operation. Information provided by institutional research is used continuously to examine and analyze programs and procedures, and it is used as a basis for proposed constructive modifications. In addition, although the institutional research function has been regularly evaluated, one outcome of this Self-Study is a superior, comprehensive self-assessment plan for the evaluation of this important function.

Return to Table of Contents

At the University of North Carolina at Wilmington, planning and evaluation at the institutional level occur in the context of membership in the University of North Carolina (UNC) system, which consists of sixteen constituent institutions. The North Carolina state legislature, the UNC Board of Governors, and the UNC General Administration all play defining roles in the nature and scope of institutional planning and evaluation. This circumstance has both advantages and disadvantages. While it results in well-formulated program-planning procedures and in the coordination of programs in the state institutions, it has apparently served as a disincentive to aggressively pursue strategic planning at the institutional level.

In its responsibility to "determine the functions, educational activities, and academic programs of the constituent institutions," the Board of Governors of the UNC system has established procedures for academic planning. The most recent edition is "Academic Program Development Procedures, The University of North Carolina, January, 1988." This 65-page booklet contains a detailed and comprehensive description of the policies and procedures for planning and establishing new academic degree programs, for planning and conducting degree related extension instruction activities, and for planning and establishing institutes and centers for research and public service. The principles of institutional effectiveness are built into these processes. The requests for authorization to plan, and for authorization to establish, new academic degree programs must be justified by prior-needs assessments. There is an exhaustive, three-step review process for approval, which includes external review. Then, following establishment of the new program, two follow-up reports are required - one after one year, the second after three years - to assess the effectiveness of its operation. All new degree-program proposals, as well as degree-related extension-program proposals, must "include an evaluation plan which includes (a) the criteria to be used to evaluate the quality and effectiveness of the program, (b) measures to be used to evaluate the program, (c) expected levels of productivity of the proposed program/track for the first four years of the program . . ., and (e) the proposed plan and schedule to evaluate the proposed new degree program prior to the completion of its fifth year of operation once fully established." In 1976 the UNC system's Board of Governors adopted an additional system of evaluation of existing programs, thereby assuring that all academic programs are reviewed periodically. Thus the process of academic-program planning and evaluation at UNCW is exemplary, by virtue of the institution's membership in the UNC system.

... each institution will be expected to analyze its performance on the assessment measures in the context of its own institutional goals or standards of achievement. When it finds itself falling short of its goals, it is expected that the institution will undertake additional studies, if necessary, to discover the factors contributing to the situation. Having diagnosed the problem, the institution will take corrective action to address perceived weaknesses. It will monitor the results, and, in future assessment reports, discuss the steps being taken to address the problem.

UNCW's "First Year Report" (dated May 3, 1991) has been completed. The report is written with the following organization: excerpt of institutional mission or goals, followed by itemized assessment measures to be applied, each followed by a narrative analysis and discussion of progress and improvement.

The approach taken for the remainder of this chapter is as follows. The primary sources of information used were the Self-Study unit reports prepared by each academic and administrative unit at the request of the Steering Committee. These were supplemented by additional written requests to selected units, and by several interviews with vice chancellors (or their representatives) and with the Director of the Office of Institutional Research. The chapter attempts to summarize the various aspects of planning processes at UNCW, analyze the quality of planning and evaluation exhibited, and supply examples of both strengths and weaknesses with respect to institutional effectiveness, thereby supporting its recommendations and suggestions for improvement. The two principal subdivisions of the report are, first, a comprehensive examination of planning and evaluation at UNCW and, second, a description and analysis of UNCW's institutional research function.

To address the essential components of planning and evaluation, this chapter is divided into three main areas: administrative planning and evaluation, educational planning and evaluation, and research and service planning and evaluation.

Historically, planning at UNCW has been carried out at the departmental and divisional levels. This situation resulted, in large measure, from the nature of the planning process within the UNC system. Much of the planning done at the divisional/departmental level came in response to planning memoranda sent from the UNC General Administration to UNCW administrators. The memoranda requested both biennial and five-year planning documents. This timetable was determined by the need to develop legislative budget proposals biennially and to develop the UNC system's five-year plan which includes projections for all of the system's sixteen member institutions. UNCW administrators in turn requested responses from divisional/departmental level administrators. The strength of such a system was that most units and departments within UNCW developed an individual planning process and periodically produced planning documents. The weakness was that there was no comprehensive planning at the institutional level. Thus, the individual plans did not necessarily contribute to the implementation of a coherent plan and a unified system of evaluation and assessment, nor did they reflect budgetary exigencies.

A review of the Self-Study reports of the four principal divisions of the University, each headed by a vice chancellor, corroborates the situation described above. Working within the system described, some divisions have developed thorough planning and evaluation procedures for internal purposes. Others have not.

The Division of Student Affairs provides an exceptionally good model of an internal planning and evaluation procedure. Departments within the division each develop long-range plans. The planning process includes a review of functional work areas, mission and goal statements, key results areas, and problem-solving objectives. The plans are developed by committees, each of which contains representatives of the administration, faculty, and staff. Each department engages in an annual review of its mission, goals, and achievements, and it attempts to revise its plan through this process. The only weakness of this highly professional planning and evaluation process is that there has been no provision for integrating planning in the Division of Student Affairs into a comprehensive University planning process.

If the planning and evaluation process of Student Affairs represents a creative and professional approach, that of the Division of Business Affairs reflects a failure to take planning and evaluation seriously. Again, departments within the division prepare plans and objectives. However, all such preparation is done internally, with no effort to obtain information or insights from other University constituencies. No special committees are charged with the responsibility for planning or evaluation. No attempt is made to create a comprehensive plan for the division; rather, the individual departmental plans are shelved as reference sources to be consulted by divisional administrators who expressed the belief that the individual departmental plans were adequate substitutes for a divisional planning approach. Evaluation procedures vary from the vague to the nonexistent. The vagueness of the evaluation procedures employed by Business Affairs can perhaps best be conveyed by the language in a memo from that office, sent to the Administrative Planning and Evaluation Subcommittee of the Committee on Institutional Effectiveness in lieu of a unit report, which the Division of Business Affairs had, at that time, failed to prepare.

It is necessary in Business Affairs that many assessments are perceived from the reaction of the university community. Numerous complaints from the faculty concerning facilities would certainly indicate that the facility function is not being carried out as it should be. This same perception is used in any other area of Business Affairs to detect areas which are notably weak in performance. In an effort to achieve a higher degree of effectiveness, the results of the evaluated and perceived assessments allow adjustments to be made throughout the five-year period.

How these "perceptions" are collected, to whom they are referred, and how they result in "adjustments" are never made clear.

The other two divisions fall between these two extremes. In Academic Affairs, departmental units are charged with developing internal planning and. evaluation procedures. While there is no planning body as such within the division, the Vice Chancellor meets twice monthly with the deans to "review their areas of responsibility and to discuss issues of significance to their agencies." The Office of Academic Affairs also "participates as appropriate" with faculty and administrative committees, the Chancellor, and the Faculty Senate.

The assessment section of Academic Affairs' unit report is brief, and its brevity indicates that that office has not developed adequate assessment measures. Basically, the evaluation process consists of the Vice Chancellor's office reviewing agency reports and plans as well as the results of a yearly survey of faculty perceptions of the performance of various administrators. This survey contains approximately 25 questions about the performance of the Vice Chancellor for Academic Affairs, only one of which deals with planning or evaluation. (It should be noted, however, that the Vice Chancellor for Academic Affairs, who is also

Provost, is the only divisional head to undergo such comprehensive faculty evaluation. Faculty evaluation of the Vice Chancellors for Business Affairs, Advancement, and Student Affairs is obtained by a single line on the survey devoted to each of these offices.)

The unit Self-Study report of the Division of University Advancement presents a similar picture. While the report contains a rather comprehensive list of goals and the plans to achieve them, it does not indicate how either were determined. The report does not identify an internal planning unit, instead noting that this function resides with the Vice Chancellor, who "consults" with other administrators within the division, though no schedule of such consultations is provided. The report also notes the "lack of process for development of needs from faculty," an acknowledgement that the division's planning process to date has excluded faculty participation. Evaluation procedures are, again, sketchy in the extreme, and are herewith quoted without further comment:

University Advancement can measure the achievements of the division by the increase in its non-restricted and restricted fund base. University Relations can measure the achievements of the office by the enhancement of the image of UNCW to be perceived as a tier one regional university.

The planning and evaluation process revealed in the Self-Study reports of the University's four major divisions is indicative of the priority assigned to planning at the University level. In fact, from 1976 to 1978, as noted in a memo of February 1, 1991, to the Self-Study Committee on Institutional Effectiveness from Charles Cahill, Provost, "The UNC General Administration takes the lead in defining the scope and format of institutional planning." Provost Cahill also observed that for almost two decades "planning has basically meant academic program planning." In 1988, however, a University Planning Committee, which originated two years earlier as a result of faculty concerns about the lack of comprehensive planning on campus, "initiated a review of all units within the university for the purpose of laying the groundwork for a more inclusive planning document." This effort, encouraged by the Office of the Provost and Vice Chancellor for Academic Affairs, resulted in 1989 in a call for each University agency to develop a five-year plan. The Planning Committee intended to begin a review of these agency plans in the fall of 1990. This would have represented the first step toward a comprehensive University planning process, since the Planning Committee did contain representatives of both faculty and administration. However, once again policies developed by the General Administration resulted in changes in the planning process. These changes have made evaluation of current institutional planning and assessment procedures more difficult because they are in the process of revision.

On March 16, 1990, an administrative memorandum from the office of the President of the UNC system called for reassessment of the policies embodied in long-range planning documents. He informed campus administrations that they were not limited by prior Board of Governors actions on institutional mission and program structure." This refocused the already ongoing reevaluation of the planning process at UNCW and reinforced concerns expressed about the planning process by the incoming Chancellor, James R. Leutze, who took office in July of 1990. (Chancellor Leutze, who describes himself as a "fanatic about planning," has in little more than one year given new vitality to institutional planning at UNCW.) Since the spring of 1990, the existing University Planning Committee has written a new institutional mission statement and adopted a revised planning-committee structure

The current University Planning Committee for the first time provides the University with a process for institutional planning. The Chancellor is now the chair of that com-mittee and clearly responsible for planning at the University level. The membership allows for broad-based faculty and administrative representation, as well as a link to the UNCW Board of Trustees through a representative appointed by that body, though faculty constitute a decided minority of the new Planning Committee.

The new planning document ("University of North Carolina at Wilmington Long-Range Planning," adopted April 25, 1991), which identifies University Planning Committee membership, also incorporates adequate assessment and evaluation procedures, both at the University level and at the divisional level. It specifically mandates the creation of divisional planning committees, the absence of which is so noticeable in divisional Self-Study reports. In addition, the new planning document seeks to insure that adequate data are available to the Planning Committee for purposes of planning and evaluation by requiring the Director of Institutional Research to serve as a staff member of the -Planning Committee. This allows committee members access to a variety of data relating to measurements of institutional effectiveness in the areas of educational program, research, and service. For example, results of student, alumni, faculty, and other survey instruments will be made available to committee members through the Office of Institutional Research.

An organizational chart, which is included in the new planning document, identifies both the responsibilities of the committee and also the subcommittee structure designed to insure that these responsibilities are met. The four subcommittees established are Institutional Mission, Academic Program, Financial Resources, and Support Services. All include faculty and administrative representation. The one weakness perceived in the new document is that it does not provide adequate mechanisms to furnish a variety of data to departments and units below the division level.

For effective planning and evaluation to occur at UNCW, there ought to be widespread recognition of the goals and planning activities of the institution. There also needs to be widespread involvement of the faculty in the planning and evaluation procedures. The surveys of faculty and students, which are a part of the Self-Study, show not only how widely recognized are the goals of the University, but also the degree to which the entire planning process is perceived to allow involvement of the faculty and other University constituencies.

On the positive side, it is clear that the faculty is aware of and supports the goals articulated in the UNCW mission statement. Ninety-four percent of the responding faculty members were aware of the University's purposes and goals. Ninety-one percent of the faculty members agreed with the purposes and goals outlined in the mission statement (a recently adopted and widely circulated document).

Faculty survey data, however, reveal that the faculty are fully aware of the inadequacy of evaluation procedures at the University level. While 70% of the faculty believed that procedures for evaluation are in place, the faculty was evenly split as to whether or not beneficial changes have occurred as a consequence of existing evaluation procedures. Sixty-five percent of the faculty, however, were supportive of the current procedures for evaluation.

One possible source for the faculty's relatively low support for the current evaluation procedures can be seen in the survey. Asked if they were involved in evaluation procedures, 58% of the respondents took the position that they were uninvolved, a position supported by the Self-Study reports of the University divisions.

The scope of the faculty's criticisms of the evaluation and planning process is evident in several specific areas of University administration, but the budget process appears to be the most significant area of faculty concern. Only 25% believed that they understand the budget-development process. Only 13% agreed that the budget development process provides for faculty input, again a position supported by the divisional Self-Study reports.

Weaknesses in the budget-development procedures may account for the perceptions that there are inadequate resources for important components of the University's mission. Only 28% agreed that financial resources support instructional activities. Only 19% agreed that financial resources support research activities. Only 13% agreed that financial resources support professional development.

This confusion is also apparent in a series of responses relating University goals with budget authority. A large majority (63%) of the faculty did not see a clear link between the goals of UNCW and the allocation of resources. Only 32% believed that the administration allocates resources effectively. This perception is again explained by the lack of faculty participation in the budgetary process.

Lack of clear planning and evaluation procedures may contribute to the faculty's negative perception of other administrative functions. Only 30% believed that the administration implements institutional policies effectively. Only five percent believed that the University has adequate funding to support programs. Only 22% perceived the University budget to be the result of comprehensive planning. Only 27% agreed that physical resources are adequate for achieving the educational mission of the University.

While it is hard to identify from the survey what the source of the problem might be, a large proportion of the faculty (41%) feel that there is ineffective coordination between UNCW and the UNC General Administration and the state legislature. Since the reports of the planning officer placed great emphasis on the constraints arising from the UNC General Administration and the state legislature, it is clear wat this is an area needing additional study.

Outside the budget process, there is one unit within the University about whose evaluation procedures the faculty has more positive perceptions. Fifty-nine percent give a grade of A or B to the procedures for evaluating library services.

There is, however, some evidence that many of the faculty do believe that University evaluation procedures are somewhat effective. Sixty-five percent feel that feedback is used at the University level for assessing planning decisions. What is not evident is what the source of the feedback might be.

There are evident areas in which the faculty believes that it does not have information on which to make judgments about the effectiveness or adequacy of a unit's evaluation procedures, even below the divisional level. Sixty percent could not assess the evaluation procedures for student government. Fifty-six percent could not assess the procedures for evaluating student health services. Sixty-two percent could not assess the procedures for evaluating intramural athletics. Fifty-two percent could not assess the procedures used to evaluate intercollegiate athletics.

In other areas, only a small fraction of the faculty agreed they have adequate input in the activities of several units. An area of major concern is academic computing services. Only 23% agreed that the policies for the allocation of computing resources and the assignments of priorities for computer use are consistent with UNCW's educational mission.

While there may be several explanations for this assessment, other responses reveal that frequent users of the computing resources believed they are uninvolved in the planning and decision making of the Office of Computer and Information Systems (OCIS). Only 21% reported they have an opportunity to have input on budget decisions of OCIS. Only 23% reported they have opportunity for input into OCIS planning activities. More generally, only 27% reported that the Faculty Senate's Computer Services Committee adequately represents the faculty.

Other areas where the faculty believes it has limited input include student development and institutional advancement. Only 31% agreed that faculty has adequate input into student-development policies. Only 16% of the faculty were satisfied with the involvement of the faculty in institutional advancement, a statistic that is supported by the Advancement Office's unit Self-Study.

In addition to the faculty's involvement in the ongoing evaluation of the University's success in achieving its goals, effective evaluation requires the participation of other members of the community. The University's students are an important group that needs to be included.

The Self-Study Survey of Students produced data similar to that obtained by the faculty survey. Fifty-three percent of the students responded that they had an opportunity to provide feedback about UNCW at the University level. Sixty-two percent had the impression that feedback from students, staff, and faculty was used in planning decisions. A much smaller proportion (38%) believed that they had seen beneficial changes at UNCW because of evaluation procedures. Only 48% responded that students were sufficiently involved in the evaluation of instruction at UNCW.

Interpreting these perceptions should be qualified by the fact that only about half (55%) of the respondents believed they understand how evaluation results are used. Also, they are somewhat more critical of the effectiveness with which problems are corrected: only 29% of the undergraduate respondents agreed that the University acts to correct instances of poor instructional performance.

Students were also asked if they had been surveyed about certain components of UNCW's mission. This feedback is necessary in order to see if UNCW is accomplishing its student-related goals. It is clear from the survey results that students are not being consulted about whether UNCW is accomplishing these goals. Fifty-seven percent reported that they had not been consulted about the adequacy of the guidance they were receiving for achieving their own goals. Only 46% agreed that UNCW provides adequate personal counseling. Ninety percent reported that they had never been asked about how their values had changed since they have been at UNCW. It seems clear that the students are not aware of the University's efforts to use student feedback in the planning and evaluation process, and, in fact, the new planning document does not provide a mechanism for considering student concerns. This is a weakness of that document.

Conclusions - A review of the Self-Study reports of the University's major divisions, summarized above, clearly indicates that in the past planning and evaluation have been assigned a low priority by the University administration. As a result, until 1988 there was no comprehensive planning process at the institutional level. The low priority assigned to planning and evaluation at the University level encouraged similar treatment of planning and evaluation at the divisional level. The planning that did occur was done within a department/unit/division and was for internal use only. There was no effort to coordinate divisional/unit planning to carry out, or even create, University goals. Most divisional planning procedures lack any mechanism for consulting constituencies of the University outside the division. For example, faculty feel that they had no influence in the decisions made by the Division of Business Affairs perhaps for the reason that there was no formal mechanism for faculty concerns to be considered by that division in its planning and evaluation processes. Assessment procedures were also inadequate and for precisely the same reasons.

Recently adopted planning structures and strategies should address these weaknesses, especially the development of a new planning document and the creation of a new Planning Committee. While the new committee has some weaknesses, especially in the area of evaluation and its lack of a mechanism for considering student concerns, it addresses the major SACS requirements. It establishes a planning procedure and an evaluation procedure with both administration and faculty participation, clearly designated lines of authority, and means of obtaining adequate data for planning purposes.

Educational Planning and Evaluation

Evaluation of educational programs is necessary but particularly difficult. The problem of assessing the effectiveness of training programs has long been acknowledged, and a number of methods have had varying success, with the greatest success in measuring very specific mental or motor skills. However, a university seeks to accomplish a much broader set of outcomes with much less clearly definable and measurable results. While one can readily count the number of widgets produced in an hour or the number of correctly identified nations on a blank map, it is much more problematic to assess a university student's ability to face a new situation, analyze its elements while bringing to bear an understanding of philosophical and historical perspectives, arrive at a rational "solution" or course of action, and implement steps to bring about a desired change. And even this lengthy sentence does not fully describe what we expect of our graduates in the coming century.

While a part of the university experience (especially in the professional programs) prepares a student for a job, "education" implies a great deal more than economic independence. It is this "great deal more" which is so resistant to being reduced to standardized, quantifiable, predictable, and measurable units. One of the great strengths, indeed part of the very concept, of a university is its teaching and learning of all the higher branches of knowledge. While such diversity is an acknowledged strength, it also is a "weakness" when one tries to bring about uniformity in structure to its processes. A method that is very effective in determining the "skill" a student has developed in generating an income statement would not only be ineffective but in fact stultifying in measuring a music student's creativity in composition or a philosophy student's ability to reason through an ethical dilemma. UNCW seeks to play an important role in the total development of its students, preparing them for life as well as for careers. Such an aim is ambitious, to say the least, and perhaps lacking in humility, yet if our universities do not do this we can expect to experience a deterioration of our very culture as well as our competitive enterprises.

Thus, evaluation of UNCW's educational program recognizes the critical importance of diversity through its wide variety of assessment measures (examples are given below). It also recognizes the value of professionalism in its faculty and administration. For example, a part of faculty professionalism is the responsibility to continually seek feedback on one's course presentations, seek current and new knowledge in one's field, seek new concepts and techniques of pedagogy, and adjust one's courses to reflect the information one has gained. Feedback at UNCW includes widespread use of student opinion surveys administered at the end of each course with results tabulated and returned to faculty members.

In addition, faculty members frequently discuss both pedagogical methods and developments in their fields with colleagues at UNCW and with peers in the profession across the country and, in some cases, around the world. Through research efforts, faculty discover current and new knowledge of their fields which they incorporate in their courses. The expertise faculty have acquired through their own study (both in schools and independently) and the experiences they have had equip them quite well to use professional judgment in presenting material to students and evaluating student performance. As indicated above, in many cases this performance may not be wholly quantifiable but rather requires a judgmental evaluation. The University has extensive procedures to assess and evaluate a new faculty member's performance, relying on feedback from students, judgments of peers, and analysis from department chairs. In order to remain with the University, a new faculty member must show satisfactory performance (including professionalism as described above). In order to have the opportunity for promotion and salary increases, this satisfactory performance must be maintained even after tenure has been granted.

Failure to rely on professionalism as a "mechanism" for assessment would defeat the whole purpose of education. Forcing faculty into a situation where they would be evaluated (and tenured, promoted, and paid) on the basis of some sort of "standardized" test of student performance or by some series of numerical data (class GPAs, number of graduates, etc.) would result in the most professional faculty leaving the University and/or students being taught "to the test," grade inflation, and similar undesirable outcomes.

The following sections examine the planning and evaluation of UNCW's educational programs. The linkage between program mission and goals at all levels in the University is examined. Current assessment procedures are described, and, finally, faculty and student perceptions of the academic programs are reviewed. Strengths and weaknesses in educational planning and evaluation are pointed out.

The institution's educational mission and goals are directly linked to the educational mission and goals of individual units. The Statement of Institutional Mission recognizes UNCW as a "community of scholars dedicated to excellence in teaching, research, artistic achievement, and service to local and global communities." This recognition of excellence in teaching as a mission and a goal, supported by dedication to scholarship and service, is also acknowledged at the departmental level. Several departments - such as Biological Sciences; English; Health, Physical Education and Recreation (HPER); and Mathematical Sciences - and the Cameron School of Business Administration (CSBA) explicitly reiterate this dedication to teaching, scholarship, and service (CSBA includes professional development as a fourth area) in their unit mission statements.

The student-centered philosophy of the University acknowledged in the institutional mission statement is also an explicitly stated goal of several departments (such as Communication Studies, Fine Arts, and HPER).

The University's commitment to providing lifelong learning opportunities receives special attention from the Office of Special Programs (OSP), recently reorganized as the Division for Public Service. Although several departments promote lifelong learning through particular courses they offer and courses offered in the evening, OSP has this goal as its main focus. Its unit report addresses this directly: "The OSP is the administrative unit through which lifelong learning, personal and professional development, and public service is rendered. . . ."

The University's mission to assist with the improvement of public school education is directly linked to the missions of the School of Education and the Departments of Biological Sciences, Chemistry, Earth Sciences, English, Fine Arts, Foreign Languages and Literatures, HPER, History, Mathematical Sciences, Physics, Political Science, and Sociology and Anthropology. The School of Education and these departments provide certification for elementary and secondary school teachers. An example of the linkage is provided in the English Department's unit report. The report states that an objective is to "serve as a campus resource for the region's schools in the improvement of English education."

The University's mission "to stimulate in its students intellectual curiosity, imagination, rational thinking, thoughtful expression, and a love of learning" is directly linked to the educational units of the institution. Similar statements can be found, for example, in the departmental mission of Earth Sciences, English, Fine Arts and in the mission of the College of Arts and Sciences and that of the Basic Studies Program.

The Department of Earth Sciences provides an excellent example. Its unit report describes a purpose "to stimulate the intellect, imagination, and emotions of its students by affording them every opportunity to achieve their educational goals in their pursuit of knowledge and understanding through course work and research." It then sets a specific goal: "to provide students with a basic understanding and an in-depth knowledge of fact, theory, and techniques within the earth sciences, and to provide undergraduate and graduate research opportunities which stimulate intellectual curiosity, imagination and rational thinking." Another goal is to "provide undergraduate degree programs which fully prepare students for entry level professional careers. . . ." The department's plan for achieving the goals is "embodied in its course offerings and degree programs [which offer a] wide variety of courses [covering] the basic areas of geography and geology." Expected results include "graduates who are fully prepared and competent to begin professional careers in earth sciences or to continue their education and training in graduate school."

Knowledge of the humanities, social and natural sciences, and fine arts is a part of the mission of the institution and is directly linked to the Basic Studies Program and to the respective departments for each area. The Basic Studies program requires students to successfully complete courses in six broad areas: English composition, physical education, humanities (at least one course each in literature, history, philosophy, and language), fine arts, natural and mathematical sciences (at least one life-science course, one physical-science course, and one mathematics course), and social and behavioral sciences. Courses must be selected from a series of offerings in each area and include, in some areas, one or more required courses.

Effective communication is a stated goal in the Institutional Mission Statement. The Departments of Communication Studies and English address this goal most directly in their focus on both oral and written communication skills for their majors and for the general student (through the Basic Studies courses). In the Department of Communication Studies' Mission Statement, for example, the following sentence is found: "Above all else, we serve to provide for our majors an undergraduate degree program in which the philosophy, practice, criticism, and study of communication are emphasized and balanced." In addition, several departments (such as Sociology and Anthropology, History, and Political Science) have effective communication as a primary objective for their majors. Political Science serves as an example; the department "puts emphasis on the ability of its students to write well by assigning written work in each class, . . . essay exams, book and article reviews, research papers, and bibliographic essays." The Basic Studies Program is also designed to promote effective communication by students. The purpose statement of Basic Studies reads in part, "Courses in Basic Studies should require reflection, reasoning, and thoughtful expression.

One of the ten guidelines for Basic Studies courses states that Basic Studies courses should "develop students' abilities to think and communicate clearly by providing instruction in and opportunities for written and oral composition." Thus students continue to receive instruction in writing beyond the required six semester hours of English Composition. In an attempt to insure that each student receives direct instruction in communication, every student must take at least one course from the category of "Language," which includes not only courses in foreign languages, but also a course in Fundamentals of Speech and one in Introduction to Logic.

Developing and improving decision-making skills is a stated goal in the Institutional Mission Statement and is an explicit goal of the Department of Communication Studies and the Department of Production and Decision Sciences (PDS). PDS has a goal of enhancing students' understanding of and developing skills in quantitative methods used to solve business problems which "will enable the student to approach and solve business problems in a structured way." In addition, several departments (such as Biology, English, and Philosophy and Religion) foster critical inquiry. One stated purpose of Basic Studies is "to develop skills of reasoning ... and analysis." To be included in the Basic Studies offerings, a course must conform to a set of guidelines designed specifically to nurture students' abilities to make decisions (including an introduction to the substantive ideas of a discipline, its accepted modes of inquiry, its historical development, its applications, and its unsolved problems). In this way, every student is educated in decision-making in a variety of contexts.

The University mission includes helping students develop an understanding of their own and other cultures. Each UNCW student is required to take at least four Basic Studies courses from the humanities, all of which have been subjected to the test that they "widen the context of ideas and experiences to which students have been exposed to include cultures, eras, values, and philosophies different from their own." Many of the Basic Studies courses in the social and behavioral sciences also contribute to the fulfillment of this part of the institutional mission. Among departments, the missions of the Departments of Foreign Languages and Literatures, Philosophy and Religion, History, and Sociology and Anthropology are most closely linked to the aim of stimulating students' appreciation of cultural and philosophical diversity.

It seems evident that the educational mission and goals of the University and its corresponding units are clearly linked at all levels. Furthermore, these various units specify how they will assess the expected results of their goals. As articulated in the following section, these results are assessed through a variety of approaches which include student evaluations, student performance, and the internal and external assessment of programs.

The approaches to assessment of educational programs within the various units Of the University vary as much as the units themselves. This summary of those approaches addresses first the undergraduate programs, and then the graduate programs.

Student Evaluations - One of the most widely employed methods for evaluation of educational programs is the use of student-opinion questionnaires administered in individual courses near the end of each semester. The questionnaire most frequently used is the student VOTE (View of Teaching Effectiveness), which quantifies student responses in a variety of categories, some directed at the students' perception of the performance of the instructor and some toward perceptions of course content.

Several departments (for example, Mathematical Sciences and Sociology and Anthropology) and the Cameron School of Business Administration have fashioned their own version so as to probe the individual needs of the department or unit, but they share the same goal as VOTE: to determine student perceptions of the course's content, rigor, level of difficulty, faculty, facility, support material, and educational value, among others. Results from the questionnaires are reportedly used in different fashions and by different persons, depending on the department. Within departments the numerical value for courses in one or more categories is examined by a committee and/or by the department chair, "poor" evaluations are noted, and faculty members are notified either formally or informally of steps that must be taken to bring the student course evaluations up to acceptable levels.

Closely related to these course-by-course evaluations are surveys administered to graduating seniors (for example, in Chemistry and Political Science). These surveys are more global in nature, and their goal is to measure the students' overall perception of their curriculum and department. The findings of such surveys are forwarded to the various curriculum committees within the schools or departments where action can be taken to address any perceived deficiencies.

Surveys with a similar mission are sent to alumni (for example, in History and Earth Sciences), but those responses are viewed differently since the respondents have the apparent advantage of time, experience, and maturity to better evaluate their educational experience. The source of these surveys is largely the Office of Institutional Research; however, several departments have initiated alumni newsletters, and these have also served as vehicles for surveys. The newsletters also stimulate unsolicited responses from alumni about their educational experiences and can be a welcome source of feedback for schools and departments.

Estimates of Student Performance - Various units have used various measures of student performance to assess their programs. Informal assessment is accomplished through interviews with employers, prospective employers, and directors of graduate schools (for example, in the Cameron School of Business Administration). Performance in capstone courses is a way chosen by some departments (such as History and Sociology and Anthropology). Another is to evaluate students' overall knowledge or training in the context of a senior seminar format (for example, in Biological Sciences) where the numbers of students are small and faculty-student interaction is maximum. In the sociology curriculum, students are required to take a practicum in which they are assigned to an outside agency or law-enforcement entity and their performance is evaluated by both a site supervisor and a faculty director. Furthermore, sociology students are asked to engage in a senior research project. Assessment of the results of these projects is seen as a general measure of the effectiveness of the program.

The performance of students on standardized tests is also used as a measure of program success. These include such voluntary tests as the Graduate Record Exam, Law School Admissions Test, the Graduate Management Admissions Test, and the Professional Knowledge and Specialty Area test batteries of the NTE. One feature of these tests that makes them particularly valuable is that performance is often broken down into categories so that specific aspects of the curriculum can be evaluated. Performance of students in licensing examinations is still another measure of program success. Such examinations include the National Comprehensive Licensing Examination for Registered Nurses (NCLEX-RN) and the North Carolina test for the licensing of geologists.

Finally, special circumstances dictate special methods to be used to evaluate students, and in the Department of Fine Arts these include portfolio evaluation in art, evaluation of technical and performance skills in theater, and juried performances in music. While serving as evaluation tools for students, they likewise serve as measures of the success of the program.

Internal Program Assessment - Assessment of educational programs is also achieved by internal evaluations within the schools and departments. These include self evaluations by individual faculty members, evaluations by department chairs based on a variety of input, and committee evaluations, the latter also with input from a variety of sources.

Evaluations can be as simple as the examination of course syllabi to make sure that all sections of a given course are covering the same material (for example, in Fine Arts and English). Choice of appropriate texts and teaching materials is also evaluated to assure that courses continue to conform to initial course descriptions. The responsibility for these assessments lies either with the department chair or with a departmental or school curriculum committee. It is at this point that alterations in curricula can also be effected, since the curriculum committees are charged with updating and revising course offerings in response to perceived needs.

The overall assessment of faculty teaching is an important measure of the success of a program. This evaluation often includes peer review by individuals and committees, but the final responsibility rests with the chair of the department. While monitoring of classroom performance might be the most direct measure of teaching competence, the time demands of such an approach dictate that more indirect measures be used, most of which have previously been mentioned in this report.

External Program Assessment - Assessments of educational programs are also made outside of departments. These can include assessments from departments who are the recipients of service courses. In those cases (for example, in Biological Sciences, CSBA, Chemistry, English, Nursing, and Physics) it is important that the performance of the service department fulfill the needs of the departments being served. Currently, there is little evidence of such important interdepartmental communication.

Evaluations are also solicited from outside the University. These include surveys of employers, cooperating primary and secondary teachers, and internship supervisors. By their evaluation of the performance of our graduates, they can provide an invaluable measure of the skills derived from our educational programs.

Assessments of programs by groups or agencies outside the University include evaluations with different time scales and with varying degrees of rigor. For example, the Department of Political Science participates in an annual survey by the American Political Science Association that allows them to compare their curriculum to departments nationwide. The Department of Chemistry undergoes a review every five years by the Committee on Professional Training of the American Chemical Society. The music division of the Department of Fine Arts adheres to the standards of the National Association of Schools of Music. And the Department of Health, Physical Education and Recreation complies with the standards of the National Council for the Accreditation of Teacher Education Programs (NCATE), the North Carolina State Department of Public Instruction, and the National Recreation and Parks Association/American Association of Leisure and Recreation.

Finally, all schools are subjected to accreditation criteria, and it is the responsibility of the appropriate dean within that school to assure that educational guidelines are met.

Specific Evaluation Exams - The preceding paragraphs have described in general the various assessment methods used throughout the University. Some specific examples will illustrate the process in detail. (Space does not permit an enumeration of every evaluation and assessment procedure in use by every department. The following examples are intended to illustrate the climate of educational planning and evaluation at UNCW and samples of both strengths and weaknesses.)

First, the Office of Special Programs (OSP), now being reorganized as the Division for Public Service, was mentioned as having a goal of providing lifelong learning opportunities. In order to accomplish this goal OSP conducts regular, systematic assessments of the educational needs in the local and regional area. It then develops, together with the faculty involved, programs based upon the results of the needs assessment, within budgetary constraints. Programs are evaluated by clientele using OSP forms to determine overall effectiveness of teaching and the OSP staff effort. Results of these clientele evaluations are used by OSP staff and program instructors or faculty to revise and improve the programs as well as the delivery system.

The Department of Mathematical Sciences has one of the most complete processes for planning and evaluation. Having stated six specific goals clearly linked to the University's mission, the department has established procedures to assess progress toward those goals, including a departmental committee structure, a long range plan, and the use of student and peer evaluations among others. Expected results, assessment procedures, and use of assessment findings are specified for each of the six goals. As an example, the first goal is to maintain a commitment to teaching excellence. One of the five expected results is that "the department shall fully support the remedial and tutorial services of the Math Lab." The Lab is to be staffed by graduate assistants, student assistants, professional staff, assigned faculty, and faculty volunteers. One of five assessment procedures reads, "The Lower Division Mathematics Committee in conjunction with the Director of the Math Lab will assess the effectiveness of the current level of departmental support for services provided by the Math Lab by examining staffing, equipment, and student satisfaction and progress. Faculty will be encouraged to voluntarily spend time tutoring in the Math Lab, especially during peak hours." One of the five procedures for administering the assessment procedures reads, "Each semester the Director of the Math Lab shall review the current Math Lab schedule and determine whether or not it is adequately staffed." One of the five "Uses of Assessment Findings" reads, "In conjunction with the Lower Division Mathematics Committee, the Director shall modify existing activities of the Math Lab and institute new programs to better serve the students in lower division mathematics courses."

The English Department has a goal to "serve as a campus resource for the region's schools in the improvement of English education." Planning is "coordinated with the department chair by the directors of composition and the Writing Place, who work with outreach programs for the region's schools. Each year public school teachers throughout the southeast region have the opportunity to familiarize themselves with current pedagogical theories and their applications in such projects as the Cape Fear Writing Project, the North Carolina Writing Awards program, the Amy Charles Writing Awards competition, the annual meeting of the North Carolina English Teachers Association, and in-service workshops at their schools. The effectiveness of these efforts is assessed by participant evaluations, by participants' application of their learning to their public school classes as evidenced by follow-up meetings and classroom visits, and by school officials. The participants are evaluated on oral presentations and research projects. Feedback from these assessment methods is used to refine, revise, and improve the programs.

The Department of Earth Sciences has a specific goal of providing "students with a basic understanding and an in-depth knowledge of fact, theory, and techniques within the earth sciences. . . ." Evaluation is accomplished in part by giving students the opportunity to evaluate the courses and instructors at the end of each semester. Results of these evaluations are shared with instructors and the senior faculty as part of the ongoing faculty-performance evaluation and curriculum assessment. Instructors adjust the content and presentation of their courses based partly on this feedback. In addition, the department publishes an alumni newsletter including a survey which allows it to assess programs relative to the overall preparation, satisfaction, placement, and success of students after graduation. Information gathered from the newsletter and other sources helps the department continually evaluate its offerings with respect to employment trends. Recently, for example, the department added a hydrogeology course as a result of information received from students and alumni. The department uses a required senior seminar to assess students' overall knowledge of geology and analytical skills; the results are used to identify strengths and weaknesses in the geology curriculum. Graduate students must pass a comprehensive oral examination to determine their overall knowledge of the discipline and progress toward completion of their program; they must also defend their thesis research before peers and faculty. While intended to test the students, these examinations also serve as assessment tools for the program itself. Finally the state of North Carolina licenses geologists by examination. The department maintains records of graduates who attempt the exam and of those who become licensed.

The Basic Studies Program is clearly linked to the University's mission of providing knowledge of the humanities, social and natural sciences, and fine arts. The Basic Studies Program was recently completely reevaluated and as a result extensively revised. A faculty committee (the University Curriculum Committee, a standing committee of the Faculty Senate) directed the effort which included the opportunity for input from all faculty. The program is now on a three-year evaluation cycle with one-third to be reevaluated each year. Currently underway is a study by an ad hoc committee of the Faculty Senate of a new component for Basic Studies, that of interdisciplinary studies. Such a component is believed necessary to reflect changes in the world in which our students will -live, work, and continue their learning.

Changes in the Basic Studies Program reflect broad involvement of faculty and administrators. Every second year the University Curriculum Committee calls for proposed changes. Each department studies its actual or potential Basic Studies offerings, adjusts course descriptions as necessary, and considers new courses. Approval by the department is followed by submission to the school or college curriculum committee whose approval is needed before submission to the University Curriculum Committee. Upon approval by the University committee, proposals are submitted to the Faculty Senate. If approved by the Senate the proposal is forwarded to the Provost and Chancellor for final approval.

The Department of Political Science provides another example. As a strategy to achieve its objective of students' learning to write well, all classes require written work, such as essay exams, book and article reviews, research papers, and bibliographic essays. Assessment is accomplished by grades awarded in classes and by efforts to monitor the performance of majors who choose to take standardized tests such as the Graduate Record Exam and the Law School Admission Test. The success of majors who have applied to and/or entered professional schools is also monitored. Similar methods are used by several departments.

A new department, Production and Decision Sciences (PDS), has an explicitly stated goal consistent with the University's goal of developing and improving -decision-making skills. Since the department is newly organized (1988) and its area of concentration/major (Business Systems) was only approved in 1990, assessment is in its formative stages. The department expects that by 1995 about twenty majors will be graduating each year and taking jobs in the areas of management information systems or production/operations management. During its first semester ten students declared Business Systems as a major. As a new department and new program, PDS is in a unique position to state clearly its expected results and a plan for assessing performance of its program and its students.

Some unit reports reveal weaknesses in the area of planning and evaluation. In some cases it is in goal setting, in others procedures for evaluating goal attainment, in still others in the use of evaluation results to improve effectiveness, and in a few cases in all these areas. Some examples will illustrate.

The College of Arts and Sciences, according to its unit report, "has not yet carried out a strategic planning process to set goals for the College." However, it does have a plan to do this after the University-wide strategic plan initiated by Chancellor Leutze has been completed. Meanwhile, the College is evaluating its effectiveness to meet previously stated goals.

The School of Education does not list "expected results" in its report but refers the reader to ". . . sections of North Carolina Accreditation Report, Volumes I and 11 (1990)." The report states that "[s]ervice activities are evaluated both formally and informally and the results of those evaluations inform subsequent practice." How the evaluations are conducted and how the results affect subsequent practice are not stated.

The Department of Management and Marketing lists a goal of achieving "teaching, research and service excellence and continuing faculty professional development" followed by a short-range goal of achieving AACSB accreditation and a longer range goal of "differentiating itself from the myriad of those accredited by AACSB." The mission of the department is "to achieve regional recognition for the department's graduates and faculty" through excellence in teaching, research, and service. The department lists eight items it considers "evidence of accomplishment of Department goals" including placement of students in "superior" jobs and graduate schools and the number of research grants received, leadership roles of faculty in professional associations, and leadership positions of faculty in organizations. The items of evidence do not seem to be related to differentiating the department from other departments accredited by AACSB, although they could be considered related to teaching, research, and service excellence. However, it is not clear how this evidence would indicate goal attainment - that is, what numbers would be acceptable. Further, the unit report states that "specific targets for graduation rates and specific employers for our graduates are not established." But it is not stated how placement of students in "superior" jobs can be evidence of success without more specific targets. The report does state that it has profiled the "product we want to graduate and his/her requisite skills" in its long-range plan, but this information is not included in the unit report. Nor is it clear how evaluation results will be used to improve the effectiveness of the department.

The Department of Physics states that its educational goal is "to prepare students for graduate school in a physics related discipline or for employment in a physics related occupation." The department evaluates its programs "every three to four years" and "revises them accordingly." The only standard of evaluation given is "the publication of the American Institute of Physics, Graduate Programs in Physics, Astronomy, and Related Fields." It is unclear how the publication would be used or how programs would be revised. The only "measure of success in achieving [the goal is] the number of graduates accepted for graduate school or employed in a physics related job." Such information' is obtained from the surveys conducted by the Office of Institutional Research. It is not clear what number of graduates in graduate school or employed would be acceptable to indicate "success" for the program. The unit report's short paragraph on planning and evaluation seems to offer little guidance to department members in planning for, conducting, or evaluating the success of their educational program.

Graduate Program Assessment - At present no general evaluation of approved, ongoing graduate-degree programs has been formally initiated, although a proposal has been made and submitted to the Provost for reviews of programs in the College of Arts and Sciences; those in Education and Business Administration already are conducted at regular intervals by the appropriate professional accrediting organization. Pending approval of the proposal, individual departments and schools have been conducting their own evaluation of programs. This is done by faculty committees and by the faculty as a whole. It is the general opinion of most graduate degree-awarding departments that the requirements for the degree lend themselves to constant program assessment. That is, oral and written comprehensive examinations serve as adequately reliable measures of the success of the educational program, as do the composition of theses. Furthermore, the intimate relationship between faculty mentors and graduate students lends itself to the exchange of informal evaluations of courses and curricular requirements. The Cameron School of Business Administration conducts exit interviews of MBA graduates along with surveys of graduates and interviews with employers.

As a part of the Self-Study project, faculty, staff, and students were surveyed to determine perceptions of the University, its programs, and its effectiveness. The results of the studies must be interpreted with care. One potential problem is with interpretation of the questionnaire. Care was taken to be very clear in asking the questions; for example, the words "the procedures it uses to evaluate/review" were underlined as were the words "evaluate the effectiveness of." In spite of this it is quite possible that some respondents were giving their perceptions of a program rather than the procedures used to evaluate it. With this caveat, data from the faculty and student surveys do provide some information about perceptions of the linkage between University goals, program evaluation, and corrective procedures. For example, nearly all faculty reported that they were aware of the goals stated in the 1990 mission statement (94%) and of goals for their department/unit (87%).

Most perceive that procedures are in place to assess both general UNCW operations (70%) and their department/unit (77%). Virtually all faculty agreed with the goals stated in the UNCW mission statement and with their departmental goals as well (91% in each case). These results indicate a broad-based consensus on the general purposes of UNCW. While a substantial majority agreed with the procedures used to assess their department/unit (75%), the smaller number agreeing with the procedures used to assess University programs (65%) may indicate an area which could be improved. A similar distinction is presented by the number of faculty agreeing that planning decisions at UNCW use feedback from assessment at the University (65%), college (64%), and department (73%) level. Faculty seem to have more confidence in department processes than in University processes, although this could be a result of less familiarity with the University processes. Responses of students to this question were quite similar. In addition, just over half of the students (53%) felt that they had the opportunity to provide feedback at the University level, and 51% of the students surveyed perceived they did not have the opportunity to provide feedback at the departmental level. It would be useful to improve this situation - that is, to provide more opportunities for student feedback at both levels or to communicate the existing opportunities to students.

Half of the faculty reported that they had seen beneficial changes as a result of assessment procedures at the University level (50%) and a stronger majority perceived such changes at the departmental level (65%). This result follows the already-established pattern regarding faculty perceptions of department- versus University-level processes.

Students generally had not seen evidence of the benefits of assessment at either the University or departmental level. This may be an expected outcome since (1) students may leave the University before changes take place, or (2) changes may not be attributed by students to assessment procedures.

More than half of the faculty thought that there was effective coordination between the legislature, the UNC system's General Administration, and accrediting bodies regarding planning and assessment (57%), and slightly more perceived a clear linkage between the goals of the University, the colleges, and individual departments (59%). The relatively modest proportion perceiving a clear linkage between University-, college /school-, and department-level goals is puzzling given the rather explicit statements found in most departmental and college/school documents. There may be weaknesses either in the documents or in the communication process.

Only 37% of the faculty agreed that they perceived linkage between the University's goals and the allocation of resources. This small percentage is troubling. Either perceptions are incorrect and should be changed, perhaps by better communications, or resource allocation policies are faulty and should be examined.

Students also perceived clear linkage between the goals of the University and their department (73%), and a smaller majority saw linkage between goals and resource allocation (58%). Faculty generally agreed that the necessary evaluation processes were in existence to improve teaching (64%), a perception which seems to be contradicted by the faculty rating in Table 3.1. Both faculty and students were asked to "grade" (on a 4-point scale) the evaluation procedures for the undergraduate and graduate programs. Results for the undergraduate program are shown in Table 3.1. Results for the graduate program reveal a few differences as shown in Table 3.2.

Clearly graduate students rate evaluation procedures higher than undergraduates do, the difference being greatest for academic advising and

Perceptions of Evaluation Procedures (on a 4-point scale)

Perceptions of Evaluation Procedures (on a 4-point scale)

instruction. Furthermore, graduate students rate procedures higher than faculty do. Undergraduate students and faculty share similar perceptions, the largest difference being in evaluation of instruction. Evaluation of instruction was rated below 3.0 by all except the graduate students - a rather low number which should be improved in light of the emphasis the University places on the quality of teaching. Effective evaluation procedures may help to demonstrate the quality of teaching desired. It also appears that both faculty and students see a need for more effective evaluation of advising, although this may be one of those areas in which respondents were evaluating the advising process rather than the evaluation procedures for it. (For a detailed description of the academic advising processes, see the section on Student Development Services on page 280.)

A high percentage of faculty (ranging from 49% to 63% in nine categories) reported that they did not know enough about the graduate program's evaluation procedures to assign a grade. This could be due to these respondents not being involved in graduate programs since such programs are relatively new on this campus. It also could indicate a weakness in communication.

Faculty were asked to rate the Office of Special Programs, but 50% or more reported that they did not know enough to provide a rating. This may indicate another area in which communications should be improved.

Finally, in response to methods of assessing undergraduate learning, it is quite clear that both students and faculty are opposed to commercially produced standardized tests. Both faculty and students do support standardized testing of students to evaluate the effectiveness of basic mathematics programs and, to a slightly lesser extent, of writing and foreign languages programs.

Methods of assessing undergraduate learning seen as desirable/undesirable and the percentage so perceiving them, are shown in Table 3.3.

Table 3.3

Perceptions of Methods of Assessing Undergraduate Student Learning

a. Respondents Seeing Method as Desirable (%)

b. Respondents Seeing Method as Undesirable (%)

Summary Planning for and evaluation of the educational program at UNCW is an area of major importance to faculty and administrators. Many strengths of the process have been suggested in the foregoing paragraphs. Goals of individual departments, schools, and units are clearly - and, in most cases, explicitly - linked to the institution's goals and mission. A strong commitment to quality education runs through the entire institution. A specific illustration of this commitment is offered by a recent development. A small group of faculty volunteers was sent to conferences on teaching excellence at Western Carolina University. Subsequently those faculty members were among those appointed to a Chancellor's Committee on Teaching Excellence. After two years of study and planning including an extensive faculty survey, that committee issued a report that included a recommendation for a Center for Faculty Development and additional Teaching Excellence awards. The recommendation was accepted, and work is underway for bringing such a Center into existence. Although the current budget crisis may delay implementation, the concept has widespread support. The Teaching Excellence awards have also been implemented. One troubling aspect involves perceptions of how resources are allocated. Only 37% of the faculty agreed that they perceived linkage between the University's goals and the allocation of resources. As mentioned above, allocation policies and communication processes should be reviewed.

Another strength lies in the broad range of measures used to assess the quality of educational programs offered by the institution. Recognizing the differences inherent in programs, the difficulties of program evaluation, and the advantages of multiple measures of effectiveness, UNCW has instituted a variety of assessment methods. Several measures are used for assessment of faculty effectiveness, program evaluation, and student performance. Graduate-degree programs are in the early stages of development, but the preliminary evidence seems to indicate that here, too, the emphasis is on quality programs and multiple assessment methods.

Weaknesses seem to be related to budgetary conditions affecting a much broader population than the University (several unit reports suggest that budget constraints hinder the accomplishment of goals and, sometimes, the evaluation of progress) and to perceptions within the faculty/student populations. As resources become available, more planning in the form of surveys to determine community/ regional needs, as well as more sophisticated assessment methods, will become possible. Perceptions may be changed by improved communications or, in some cases, by revisions in certain administrative policies.

Some of the possible communication problems include: (1) faculty perceptions of procedures to assess University programs, (2) student perceptions of opportunities for their feedback, (3) faculty awareness of graduate program procedures, and (4) faculty awareness of procedures of the Division for Public Service.

Other areas for improvement include: (1) the use of assessment results for change at the University level, (2) the linkage of goals at the University, college/ school, and department levels, (3) evaluation procedures for teaching, (4) evaluation procedures for academic advising, and (5) isolated weaknesses in planning and evaluation as indicated by the Self-Study reports of several units (more specific statements are needed, including specific goals, expected results, and methods of assessment).

Suggestions for using the results of this Self-Study in planning and evaluation follow from the weaknesses identified. A commitment to more effective communication between levels in the University and among administrators, faculty, and students is sorely needed. Rarely in the unit reports is a recognition of the need for such communication noted, especially with respect to evaluation procedures (as distinct from programs and policies themselves).

In order to make most effective use of planning and evaluation efforts, the University community should become more knowledgeable of the variety of methods for, and the benefits of, assessment. It is clear that what is needed is not more assessment procedures or techniques but rather a better understanding of the potential for effective use (and possible modifications) of existing ones. Some training in the planning and evaluation process could be beneficial. This could be done through on-campus teleconferences, workshops, and newsletters - in short, education in the concepts of institutional effectiveness.

A commitment to high quality self-assessment should be communicated to all members of the University community. It is not now clear that a high level of commitment to the process exists uniformly across the campus. The diversity of programs and the diversity of evaluation methods represented in the unit reports is a strength of the University, but the deficiencies and inconsistencies in some of the unit reports are a weakness. In order for the evaluation process to be effective, individuals must understand where they want to go, how they will get there, how they will measure their progress along the way, and how they will determine that they have arrived at the destination. Although there need not be unanimity throughout the University, there should be-a fairly broad consensus within each unit. And goals of units must, of course, be consistent with University goals.

In order to stimulate faculty to become more aware of, committed to, and involved in planning and evaluation, leadership is essential. At a minimum, there should be an explicit, publicized expectation that requests for funding, staffing, and program change be justified by prior evaluation and assessment. This leadership must start at, but not be limited to, the chancellor level of the University. It must flow throughout each level of administration down to and including departmental chairs.

Research and Service Planning and Evaluation

Academic research activities within the UNC system are classified as either departmental research or organized research. According to "The University of North Carolina Institutional Assessment Plans 1991-1995," departmental research is carried on by faculty members as a part of their regular professional pursuits. Organized research consists of those research activities that are conducted through academic departments (especially externally funded research) or through special institutes and centers established to facilitate research. Although service is not explicitly classified in this way, in practice there is a similar dual contribution. Assessment of the progress toward fulfilling the institution's research and service missions must recognize this quality.

Academic research at UNCW is organized both institutionally and departmentally. Several components within the University operate for the express or major purpose of conducting academic research, such as the Center for Marine Science Research, the Center for Business and Economics Services, and the Center for Small Business Technology. The Office of Research Administration and the Research Committee of the Faculty Senate are examples of research organizations which identify and administer funding resources for research. There is apparently no formal overall planning of organized research at UNCW, nor is there any assessment of the cumulative effect of its organized research activities (other than keeping track of grant and contract applications and awards, and their dollar value). This is likely to be remedied by the oversight of the newly restructured University Planning Committee.

A diversity of methods for evaluating and assessing research effectiveness exists among the academic units in the College of Arts and Sciences, in the Schools of Business, Education, and Nursing, and in the Graduate School. This diversity is a strength. Such diversity is appropriate and, moreover, necessary in order for participants in the academic areas to.maintain research excellence in accordance with standards of their disciplines. However, different standards exist across the disciplines.

The uniform assessment method for all departments is the annual report, which is submitted to the deans by each unit. In preparation for the annual report, all full-time faculty prepare their own reports of activities, including academic research and scholarship. In the case of the College of Arts and Sciences, the annual report must include information on published, performed, or exhibited research; presentations to professional audiences; grants submitted and awarded; and service to the department, the college or University, the community, and professional organizations. Policy requires that the faculty report be reviewed by the appropriate unit head, and assessment is made of progress viewed in relation to the goals planned in the previous year. Further planning is then made in view of the assessment of past planning and the evaluation of its effectiveness. Unit heads prepare a written report on all faculty members to be used in their yearly review of merit; in some cases, interviews between faculty and unit heads supplement these reviews (for example, in Philosophy and Religion); in other cases, peer review by department members is an element of the unit head's assessment (for example, in Chemistry and Earth Sciences).

The faculty annual report and the unit annual report are specifically used for assessment and planning, and they assist in assessment and planning for reappointment, promotion, and tenure. These reports are a strength.

Planning was not reported as a specific, formal procedure for entering faculty. However, the campus interview of faculty job applicants is a possible forum for such planning efforts.

A diversity of methods exists in the various academic units for determining research effectiveness and for making appropriate planning efforts based on these determinations. A strength is that each unit requires conformity to a standard of excellence in academic research, in terms of both its quality and its quantity. However, a weakness is that this standard is not made clear in assessment and planning from the start of a faculty member's relationship with UNCW.

Therefore, the committee suggests that members of the units, including, but not limited to, administrative heads, conduct interviews with incoming faculty in which they outline the parameters for excellence in academic research within the disciplines. The units themselves must establish guidelines as to what they consider excellence in academic research within the disciplines as part of an ongoing effort to incorporate assessment and planning into research. Further, these communications should also emphasize the connection between assessment and the planning of future research activities. In other words, faculty should have a formally instituted means of being informed of criteria of assessment in order to plan, and departments should have a clearly understandable set of criteria to communicate to faculty. It is understood that criteria will differ from unit to unit and will also be in need of review and revision on a regular basis. The problem of making clear to new faculty members expectations for academic research lies in better communication of disciplinary standards, not in revision of the University's reappointment, promotion, and tenure guidelines.

In addition, there are two weaknesses in the relationship of academic units to the upper administration:

First, the faculty's perception of deficiencies in planning opportunities as a restilt of economic restrictions is evident from responses to the Self-Study Survey of Faculty. Only 22% of the faculty respondents agreed that budgets at UNCW are a result of a comprehensive planning process. Moreover, only 19% agreed that financial resources are adequate to support their research activities, and just 13% agreed that financial resources are adequate to support their continued professional development. Economic restrictions on productivity in research, inasmuch as they are effects of the national and state economies, are a part of life. However, the uncertain reliability of information concerning University funding hampers the planning of research activities (such as travel to conferences, purchase of equipment and supplies, and access to collections), which compounds the challenges to assessment inherent in academic research. Economic restrictions have a direct effect on assessment and planning. Insofar as faculty members are limited in the conferences they may attend, their opportunities for peer assessment are restricted. One cannot plan to publish a paper as effectively under these circumstances if one's paper has not been exposed to sufficient constructive criticism. The lack of reliable information about whether or not a University funding source will support research activity is a restriction on research planning, insofar as faculty must continuously make contingency plans about research activity, based on ignorance of whether or not a planned activity is even possible economically.

Second, faculty can be made better aware of expectations and of opportunities for professional development. Thirty-two percent of the faculty responding to the Self-Study Survey of Faculty disagreed (41% agreed) that the general tone and policies of the University are supportive of faculty members' professional development initiatives. Fifty-nine percent of the faculty respondents were unaware (only 23% were aware) of the criteria used by the University for granting a leave of absence for professional activities. Faculty were asked whether the expectations for teaching, scholarship, research, and service required for promotion and tenure are made clear. Forty percent of the faculty agreed (3 1 % disagreed) that the University makes its expectations clear, while 60% agreed (21% disagreed) that the expectations of their department are made clear. The perception of clearer departmental expectations supports the concept of departmental self-determination; however, uncertainty about the precise nature of the University's expectations is compounded by uncertainty of precisely how the University's expectations correspond to each department's expectations.

While it is a weakness that this uncertainty exists and that planning is restricted at the departmental level insofar as a system of second-guessing is no substitute for real knowledge of precise University aims, it is certainly the case that the strength of the UNCW Mission Statement is its breadth and flexibility and thus its suitability for a variety of departmental plans. In view of this strength, it is suggested that departments and units determine - and communicate - proportional expectations for their members.

Two additional suggestions are presented as general incentives for the advancement of research planning:

A substantial proportion (50%) of the faculty does not believe that UNCW provides sufficient opportunity for temporary reductions in teaching loads and other assignments to foster research. In view of UNCW's expectation of research from its faculty, creative opportunities to revitalize research should be developed and offered as part of the University's long-term planning efforts. University recognition of research efforts must be escalated at every level. For example, funding for the express purpose of research leaves, short courses, conferences, research equipment, and international, national, and regional conferences should be sought. The topmost levels of achievement must not be overlooked in planning efforts, and faculty obtaining research grants should be supported with job benefits so that the acceptance of an award does not lead to an economic penalty or an increase in workload. Funding for endowed chairs for outstanding researchers and scholars on campus should be sought. Such efforts would enhance recruitment of the nation's top scholars. In recognition of the fact that the weaknesses in research planning in general are caused by limitations of the state's budget for UNCW and the UNC system, it is further suggested that, in the capital-funding program to be adopted by the University Planning Committee, academic research be given a high priority. Obtaining non-state funding - that is, a source of funding independent from the vicissitudes of the state budget process - would insure the ability to plan to meet the University's research needs.

According to the Self-Study Survey of Faculty, fully 52% of the respondents believe that their department is not competitive for new faculty on the basis of teaching load, and 11% of the respondents identified teaching load as the single most important reason for faculty resignations in their department over the past five years.

This would be a step toward balancing the multitude of demands made upon the faculty and allow the pursuit of research to be considered a bona fide factor in planning.

As in the assessment and planning of research, service activities are universally evaluated through the unit and faculty annual reports. While the mission statements of UNCW and of the UNC system in general commit the University to public service, the term "service" is used to signify several categories of activities: (1) service to the community, (2) service to the University system, to UNCW, to the colleges, to the departments, and to students, and (3) service to professions in the individual academic disciplines. The first two categories may be said to give a "public" benefit in that the public is constituted by the regional or state community outside the University, and in the second, the "public" (state-supported) nature of the University, guarantees that any service within the institution is a "public" one. The third category is not "public" in any sense but is just as important an aspect of the duties and responsibilities of faculty as are the other two categories. These distinctions are not explicitly rendered in any planning document. Moreover, the term "public service" is used when "service" is intended, and vice versa, in a variety of documents.

The concept of "service" as a part of the University mission is understood, but its relation to "public service" is less clear with the exception of strongly serviceoriented organizations within the University. Examples are the Consortium for the Advancement of Public Education (CAPE), the Writing Place, Randall Library, the Science and Mathematics Education Center (which provides professional development opportunities for in-service middle school and high school teachers of science and mathematics in the region), Athletics, the four centers and institutes of the Cameron School of Business Administration, and the Office of Special Programs (now being reorganized as the Division for Public Service). While these service organizations evaluate and plan their service activities (for example, the Randall Library documents specific expectations for service not only for the library as a whole but for its individual faculty and staff members), the University lacks a campus-wide service evaluation and planning organization to coordinate University resources (for example, the particular abilities, knowledge, and availability of its faculty and staff) with public needs.

Additionally, it does not follow from the University's requirement for public service how any particular faculty member must perform service or the amount of time and effort to be expended in this endeavor. It is possible for the University to meet its cornmitment to service in a plurality of ways. The collective accomplishments of the University community meet this commitment; however, it is difficult to ascertain how individuals meet this commitment.

A strength of the University is the flexibility allowed by its mission statement toward an array of creative and innovative choices for service. A corresponding weakness is that institutional expectations of individual faculty members remain unclear, especially as they concern the evaluation of faculty for merit awards, reappointment, promotion, and tenure. By contrast, a strength of individual departments is that their expectations are generaUy more clear and there is across the spectrum of departments a wide variety of interpretations of faculty service roles.

It is therefore suggested that "service" be clearly defined not only in terms of "public service" (categories 1 and 2 on page 15 1) but also in terms of service to the professions as well (category 3).

Insofar as the autonomy and self-determination of individual faculty are necessary for creativity and innovative effort in designing and carrying out service activities, it is suggested that faculty be encouraged to develop plans for their own service activities.

Further, it is suggested that the new Division for Public Service make the following contribution:

INSTITUTIONAL RESEARCH (3.2)

The Office of Institutional Research (OIR) has served UNCW well since early in the existence of the University. OIR has matched the growth of the institution by providing an expanding number of services to numerous operating units, focused toward administrative, academic, and student affairs. If there is any concern, it is that in expanding their services they may have stretched themselves too thin. The staffing of OIR is barely adequate, and, as a result, there has not been time (or staff) to adequately document the details of their many activities. This situation will now be compounded as the Director of OIR begins to assume other official duties for UNCW (in April 1991, his title was augmented to become Assistant to the Chancellor for Planning/ Director of Institutional Research). It would seem important to the continued success of OIR that its staffing be given a high priority in the development of the University's budget and that some catalog of its existing procedures be developed, so that new staff can function without heavy dependence upon the acquired experience and knowledge of existing staff.

Another suggestion has to do with the oversight function that is meant to be provided by the Institutional Research Council, as described in the following section on the organization and function of institutional research. This oversight should be transferred to the University Planning Committee.

Finally, from the section below on the evaluation of the institutional-research function, one can see that, although OIR does provide valuable services to many University constituents, those constituents - as persons - are a minority at UNCW, and the majority of faculty are unaware of the role of OIR on campus. Currently every semester each member of the faculty receives a publication giving many vital statistics about the University known as the "Factsheet." Perhaps the cover of the "Factsheet" could be used to publicize what OIR does and the services it can provide, of which the "Factsheet" itself is a very modest example.

Organization and Function of Institutional Research at UNCW

The Office of Institutional Research was established in the mid-1960's. Since 1973, it has occupied a variety of facilities but has always been located near the University's computer center. The staff of OIR currently includes the Director, the Assistant Director, a secretary, and a statistical assistant, as well as student assistants and other temporary employees. One other staff position, a data coordinator, is presently (at the time of the Self-Study examination) unfilled.

At present the Institutional Research Advisory Council is chaired by the OIR Director and includes the Assistant Director and representatives from the Divisions of Academic Affairs, Student Affairs, Business Affairs, and University Advancement. The purpose of the Council is to make recommendations to the Director in: (1) determining the University's needs for institutional research; (2) establishing and maintaining lines of communication where the need for OIR services crosses organizational lines; (3) planning and scheduling institutional-research activities; (4) using available OIR resources and staff; (5) developing operational guidelines for OIR activities; (6) enforcing institutional-research and -publication standards; and (7) responding to institutional-research needs as quickly as possible. Since the Council has not, in fact, been serving this role as intended, it would be more helpful to link the oversight of OIR to the activities of the University Planning Committee. - with which M is already intimately linked.

One of the distinguishing aspects of OIR's organization is that it reports directly to the Chancellor, providing a level of mobility that other organizational structures would not permit. This mobility permits OIR to work with a variety of University constituents. However, the lack of an institutional strategic-planning process has hindered the ability of OIR to anticipate new organizational needs and directions.

Resources have mostly been adequate for the past decade. OIR staff have also had direct access to both student and personnel databases without having to wait for Information Systems to generate relevant data files. Since 1986, OIR has become increasingly committed to microcomputer-based applications. The added equipment and software have enhanced their capabilities in the areas of word processing, spreadsheets, document scanning, color printing, computer modeling, and 3-D graphics.

During the life of OIR, activities have been planned, initiated, and occasionally terminated as the institution's focus has shifted. At present, OIR is actively engaged in providing a number of institutional-research services for both external and on-campus use; examples include monitoring credit-hour production, conducting surveys, assessing departmental curricular programs, and providing data for Faculty Senate activities. Moreover, OIR also provides services to other University constituents, such as the General College Advising Center, the Office of Minority Affairs, and several offices within the Division of Student Affairs.

Use of Institutional Research

As cited in the preceding section, OIR provides useful services to many operating units, both externally and on campus. Other examples of its output include: data on student admissions, enrollments, and retention; analysis of credithour production; longitudinal assessments of educational programs for many departments, such as Biological Sciences, English, HPER, and Mathematical Sciences; data for Faculty Senate elections and committee selection; planning and assessment data for several Student Affairs offices, including the Dean of Students, the Health and Wellness Program, and Residential Life; an annual analysis of faculty salaries; data for various administrative offices, for planning and evaluation purposes; data for individual members of the faculty, as material for their research; and the University "Factsheet," produced every semester.

Although the level of activity of OIR is impressive, it is important to discover whether the various units of the University do, in fact, make use of these data. So far as the administration is concerned, the answer clearly seems to be yes. In addition to regular reporting for the administration, OIR fields a large number of special requests every month, some from the Chancellor himself Among examples within the Division of Academic Affairs are the following:

• The General College Advising Center uses routine and special reports to evaluate existing programs and to plan new programs. Examples of these programs are minority retention, the Freshman Seminar, and comparisons of retention over a five-year period.
• The Department of Biological Sciences uses data from OIR to predict course enrollments, to plan space allocation, and to analyze salaries.
• The Department of English uses OIR data to schedule courses for nontraditional students.
• The Department of Mathematical Sciences depends upon OIR to help validate the placement examination given to all incoming students and to compare the performance of transfer students with regular students. The department also relies on OIR to identify candidates for membership in Pi Mu Epsilon, the national mathematics honorary society, and in Upsilon Pi Epsilon, the computer-science honorary society. The department plans to call upon OIR in following up on data from the Self-Study surveys - namely, to assess consequences of the expressed need for computer literacy as a Basic Studies requirement.

Investigation of satisfaction on the part of individual users of OIR services exposes two inconsistent points of view. On the one hand, according to the results of the Self-Study Survey of Faculty, about two thirds of the faculty are unaware of how OIR reports are used to improve operations in the University ("don't know" responses range from 59% to 69% in eight categories). On the other hand, interviews with persons who have used the services of OIR indicate that they are uniformly very positive about the value of those services.

Evaluation of the Institutional Research Function Itself

Evaluation of OIR itself has been largely an informal process to this point. One instrument has been the annual "Faculty Perception of Administrator Performance." For another, the Director has had one-on-one interviews with department chairs to determine what types of information would be most helpful and to obtain feedback on the services OIR was providing. In the future, this interview process will be expanded, and a log of needs and suggestions will be maintained. This log will be used in internal discussions of both office and staff performance. As another instance of informal self-evaluation, OIR has monitored requests for services, the timeliness of its responses to these requests, and the quality of the responses. In the future, these data will be kept more formally in a log. Beyond the logging activities just cited, OIR plans to implement the following additional self-assessment measures:

1. A needs-assessment survey to be administered to appropriate members of the campus community to determine future needs; results will be used in setting goals and determining resource needs. .
2. A user-satisfaction survey to be administered to members of the campus community who have used the services of OIR; results will be used to measure satisfaction with services provided.
3. An office- and staff-performance evaluation survey to be conducted periodically; results will be incorporated in the planning and assessment process.
4. External evaluation of OIR to be conducted by an external evaluator beginning in the fall of 1992 and every five years thereafter; this external evaluator will also be expected to make recommendations for correcting any deficiencies noted in the operations of OIR.

Overall OIR seems to be doing an excellent job. Implementation of the self assessment plan described above will further strengthen the institutional-research function at UNCW.

OIR should implement its proposed assessment plan.

1. The new planning process described in the recently adopted planning document ("University of North Carolina at Wilmington Long-Range Planning") must be implemented immediately. In particular, each of the University's divisions must immediately establish broad-based planning committees as required by the Long-Range Planning document.

1. The University Planning Committee should develop means to provide departments, and other units beneath the divisional level, with plans adopted for the institution, including the financial and demographic data upon which they are based, and upon which successful evaluation depends.
2. The Evaluation Committee of the Faculty Senate should revise the current instrument used for obtaining faculty perceptions of administrative performance to allow more specific evaluation of the performance of additional individual members of the administration, including the Vice Chancellors of Business Affairs, University Advancement, and Student Affairs and such significant officers as the Registrar, and the Directors of Undergraduate Admissions, the Randall Library, and Computing and Information Systems.
3. The University Planning Committee should insure that better methods are developed to inform faculty of the planning process at the University level, and of the outcomes of that process.
4. Means should be developed to increase communication between departments offering service courses and those departments receiving those services, especially to ensure that feedback on the effectiveness of the service courses is available to the service department. Because this communication must include the College of Arts and Sciences and the Schools, the Office of the Provost should be responsible for establishing the procedures to accomplish this.
5. Steps should be taken to educate the University community about the variety of methods for and the benefits of assessment. This could be done through on-campus teleconferences, workshops, or newsletters. The Assistant to the Chancellor for Planning should be responsible for organizing this effort.
6. The University Planning Committee should assure that administrators maintain an explicit and publicized expectation that requests for funding, staffing, and program changes be justified by prior evaluation and assessment.
7. Department chairs should assure that applicants for faculty positions be apprised of departmental criteria for assessment of research and that research planning be discussed at the time of the campus interview.
8. Each department should establish guidelines for what is to be considered excellence in research within the discipline involved. Such guidelines would facilitate research planning and evaluation and would provide clear expectations and goals for both current and prospective faculty members. The formulation and application of research-assessment criteria should involve faculty at all levels of seniority in each department.
9. Administrators responsible for determining funding for research should develop a specific, formal method for informing faculty, in a timely manner, of the availability of funds. Faculty should be informed, at the time of the annual review, of the connection between their individual research planning and the budget. The limitations of funding should be considered as a factor in the evaluation of individual research performance.
10. Academic departments and units should determine proportional expectations for teaching, scholarship, research, and service for their members, and these expectations should be clearly communicated to their faculty.
11. As part of its long-term planning efforts, the University should develop and offer creative and major opportunities to enhance faculty research, including increased recognition of research efforts and assignment of a high priority to academic research in capital funding programs. Responsibility for the implementation of this suggestion should be shared by the Provost and the University Planning Committee.
12. The Faculty Senate should investigate the possibility of a "faculty workload policy" which would recognize research, service, and advising responsibilities of the faculty, as well as teaching load, all as parts of a faculty member's job description.
13. "Service" should be clearly defined by the University not only in terms of "public service" (to the University and to the community) but in terms of service to the professional fields as well.
14. Faculty should be encouraged by their departments to develop plans, corresponding to their specific talents, abilities, and professional training, for their own service activities, and the service component of the assessment of individual faculty should be based on these plans.
15. The Division for Public Service should develop and disseminate listings of

1. public-service opportunities of possible interest to the faculty and
2. faculty members' professional expertise which might be of interest to the community.