Response to John Rosenberg
Dear Professor Rosenberg,
Members of the Deakin Branch of the NTEU have asked that I deliver their feedback to your recent document Improving Teaching and Learning at Deakin through the Student Evaluation of Teaching and Units.
The NTEU has received a significant number of responses, which are reproduced, largely unedited (except to remove identifying information), below.
The responses from staff show an overwhelming dissatisfaction with SETU. In fact, I received no wholly positive responses, as you can see from the material reproduced below. (One response, “Seems like a good idea”, appears to be positive, but as they were the only words in that particular response, I have not included it in the responses below).
The NTEU has a number of concerns about the SETU system as proposed in your document, and these are outlined below:
1. In your paper you claim that there has been considerable consultation over the new SETU instrument. Given our feedback has found universal dissatisfaction with it, it is difficult to see how this consultation claim can be substantiated. Certainly I am not aware of any staff actually engaged in teaching who have been consulted. If the University does intend to establish a proper student feedback system, it would be appropriate to engage in serious consultation.
2. We note your admission that the previous SETU measure was ‘far from perfect’. Yet results of these surveys have been and are still being used in staff members’ PPRs, promotion applications and probation assessments. We believe that, given your admissions, this is completely inappropriate, and the practice should be stopped immediately. The use of SETUs in PPRs, promotions and probation processes should be suspended until an agreed-upon SETU measure is established. I remind you that the Enterprise Bargaining Agreement specifies that the contents of the PPR needs to be agreed by both the staff member and his or her supervisor.
3. Your paper begins with a discussion of Deakin’s low CEQ ranking. Despite the wide range of issues that could affect the measurement of learning outcomes at the university – poor staff-student ratios, excessive workloads, IT problems, reductions in student services, reductions in budgets in Faculties and Divisions etc – you choose to concentrate on the efforts of individual academic staff in a way that most staff find intimidating, ineffective and unfair. The NTEU would like to see a clear plan that deals with all these other influences on teaching and learning at the same time as a measure of teaching performance is developed.
4. The attitude to SETU varies dramatically across Faculties. The Dean of Business and Law, for instance, has announced that the required score to avoid investigation is 3.8 or 4.0. Yet, in your paper you give a score of 3.5. How can this be justified? This means that the promotion and career opportunities vary across faculties based on the whim of particular Deans. The University must develop a policy on SETUs that is enforced across all Faculties in a uniform way. In particular, the Faculty of Business and Law must be brought into line with other Faculties and with the policies outlined in your paper.
The NTEU would like you to establish a committee involving “coal face” academics – not Heads of Schools or Deans – to review the whole SETU process and devise a teaching and learning assessment scheme that meets with a higher level of approval from staff and actually makes some semblance of assessing all the variables that contribute to good teaching – not just a “blunt instrument” that is used to club individual staff with.
The number of concerns raised by staff is very high. I do not want to list all of them here. I urge you to read the following pages to get a real sense of what staff think about SETU.
Yours sincerely,
Dr Colin Long
President, NTEU Deakin Branch
STAFF FEEDBACK BEGINS (the submissions are divided by a dotted line)
The proposed form and the accompanying letter (dated 4 October 2006) from John Rosenberg both assume that Deakin’s ‘poor’ results in the Learning and Teaching Performance Fund are caused by poor teaching. No other cause is canvassed, let alone assessed. Below, I have commented on each document separately.
The proposed SETU form.
This addresses none of the problems with the present form that I have identified after consultation with colleagues (see attached, ‘”Some problems with the Arts Faculty’s current Student Evaluation (SETU) system.” June 2006’.) The proposed form repeats the faults of the present one, but I won’t address each of those faults, as this, too, would be repetitious.
The proposed form’s foundational assumption is that students’ learning is the responsibility solely of academic staff. (The only exception is Q8, which asks whether the technologies ‘performed satisfactorily’.) Every question assumes that learning is a process of transmitting knowledge ‘from one to many’ and, therefore, that any fault in the learning must be the fault of the transmitter (the academic staff). Anyone who’s ever taught knows that this model doesn’t work. Unless students participate actively, they won’t learn anything. This proposed form, however, pays no attention to the student’s role in learning. Like its predecessor, it presents learning as equivalent to shopping at a supermarket, in which the only role of ‘the consumer’ is to take items off the shelf and complain if something’s not in stock.
The proposed form introduces some different faults.
1. The ‘NA’ response is not explained at the start of the survey.
2. The survey refers throughout to ‘questions’, yet it consists of statements. This sort of basic error reduces the instrument’s credibility.
3. Q3 ‘The workload in this unit was manageable’. This assumes that workload is the unit chair’s responsibility, but in the Arts Faculty, at least, the workload is prescribed by Faculty Board, so it’s inappropriate to ask this question. Again, this sort of basic error reduces the instrument’s credibility.
4. Q9 ‘The on-line teaching and (sic) resources in this unit enhanced my learning experience.’ Whatever a ‘learning experience’ is, you can only assess whether it’s been ‘enhanced’ by doing the same unit twice - one with the on-line resources and once without. This simple fact of scientific method renders this question meaningless.
5. p8. ‘Please answer the following additional questions …’. These statements aren’t even numbered, so they can’t be identified in analysis.
(i) What is the distinction between a ‘tutorial’ and a ‘workshop’? Is there evidence that all students will recognise this distinction?
(ii) What is the distinction between a ‘subject matter’ and ‘discipline’? Is there evidence that all students will recognise this distinction?
(iii) Tutorials aren’t ‘taught’, in that mostly they are occasions when students demonstrate their degree of understanding of materials presented to them recently (e.g. in that week’s lecture). This error is indicative of the ‘shopping’ view of learning that underpins the present form and is repeated in the letter from Prof. Rosenberg that accompanies this form.
The accompanying letter (dated 4 October 2006) from John Rosenberg
This is marked by a combination of assertion and a lack of understanding of how surveys work.
1. Page 1 Para 2. The letter states that ‘In 2005 Deakin ranked 21st overall …’, but it doesn’t state how many were in the ranking. 21st out of 100 isn’t bad, 21st out of 21 is.
2. Page 1 Para 3, beginning, ‘There is no doubt …’ is a series of assertions. No evidence is provided or even cited. We fail students for such laxity.
3. Page 1 Para 6 states that, ‘It was clear in 2004 that the survey instrument was far from perfect. However, it was agreed that the format would not be changed for at least two years to allow longitudinal data to be collected and to allow comparisons across years.’ This means that having realised that the instrument was faulty, management nonetheless continued to use it to collect faulty data and compounded their error by attempting to make comparisons within this faulty data, thus compounding its faults.
4. Page 1 Para 7 lists several ‘problems’ with the existing form. How were these ‘problems’ identified? Who identified them? Were any more ‘problems’ identified? In other words: where is the evidence that anyone other than Prof. Rosenberg has been consulted about the form? The bland statement that, ‘The revised survey was developed in consultation with the Associate Deans (Teaching) from each faculty who, in turn, sought feedback from the staff of their faculty’ (page 2 para 2) is insufficient assurance that there has been broad consultation, especially with staff with expertise in survey design and analysis, and that the results of those consultations have been acted upon. Has the new form been designed by anyone associated with the previous ‘far from perfect’ form?
5. Page 2 Para 7 onwards (‘Response rates) relies on assertion to criticise academic staff. ‘We almost never tell them what we do with the results’: where’s the evidence? ‘Most academic staff do use the survey results to guide improvements and alterations to their units’: where’s the evidence? (Several colleagues are so critical of the whole SETU exercise that they approach the results with extreme caution and certainly don’t immediately translate the results into changes in their practice.) This whole section assumes that students are always right in their assessment of a unit and of staff associated with it and that staff’s views are irrelevant. ‘Changes that were made as a result of the last survey for (sic) the unit’ and ‘indicate how they have responded’ each assumes a simple equation between answers and changes, allowing no room for interpretation, let alone disagreement.
(Incidentally, the proposal to hand reminder cards to students taking exams will - if it works as intended - skew any results towards units with exams. Thus, the survey will be faulty before it is administered!)
6. Page 3 ‘Action as a result …’ refers to ‘the lack of evidence, in many cases, of concerted actions being taken as a result of the analysis of the SETU results.’ This statement again assumes that one can treat the results as the truth, which they manifestly aren’t; it contradicts the earlier statement that ‘most academic staff do use the survey results to guide improvements and alterations to their units’; and, ironically, both statements are marked by their ‘lack of evidence’!
Similar contradiction marks the next paragraph: ‘It is recognised that SETU is a blunt instrument … However, it does provide real feedback and I am convinced that it can provide a broad indication of teaching performance as perceived by students.’ If it’s a ‘blunt instrument’, how can you claim that it provides ‘real feedback’? While Prof. Rosenberg may be ‘convinced’ of the value of SETU, he provides no evidence - here or anywhere else - on which he bases his conviction and with which to convince anyone else. He admitted earlier that, ‘It was clear in 2004 that the survey instrument was far from perfect’, yet he is ‘convinced that it can provide a broad indication of teaching performance as perceived by students.’ Thus, while giving no evidence to support his conviction, Prof. Rosenberg ignores such evidence as does exist - that the SETU form is flawed.
7. The letter reinforces current policy to include the SETU results in the PPR process and buttresses it by including Deans and DVCs, yet when PPR was introduced we were told (and the NTEU was told formally) that it would NOT be used in any form of disciplinary process. It’s quite clear that academic staff will face still further demands from management that they prove their innocence; and equally clear that management will follow this line rather than accept their responsibility to introduce fair and transparent appeal processes.
Some problems with the Arts Faculty’s current Student Evaluation (SETU) system.
Student evaluations can be valuable to staff - and valued by us - if they're developed and administered appropriately. At their best, they can indicate whether and to what extent one's teaching meets the different learning styles of the students in our units. However, the current mandatory system is useless as a measuring instrument, yet management relies on it increasingly to measure staff ability, quality and value.
1. The current system ignores the differences in ENTRY SCORES between different courses and, within courses, between campuses. Instead, it is used in a uniform way across courses, units and campuses.
2. The current system fails to acknowledge the variation in the NUMBERS OF RESPONSES, presenting units' results in percentage terms. E.g. two units may have the same rate of response (e.g. 10%), yet in one unit this means sixty students and in the other it means just two students.
3. The current system is used by management on a semester-by-semester basis, which fails to acknowledge differences between successive COHORTS OF STUDENTS. I’ve seen widely differing responses by different cohorts of students to a unit whose content remained essentially the same over a period of years.
4. The current evaluation form fails to distinguish between TYPES OF STUDENT - especially on-campus and off-campus students. I've seen forms from off-campus students who have responded to all the questions about on-campus classes - rendering their responses invalid.
5. The current evaluation form fails to acknowledge that the RATE OF RESPONSE can vary enormously between years, between units, between campuses and between on-campus and off-campus students. Each variation makes it extremely hard to make valid and meaningful comparisons and contrasts between any two sets of data.
6. The current evaluation form lacks INTERNAL VALIDITY CHECKS. Consequently, each response to each question can only be taken at face value, because there are no other questions against which to check it. E.g. one question might ask, "Did the Lecturer prepare well for each class?", while another might ask, "Did the Lecturer know her/his subject well?" Both questions concern the same issue, so if a student responded differently to each question, the validity of their responses would be in doubt.
7. The current evaluation form fails to acknowledge RESPONDENTS’ INVOLVEMENT with the unit. This determines the experiential basis - and, therefore, the validity - of their comments. E.g. praise or criticism from a student who had extensive experience of a unit through participating extensively in classes and/or in online discussions should carry greater weight than if it comes from a student who had little experience of the unit because they'd participated in it only rarely.
8. The current evaluation form assumes that students' responses are RATIONAL and DISINTERESTED. However, a student who does poorly in a unit doesn't respond rationally and disinterestedly; instead, s/he resents what s/he sees as a hindrance to her/his job, career and wealth goals. This is happening because successive governments (with the willing compliance of Vice Chancellors and senior university managers) have led students to regard university as a stepping-stone to a job, career and wealth and have pressured universities to become more 'vocational'. Resentment at poor results increases with each increase in fees - "I've paid my fee, where's my degree?"
9. The current evaluation form EQUATES EDUCATION and SHOPPING. It assumes that you can ask university students about the quality of their education in the same way that you can ask supermarket customers about the quality of their shopping. However, education isn't a matter of acquiring things (knowledge) off shelves; students aren't customers in knowledge supermarkets; and in education, the customer isn't always right - especially when s/he assesses the value of the education on offer. This ISN'T a defence of elitism and it DOESN'T assume that students' views are valueless. Instead, it says that the quality of education can't be assessed accurately solely by asking the opinions of people with a vested interest in the outcomes of that assessment.
In summary, the DVC’s argument appears to be:
- Teaching excellence=high SETU score=high CEQ and similar ratings
- Higher SETU scores will lead to an enhanced reputation of Deakin and in tun an increase in student numbers.
- The emergent corporate university manager therefore needs to monitor SETU scores of academics and this will lead to an increase of SETU and CEQ scores and ultimately increased student numbers.
It is my view that this argument is flawed and demonstrates not only a needlessly authoritarian management approach but an approach that is derelict to its duty to the university. I will deal with each aspect of the DVC’s argument in turn:
Is teaching excellence related to a high SETU score?
Quite plainly, a high SETU score is not and it cannot be necessarily related to teaching excellence. A student is neither in a position nor is qualified to make a judgement on teaching excellence in a particular area. At best, a student can answer that the teaching was appealing/engaging/satisfying/challenging and so on. We would not judge the quality of art presented in a gallery based solely on a public survey of those who visited the gallery. We may be able to measure how much the student liked the unit and the lecturer but we cannot use the data to comment on whether the teaching was excellent.
It may be countered that teacher excellence includes engendering some level of enthusiasm, satisfaction etc. in the student by the teacher. Thus, while something like the SETU may be in principle an incomplete measure, it may capture some critical aspect of quality. Such an approach assumes either homogeneity of the circumstances of the entire student class or a Herculean effort on behalf of the teacher to reach all students. The matter is that within a classroom there is generally a heterogeneity of expectations, commitments, abilities and circumstances of students. An excellent teacher might for example take the top third students in terms of ability and turn them into excellent scholars some of whom go on to be leaders in their field whilst alienating or at least not ‘satisfying’ the less talented, less able or less committed students. On the other hand, the teacher may focus their attention on the latter, larger group while failing to get the most out of their better students. Sheer demography suggests that the latter teacher will gain better SETU scores than the former teacher – and teachers have already began to do these numbers. What SETU will most likely end up measuring is the ability to cajole, coax and assuage the majority rather than challenge the minority.
Such a measure is a recipe for mediocrity in student outcomes. Furthermore, where we can equate student satisfaction to SETU we still cannot be sure that a relationship between student satisfaction and excellence obtains. A popular lecturer may not necessarily be an excellent lecturer. Conversely a lecturer who is demanding and unpopular in one year may set up students to do well in subsequent years. I am sure that we can all think of lecturers we had in the past who we thought highly of who turned out to be charlatans or to have feet of clay, while other lecturers whom we found uninspiring or difficult we later realise laid down some very important foundations.
The whole approach of predicating excellence on student satisfaction suggests that all parts of study and all disciplines are more or less as accessible and as fun to learn and to teach as each other. Sorry, but every discipline has areas that are just plain difficult. Woe betides the poor lecturer whose lot it is to teach them. In my experience at least, Socratic methods were more fun in class than following a lecturer develop a mathematical theorem which was often a fraught or dull experience. It was dull when I saw the steps, fraught when I couldn’t. I loved both subjects. I remember the names of my philosopher lecturers and tutors but only a couple from mathematics. Past some basic initial mathematical intuition I can’t imagine how one would teach mathematics socratically. Cross discipline comparison should not be attempted. However, they are and will be: all units will be required to reach the same target. The same argument applies to units within disciplines. This puts pressure on staff teaching difficult, unpopular areas to jazz up the show and dumb down the content or face the ‘counselling’.
One might suggest we have the ‘tragedy of the discipline’ or’ tragedy of the degree’ in the making where everyone looks after their SETU but the intellectual commons is lost.
Is teaching excellence related to a high CEQ score?
For similar reasons, any relationship between teaching excellence, or more accurately high SETU, and high CEQ is likely to be weak. There is much anecdotal evidence to suggest that course experience is more likely to be related to quality of campus life, the standing of the degree in the community and the employment they get and the relationship they formed with one or two teachers that inspired them, than it will be to SETU.
In the market place for students it seems likely that were CEQ scores to improve then student numbers would also increase. It would undoubtedly be used for marketing to the students. “Be content with Deakin” might be an unusually accurate future advertising slogan. It is clear, however, from what the DVC states that the prime responsibility for increasing CEQ scores lies with the lecturer in front of the class. It does not appear to lie at all in the provision of more and better resources to students: such as increased student services, lower class numbers, increased tutorials, better facilities and fresh and energised staff who are supported by their senior colleagues. Above all, it is not his or his immediate colleagues’ responsibility – it is ours.
This shifting of responsibility is a management choice. It is not a logical or empirical inevitability. Faced with the choice between increasing resources to better student experience and sheeting responsibility to academics, the Executive has made a resounding decision.
Monitoring is the only or best way to achieve higher SETU scores
This authoritarian approach is made clear with the final part of the argument – that the monitoring of SETU of individual academics will lead to higher SETU scores. This (unfortunately for teaching quality) may be true, but it is not the only and probably not the best way to raise SETU scores. Giving staff more time, decreasing the rigid timelines for study guide preparation, relaxing the link between off and on campus so that staff can deliver to the strengths of the media, providing pep talks, tricks and showmanship to increase satisfaction, and above all consulting those who actually do have contact with students: talking up staff not talking down staff are also likely to be effective. I am sure there are many more and better ideas too.
The management choice is, however, monitoring, surveillance and ‘counselling’ sessions for those with low SETU scores. One wonders whether we are constructing a re-education camp, not a university. Academic staff may be some of the brightest people around who have committed themselves to the university but they are treated as if the only thing that will work is to monitor and hound. No reason is given as to why this level of supervision and monitoring is a necessity or why it will not prove counterproductive. Faced with the choice of the carrot and the stick, between treating the lecturing staff as professionals and treating them as subordinate employees unworthy of trust, the latter is chosen.
The DVC suggests that all these measures such as SETU are questionable; however he fails to discuss any policy to counter the use or interpretation of such questionable measures. He appears to have resigned to the view that such measures will dominate student choice and university reputation. I wonder whether such defeatism and cynicism is the appropriate attitude for a DVC. The DVC’s duty is to defend the university and to not do so is to be a Quisling. For what we see here is the autonomy of the university being destroyed. Perception of students, however erroneous, appears to be dictating teaching policy and practice rather than disciplinary and vocational needs.
The foregoing is not to conclude that student views are not important or that they should not be considered. Indeed, I would suggest that they should have a greater role in the constitution of the university: a democratic, collegial university. Our senior colleagues now see themselves as ‘corporate managers’. What we have is the ongoing takeover of the reins of the university by a corporatism that views itself as bosses or ‘managers’ not senior colleagues and as corporate managers they quite happily and cynically apply different rules to themselves than to their emerging ‘subordinates’. Students are shunted out of the university and have become consumers of education, their participatory role in the direction and political life of the university reduced to customer feedback forms such as the SETU. Academics are ‘service deliverers’ and are now increasingly monitored and supervised – student feed back is just one obvious means of asserting control. Our senior colleagues are not asking us to monitor their decisions; there is no provision of an equivalent survey where we tick off such items as “the DVC has shown courage and vision” and where we evaluate and monitor their actions and counsel them when their performance is in our opinion poor. The pigs really have taken over control of the farm. Did I hear the echo of Squealer: “four legs good, two legs better” in that DVC’s statement?
One of the big problems with SETU is that the University reports and relies on MEAN scores. But all it takes to produce an appreciably lower mean is for 10% of students who may be disgruntled (e.g., because they didn't do well on the mid-semester assignment) to give a very low rating.
It would be much much more sensible to report the PERCENTAGES who (a) gave a disapproval rating (b) an approval rating; and (c) a neutral rating.
SETUs may be useful in some sense, but they are at best a general guide, and contain within them many problems.
A key problem is that areas of concern to students, such as poor turn around times of assignments (by the mailroom or assignment tracking), or poor production of reading material, are allocated to the teacher. This is clearly inappropriate, Indeed, I have had students saying that while they recognise the teacher is not responsible for all facets of the university, they are the first point of contact and reference. Moreover, fee paying students expect all aspects of 'customer delivery' to be centralised around the teacher, who is then held responsible for such failures.
It is also worth noting that many students who do fill in the surveys are those who are either very happy or very unhappy with their courses. As a consequence, the results are sometimes skewed, particularly with small returns from large cohorts.
I am further disturbed by Professor Rosenberg's suggestion that 'every DSO site now has a mandatory section indicating changes that were made as a result of the last survey for the unit'. This implies that the surveys provide such detailed information, that the information provided is consistent across students, that one-off factors must be included (even though they are redundent as one-offs), and that there must be an annual update of all courses. While I and my team regularly update courses, including annually, a mandatory requirement to do so makes a mockery of the existing five-year rule.
The following point about referring to how teachers respond to surveys in the first following lecture also seems highly inappropriate, especially given that there will necessarily be a completely new cohort of students who won't have any idea what is being referred to. It is further complicated by DSO teaching, where an explanation must be given to students about changes before they even start the course.
In terms of 'concrete action' the Faculty of Arts has an active assessment program in place, and each semester addresses problematic SETUs with concrete changes. To suggest this doesn’t happen is just plain wrong.
Finally, Professor Rosenberg notes that SETUs are a 'blunt instrument'. Indeed they are, which is why care needs to be taken in responding to them, and in drawing 'mandatory' conclusions from them. And I offer this as one who acutely responds to SETUs, including annualised course updates.
1. It focuses, almost exclusively, on "on campus" "Face to Face" teaching. I teach an Online Unit and many of the questions can only be responded to as N/A.
2. I like that it identifies and enables the measurement of individuals - many others will not (but the reality is that they have been dragging "my" SETU down for years).
Otherwise it is better than we had.
I would like to see a firm policy statement from the University (backed up by senior management stopping Deans misusing the SETU results) as to the use of these individual comments on teachers, who will see them and each individual teacher’s right to see their own comments. As you know, these comments can be destructive and biased against an individual, so the use that is made of them is a major concern. The students can comment on teachers individually but you can bet that it will not always been constructively, - we mark their work anonymously (rightly so, we should not let personalities affect the marking) and will not be able to see who has made the comments. You could ask if the University will pay for and provide counselling for
staff when they receive the comments and have to go through them with their HOS or Dean and justify why they may have received a negative comment!
My SETU results are generally pretty good, and our HOS handles SETU well, but I still recognise a flawed process when I see one.
There does seem to be some improvements in the SETU survey
but they have still not overcome the major deficiencies in the
instrument:
1. The sample is self-selective and not random. So the results
cannot be generalised. Moreover, students who don't attend
classes or participate in DSO (and hence don't have a clue
how well the unit is taught) are able to fill out the
survey.
2. The University insists on using the mean of the responses in
their analysis. Given that the data is from a likert scale they
should really be calculating the median and/or "percentage
in general agreement/disagreement". Unless they can justify
the use of the mean (which I don't believe they can) they
should not be using it.
The question “The workload for this unit was manageable” – what incentive is there for students to answer this honestly? How are students to interpret “manageable”? If an individual finds a unit too difficult, is that a problem of the unit or the student? Pass rates should be able to indicate if the course is too strenuous.
The question “The unit was well taught” - This question should be rephrased again later in the survey to validate that the Student is actually fairly evaluating the unit. For example: “The instructional component of the course was?” - Nowhere do I see questions being used to back up prior questions.
Many of us are wondering how students can reasonably evaluate a unit when they do not attend lectures/classes.
This year many staff have reported astonishingly low attendance and an unprecedented number of applications for extensions.
I get a large number of "the question is not clear" comments when I spend a lot of time trying to explain IN CLASS. I guess I am wondering in all this how the University views attendance by students these days ....and how that intersects with teaching evaluations. I find it very distressing to have no "right of reply" for the perceptions of students who did not even begin to engage with the unit. It can't be just me being "boring" as attendance was low from lecture 1 this semester.
I think student attendance is a major issue the University needs to address. If the unit is demonstrably Off-Campus teaching strategies are in place........but it is pretty hard to teach an ON-CAMPUS unit if we never see them ! I have had a request for special consideration from a student today asking for consideration for the whole semester (with medical certificate) and 3 assessment tasks completed by class so far who I have never seen in a class at all and who has never asked for an extension for anything. How can they reasonably evaluate the ON-CAMPUS unit ?
I am sure you have heard all this before. The University seems to be very slow about recognizing change in attendance patterns and student expectation of one-to-one responses by email to individually catch up....the old "I have missed three weeks of class.......did I miss anything" approach.
I have always enjoyed teaching, I get high scores and have won Teaching Excellence Awards - but I think there has been a big change in students and staff are going to have a pretty hard time as a result of student expectation. We don't seem to expect any commitment by students - they can comment as they seem fit with no need to justify.
Are there to be any instructions to students as to their responsibility in filling in the anonymous form? Any instructions re the nature of constructive feedback as opposed to personal comments about individuals?
The new SETU questionnaire is certainly an improvement over the ones used in the past.
My primary concern is still in the use of statistical instruments to analyse the collected data in view of incredibly low response rates. I can see two possible solutions to this problem:
1) Collect the survey data in class-time (by suitable and independent person);
2) Release the unit marks to students on the condition that the on-line survey has been completed (even allowing the response "I refuse to fill in this questionnaire").
Many students would be willing to fill in the survey but this is simply inconvenient for them.
My other concern is that in most cases, teaching staff will be unaware of developing problems until the end of semester until their unit is surveyed and they receive bad feedback. As is the case in some schools, the staff rotation between units is quite high and the organisational memory is incredibly low (e.g. in my School). My suggestion in dealing with this problem is to formally provide additional feedback loops to collect, assess and act upon the emerging trends in student views and opinions. In other universities where I had worked before joining Deakin, the following practices were adopted:
A) Half-way through the semester a "fix-it" questionnaire was given to students to determine any emerging problems, students' wishes and their advice on fixing the problems. The identified problems at this early stage were "fixed" before the end of year survey was served.
B) Student representatives were elected for each year / unit. Their responsibility was to act as mediators between the students and teaching staff, take part in the unit planning and in dissemination of information to students. Again, the unit representatives' continuous involvement allowed staff to react to any emerging concerns.
C) Focus groups of students drawn from the units, years or courses were set up to establish any problems and their possible solutions. Students were consulted on various issues ranging from deadlines, structure of projects, dealing with problems, meeting expectations, etc.
D) Lecturers would report findings of A-B-C in their lectures and advise students on the action taken (with full explanation).
There are many other ways of engaging students in this process. In either way we achieve the following objectives:
i) Students feel they are treated seriously and their opinions are valued.
ii) Individual students know what are the other students' concerns and how they are being addressed by teaching staff.
iii) Teaching staff have opportunity to address students' concerns before the final student feedback is collected. Knowing that their problems are being resolved, students give positive feedback.
How can SETU be used to identify quality teaching at the
top end? If it can't be used for that (as I'm sure most
academics will agree), how can it be used to identify concerns at
the bottom end? Is this really about teaching quality or about
course popularity and marketing?
I think this has the potential to misdirect attention away from
appropriate areas, and create a culture of fear and stress,
possibly lowering teaching standards as staff drastically try to
ensure they get good scores.
I have grave concerns with such evaluations. It appears there is a disconnect between the personal comments, notes, emails of thanks and invitations to attend award ceremonies as a valued teacher that I receive from many of my students and the more uneven formal data perused by my leaders.
I can only assume that students who are unhappy with a result or have failed are the ones motivated to vent their spleen via the on-line SETU, cf with personal communications as mentioned. Of course, there are those not motivated to do either.
My experience has been that the tick the box questions give
you nothing to base decisions about changes to the unit. The
open-ended questions tend to produce a completely polarised set
of returns--one half (I suspect those students who have done
well) think the unit is fantastic; the other half (who have not
done well?) think it stinks! Consequently, I would find it
difficult to refer to changes based on last year's SETU
results.
Clearly, there are big issues re the new sections which allow
students to (anonymously presumably?) make potentially scathing
comments about individual staff, these people subsequently having
to defend themselves.
The most obvious thing is that they are called EVALUATIONS. At some other universities they are called "Student Perceptions of Teaching", which is must more accurate. Students do not have enough information or knowledge of the discipline area, of pedagogy, of the expectation of professional bodies, or of community/employer expectations to "evaluate" teaching. A few years after they graduate, they may be able to look back and evaluate.
1. It appals me that students are given the right to say anything they like about staff members without giving their names. Students who wish to write particularly injurious comments about a unit should be expected to give their names.
2. It is mentioned that Staff names are not included on the web but it is not difficult to trace a unit to a unit chair. Even heads of schools and co-ordinators receiving sometimes nasty and mischievous comments by failed or difficult students is a potentially professionally undermining situation.
3. Most of my colleagues are upset about the way SETU is being handled. As someone in a position to know about this, I speak with the experience of dealing with devastated staff members. I would like to know why students are given an opportunity to undermine (as well as praise of course) staff members without proffering their names and why there is little protection of staff in this instance. I am wondering if this is legal.
4. Despite my attempt at several fac.exec. meetings to point out that the SETU is filled in by a minority of students and that one of the reasons is that most students do not use the Deakin email system, and therefore do not get the pop-up reminders or access the relevant form, my insight here is ignored. I believe that it is a matter of urgency that all students know when and how to fill in these forms and that the net address is made available to all staff to inform students where they can fill in the data
5. Names of tutors put on the forms goes against principles of fairness if the students are not required to sign their complaints or criticisms. They should be responsible for what they think. Given that staff do not receive the results until after work is assessed there is no problem about assessments being 'vengeful' though I hasten to add I do not know one colleague who would mark down a student who they knew disapproved of him/her.
I feel very strongly about the above. As a person in a position of responsibility I have had to deal with some very upset staff this semester. I do not think they have deserved the comments written about them by students- particularly ones who, it seems, had not expressed their views throughout the unit in order that problems could be dealt with. I happen to always get excellent SETU scores so my criticism of this system is not dictated by personal experience. I am mostly angered by the way the students are encouraged here to have control over situations that they are not necessarily equipped to understand. I do not understand why the management wants its staff to be 'frightened' by students and totally vulnerable to their view of them. This is not a healthy environment in which learning might take place. It seems that this is a system in which academic staff are further undermined and stripped of a little more dignity.
This appears to be a real improvement over past forms of the
SETU. The university also seems to be adopting a more realistic
attitude to dealing with poor evaluations. I think the approach
of using PPRs and mainly dealing with issues within schools is
appropriate. As long as HOS and Deans behave themselves this
could be constructive and not punitive, of course, this is where
the union comes in, as some Faculties do have HOS who
'misbehave'.
Overall I think this is a much more rational approach and a
better survey instrument - I think we'll get better and more
meaningful results. I still have reservations about the potential
for bullying of staff by students through inappropriate comments
in the SETU. I have been told that while swearing could be
filtered out, the comments could not be edited to remove
inappropriate (racist, sexist etc) statements before distribution
to staff. (Here I am not talking about negative comments but
about offensive comments to do with race, sex or the personal
attributes of staff. For example, we have often had comments like
"all women lecturers are terrible and X is no exception). I
have been told that staff who were upset or offended by comments
could complain, and if students could be identified a warning
letter could be sent - using the university solicitor if
necessary. I think that most staff are unaware of this option and
I also think that we should be using it. Just like plagiarism,
word will get around among students that this kind of abusive
language is not OK. I think the Union could get involved here by
asking Alan Farley to clarify the procedure and advertise it to
staff.
On glancing through the narrative from John Rosenberg it is obvious that no one wants to address the issue of students whose assignments are failed (in many cases because the students joined the unit late/have no work experience/have no prior experience writing university-level essays /have poor maths skills/have limited English. Such students tend to feel very aggrieved at an assignment failure (or even a bare pass), and are more likely to respond to the survey than the average student. Some teachers get over this issue by handing back assignments after the evaluations, but that defeats the educational goals of having feedback in time for the students to do something about it. Nor does his discussion address the issue that if staff report students for plagiarism the same effect occurs (only the students are even more aggrieved and want to lash out).
Monash's Grad Business School has only just finished a very large survey of students who discontinued from their masters courses, and they used an unusual design protocol that chased up the students by phone up to 6 times. They found a distinct differences between those small minority who responded the first time (very negative responses) and the overall average responses (which were positive on average) once they had gone to the trouble of increasing the response rates to avoid this effect.
The issue could be fixed, at least for the oncampus students, by doing a paper-based survey in Week 12 or 13 as is still done at Monash where staff have insisted on it. At least that way we would get response rates of around 60-70% with more of the non-outlyers included, not 15-30% . Of course that costs money and shows up how inaccurate are the online surveys!
It is interesting that Deakin prides itself on the quality of its teaching but scores poorly on the national surveys. I would have thought that an obvious research project for the Institute of Teaching would be just that - namely who answers these surveys and do they align with or don't with our internal surveys. From the Rosenburg paper there is an assumption made that I have not seen tested, that our national performance mirrors our internal one. I suspect this is not the case but it needs researching. If there is not a strong correlation then a lot of the logic in his paper falls apart. It would also be worthwhile to actually do some more work on DU's performance in these national surveys.
On the question of the SETU, I agree that it needed reforming and the new version seems much better ie shorter, clearer and more focused on individuals. There needs to be a clarification of who gets the data and a confirmation that only the HOS and the individuals concerned get their own data while the unit stuff can be circulated more widely.
The one part of the Rosenberg paper that really disturbs me is the section on P3 that deals with what happens when scores do not measure up. Why is 3.5 taken as the trigger? What proportion of existing units fall above vs below this magic number? Also, the focus of actions is very much on the individual academic via the PPR process. In one way this is fine, but there are many instances, noted by Rosenberg where the issue is structural eg Library, Mailouts, Text book supply, ITS, teaching facilities in some spaces etc. However the only ones who have to report are indivudal academics. In a previous position I had some poor results that were related to mail outs and book ordering and wrote letters to the appropriate people about that. But I had no idea if anything changed as a result ie did these areas of the University improve their performance and what was their accountability? There is a lot about improving individuals but little about dealing with other, structural and divisional limitations. These needs to be addressed.
In a punitive, down sizing climate I am also anxious that SETU can always be used for some sort of punishment of individuals. So I think to allay that fear, while also supporting the principle of listening to students and to improving our teaching, this needs to be made explicit ie that a poor SETU result cannot be used as the basis for disciplinary action or dismissal. This raises the tricky question of what we should do with academics who are hopeless teachers (and there are a few).In these instances there should be other forms of evaluation used in conjunction with a separate process and these should only come into play after a number of poor SETU results and other actions (eg professional development, time out etc).
The other point that I had real trouble with as a HOS was: what do you do when the results are uniformly good. ie we often had most individuals and units scoring very highly but I had to write endless reports about what I/we were going to do to improve things further. And it really is pretty hard to go beyond lots of 4s to 5s. Many students just don't give 5s in the same way that we don't give many HDs! So while I cannot argue against continuous reflection and improvement, there should also be due recognition given to good teaching that may not need endless reforming!
We are concerned about the direction of the revised SETU questionnaire. I feel that it would be more constructive for students to comment upon how the unit is taught rather than comment upon individual teachers. In other words we should be concerned about the unit material rather than the popularity of individuals. Individuals are constrained by the number of staff, workloads, structures for course delivery that are economically driven by the broader University, issues that we at the 'coal face' have little control over.
However, if the university does go ahead with the additions I would like to have questions that relate:
· to the % of lec/tut/workshop/labs that the student attended if an on campus unit;
· the number of hours per week that students spent studying the unit;
· the manner in which students engaged with the unit material;
· an indication of how well the student is progressing in the unit at the time of the evaluation;
· if in fact the all the student's unit requirements have been fulfilled at the time of evaluation. For example we have one unit that is not completed until mid Dec. and yet students are asked to evaluate this unit prior to the first week of Nov.
I am opposed to the HOS sending personalised reports to the DVC naming staff as the measures are so inaccurate but the results will take on a life of their own.
eg where staff are involved but more in a coordinating role with some lectures, will they get a low score because of limited contact with students, not poor quality?
The new SETU for Semester 2 2006 is very harsh in regards to isolating, victimizing and punishing individuals. This is a perspective from an Associate Lecturer, who has no control over workload distribution.
For example, I could be a very good tutor, but if one student decides the mark I gave them was harsh, or even worse the mark another assessor external to my tutorial gave them, lets say a low mark or fail [might be reflective of student’s performance or for many other reasons], the students can now directly target individuals. This then allows the University to target those same individuals for the PPR process and promotions, as well take other punitive measures. Some heads of schools could also use these to target staff, by putting them in tutorials in areas not associated with their discipline.
The lesser of two evils. If we take the S1 SETU method, it at least works on the basis that in a University "we" are teams in the units/subjects taught. In S1, some staff had received some unwanted comments from students, but through S1 SETU process these can be moderated because of the 'team'. Imagine what it would be like if those comments targeted an individual? The head of school then would wish to discuss such, it goes to the Dean because you are below the SETU benchmark - how would you feel? Because you are not only being targeted by the student but questioned by management? You have to justify your position at every level of the Uni, why? I am here to teach students not make it "fun" for generation Y. Discipline goes out of the window if I have to cow-tow to SETU.
This is even harsher considering that the number of students who complete surveys is very, very, very low.
SETU remains with these changes a blunt instrument. When I first arrived at Deakin a couple of years ago someone I spoke with on the 'phone at that time suggested to me that Deakin was moving to adopt a SETU based on the model in place at the Uni of Qld. I had just come from UQ and am aware of the instrument in place at UQ, which is preferable to SETU. The UQ instrument permits, apart form other options, staff to select banks of questions specific to their units/pedagogy. The provision of choice, apart from providing unit specificity, is appreciated by staff- and would meet Rosenberg's needs in a more precise way than the current blunt instrument (which in terms of 'compliance' we must follow- or else; irrespective of whether the instrument is good, bad, or indifferent).
One point worth making I think, a point that from what I can see
John Rosenberg has failed to appreciate, is not simply the low
response numbers. A problem is that the way SETU is administered
is self-selection to participate by students. Given that this is
the case, the sample is simply not random, and as such must be
treated with great caution. It is concerning therefore that
apparently these results are now being used to make budgetary
decisions within the university (as well as the on-going problem
of them being written into PPRs).
From what I can see, the changes have not addressed this basic
problem. If the survey were appropriately administered, using for
example a telephone survey of students in a way that led to a
truly randomised sample, the credibility of the SETU results
would be raised, so improving response rates, and the credibility
of these numbers in the eyes of staff.
Generally the revised SETU instrument is an improvement on the last, particularly as there is now some way of discriminating between members of the unit teaching staff. I am also in a favour of any system that helps us to ensure that the content and delivery of our units is relevant and engaging. Providing the SETU feedback is used in a constructive (rather than a punitive way) I agree that the system can lead to positive outcomes for both staff and students.
However, my biggest concern is not with the instrument or the follow-up, but more with the data collection process. The response rates for the SETU questionnaires are routinely 15-25% of the cohort. There is not a journal ranked in the top 3 tiers of any discipline that would accept a piece of research based on less t