JOURNAL OF NEUROSCIENCE AND NEUROSURGERY (ISSN:2517-7400)

Needs-specific goal setting is a key tool used in high-performing organizations, such asworld-class athletes and professionals, to maximize performance and the probability of achieving a desired goal or success criteria [1]. As with elite athletes, medical students have a huge volume of components to internalize and perfect to perform clinical procedures and examinations to a proficient standard. An emerging area of professional career development is the use of goal-setting using computer software [2,3]. This project investigates the potential benefits of implementing of goal setting software in the clinical skills setting to increase medical students’ performance in Objective Structured Clinical Examinations (OSCEs).

Goal setting aids the identification of components that underpin success in a given skill. Knowing the progress in each of these components and the relative priority for their improvement is crucial to allow forward progress of the skill in question, as per the ‘SMART’ goal-setting process [4]. In addition to an objective measure of performance, goalsetting increases self-awareness of the priorities needed for success [5]. The mere act of evaluating performance can also have positive benefits, as per the Hawthorne effect [6,7]. Furthermore, mapping progress objectively using goal-setting techniques has a positive effect on motivation to improve further [8]. Despite such suggested benefits of implementation, no robust goal setting method used by medical students to improve performance exists in the current literature [9].

The software used in this study was Goalscape^TM, a goal setting software programme developed by the German Olympic sailing team that in recent years has been released into the public domain. The software uses a multi-level pie chart display involving a central, ultimate goal (Figure 1). The user can break this goal down into an infinite number of ‘subgoals’ that together constitute the components required to achieve the ultimate goal (Figure 2). Subgoals can also be broken down into further, infinite levels of detail in order to allow the user to target improvement interventions on small, detailed, measurable components (Figure 3) that when aggregated together, contribute significantly to the cumulative achievement towards the central ultimate goal. The software allows effective visualization of priorities and progress for each subgoal, thus allowing the user to direct revision priority to the components of most importance. In complex systems that require high levels of competency in each component (such as clinical skills training in medical undergraduates), priorities can often become unclear due to the sheer number of components involved in the skills they must master. The use of goal setting software clarifies the respective priorities of components so that the user can adopt a methodical approach to improvement that is both goal-specific and time-effective [10]. By changing the priority of subgoals on the software, the percentage area of the pie chart each subgoal corresponds to can be changed. The progress of each sub goal can be changed and logged by the user, and the overall accumulation of progress towards the ultimate goal can be measured.

Ethical approval for the study was granted by the University of Dundee Research Ethics Committee (UREC). The study followed an RCT design, whereby 42 medical students in their third year at Dundee Medical School undertook a mock Neurology OSCE station (cerebellar examination, total 12 marks). Fourteen days prior to the OSCE, students were randomly allocated into two groups. Group 1 was given Goalscape^TM software with a present template designed by Dundee neurologists to help direct study focus. Group 2 used traditional study methods. The difference in mock OSCE score between group 1 and 2 was compared by student t-test using SPSS 22. A questionnaire evaluating students’ perceptions of goal setting was given to participants in Group 1 after completing the OSCE station (Figure 4).

The OSCE scores (mean, median [SD, IQR]) were 10.26, 10.00 [1.28, 1.50] in the software group and 9.59, 10.00 [1.56, 2.00] in the non-software group (p= 0.186, Mann Whitney U Test) (Figure 5). Questionnaire feedback from students revealed a mean participantreported score of 7.00 and 6.67 on a 0-10 analogue scale when students were asked ‘How useful is Goalscape^TM for visualising components on neurology clinical skills?’ and ‘How useful is goalsetting to medical students?’ respectively (Table 1, Figure 6,7).

Our study was designed to evaluate if the use of Goalscape^TM technology may be applied to the field of Medical Education. To this end we feel there has been some degree of appropriate application. Goalscape^TM allows the user to identify and evaluate performance in subcomponents that make up a given skill, and allows them to prioritise what subcomponents should be improved first in order to maximise performance increases before the skill is assessed. GoalscapeTM was initially developed for goal-setting in performance athletes, however wider applications of the software in business are now also being trialled. This is the first attempt to use user-led, structured goal setting software in medical education to evaluate the benefits that structured goal-setting may provide to medical students who want to maximise performance in OSCEs.

We found it possible to apply this technology to the concept of a neurological examination, as the numerous aspects of neurological examination that often proves difficult to teach and learn, lends themselves to being presented as focussed subdivisions, as displayed by Goalscape^TM software.

In addition, it was possible to apply this technology is a unified way to a substantial number of medical students, and progress on individual users’ software can be tracked and analysed by a tutor. The overall feedback from the study was positive. The subject-reported questionnaire feedback demonstrated that the majority of students agreed that goal-setting was useful for visualising components of neurology clinical skills and that overall, goal setting techniques were beneficial to medical students.

Despite these findings, our study found insignificant differences between OSCE scores in the goal-setting and non- goal setting groups respectively when the software was used over a two-week period prior to a standardised mOSCE station testing the neurological examination of the cerebellum. Thus, from our results it is inconclusive whether use of goal setting software could impact exam performance.

There were some limitations. Short study duration- time constraints on the third year university students, availability of enough neurology clinicians simultaneously, and resources available to conduct the study resulted in significant constraints to run the study from participant sign-up to completion of the OSCE. The total study duration from allocation of goalsetting software to completion of the mOSCE was two weeks. It is widely accepted that performance improvement from using structured goalsetting is a long-term process, and often involves temporal failures before success is ultimately reached. [11] Thus, it is likely that minimal performance benefit could be obtained from use of the software in such a short time frame.

Lack of detailed subgoals- the software was pre-programmed by neurologists involved in the teaching delivery of the neurology curriculum at the University of Dundee medical school. The pre-set goals set in the software programme were set at an appropriate level for students who had recently been taught the neurology course for the first time, and thus smaller detailed goals were not included as it was inappropriate for the participants’ stage of neurology training. In addition, there is evidence to suggest that performance improvement through marginal gains (i.e. the cumulative improvement of many small, detailed goals) only works in those who have sound comprehension of the fundamentals in their field to start with, and that the higher the specificity of a goal (as in the marginal gains theory), the higher the performance level required of the individuals executing the task. Thus, in our study population, it is possible that subjects were too inexperienced in neurology to benefit from gaining knowledge in marginal-gain-based goal setting, and instead would benefit more from understanding basic neurology principles at this stage, where improvements can be made globally rather than limited to predefined, narrow areas [12,13]. It is also possible that goals set were too broad for subjects to notice tangible differences to performance when they tried to improve them [14], thus affecting students’ ‘self-reaction’ and thus continued commitment to use the software [15].

Lack of control over how much software is used- once the software was activated for those in the goalsetting software group, subjects were left to use the software unsupervised. Thus, there is likely to be a large range of total time spent using the software over the study duration.

Difficulty in helping users trouble shoot in such a short study duration- some subjects allocated to the goalsetting software group struggled to use the software themselves and required extra assistance in familiarising themselves with the software programme. This could have decreased the total time students were able to spend using the software, and also may have led to a reduced willingness to use the software.

Number and content of mOSCE stations- since only one station was assessed in the mOSCE, it is likely that some students will have underperformed simply due to the particular task they were asked to perform. Thus, this may have masked the true benefit that could have been provided through the use of structured goal setting software had a wider variety and number of stations been assessed. Had there been multiple stations, this would undoubtedly have provided a better understanding as to the potential effect of Goalscape^TM applications. In relation to the station itself, this may have been limited as it focused on a single aspect. If we had two further stations, it may have been possible to assess students’ skill in finer detail, thus testing whether Goalscape^TM helps increase performance of specific,detailed components of the skill. From the list above, the first four out of five points are with regards to how the software is utilised. Thus, with small corrections to the methodology, a clearer difference between software and non-software groups may be observed.

There were also a number of confounding factors. Small study number- A larger population would allow greater confidence in the p-value obtained. This study may interest only keen, performancedriven students- since study participation was completed using voluntary sign-up, it is possible that only keen students who have strong interests in improving performance would sign up to the study.

Differences in the study duration available to each group- after signing up to the study, participants allocated to the goalsetting group were required to sign up to the software programme. There was an element of troubleshooting in signing up students, and thus small differences in the exact duration available to use the software existed within the goal setting software group. Thus, it is possible that the non-software group had on average, an increased amount of time available to study for the mOSCE.

This was a pilot study and as such demonstrated that the application of GoalscapeTM technology the Medical Education is possible. We plan to extrapolate this study with greater numbers and a greater separation of variables of the neurological examination. This study demonstrated a trend towards improvement when utilising a structured goal setting software programme, however our results are statistically inconclusive in determining whether goal setting has an effect on medical students’ performance when studying for OSCEs, which we assume is a result of various limiting and confounding factors that were present in the study. Thus, larger population studies with more robust methodology may quantify the potential benefits of structured goal setting further. Goal setting software has proven popular with medical students who took part in this study, and its role within medical education will be investigated further 

OSCE Station

Instruction for students:

“You have been asked to examine a patient with poor balance. Please perform a cerebellar examination. You will only be marked for performing the examination itself- assume that you have already gained consent, washed hands and explained examination to the patient”.

Instructions for examiners: 

• Exam date: Monday 21st March 12.45pm-2.00pm (aiming for 1.00pm start).
• 7 minute station, 8 students per examiner per hour (6 examiners therefore 48 students).
• Examination rooms: Library teaching rooms 2, 3, 4, 6, 8, 9, ITS Suite (12.45-2.00pm) .
• Cerebellar examination.
• Only mark students for their practical examination skills only. Introductions, consent, explanation of procedure, history taking etc. should NOT be scored.
• Please give as much formative feedback in writing on student’s marking sheet as possible, as this is what is most valuable to them.
• Ensure study ID number is written in top right corner of marking sheet.
• Keep marking sheets, do not give them to student (students will receive a photocopy later).

Goal setting software details: 

Software requires personal username and password that is accessible only on signing up/paying for software, however most info available at www.goalscape.com.

mOSCE details:

• Timings: Monday 21st March 12.45pm-2.00pm (aiming for 1.00pm start). Students asked to meet in ITS Suite as close to 12.45pm as possible to allow for a 1.00pm start time.
• Examinations themselves take place in library teaching rooms 2, 3, 4, 6, 8, and 9.
• Students await for their mOSCE run in ITS, where the student neurology society is holding a tutorial session in between students sitting their mOSCE over the course of the hour.
• Station details:

a. Each student sits one 7 minute station, 8 students per examiner per hour.

b. Station: Cerebellar examination.

c. Marking sheets have been printed off and will given to examiners before start at 12.45pm (draft of marking scheme attached)

• Marking sheets:

a. The goal setting software breaks complex skills into their smallest details, allowing clear assessment of performance in each part of the skill. Thus, the marking sheet focuses on the practical ‘performing’ aspects of the examination skill only i.e. non-technical skills such as introductions, consent, explanation of procedure, history taking etc should NOT be scored.

b. Marking sheets burden on formative written feedback as students appreciate that professional’s own comments a lot more valuable than a marking sheet ‘checklist’ alone.

c. Please ensure study ID number is written in top right corner of marking sheet (this will be written on a sticky badge worn by each student and students will be reminded to state this number to examiner just in case).

d. Keep marking sheets, please do not give them to student (students will receive a photocopy later). 

This study investigates how well students perform practical components of neurological examination. Please assess the practical aspect of the examination only, leaving any comment on nonpractical/communication skills in the overall comments box at the end of the marking sheet.

1. Locke EA, Shaw KN, Saari LM, Latham GP. Goal setting and taskperformance. Psychological Bulletin. 1981 Jul;90(1):125-152.
2. Hannon FB. A national medical education needs’ assessment ofinterns and the development of an intern education and trainingprogramme. Med Educ. 2000 Apr;34(4):275-284.
3. General Medical Council. Outcomes for graduates (Tomorrow’sDoctors). 2015.
4. Klein HJ, Whitener EM, Ilgen DR. The role of goal specificity inthe goal-setting process. Movitation and Emotion. 1990 Sep,14(3):179-193.
5. Lawlor KB, Hornyak MJ. SMART Goals: how the application ofSMART goals can contribute to achievement of student learningoutcomes. Developments in business simulation and experientiallearning. 2012;39:259-267.
6. Ridley SD, Schutz PA, Gianz RS, Weinstein CE. Self- regulatedLearning: The Interactive Influence of Metacognitive Awarenessand Goal- Setting. The Journal of Experimental Education.1992;60(4):293-306.
7. MacLeod L. Making SMART goals smarter. Physician Exec. 2012Mar-Apr;38(2):68-70.
8. Gale EAM. The Hawthorne studies- a fable for our times?, QJM.2004 Jul;97(7):439-449.
9. Elliot AJ, Harackiewicz JM. Goal setting, achievement orientation,and intrinsic motivation: A meditational analysis. J Pers SocPsychol. 1994 May;66(5):968-980.
10. Yussof M, Baba A. Prevalence an associated factors of stress,anxiety and depression among prospective medical students.Asian J of Psychiatr. 2013 Apr;59(2):128-133.
11. Fried Y, Slowik LH. Enriching goal-setting theory with time: anintegrated approach. Academy of Management Review. 2004Jul;29(3):404-422.
12. Bandura A, Cervone D. Self-evaluative and self-efficacymechanisms governing the motivational effects of goal systems.Journal of Personality and Social Psychology. 1983;45(5):1017-1028.
13. Locke EA. Motivation through conscious goal setting. Applied andPreventive Psychology. 1996;5(2):117-124.
14. Ordonez LD, Schweitzer ME, Galinsky AD, Bazerman MH. Goalsgone wild: the systematic side effects of over-prescribing goalsetting. Academy of Management Perspectives. 2009;23(1):6-16.
15. Schunk DH. Goal setting and self-efficacy during self-regulatedlearning. Educational Psychologist. 1990;25:71-86.