Evaluations are a critical component of the GSoC program. Throughout the program evaluations serve to expose process flaws, assess performance, and precipitate pass/fail decisions. Taking time to evaluate the progress and workflow of the project provides an opportunity to correct course and address underlying issues. Most importantly, it provides a structure in which to give valuable criticism and praise to the GSoC contributors. After all, they are supposed to be actively learning (not just working) and effective learning requires evaluation.
Unlike code critique, which happens openly on mailing lists or other public forums, performance evaluations are personal in nature. Delivering the mentor-GSoC contributor evaluations should be done privately. And more generally, remember the maxim: criticize in private, praise in public.
There are two types of evaluations:
GSoC contributor evaluations: Each GSoC contributor fills out two evaluations: at the halfway point of coding and at the end of the program, pertaining to their experience and their interaction with mentors.
Mentor evaluations: Mentor fills out two evaluations: at the halfway point of coding and at the end of the coding period after the GSoC contributor has submitted their final work product. Both evaluations cover their participation and their GSoC contributor’s performance. If there are multiple mentors, only one mentor will complete the evaluation (decide amongst yourselves who that will be beforehand).
These are provided online by Google at specified times with deadlines. The deadlines are important for organizing the payments to GSoC contributors based on the pass/fail decisions by their mentors, so you must ensure that you complete your evaluations on time. Remember, if a GSoC contributor fails any evaluation, they will no longer be part of the program and will not receive any more payments from Google.
We have added more flexibility into the scheduling for GSoC contributor projects - the coding period can be the standard 12 weeks or with GSoC contributor, mentor and org admin agreement it can be extended to a maximum of 22 weeks. Mentors and GSoC contributors should discuss early on when GSoC contributors may need to step away for a bit for a planned break (exams, vacations, etc.). Ideally these will be discussed and agreed upon during the community bonding period to make sure they are on the same page.
GSoC contributors will be expected to complete 40-50% of their project by the first evaluation which is the halfway point of the program (at the end of 6 weeks for a standard timed project) if they want to pass their evaluation. Also if a GSoC contributor completes their project 8 weeks into the 12 week program, that’s great, but they still have to log back into their GSoC dashboard during the final evaluation period to submit their final evaluation and work product URL in order to pass the program.
They Won’t Know Unless You Tell Them
It is important to note that other than the pass/fail status, GSoC contributors do not have direct access to the evaluations of their performance. Mentors are welcome to make a copy of their evaluation and share it with GSoC contributors directly. GSoC contributors are welcome to do the same, but it is not required for either the mentor or GSoC contributor to share their evaluations with the other. Make sure to thoroughly check the topics covered by the evaluations regardless of your choice to share them with one another.
Org admins have access to all evaluations submitted by their org’s mentors. Mentors have access only to their own evaluations, not to the evaluations submitted by their GSoC contributor. There is a field on the evaluation form where you can make a direct comment that will be sent to the GSoC contributor’s email. It is important for mentors and org admins to coordinate in assessing the GSoC contributor and mentor evaluations and taking appropriate follow-up action. There are a few questions on the GSoC contributor evaluations that can be seen only by org admins (ratings of their mentor, etc.). These fields are clearly marked on the GSoC contributor dashboard so they know who can or can not see them.
Evaluations should result in a direct review of the GSoC contributor’s progress and should be conducted using a real-time communication medium. This can be by phone, your favorite voice-over-IP service, or in person. The discussion should be frank and in the context of periodic review so that the student is prepared for criticism and to work with you to revise workflows, timelines and habits. Take time to explain the value of learning how to take criticism and praise in a professional setting.
When delivering reviews of GSoC contributor performance, be specific about both positive and negative aspects of a GSoC contributor’s performance. Make the suggestions for change or improvement relevant to what your GSoC contributor is currently working on, and provide specific examples. As you prepare for your GSoC contributor’s review, you might write it out as though you were sending an email; this will help you to frame your thoughts and to ensure that you are providing a balanced perspective. Make the GSoC contributor aware of the review schedule well in advance. If you provide a written copy of the review, schedule time for discussion immediately after the GSoC contributor reads it.
When in Doubt, Fail the GSoC contributor Early
This is harsh-sounding advice. However, GSoC stats show that more than 80% of the GSoC contributors who are reported as marginal at or before the first evaluation eventually fail or withdraw from GSoC. Whatever problems your GSoC contributor is currently having, they are likely to be worse than you currently appreciate, and to get worse rather than better over time. You are not doing your GSoC contributor, much less yourself, a service by prolonging the agony. Most GSoC contributors and their mentors have a great time and get a lot done. If you are having the other kind of experience, cut your losses and try again next year. Remember there are hundreds of GSoC contributors diligently working on their projects and if you give a GSoC contributor a pass simply because you feel guilty failing them (even though they are not meeting deadlines or communicating) then you are doing a disservice to all of the other hardworking GSoC contributors.
Note that GSoC gives mentoring orgs quite a bit of flexibility and cuts them a lot of slack. GSoC contributors need to understand that they are being evaluated from before they are accepted to the end of the program, and that you take the GSoC experience seriously and expect them to do likewise.
It’s Not Impossible to Turn Things Around
A true story: Once upon a time there was a student with limited English. The mentor considered the communication difficulty as a potential issue, but the student was enthusiastic, and hoped to use GSoC as an opportunity to improve his English.
The community bonding went well, but during the first half of coding, progress drifted off track. The student’s mentor was much busier with work than was anticipated, and the student became stuck several times with various issues. The student failed to proactively ask for help, and the mentor didn’t catch this, so the student became disheartened.
At the midterm, progress was disappointing, and the project came close to failing the student. But as the mentors looked at how they might try to rescue the situation, and after discussion with the student, the project came up with a concrete plan. The student had a job in the lab for about 8 hours a week, which he agreed to put on hold until the end of GSoC. Each day the student updated a Google document with what he was working on. This included reviewing any outstanding issues which might block progress. The student would camp on IRC during the hours he worked on his project.
Additionally, a second mentor was brought in, so that a mentor would always be available when the student might be working. Progress was discussed daily. The organization also made it clear that if the plan didn’t succeed, that the failure would reflect badly on both the organization and the student. All those involved were happy to work to address any additional issues that might come up.
By the final evaluation, the student was almost back on his original project plan, and had completed all the required goals.
It is possible to rescue a failing GSoC contributor, but you need to really consider the issues, and come up with a plan to address them which everyone involved buys into. You also need to be prepared for the project to fail, and be okay with the extra energy you have invested not resulting in the outcome you wanted.
Mentor, Heal Thyself
This is also an important time for self-evaluation. Are you managing your time adequately? Do you know where the project is at and where it is going? Are you enforcing the deadlines you set? Are you integrating your GSoC contributor into your community?
Take the evaluation period as an opportunity to get feedback from GSoC contributors. Is there any way that you could have helped the GSoC contributor more? How does the GSoC contributor think you might be more effective as a mentor? Ask your GSoC contributor directly for feedback.