SchoolStackMany school pilots end the same way. A few enthusiastic teachers try a product for a term, they say they liked it, and the school either buys it on goodwill or forgets about it. Neither outcome is a decision. A good pilot answers one question: should we pay for this for everyone, yes or no?
Write the decision down before you start
Before anyone logs in, write one paragraph that answers three things:
- What problem are we trying to solve? Be specific. “Improve maths” is not a problem statement. “Year 7 students who arrive below expected level don’t catch up by the end of the year” is.
- What would we see if the product worked? Pick one or two measures you can actually collect.
- What result means we buy it, and what result means we don’t?
Agree on this with the people who sign the budget. If you can’t agree on what success looks like now, you won’t agree at the end either.
Pick measures you can collect
Good pilot measures are already collected or cheap to collect. Some examples:
| Problem | Measure |
|---|---|
| Homework not handed in | Completion rate per class, from the LMS or homework app |
| Parents not reading messages | Read rate per message, from the parent app’s delivery reports |
| Slow marking | Teacher-reported minutes per set of books, before and during the pilot |
| Low engagement in practice | Minutes of practice per student per week, from the app |
| Gaps in core skills | Scores on a short assessment at the start and the end |
Usage data is useful but it’s not the same as impact. A product students log into every day can still make no difference to learning. Use usage data to check whether the pilot actually happened, and use an outcome measure to judge it.
Choose pilot classes carefully
If you only pilot with your most enthusiastic teachers, you learn how the product works for your most enthusiastic teachers. Mix the group:
- Include at least one teacher who is sceptical about technology.
- Include classes with different attainment levels.
- If you can, keep a similar class that doesn’t use the product, so you have something to compare against.
A comparison class doesn’t need to be a formal trial. Two parallel year 8 classes taught the same unit, one with the product and one without, already tell you much more than one class on its own.
Run it long enough
A two-week pilot measures novelty. Most products need at least six to eight weeks before the novelty wears off and routines settle. A full term is better for anything that claims to change learning outcomes.
Set two checkpoints:
- Week 2. Is everyone set up? Are there technical problems? Fix logins, rostering and device issues now, or they’ll swamp the results.
- Midpoint. Are teachers using it as planned? If usage has collapsed, find out why before the end.
Ask teachers the right questions
“Did you like it?” produces polite answers. Ask questions that reveal cost and behaviour:
- How many minutes a week did it take you to prepare?
- What did you stop doing to make time for it?
- Would you keep using it if it cost your department budget?
- What would make you stop using it?
Ask students too. Short anonymous surveys with two or three questions work best.
Get the commercial terms in writing first
Before the pilot starts, ask the vendor for the price you would pay for a full rollout, for at least the first three years. A product that is great at a pilot discount may not be worth it at full price. Also confirm that pilot data will be deleted if you don’t buy.
Make the decision
At the end, go back to the paragraph you wrote at the start. Compare the results with what you said success would look like. There are three honest outcomes:
- Buy. The measure moved, teachers will keep using it, and the cost is justified.
- Don’t buy. The measure didn’t move, or teachers won’t use it without heavy support.
- Extend. Only if a specific problem, such as rostering, stopped the pilot from being a fair test. Set a new end date and stick to it.
Write a one-page summary of what you tested, what happened and what you decided. Next year, when another vendor pitches a similar product, you will be glad you did.