Almost every product in interview scheduling now describes itself as AI-powered. To a large extent, that is quite accurate.
The question of whether the software is autonomous though is very difficult to answer. While this can’t be evident in a sales demo, asking the right questions can surely resolve it.
The five questions below are all requests to see something, and they are ordered so that each one is harder to satisfy than the last.
1. Ask to see a booking that nobody approved
Request a specific instance in which the software selected a time, sent it, and no person touched the decision. Ask to see the audit trail alongside it.
This establishes whether autonomy exists at all, or whether human approval has simply moved to an earlier point in the process. A good number of products in this category act independently only after somebody has confirmed a shortlist of times, which is a genuine convenience and a different thing from what the marketing describes.
If the response is an offer of flexibility, along the lines of being able to configure how much oversight you want, the useful follow-up is to ask what the system does on its default setting and how many customers change that setting. Configurability tells you what the software can be told to do. It does not tell you what it does.
2. Ask to see a thread with an executive assistant
Request a real exchange between the software and an EA who manages a senior’s calendar. Ask for the whole thread, from first contact through to a confirmed time.
This is where senior hiring is decided. An executive assistant will not create an account in a vendor’s portal in order to release their person’s time, and they tend not to answer with availability. They answer with terms and with questions of their own, such as which colleagues will be in the room and how long the meeting needs to run. A system that cannot conduct that exchange in ordinary email will be worked around within a month, whatever the product roadmap says.
Examine the thread for two things. Did the software answer the assistant’s questions accurately, and did it hold the assistant’s conditions when it went on to arrange everybody else?
3. Ask to see the exception log
Request a summary of everything the system escalated to a human being over the last thirty days, along with the reason for each escalation.
This is the single most revealing document in an evaluation, and it is the one least likely to be volunteered. Every autonomous system has a boundary, and the escalation record is where that boundary is written down. A vendor who cannot produce such a record is telling you either that the escalations are not measured or that they are not something the vendor wishes to discuss, and both answers are informative.
4. Ask to see a loop that broke in the middle
Request a case in which a multi-round sequence had to be re-planned, and ask specifically what happened to the rounds that came after the one that moved.
Interview loops carry structure that is invisible in a demonstration and expensive in production. There may be a panel that cannot convene until a technical assessment is complete, an interviewer who must not meet the candidate before the debrief, or buffers between rounds that stop meaning anything once a round crosses a timezone boundary. When the fourth round of nine slips by a week, the question is whether the system understood the remaining five as a structure that needed re-planning or as five independent bookings that happened to concern the same person.
5. Ask to see what the software wrote in the ATS
Request the system-of-record entries generated for one candidate through a complete multi-round process, unedited.
Autonomous action that nobody can audit creates a new problem in place of the old one. In regulated hiring, where the process itself may be reviewed, an action without a record is worse than an action taken slowly. What you are looking for is whether the bookings, the changes, the reasons behind those changes, and any feedback gathered afterwards arrived in the system without a person transcribing them.
An answer that describes an integration is not an answer to this question. An integration describes a pipe, so examine what travels through it, in which direction, and which side initiates.
When a vendor cannot share any of this
Some of these requests touch customer data, and a refusal on those grounds is legitimate rather than evasive. There are three reasonable ways around it, and a vendor who wants to be evaluated properly will offer one before you have to ask.
They can redact and anonymise, which works well for the assistant thread and for the ATS entries. They can run the exercise in a sandbox seeded with real complexity. Or they can introduce you to a reference customer who is willing to open their own account and walk you through the last month of it.
What should alarm you is a vendor who offers none of these and returns to the tailored demo instead.
Our own exception log
Evie handles the first, second, fourth, and fifth requests without difficulty, and those are the questions we often get asked.
On the third, our logistics escalations are limited by constraints that nobody has recorded anywhere. If a director informally avoids Friday afternoons, or two colleagues should not sit on the same panel, the system does not know until somebody tells it, and until then it will keep proposing arrangements that are technically valid yet socially wrong.
We believe that is the right boundary to have, because it is a gap in knowledge rather than a gap in authority. A knowledge boundary moves as the system is told more. A boundary built into the design does not move at all. It remains a real limitation, though, and a vendor who claims to have no equivalent is worth a longer conversation rather than a shorter one.
Why this is a better use of an hour than a demo
Evidence of past behaviour predicts if the software is a right fit. Questions like the above will tell you more about the next year of your coordinators’ working lives than any feature comparison, and they take less time to make than the demonstration they interrupt.
We built Evie as an autonomous coordinator rather than a scheduling assistant, which is why we have often faced questions like these on our sales calls. Companies using Evie report scheduling roughly 10× faster, around 50% fewer reschedules, and close to three months of recruiter time returned each year.
Leave a Reply