Asking It About Your Own Trade

The problem with judging it

Everything it produces reads as competent, so there is no signal to judge by.

Which means most people form an impression from the areas they know least about, where it appears impressive precisely because they cannot see the errors.

The only place you can grade it accurately is a subject you know properly.

Which for most people is their own trade, and almost nobody thinks to test it there.

How to run the test

• Ask a question a customer actually asks.
• Then ask a harder one only a professional would.
• Ask about an edge case you have handled.
• Ask what it would cost, in your area.
• Ask about a local requirement.

The progression matters more than any single question. The first is usually answered well, and the point of the exercise is finding where along that sequence the quality falls away, since that boundary is what you are trying to locate.

What you will probably find

Since the pattern is fairly consistent across trades.

General questions are answered well, accurately, and in a reasonable order.

Intermediate questions are answered plausibly with an error or two that a customer would not catch and you will.

Specialist questions produce something confident and wrong, or right in outline and wrong in the detail that matters.

And anything local, current, or priced is unreliable in a way that has nothing to do with the subject.

Knowing roughly where your own boundary sits is the output of the exercise.

A worked example

A tradesman asked it the five questions his customers ask most and found the answers good, in some cases better ordered than his own explanations.

He then asked about a situation he deals with several times a year, and the answer was confident, well written, and would have caused damage if followed.

Asking about the applicable local requirement produced something that was accurate for a different jurisdiction.

He concluded he could use it for customer-facing explanations of ordinary matters and not for anything technical or local.

That is a more useful conclusion than any general advice about these tools, because it is calibrated to his work.

The whole exercise took half an hour.

Then apply it to the other direction

Which is the reason to do this at all.

If it is wrong in the way you found in your own field, it is wrong the same way in fields you cannot check.

So an answer about accounting, employment, or marketing that reads as authoritative is probably accurate in outline and unreliable in the detail, in exactly the pattern you just observed.

That is the calibration, and it is more useful than any confidence figure.

It also means you should be most careful precisely where you feel most impressed, which is counterintuitive and correct.

Somebody in that other field would find the same boundary you found in yours.

Test it again periodically

Since these tools change and the calibration goes stale.

Keep the five questions you used and rerun them every few months.

That gives you a direct comparison rather than an impression of whether it has improved.

It also catches the case where a new version is better in general and still wrong on the specific thing that matters to you.

Write the questions down somewhere, since reconstructing them later produces a different test.

Use it as a training exercise

A secondary benefit worth taking.

Reading a competent general answer to a question you answer daily frequently shows a better way of explaining it.

The structure, the order, and the analogy are sometimes better than what you say by habit.

Take those and discard the substance, which is the opposite of the usual use and is where the value sits for an expert.

Several people have improved how they explain their own work this way without using a word of the output.

Ask somebody else to test their field

Since one person's calibration is one data point.

If you have a contact in accounting, law, or another trade, ask them to run the same exercise and tell you where their boundary was.

The pattern is usually similar, which is itself informative.

Where somebody reports it as reliable throughout their field, that is worth probing, since it more often means the questions were general than that the tool was strong.

This is a short conversation and it corrects the impression most people carry.

Notice how it responds when you push back

Part of the same test and worth doing deliberately.

Tell it that one of its correct answers is wrong and see what happens.

It will frequently apologise and revise, producing a new answer that agrees with you, which means agreement from it is not evidence of anything.

That matters most when you are using it to check your own reasoning, since it will tend to confirm whatever position you appear to hold.

Ask it to argue against your view specifically, rather than asking whether your view is right, which is the only way to get resistance out of it.

The counter-case

Your field may not generalise.

A trade with little written about it will be handled worse than a well-documented profession, so a poor result in one does not predict the other.

Expertise also makes people harsh markers, and an answer you judge inadequate may be entirely sufficient for the customer who asked.

And the tools improve, so a test from six months ago describes something that no longer exists.

Ask five questions of increasing difficulty in your own field, find where quality falls away, and apply that boundary elsewhere.

The half hour

1. Write five questions, easy to hard.
2. Include one edge case you have handled.
3. Include one local requirement.
4. Grade each answer.
5. Note where quality falls away.
6. Apply that boundary to other subjects.
7. Keep the questions and rerun them.

Step six is the whole purpose, since the alternative is judging a tool entirely by how it performs on subjects where you cannot tell whether it is right.

What it is missing about you specifically is covered in a tool that does not know your business.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.