For years I carried one rule into every AI conversation: never put personal or sensitive data into an AI tool. It served me well. It's still the first thing I tell a small organisation that's just discovered what a chatbot can do.

But after years of building AI systems for large, regulated organisations - and more recently sitting across the table from charities and small business owners - I've learnt that the data rule is only the first lesson. The mistakes that actually hurt people are quieter. They're the things nobody mentions in the demo.

Here are the ones I see again and again, on both sides of the Tasman.

Your data is already out of order - AI just reads it faster

The folder everyone could see. One of the first AI assistants I helped roll out did exactly what it promised: staff asked a question and got an answer from across the organisation's documents. Within days, someone asked a harmless question and got back a line from a salary spreadsheet. The AI hadn't broken any rule. The spreadsheet had been shared with "everyone" years earlier and forgotten. AI doesn't create your permissions problem; it finds it for you, in seconds.

Lesson: before switching on any AI that searches your files, clean up who can see what. Old "share with everyone" links are the first job.

The helpful free tool. A volunteer coordinator once showed me, proudly, how she'd summarised a month of client notes by pasting them into a free chatbot on her phone. She'd saved hours. She'd also sent names, health details and addresses to a service with no contract, no idea where the data would be stored, and settings that allowed it to be kept and reviewed. She wasn't careless - nobody had ever told her the difference between a free tool and an approved one.

Lesson: free and personal AI accounts are not work tools. Pasting is only one leak; watch uploaded files, meeting-recording bots, email summarisers and the "new AI feature" quietly switched on in software you already pay for.

The law that changed while nobody was looking. When I ask small organisations what's changed in privacy law this year, I usually get a blank look. Yet the rules on telling people when you collect their information from someone else - exactly what AI enrichment and "customer insight" tools do - have tightened in Aotearoa this year, and Australia is part-way through its own reforms.

Lesson: using an AI tool doesn't move the responsibility. If it handles someone's personal information, you are still accountable for it - wherever in the world it's processed.

The machine is confident. You are accountable.

The policy clause that never existed. I once reviewed a draft volunteer policy a committee was about to adopt. It was well written, tidy, and cited a section of legislation that doesn't exist. The AI had invented it with total confidence, and three busy people had skimmed past it. Nobody was at fault, exactly — but if that policy had gone to a funder or a regulator, the organisation would have owned the error, not the software.

Lesson: anything AI writes that leaves your building - advice, policies, reports, emails to clients — is your work. Someone has to read it properly before it goes.

The tool nobody owned. In one organisation I asked a simple question: "Who's responsible for the AI tools you use?" The answer was a long pause and three different names. There was no list of which tools were in use, who'd signed up, what data went in, or how to turn them off. When a staff member left, their AI account - and everything in it - left with them.

Lesson: name one person accountable for AI use, even in a team of five. Keep a simple list: tool, owner, what data it touches, how to switch it off.

Decisions about people. The most uncomfortable conversations I've had are about AI screening job applicants, prioritising who gets a service, or flagging "risky" clients. It feels efficient. It's also exactly where bias, privacy complaints and human harm land.

Lesson: use AI to prepare and summarise, but keep a person making the decision whenever the outcome affects someone's job, money, safety or access to help - and be able to explain how the decision was made.

New tricks, old weaknesses

The email that gave orders. When we started testing AI assistants that could read email and take actions, one test stopped the room. A message arrived containing a line of hidden text - invisible to a person, perfectly readable to the AI - telling the assistant to forward the inbox summary to an outside address. The assistant tried to comply. It wasn't hacked in the usual sense; it simply followed instructions it found in the content it was reading.

Lesson: AI that reads outside content - emails, documents, web pages - can be manipulated by that content. The more an AI agent can do (send, pay, change records), the tighter its limits need to be, and the more its actions need a human to approve them.

The voice on the phone. A small business owner told me about the afternoon her bookkeeper got a call from "her" - same voice, same way of speaking - asking for an urgent supplier payment before the weekend. The bookkeeper paused only because the request felt rushed. The voice was cloned from a short video on social media.

Lesson: agree a call-back rule today. Any urgent request to move money or change bank details gets verified on a known number, no exceptions. It costs nothing and beats the most convincing fake.

The boring basics still matter most. For all the attention on AI risk, most of the breaches I've seen started the old way: a reused password, no two-step sign-in, a former volunteer who still had access, or no working backup.

Lesson: fix the basics before you buy anything clever. Two-step sign-in everywhere, backups you've actually tested restoring, and removing access the day someone leaves.

The costs that arrive later

The bill in year two. I've watched more than one team fall in love with an AI trial, then get a shock when the real invoice arrived: per-person licences for everyone, not just the enthusiasts; usage charges that spiked in a busy month; and a free tier that quietly disappeared.

Lesson: budget for year two, not the trial. Ask how pricing works at full use, and what happens if the free version is withdrawn.

The workflow you couldn't take with you. One small firm built its entire client intake around a single provider's AI agent. It worked beautifully - until prices rose and they wanted to switch. Their prompts, their data and their process lived inside someone else's product, in someone else's format.

Lesson: keep your data, your templates and your instructions somewhere you control, in formats you can move. Choose tools you could leave.

The skills that faded. The subtlest cost I've seen is human. A team that let AI draft every report for a year found they'd lost the knack of spotting when the report was wrong. The AI hadn't got worse. They had stopped practising the judgement it relied on.

Lesson: use AI to speed people up, not to replace their thinking. Keep people doing the reasoning on work that matters, and treat AI output as a first draft to challenge.

What this means in Aotearoa and Australia

The principle is the same on both sides of the Tasman: you stay responsible for personal information, even when an AI provider overseas processes it for you. The details differ.

In Aotearoa New Zealand

• The Privacy Act 2020 applies to AI use like any other tool. The Privacy Commissioner has published guidance on how the information privacy principles apply to AI.
• Sending data to an overseas provider that only stores or processes it on your behalf usually isn't treated as a "disclosure" — but you remain accountable for what happens to it.
• Since 1 May 2026, a new principle (IPP 3A) requires you, in many cases, to tell people when you collect their personal information from someone other than them. AI tools that enrich contact lists or build "insights" from third-party data are squarely in scope.

In Australia

• The Australian Privacy Principles apply to AI use, including the rules on sending personal information overseas (APP 8) and keeping it secure (APP 11). The privacy regulator has issued guidance on using commercially available AI products.
• From December 2026, organisations covered by the Privacy Act must explain in their privacy policy when they use automated decision-making that significantly affects people.
• Many small businesses under the turnover threshold are exempt from the Privacy Act — but not all. Health service providers, for example, are covered regardless of size. Check before assuming you're out.

Monday-morning checklist

1. Write down every AI tool your team uses, including free ones on personal phones.
2. Name one person accountable for AI use.
3. Sort your information into three groups: public, internal, and personal or sensitive. Decide which tools each group may go into.
4. Move work onto approved business accounts with training on your data switched off and retention set as short as possible.
5. Tidy file sharing before switching on any AI that searches your documents.
6. Require a human check on anything AI produces that leaves the organisation.
7. Keep a person making decisions about people's jobs, money, safety or services.
8. Limit what AI agents can do on their own; require approval for sending, paying or deleting.
9. Agree a call-back rule for urgent payment or bank-detail requests.
10. Turn on two-step sign-in everywhere and test restoring a backup.
11. Budget for year two, and keep your data and templates in formats you can move.
12. Review your privacy notice against the law in your country this year.

The real rule

I still believe in my old rule of thumb. For a small team using free tools with no contracts in place, "no personal data in AI" is exactly right.

But the fuller lesson took me longer to learn. AI rarely fails in spectacular ways. It fails in ordinary ones - a forgotten shared folder, a rushed skim of a confident answer, a tool nobody owned, a bill nobody budgeted for. The organisations that do well with AI aren't the ones with the cleverest tools. They're the ones that decided, before they started, who is responsible, what information goes where, and when a human has the final say.

That's not a technology decision. It's a leadership one.

Sources

• Office of the Privacy Commissioner (NZ): AI and the Information Privacy Principles
• Office of the Privacy Commissioner (NZ): Privacy Amendment Act passes
• Bell Gully: Preparing for IPP 3A — effective 1 May 2026
• OAIC (Australia): Consultation on guidance for transparency in automated decision-making
• MinterEllison: Preparing for the new ADM obligation