Facilitator notes
Running this as a live session? Here's a simple way to structure the session so people leave knowing how to use AI on data without getting tripped up by a confident wrong answer.
Before you start
- Have a working data warehouse connection ready so you can demo a real query live — a working example beats slides.
- Ask each person to bring one real data question they'd normally hand off to the data team.
- Pick one shared question up front to walk through together, including a known metric to anchor on.
Self-serve data responsibly
Atlassian AI Builders Week curriculum
Take what's relevant and make it your own.
The honest starting point
Out-of-the-box AI — even the best frontier models — is not that great at analytics. The good news: with the right skills and guardrails, accuracy improves dramatically. But it requires understanding why AI fails at data — and building deliberately to close that gap.
Why AI gets data wrong
It's designed to answer, not to be right
LLMs are optimized to produce a response. Accuracy is a lower priority for the algorithm than confidence.
Your question is almost never as clear as you think
A term like "active" can mean dozens of different things across thousands of tables in a large warehouse. The agent doesn't know which definition you mean — so it picks one and runs with it.
It doesn't show its work unless you make it
AI will leap from assumption to assumption without disclosing what it assumed. Once you're chasing its logic, it's nearly impossible to reverse engineer.
Plausible-looking wrong answers are the hardest failure
The answer that's directionally off but sounds credible is the one that gets used without question. There is currently no robust catch-all solution for this — which is why human judgment remains the differentiator.
The stakes spectrum
Before you touch any data question with AI, know where you sit on this spectrum:
| Directional exploration | Mission-critical accuracy |
|---|---|
| Standups, ad hoc questions | Irreversible decisions, exec reporting |
| Decisions can be revisited | Big dollar signs, board-level numbers |
| Lower precision is acceptable | Requires nuanced business context not written anywhere |
| AI + best practices works here | Don't start with AI here |
Most of the work PMs and designers do lives on the left side. The skills and best practices in this guide are designed for that context. As you move right, the requirements change — domain experts and human accountability need to be in the loop from the start.
Core principles
Always ground on a source of truth
LLMs are designed to answer questions, not to be accurate. Keep it honest by anchoring on data you already know is correct before asking the next question.
Force AI to show its work
AI is lazy — it jumps to the answer without documenting its reasoning. Guide your agent to echo its logic and SQL into a notebook or documentation file. This creates a record you can actually trace.
Own the last mile
AI is powerful but flawed for data analytics. Human judgment and vigilance — your domain intuition, your understanding of what a number actually means — is the competitive differentiator. That cannot be delegated.
Best practices for querying your data warehouse
- Point it at the right table first. A large data warehouse can have tens of thousands of tables. If the agent picks the wrong one, everything after that is worthless. Help it find the right table before asking the real question.
- Anchor on a number you already know is true. Once the agent can replicate a known metric exactly, you have confirmation it's pulling the right table with the right logic. Only then ask the harder question.
- Build a self-serve analytics skill. Package your data team's hard-won best practices into a reusable skill so every agent gets the guardrails out of the box.
Best practices for validating results
- Ask Rovo to independently check its own work. AI won't self-audit unless you trigger it to. When prompted to double-check, it often surfaces assumptions it glossed over — and the rest of the session inherits that additional rigor.
- Reconcile against an existing asset. If the result looks slightly off, ask the agent to reconcile it against a dashboard or known metric. In the process of reconciliation, it will typically surface the assumption it got wrong.
- Use an independent judge. Spin up a separate agent or session to critique the first agent's output. Watching AI critique itself is surprisingly effective — and it catches the cases where telling your agent "don't do X" didn't stop it from doing X.
What the self-serve analytics skill does
The skill does two things that meaningfully change the quality of your output:
Question crafting — it slows the agent down
Without the skill, the agent immediately runs queries and returns an answer with confidence. With the skill loaded, the agent pauses to clarify ambiguities before proceeding:
- Is "tenant" the right granularity?
- What does "active" actually mean in this context?
- Which time window applies?
This iteration loop — going back and forth with the agent to resolve ambiguity — dramatically increases the likelihood of getting to what you actually wanted to know.
Transparent documentation — it shows the work
When the agent publishes a result, it automatically adds standardized panels:
- The SQL queries used, written to a notebook and uploaded to Databricks (with a linked URL).
- A risk assessment flagging how much of the analysis was driven by the agent's own assumptions versus user-confirmed definitions.
- Clear provenance so anyone reading the page can understand exactly how the numbers were produced.
Domain-specific skills: closing the last mile
The self-serve analytics skill raises the floor for everyone. But the closer you get to mission-critical accuracy, the more you need domain-specific context that isn't written in any data table. A domain skill for a specific team adds:
- Exact metric definitions — what "active" means for this product, specifically.
- Blessed data tables — the ones you're authorized to use and that are validated as sources of truth.
- Scoping rules — what time windows apply, what entitlements to include or exclude.
- Questions it won't answer — explicit guardrails to stop the agent from getting creative outside its domain.
- Depth modes — brief, standard, or deep analysis, so you can choose the level of rigor for your stakes.
Important guardrails
Do not provide the following directly into any AI agent:
- UGC (user-generated content)
- PII (personally identifiable information)
- Fintech / financial data
For guidance on specific use case scenarios, refer to the responsible data use with AI post from the Legal team.