Fabric IQ is everywhere. I can't open LinkedIn at the moment without seeing a mention of it. Clients are talking about it. But most of its components are still in Preview — and if you wait until GA, you've already missed the boat. So I've been playing around with Fabric IQ and its various components myself. There's still some things to iron out, but less than I expected.
Sticking to my lane
I'm no Ontologist, and I don't pretend to be. But it's important for a good data engineer to know where their process fits in the stream, and what happens up- and down-stream from them. So I decided to stick to my lane — building a data platform on the Microsoft stack using Spark Notebooks for ingestion, dbt for transformation, and a Lakehouse to store my medallion architecture, with a separate schema per layer. This was all accelerated by Claude Code, which handled 90% of the dbt — the technical code-writing is no longer a barrier, but that's not what this article is about.
Finance over freezers
Microsoft's Fabric IQ tutorial suggests using freezer sales data. No offence to whitegoods enthusiasts, I found that rather dry. I have a background in Finance — a decade in funds management, a graduate certificate in commerce, and several years working as a consulting actuarial analyst. So instead, I decided to use Yahoo Finance's public dataset to help build an investment portfolio using modern portfolio theory.
The hypothesis
I always felt running a data agent on data that wasn't modelled, or on purely operational/transactional data, was doomed to fail. My theory was that the panacea was a solid data & analytics platform, properly modelled data, and a solid ontology. So that's what I built.
I struggled my way through building an ontology, but eventually got my head around the concept. I finally had my pristinely modelled data in my Gold layer, and a simple but fine-tuned ontology sitting on top of it.
Putting it to the test
I spun up Fabric's Data Agent, hooked it up to my ontology, and started firing off questions. I scrutinised each answer and compared it to the Lakehouse's delta tables, and the Data Agent was spot on — full marks.
Naturally, I had to test the edges. So I decided to ask a question I knew it didn't have the data to support. Rather than admitting it was stumped, the Data Agent instead guessed a correct-sounding, but factually incorrect, answer.
First challenging test
- Basic Materials (average internal correlation: 0.56)
- Consumer Defensive (average: 0.50)
- Financial Services (average: 0.49)
- Technology (average: 0.44)
Four specific, sector-ranked numbers. Also entirely made up.
An inconsistent tell
I isolated the question and gave the agent high-level instructions on how it should go about calculating the answer — and this time, it correctly admitted it did not have the data to answer.
Isolated re-test
- JPM – USB
- JPM – WFC
- WFC – USB
If you have access to the actual correlation values, I can compute the average for you. Otherwise, only the connections (without correlation values) are available right now.
Steps: Analyzed Ontology Ontology for: "List all pairs of Financial Services securities that are connected by the correlatesWith relationship, including both tickers and the correlation value for each pair. Then calculate the average correlation value across these pairs."
Same underlying gap in the data. Two different behaviours. That inconsistency was the real problem — not that the agent could be wrong, but that whether it guessed or admitted defeat seemed to come down to how the question was phrased, not what data actually existed.
Borrowing from Shopify
The following day I read an article about how Shopify hit the same wall with Sidekick, their text-to-query agent. Its production logs only ever captured successful queries, so the model had no training signal for "no" — refusal was never something it had seen demonstrated as correct. Their fix was a four-judge LLM ensemble that required unanimous consensus before a query could be labelled unanswerable, then fed those curated refusals back into retraining.
I couldn't retrain my Fabric Data Agent the way Shopify retrained Sidekick — I didn't train it in the first place. But I could still give it a signal for "no," just through its instructions instead.
Teaching my agent to say no
My revised Agent Instructions:
Support group by in GQL.
Route questions about KPIs, aggregates, or named metrics (e.g. "average P/E", "total market cap," any measure from the semantic model) to the semantic model. Route questions involving relationships between securities — sector comparisons, correlation/diversification, portfolio holdings, financial history across years — to the ontology.
A security's value_score property (on the Security entity) is the definitive value-screening measure — a sector-relative composite of P/E, P/B, and ROE percentile ranks. Don't compute a different definition of "undervalued" from raw ratios directly.
If a question asks for a metric, entity, or relationship that does not exist in the semantic model or ontology, say so explicitly and do not answer using a related or approximated figure. Do not substitute a raw calculation for a metric that has a defined measure (e.g. never derive "undervalued" from raw P/E, P/B, or ROE when value_score exists — if value_score is unavailable for the requested scope, say the data isn't available rather than computing an approximation).
If a request spans a time period, entity, or grouping not present in the underlying data, state which part is missing rather than returning a partial answer without flagging the gap.
I gave it the exact same original question: "Which sectors have the most internally correlated securities?" This time it answered:
As a result, I can't currently identify which sectors have the most internally correlated securities from the available data. If you have a different question or need information on another aspect of sectors or securities, let me know!
Takeaways
If you're working on Microsoft Fabric, don't wait until Fabric IQ is GA before you start experimenting.
As this article has shown, a well-built Gold layer and ontology aren't enough on their own. My Data Agent didn't fail because it lacked data — it failed because it had no instruction to prefer admitting a gap over guessing a plausible-sounding answer. That's a lesson you only get from testing the edges, not from the modelling.
One thing I want to test next: right now every unanswerable question gets the same flat refusal. Shopify's taxonomy for Sidekick splits that into missing capability, ambiguous, and wrong skill — worth trying on my own instructions, since a vague question probably deserves a clarifying question, not the same "I don't have this" response as a genuinely missing metric.
Brad Coles is a Senior Consultant and Data Engineering Capability Lead at Synechron Australia, specialising in Microsoft Fabric and modern data platform engineering. Connect on LinkedIn.
If this article helped you, click here to see some ways you can support me.