Workflows & Automation

Databricks Genie helps you practice fast and slow thinking.

Databricks Genie Flash mode gives fast answers but can miss hidden data traps; standard Agent mode is slower and more careful, changing conclusions.

Databricks Genie helps you practice fast and slow thinking.

Key Takeaways

  • Databricks Genie’s Flash mode answers in about 10 seconds, while standard Agent mode takes longer but checks more data details.
  • The same question can receive opposite answers depending on mode: Flash may miss traps like incomplete monthly data, while Standard can detect them.
  • Flash works best when users explicitly name the right check, such as asking for volume per day rather than total volume.
  • Standard mode improves reliability on open-ended questions, but it can also overthink, add unsupported claims, and produce more mistakes.
  • The practical choice is not simply fast versus slow: use Flash for clear, targeted queries and Standard for exploratory analysis with hidden data risks.

Meta description: Databricks Genie Flash mode answers in about 10 seconds but can miss partial-month data traps; standard Agent mode checks more and takes longer.

Databricks Genie helps you practice fast and slow thinking.

Tags: Databricks Genie, Flash, data analysis, AI agents, business intelligence

Featured image: abstract friendly illustration of a lightning bolt and magnifying glass over simple data bars, no real people, no logos, no text.

A fast button changed the conclusion

Databricks Genie lets users ask questions about company data in plain English and then writes and runs database queries. A test of the tool's choice between Flash and standard Agent mode showed that the same question can receive two opposite answers depending on whether the user waits for slower reasoning.

The question asked how shipment volume in one delivery network changed from September to October. With Flash switched on, Genie answered in about 10 seconds. It reported a decrease of 300 shipments, from 900 in September 2026 to 600 in October 2026. With Flash switched off, Genie took 13 to 32 seconds and reached the opposite conclusion. The October data stops on 15 October, so September and October cover different numbers of days. Measured per day, volume rose from 30 shipments in September to 40 in October.

Both answers came from the same agent reading the same data. The only difference was the mode. The question was asked five times in each mode, and the results split the same way every time. In one run, Flash wrote that October was essentially a complete month, without checking the date range. Databricks recommends Flash for simpler tasks, but this volume question looked simple and did not say that half of October was missing.

The test hid common data traps

The tester created a fictional delivery company, Meridian Parcel, and placed traps in its data. Each trap showed a wrong number at first glance and had a check that revealed the truth. The traps included a month of data stopping halfway, every row for a month being loaded twice, and an average falling because the mix of parcels changed. A control network was built for each trap so the first number was correct; a good answer should not invent a problem there.

Questions were asked in three ways: a lookup asking for a figure, a directed question naming the check, and an open question leaving the check to Genie. The test ran two rounds. The first round used four everyday cases, such as a complaint count rising while the complaint rate falls. The second round used four harder traps that the agent's instructions never mention, such as the half month of data. Every question was asked five times in each mode in fresh conversations, producing 480 answers.

Databricks Genie helps you practice fast and slow thinking.

Before each round, the tester listed what a correct answer had to contain. Another AI model graded every answer against that list without knowing which mode wrote it, and a sample was checked by hand. A review found two mistakes in the first round list, so they were fixed, everything was regraded, and some uneven grades were settled by hand. The change log lists each change.

When Flash is fast enough and when it is not

When the question named the check, Flash did almost as well as Standard. Both reached the right conclusion in all 40 directed answers in the first round. In the second round, Flash missed 2 of 40 and Standard missed none. On first-round lookups, Flash got every one right. This fits the idea from Daniel Kahneman's Thinking, Fast and Slow: a fast system can work well when it is pointed at the right thing. If the user asks for volume per day, Flash can compute volume per day without needing to notice the missing half-month.

The gap came from detail. Flash sometimes gave the right answer without the numbers behind it. One second-round miss was stranger: Flash found that the complaint rate in remote deliveries fell from 8% to 7%, then wrote that it had nearly quadrupled. Even when the check is named, the numbers still need to be read.

The split was largest on open questions. In the second round, Flash reached the right conclusion on the partial month, duplicate and mix traps in 0 of 15 runs. Standard reached it in 10. Flash usually ran one query and reported what it saw. When every October row had been loaded twice, Flash said volume doubled in all five runs. When a complaint rate rose overall but fell in both delivery zones, Flash said it got worse every time.

Standard ran more queries before answering. On the volume question, it checked which dates the October data covered each time. On a first-round delivery question, it split parcels into local and remote deliveries in four of five runs and found that both had got slower. The overall average had fallen from 6.8 days to 4.2 days only because October shipped far more local parcels, and local parcels are quick.

The checks cost time. On open questions in the second round, Flash took a median of 11 seconds and Standard took 26.

Databricks Genie helps you practice fast and slow thinking.

Slow reasoning adds checks but also adds mistakes

Standard was not flawless. It found duplicate rows in only two of five runs, and it sometimes overthought. One lookup asked only for two averages, but Standard added "Delivery time improved by 2.6 days month-over-month", which is the misreading the trap was built to catch. On a control where the complaint rate did get worse, Standard twice ran a significance test and declined to call it worse. Every other answer, from both modes, said worse without looking at the zones, so none of them passed the tester's checks.

Standard also wrote more. Its median open answer in the first round had 233 words, against 63 for Flash. In those open answers, the grading model found 25 claims that the data did not support in Standard's answers, against 8 in Flash's answers. More reasoning gave Standard more room to be useful and more room to be wrong.

The practical choice is therefore not simply fast versus slow. Flash works when the user can name the check, such as asking for volume per day rather than volume, and when the user asks for supporting numbers because Flash tends to leave them out. Standard is worth the wait when the obvious number could be wrong for a reason the user has not thought of. Partial months, duplicate rows and shifts in mix caught Flash in every open run.

In Kahneman's terms, Flash behaves like the fast, automatic system, while the person asking the question must provide the slow, checking system. The data, questions, runner, raw responses and grades are published at github.com/Praneeth16/genie-flash-vs-standard, so users can run the same test on their own Genie agent.

Sources

Daniel Kahneman. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011.

Databricks. AI/BI and Genie release notes 2026 (15 October 2026: Flash Beta).

Databricks. Genie Agents concepts (Flash and standard Agent mode).

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.