Pipelines Get Smart: Price Hikes, Intelligent Pipelines, and GenAI That Diagrams

Biweekly Data & Analytics Digest: Cliffside Chronicle

Author

Josh Miramant
09 Apr • Estimated Reading Time: 7 minutes

Power BI Hikes Price

Microsoft recently announced new pricing for Power BI Premium (both per user and per capacity). The announced price increase for Pro and PPU licenses will start April 1, 2025, with Pro rising to USD 14 per user per month and PPU to USD 24. This change will inevitably have a ripple effect on how organizations plan and allocate their analytics budgets.

This shift is part of a broader trend we’re seeing across the analytics landscape: as platforms add sophisticated features (like AI and advanced reporting), prices are increasing. The entire BI industry is in a high-stakes race to deliver cutting-edge capabilities, and that R&D investment often shows up as increased subscription costs for customers. Keep an eye on how other BI vendors may respond with their own adjustments or potentially use this as an opportunity to stay competitive.

Lakehouse to Dashboard: Databricks Automates Power BI Refresh

In a welcome marriage of data platform and BI tool, Databricks announced a new integration to automatically publish data updates to Microsoft Power BI​. Essentially, Databricks Workflows can now refresh Power BI datasets and semantic models whenever your lakehouse data changes.

The immediate win: no more exporting CSVs or manual refreshes…your Power BI reports stay in sync with the source in near real-time. This not only saves time but also cuts down on unnecessary compute costs by updating Power BI only when needed​.

It’s a signal of cloud platforms playing nicer together. For mid-market analytics teams, that means a smoother path from raw data in a lakehouse to polished insights on the BI dashboard.

Fabric Keeps Shipping: FabConf 2025

At the Microsoft Fabric Conference 2025, Microsoft pulled back the curtain on major updates that unify data, AI, and DevOps like never before. Fabric data agents now tap into Azure AI Foundry to create custom conversational AI grounded in your own data, Copilot features are rolling out to more pricing tiers, and a built-in Synapse migration tool makes jumping to Fabric a breeze. New DevOps goodies—like a Fabric CLI, CI/CD enhancements, and Terraform support—mean automation and version control come standard. On the analytics front, real-time just got real: Apache Airflow is now native in Fabric, and Power BI’s new “Direct Lake” mode promises lightning-fast, always-fresh reports without complex refresh cycles.

Meanwhile, OneLake upgrades deliver easier multi-cloud and on-prem integration, plus direct Excel connectivity for frictionless data exploration. Purview and DLP policies now extend across Fabric to keep your data compliant and secure—even in AI scenarios, thanks to Purview’s upcoming Copilot governance features. Add in autoscaling Spark, built-in AI Functions, and a fast-expanding ecosystem of partner solutions, and it’s clear Microsoft is raising the bar for “all-in-one” data and AI.

Microsoft is working at a pretty feverish pace to ship features in Fabric.

Improve Your Technical Diagrams with GenAI

This article was very helpful on how to use AI as a writing partner for technical content. He walks through a simple but powerful workflow: start with an outline, co-write with AI, then aggressively edit to bring your own voice and expertise back in. It’s not about letting AI write for you; it’s about accelerating the messy middle and getting to a polished piece faster.

This hits home for anyone writing technical posts, especially if you’ve wrestled with structure, clarity, or just getting started. AI won’t replace your insights, but it will help you organize, iterate, and ship faster. Mehdi’s approach is practical and focused on keeping your voice intact while using AI to do the heavy lifting behind the scenes.

Perfect if you’re trying to scale your content without sacrificing depth or authenticity.

Databricks Cost Optimization: Practical Tips

Data Darvish lays out a no-fluff guide to saving money on Databricks workloads without killing performance. The post covers key levers like choosing the right cluster type, using autoscaling smartly, caching with intention, and leveraging Photon for faster, cheaper compute. Bonus: practical notes on monitoring usage and finding hidden inefficiencies through the cost analysis UI.

Some key considerations:

  • Cluster Types Matter: Use Job Clusters for scheduled workloads and Shared Clusters only when necessary. Keep lifecycles short to avoid idle costs.

  • Autoscaling ≠ Free Lunch: Autoscaling is great—but only if you tune min/max nodes and understand how jobs scale. Misuse leads to serious overprovisioning.

  • Leverage Photon: Photon runtimes offer massive performance boosts (up to 3x) for SQL-heavy workloads and reduce your compute costs.

  • Delta Caching + File Size Tuning: Cache wisely (don’t blindly enable it), and optimize Parquet file sizes (around 100–250MB) for better read efficiency.

  • Query Profile Everything: Use the Query Profile UI to find slow operations, scan-heavy reads, or skewed joins that balloon your costs.

  • Use Spot Instances Thoughtfully: They’re cheaper, but risky. Use for non-critical, retryable jobs—especially ETL pipelines.

Cloud platforms make it easy to overspend if you’re not paying attention. This post is gold for any data team trying to scale responsibly. It’s not just theory; it’s actionable stuff you can try today. Worth a read even if you think your costs are under control—there’s probably some low-hanging fruit you’re missing.

Blog Spotlight: AI in Commercial Real Estate

The commercial real estate (CRE) industry is abuzz with talk of Artificial Intelligence (AI) and Large Language Models (LLMs). But how far along are we really in leveraging these powerful technologies? A recent webinar, “From Hype to Impact: How AI and LLMs are Driving Commercial Value in Corporate Real Estate,” brought together experts to discuss the current state, challenges, and opportunities of AI in CRE.

What topics interest you most in AI & Data?

We’d love your input to help us better understand your needs and prioritize the topics that matter most to you in future newsletters.

Login or Subscribe to participate in polls.

“All models are wrong, but some are useful”

– George Box