5 Data & AI Trends for Data Leaders in 2024

From the rise of the Gen AI to the increasing costs of data debt, data leaders need to stay on top of the latest trends and challenges. In this blog post, we will explore five of them and how data leaders can navigate them successfully.

1. Gen AI: The Rise of Augmented Intelligence

With the record-breaking adoption of chat-GPT, AI became first-page news in 2023 (and triggered urgent regulations worldwide). It’s now on seemingly every company’s agenda to implement AI use cases to drive business value.

Gen AI, short for “General Artificial Intelligence,” represents the next phase of AI evolution. Unlike narrow AI, which excels in specific tasks, Gen AI possesses the ability to generalize across domains. For data teams, this means embracing AI as a collaborative partner rather than a mere tool. Here’s how:

On top of the ethical challenges, companies also need to ensure that data is accurate and reliable enough to be used by AI models. As data quality becomes the bedrock of successful AI, data teams are facing significant effort and responsibility this year.

2. Data Tool Consolidation: Streamlining the Toolbox

The tools of the modern data stack have been great for enabling data-driven decisions and empowering data teams. But it’s not all roses and rainbows. It also comes with a cost of complexity and fragmentation. With so many different tools and platforms to choose from, data teams often end up using multiple solutions that don’t integrate well with each other, slowing down progress and increasing costs. 

Consolidating data tools is a strategic move that can help data teams streamline their workflows and save money. By consolidating tools, data teams can aim for fewer layers that still meet all their data needs, from data discovery and management to monitoring and validation. More and more data-led businesses are starting to realize this, eager to get away from tool sprawl.

3. Data Prioritization: The Secret to Cutting Data Debt and Maximizing Value

Many data leaders fail to reach their goals because they have too much data debt. Data debt is the cost of bad data quality and management that makes it hard to get value from data. The main cause of data debt is not knowing which data assets are important for the business—and which ones are less important.

If organizations truly want to become data-led and maximize business value from their data, they first need to find out what data they have, how they use it, and how it helps the business. Then they need to rank all data by importance—and data teams and business teams need to agree on the ranking of data by business value. 

For unimportant data assets, consider getting rid of them to save time as well as computation and storage costs.

When you have identified your prioritized data assets, they need to be validated and improved in order to maximize ROI for your data initiatives.

4. The Semantic Layer: Bridging the Gap

Ok, so your business has a lot of data. But can you use it well? Data can come from different sources, formats, and business contexts, making it hard to understand and analyze for everyone. Data can also be complex and technical, needing special skills and tools. This creates a gap between data producers, who create and store data, and data consumers, who need to access and understand it. How can we bridge this gap?

The semantic layer has been buzzing in the data community since last year and many think it will be the bridge in the gap. It’s meant to be a transition layer between data users and business users, simplifying data metrics to be equally understood by all. Data in the semantic layer should be shown clearly and consistently for everyone in the organization, letting data consumers retrieve and understand data on their own. No more waiting, confusion, or frustration.

The growing popularity of Semantic Layers shows no signs of slowing down in 2024. After all, making data accessible and meaningful for everyone empowers data-driven decision making and innovation. That should be a goal for most data-led organizations, right?

5. Text-to-video models: A Double-Edged Sword for Data and Society

Yesterday, OpenAI released its astonishing new text-to-video model (an example of narrow AI) called Sora. It turns user prompts into high-quality video scenes.

Here is an example video from the prompt "Reflections in the window of a train traveling through the Tokyo suburbs"

Amazing, right? But don’t get too excited yet. Sora is only available to a few safety testers, who check it for issues like misinformation and bias.

Sora is not the first text-to-video model, but it says it is the best. Google, Meta, and Runway have had similar models since 2022, but they were not very good. They were low-quality, buggy, or inconsistent. Runway’s Gen-2 model, last year, was better, but still limited. It could only make short clips, and struggled with complex scenes and movements. 

Text-to-video models are a fascinating but hard problem in AI and computer vision. They need a lot of data, computation, and creativity. They have to understand the text, make realistic and diverse images, and move them smoothly and coherently. They also have to handle ethical and legal issues, like privacy, consent, and authenticity. 

Will Sora meet the expectations, or let us down? Will other models catch up or beat Sora? Will text-to-video models be widely useful, or stay a niche and risky technology? The latter becomes especially relevant in 2024, as the world prepares for major elections and many worry about misinformation.

In Closing

Remember, these trends and challenges aren’t isolated; they intersect and influence one another. Data teams that embrace collaboration and innovation to make data trusted and valuable will thrive in this dynamic landscape. 

Here’s to an exciting and data-led 2024!

Related reads