RAG-Based AI Startups: The Fragile Foundations Behind Their Success

Owning the Foundations

At the forefront are companies that have invested heavily in building their own foundation models:

• OpenAI: Creator of GPT-4, developed from the ground up using proprietary architectures and vast datasets.
• Google: Developer of Gemini, leveraging decades of search data and robust infrastructure.
• Meta: Developer of the LLaMA series, offering powerful, (near)-open-access models designed for research and commercial applications.
• Anthropic: Innovators behind Claude, focusing on safety and interpretability in AI.
• xAI: Pioneering their own models and infrastructure to maintain autonomy and scalability.

These entities not only provide the backbone for their own products but also serve as the primary sources for many RAG-based startups.

A Look at Perplexity

Perplexity, with an estimated valuation of $9 billion, serves as a notable case study. Perplexity operates by fine-tuning models like Llama and integrating them with search functionalities. With a user base of around 15 million, it showcases impressive growth and technological prowess. However, its reliance on third-party foundation models poses strategic vulnerabilities:

• Dependency Risk: Without owning the foundational technology, Perplexity is subject to the pricing and access policies of its model providers. Should companies like OpenAI, xAI, and Claude alter their terms, Perplexity’s operational costs could escalate, or worse, access could be restricted.
• Competitive Moat: The primary differentiator for RAG-based companies is their application layer. However, if foundation model providers decide to incorporate similar features, the unique selling proposition of these startups diminishes. Essentially, they risk being overshadowed by the very giants that power their technology.

Can RAG Applications Stand Alone? Is There Enough of A Moat?

I think a central concern is whether RAG applications can establish a sustainable competitive moat without owning their foundation models.

Critics argue that building on models owned by others is akin to constructing skyscrapers on rented land—where the landlord (e.g., OpenAI, Meta etc) holds the power to change the terms or evict tenants.

This analogy underscores the precariousness of relying on external foundations for long-term business stability.

Diverse Perspectives from the AI Community

Through doing research on discussion forums around this, the discourse isn’t one-sided. Supporters of RAG-based companies highlight several strengths:

• Agility and Innovation: Smaller firms like Perplexity can move swiftly, adapting to market needs without the bureaucratic inertia that larger corporations face.
• Data Acquisition: By integrating search capabilities, these companies gather valuable user interaction data, enhancing their models and refining user experiences.
• Potential for Disruption: There’s optimism that RAG-based firms can evolve into comprehensive information ecosystems (e.g., PerplexityOS), seamlessly integrating with various aspects of digital life.

Conversely, sceptics emphasise the strategic risks:

• Scalability and Sustainability: Competing with tech giants that control both the foundation models and user-facing applications is daunting. These incumbents possess vast resources, extensive data, and established user trust. A lot of these companies already have the distribution when looking at Google (Gmail, YouTube, Google Search, Google Docs etc) and xAI with X (formerly known as Twitter).
• Market Saturation: With an estimated 89% of AI startups utilising some version of GPT and 54% employing multiple foundation models, differentiation becomes increasingly challenging.
• Potential for Replication: Giants like OpenAI and Google can replicate innovative features introduced by RAG-based companies, leveraging their superior resources to outpace competitors.

I believe RAG-based companies can succeed by focusing on defensible niches rather than competing in broader, more lucrative markets dominated by Google, OpenAI, and xAI. These giants have the resources to replicate and dominate any general or high-value area they find attractive.

The key is to target specific industries or problems that are too small or specialised for the big players to prioritise, like legal research tools, rare disease diagnostics, or hyper-local logistics. Companies must secure proprietary data, tailored models, and deep customer integration to build a strong competitive moat.

Even then, the risk remains that larger players will partner with or acquire others to enter these niches. RAG startups must ensure independence and long-term value to avoid becoming stepping stones for the giants.

The Path Forward: Leveraging Proprietary Data

In my opinion, the future of RAG-based AI firms depends heavily on their ability to harness data that offers value beyond what is readily available online. Proprietary data—often inaccessible on the web or not structured in a way that delivers unique insights—can serve as a significant differentiator.

To mitigate dependency risks and secure competitive advantages, these companies may need to:

1. Invest in Proprietary Data Assets: Acquiring and structuring exclusive datasets that are unavailable to competitors can enhance the specificity and utility of AI applications. Examples include anonymised healthcare records, such as medical imaging and clinical trial data, or proprietary financial market insights like alternative datasets derived from satellite imagery or niche transactions.
2. Develop Proprietary Models: Building in-house models fine-tuned on unique datasets ensures reduced reliance on third-party providers. For instance, a company could use industry-specific operational data, like manufacturing efficiency reports or IoT-enabled maintenance logs, to create tailored predictive tools.
3. Specialise in Data Integration: By transforming unorganised datasets into usable formats, firms can address niche industry needs that general-purpose AI tools overlook. This might include turning fragmented legal case precedents into a searchable database for law firms or structuring private network analytics to improve cybersecurity.
4. Diversify Foundations: Companies should also maintain flexibility to switch between multiple foundation models, minimising the risks posed by dependency on a single provider.

Examples of high-value proprietary data include customer feedback aggregated from private surveys, experimental logs from R&D efforts, geospatial data for logistics optimisation, and specialised training data for hard-to-automate problems, such as rare medical conditions (billions of dollars are already being poured in here).

By focusing on these strategies, RAG-based firms can carve out defensible niches, delivering unique insights and applications that stand apart in a crowded AI ecosystem.

Data as the Digital Currency

In today’s digital economy, data is the new digital currency, and its value lies in its exclusivity, structure, and timeliness. For RAG-based startups like Perplexity, integrating commonly available web data with hard-to-access or real-time proprietary datasets is critical for building a defensible advantage.

Perplexity’s recent integration with Tripadvisor highlights this perfectly. Tripadvisor’s data ‘currency’ lies in its real-time hotel options and structured insights, which Perplexity now uses to provide curated, context-rich recommendations. This level of specificity—combining relevance, trustworthiness, and timeliness—isn’t easily replicable by competitors relying solely on general web data.

Scaling these solutions isn’t trivial. Ensuring reliability through chunking, embedding, and data hygiene is essential to building trust and usability. By blending open web data with proprietary sources like Tripadvisor’s, RAG-based companies turn data into a powerful currency that sets them apart.

For these startups, proprietary data isn’t just an advantage—it’s the foundation for long-term success, enabling them to carve out sustainable niches in an increasingly competitive AI landscape.

Conclusion: Building on Shifting Sands?

RAG-based AI companies like Perplexity showcase remarkable innovation and user growth, yet their reliance on foundation models developed by industry giants and easily accessible data (the open web) introduces significant strategic vulnerabilities.

To succeed, these startups must secure and optimise proprietary data that adds value not available through general web sources. As the AI ecosystem continues to mature, the sustainability of these companies will depend on their ability to carve out defensible niches and deliver specialised insights, answering the critical question of whether they can build resilient businesses on “shifting sands.”