“Do we need to move to the cloud before we can do anything with AI?” I hear this most from organizations that still run a lot on premises: manufacturers, healthcare providers, firms with a server room they’re quietly proud of. The question usually comes with some anxiety, because a cloud migration sounds like a multi-year project that would push AI out to the next decade.
The short answer is no. For most mid-size organizations, cloud readiness for AI workloads has very little to do with where your servers are. Your first AI capabilities will almost certainly arrive as SaaS: an assistant inside your productivity suite, AI features in your CRM or service desk. Those need a few specific things from your cloud posture, and moving your ERP isn’t one of them.
Three ways AI actually reaches a mid-size organization
It helps to separate the ways you’ll consume AI, because each asks different things of your infrastructure:
- AI inside SaaS you already use. Assistants in Microsoft 365 or Google Workspace, AI features in your CRM, ticketing, or HR system. These need cloud identity and a clean permission model. They don’t need you to run anything.
- AI services you build on. Cloud platforms such as Amazon Bedrock, Azure AI, and Google Vertex AI, called through APIs to build your own tools and automations. These need a governed cloud account, cost controls, and a way to reach your data securely.
- Models you host yourself. Running models on your own GPUs, on premises or in a private cloud. It has real uses, but it’s rarely the right starting point for a mid-size team.
Most teams start with the first, move into the second within a year or two, and consider the third only for specific reasons. Your cloud readiness work should follow the same order.
Here’s how the assessment asks the question, and the 0 to 4 ladder I score it against:
I1. Which best describes your infrastructure today?
- Mostly on-premises, no cloud plans
- Mostly on-premises, exploring cloud
- Hybrid: key SaaS apps plus some cloud workloads
- Majority cloud/SaaS with a defined cloud strategy
- Cloud-first with a governed landing zone and infrastructure as code
You can do meaningful AI work from level 1 or 2. What changes as you climb is how easily and safely you can build your own solutions, not whether you can use AI at all.
What actually matters
1. Cloud identity
Every AI service you’ll use signs people in through a cloud identity provider. If your users still live only in on-premises Active Directory, synchronizing them to your cloud identity platform with MFA and SSO is the real prerequisite. It’s also the foundation for everything in identity as your AI control plane.
2. Reaching data where it lives
You don’t have to migrate data to the cloud for AI to use it. Connectors, data gateways, and APIs can reach on-premises systems securely. The question is whether the specific data your use case needs is reachable, and through which system. The source-of-truth map answers that per use case, and APIs before agents covers how to open that access safely.
3. A governed place to build
The moment someone wants to build with a cloud AI service, you need an account structure with guardrails. The major cloud providers call this a landing zone, and each publishes reference guidance for one. For AI, a minimal version is enough to start; the checklist below covers it.
4. Cost controls
AI services are billed by usage. A test that loops overnight can produce a bill nobody expected. Budgets and alerts need to exist before the first build, not after the first invoice. The details are in managing usage-based AI spend.
5. Network, mostly for scale
Chat-style use barely touches your bandwidth. Bulk ingestion of documents into an AI service, or moving large datasets between clouds, can. Watch data egress charges in particular, since they’re easy to overlook in planning.
A decision rule for migration
Don’t migrate workloads for AI’s sake. Move a system or its data only when a specific AI use case can’t reach it where it is, or when you’d be moving it anyway for other reasons. AI is a good reason to finish a cloud strategy you already have. It’s a poor reason to start a lift-and-shift you weren’t planning.
The same logic applies to self-hosted models. They can make sense when data genuinely can’t leave your premises, when volume is high and steady enough that owning capacity is cheaper than renting it, or when you need low latency at the edge, such as on a factory floor. For a first AI program at a mid-size organization, those conditions are unusual. Start with managed services and revisit when a use case proves otherwise.
The asset: a minimum AI landing-zone checklist
Before anyone builds on a cloud AI service, put these ten things in place. Most take hours, not weeks, and together they move you toward level 4 without a big program.
- A dedicated account or subscription for AI work, separate from production, so experiments can’t touch live systems.
- Console access through SSO only, with no day-to-day use of root or owner credentials.
- Approved regions defined, matching your data residency commitments.
- Which models are enabled, and who approved them. Most platforms require you to turn on access to each model; treat that as an approval, not a formality.
- A budget with alerts at sensible thresholds, sent to someone who will act on them.
- Activity logging on and retained, flowing to wherever your security team already looks.
- Policy guardrails that restrict services and regions to the approved list.
- A tagging standard for owner, project, and cost center, enforced on creation.
- Private or controlled access to data sources, so AI services reach your data without exposing it publicly.
- A named owner for the AI environment who reviews spend, access, and enabled models monthly.
Write it as infrastructure as code if your team already works that way. If not, a documented manual build is fine to start. Consistency matters more than tooling at this stage.
Mistakes I see at this stage
Building in someone’s personal cloud account. An engineer signs up with a company card to “just try something.” Six months later it’s running a business process, outside every control you have. Give builders a governed account early so they don’t need to improvise.
Enabling every model “to see what’s there.” Each enabled model is a decision about data and cost. Enable the few you’ve evaluated, and add more on request.
Treating AI as a reason to rush a migration. A rushed migration creates security and cost problems that make AI harder, not easier. Keep AI and migration timelines separate unless one genuinely depends on the other.
No one with cloud skills. The landing zone, the guardrails, and the cost controls all need someone who knows the platform. If that person doesn’t exist on your team yet, which AI certification to get is a practical place to start planning the skills.
What moving up one level looks like
From 0 to 1: put cloud identity in place and license a governed AI assistant, even if nothing else moves. From 1 to 2: connect the systems your first use cases need and stand up the minimum landing zone above. From 2 to 3: write down a cloud strategy that includes where AI workloads will run. From 3 to 4: codify the landing zone and manage it through version control, like the rest of your infrastructure. For the full infrastructure picture, see is your infrastructure ready for AI?
Where does your team actually stand?
Infrastructure posture is one of 24 questions in the AI Readiness assessment, which covers six dimensions: data, security, infrastructure, skills, use cases, and governance. The free version is 10 questions and gives you a score in a few minutes.
Get your free AI Readiness Score →
Want to see what the full assessment covers first? Flip through a complete 38-page sample report.
Related guides
- Identity Is Your AI Control Plane: SSO, RBAC, and Access Reviews
- Where Should Your Business Data Live Before AI?
- APIs Before Agents: Giving AI Access to Your Systems Safely
- Budgets and Alerts for Usage-Based AI Spend
- Is Your Infrastructure Ready for AI? A Platform-by-Platform Check
Frequently asked questions
Do we need to move to the cloud before using AI?
No. For most mid-size organizations, the first AI capabilities arrive as SaaS assistants and AI features in tools you already use. What you need is cloud identity with MFA and SSO, secure ways to reach the data your use case needs, and a governed place to build later.
What is an AI landing zone?
A governed cloud environment for AI work: a separate account or subscription, SSO-only console access, approved regions, approved models, budgets with alerts, activity logging, policy guardrails, a tagging standard, controlled access to data sources, and a named owner. A minimal version takes hours, not weeks.
When does hosting our own AI models make sense?
Rarely for a first AI program. It can make sense when data genuinely can't leave your premises, when usage is high and steady enough that owning capacity is cheaper than renting it, or when you need low latency at the edge, such as on a factory floor.
Can cloud AI services use data that stays on premises?
Yes. Connectors, data gateways, and APIs can reach on-premises systems securely, so you don't have to migrate data first. The question is whether the specific data your use case needs is reachable, through which system, and under which identity.
