Experimenting with AI for Cloud Administration

There's lots of talk these days about using AI to accelerate software development via tools like GitHub Copilot and Amazon CodeWhisperer.

But what about cloud administration? Can AI also help cloud admins work more efficiently?

To find out, I put the platform from OpenAI — the AI company whose software also powers GitHub Copilot — to the test. Using the OpenAI playground's text-davinci-003 model, I posed several questions that a cloud admin might need to answer.

Download our free report: AI-Assisted Coding: What Software Developers Need to Know

Here are the results, lightly edited for readability and style, along with some analysis of what OpenAI's AI models do well — and not so well — when it comes to cloud administration.

Cloud Architecture Design

I started by asking OpenAI one of the very first questions anyone would ask when planning a cloud strategy: "Should I use AWS, Azure, or GCP?"

It responded:

The answer to this question depends on the specific requirements of your project. All three cloud providers offer different services, features and pricing models. Each platform has its own strengths and weaknesses, so it is important to consider the specific needs of your project and evaluate the different options before making a decision.

That's a fair answer, but it's not very detailed or specific. So I asked a more focused question: "What are the main advantages of AWS compared to Azure and GCP?"

OpenAI responded:

1. Cost: AWS is often the most cost-effective solution, offering the widest range of pricing and discounting options for businesses of all sizes.

2. Security and Compliance: AWS has the most comprehensive security and compliance offerings across all of the major cloud providers.

3. Flexibility: AWS gives you the most flexibility when it comes to deploying and managing applications. You can choose from a wide range of services and mix and match them as needed.

4. Support: AWS provides the most comprehensive customer support options, including 24/7 support and a global network of Certified Solutions Architects and Partners.

5. Speed of Deployment: AWS can help you quickly deploy applications and services, allowing you to focus on innovation instead of infrastructure.

6. Scalability: With AWS, you can quickly and easily scale your applications and services to meet your business needs.

Most of these answers are either subjective or too generic to prove right or wrong. Is AWS the "most cost-effective" cloud? Sometimes, but it depends on what you're using it for. Does AWS have the "most comprehensive security and compliance offerings"? Maybe, but that's highly debatable.

Related: Can AI Help Automate IT Operations?

Interestingly, when I asked OpenAI, "What are the main advantages of Azure compared to AWS and GCP?" it gave a pretty similar answer to when I asked what AWS' advantages were, except in this case it attributed the standout features to Azure:

1. Cost: Azure offers a more cost-effective solution than AWS and GCP for many services, including its Infrastructure as a Service (IaaS) and Platform as a Service (PaaS).

2. Performance: Azure offers a more reliable performance than AWS and GCP, with better availability and faster load times for applications hosted on its cloud.

3. Managed Services: Azure offers comprehensive managed services that allow businesses to focus on core activities instead of managing infrastructure. This includes the ability to deploy and manage applications, databases, storage, and networking.

4. Flexibility: Azure offers a wide range of services, allowing businesses to customize their cloud environment to meet their unique needs.

5. Security: Azure offers a world-class security model, including encryption, identity, and access management, and threat protection.

6. Scalability: Azure offers scalable infrastructure, allowing businesses to quickly add resources as needed.

The bottom line here is that OpenAI doesn't seem to understand the major differences between AWS, Azure, and GCP with a great deal of specificity. It doesn't say things that are wrong, exactly, when you ask it to compare cloud platforms, but it doesn't give answers that are very useful, either.

Understanding Cloud Services

OpenAI seems to perform a bit better when you ask it for guidelines related to specific cloud services, as opposed to cloud platforms.

For instance, when I asked, "What is the best EC2 instance type for a simple web application?" it answered:

The best EC2 instance type for a simple web application would depend on the specific requirements of the application, but a general purpose instance such as an Amazon EC2 t2.micro is often a good starting point. The t2.micro instance is cost-effective and offers enough resources to handle small workloads.

That's a perfectly valid answer that includes enough specificity to be valuable to a cloud admin, while avoiding making any recommendations that are subjective or overly generic.

On the other hand, when I asked OpenAI whether it was better to use EC2 or Lambda to host a web app, it said:

The answer to this question depends on your individual needs and preferences. Both Lambda and EC2/EC3 have their own advantages and disadvantages. Lambda is a serverless compute service that allows you to run code without managing any infrastructure. It is well-suited for applications that require quick response times and high scalability, as it can handle sudden peaks in demand without any extra effort. However, it can be more expensive than EC2/EC3 and may not be suitable for more complex tasks.

Most of that is true. The problem, though, is that there is no such thing as EC3 in the context of cloud computing. This appears to be a term that OpenAI completely made up — although if you weren't familiar with AWS cloud service names, you might not know the term is made up because the answer seems reasonable in most other respects.

Cloud Security Rules

OpenAI might understand the basics of how to use tools like IAM frameworks to secure cloud workloads, but its depth is limited.

When I asked it to "Write an IAM policy to secure an S3 bucket," it gave me this:

            "Version": "2012-10-17",            "Statement": [            {            "Effect": "Allow",            "Action": "s3:ListBucket",            "Resource": "arn:aws:s3:::example-bucket"            },            {            "Effect": "Allow",            "Action": [                        "s3:PutObject",                        "s3:GetObject",                        "s3:DeleteObject"            ],            "Resource": "arn:aws:s3:::example-bucket/*"            }            ]}

The policy appears valid, but whether this is a highly secure IAM policy is debatable. It doesn't lock down access to the data in the bucket in a particularly rigorous way.

That said, OpenAI has a fairly decent understanding of IAM security best practices. When I asked, "What are the most common configuration mistakes in IAM policies?" it said:

1. Not using IAM roles for cross-account access
2. Not using IAM groups to assign permissions
3. Not using IAM policies to limit access to specific resources
4. Not using least privileges when assigning IAM policies
5. Allowing unrestricted access to IAM users
6. Not using IAM conditions to restrict access
7. Not using AWS Organizations for centralized account management
8. Not using MFA for privileged IAM users
9. Not enforcing password policies for IAM users
10. Not regularly auditing IAM policies and users

There is much to debate here, but on the whole this is a valid and relatively specific answer.

Can AI Help with Cloud Administration?

So, can AI models like those from OpenAI help cloud admins?

The answer is "yes, but with limits." AI can provide some basic recommendations about how to configure cloud workloads or understand the differences between cloud platforms and services. But it sometimes states its recommendations with more confidence than it should, given that the recommendations are debatable. It also occasionally says things that are totally bogus, like the "EC2/EC3" answer from above.

Because of this, it's hard to say that AI in its current state is as useful for assisting with cloud administration tasks as it is for assisting with software development. AI-assisted coding tools work surprisingly well. But for now, AI-assisted cloud administration tools aren't even a thing, and if they want to become one, the AI models have some learning to do.