DECISION NODE

In the halcyon days of back-end development, choices were straightforward, and the biggest debates were tabs versus spaces. But then, a deluge of tools—Kafka, Redis, Elasticsearch, and a buffet of SQLs and NoSQL databases—turned our development landscape into a complex jungle.

No sooner had we navigated this, the front-end realm decided to up the ante. Node.js, Webpack, sass, and JSX transformed the once straightforward world of HTML and CSS into a maze of frameworks and preprocessors, taking a short list of choice and growing it exponentially.

And now, DevOps takes the stage, introducing a new era of complexity. Gone are the days when server architecture decisions were as simple as choosing between buying hardware or renting a server. Today, we're faced with lambdas, managed container services, virtual machines, Kubernetes, and more. Amidst these, the option of buying your own hardware still survives into the new era.

Each choice comes with its own set of pros and cons, and, more often than not, a modern project will require a blend of these options to truly succeed.

So, let's embark on a journey through the realms of lambdas, managed containers, virtual machines, and BYO hardware. By the end of this guide, you'll be equipped to navigate the modern landscape of infrastructure with confidence, ready to make informed decisions for your next project. This article contains:

• An overview of lambdas, managed containers, VMs, Kubernetes, and BYO/Bare Metal hardware
• Factors you should consider when choosing a solution
• Detailed pros and cons of each option
• A few common use cases with recommendations

While this article doesn't provide a one-size-fits-all answer, it aims to equip you with the knowledge to make informed decisions.

This guide does not cover database scaling nor static hosting. Scaling a database has its own concerns that may be the subject of a future Decision Node.  For static hosting there are many solutions such as GitHub pages that can serve this purpose brilliantly.

Understanding Common Application Hosting Options

Before diving into the pros and cons of each hosting option, let's first understand what these options are and what they offer. Every provider has a different name for their services, each with similar feature sets.

Lambdas or Serverless Functions

A lambda typically represents a discrete task, like sending an email, resizing an image, or handling a specific request. By default, lambdas can scale down to zero and scale up as needed. Billing for lambdas is per instance, based on the active duration of the task, often calculated to the millisecond. Control over the hardware is minimal, the choice of languages is limited, and each instance only processes a single request at a time.

Managed Containers

A managed container service runs a Docker container that you create. While you retain complete control over the application, the service handles scaling and load balancing. These services have the ability to scale down to zero, although this may not be the default setting. Billing is determined by the running time of your container and the number of instances.

Kubernetes

Kubernetes is a platform designed to automate deploying, scaling, and operating application containers. It groups containers that make up an application into logical units for easy management. It allows for manual scaling as well as automatic scaling based many metrics. While Kubernetes itself is free, running it on cloud platforms may incur costs based on the compute resources consumed by the containers and the nodes (VMs) that host them. Kubernetes is a very popular cross-cloud option, meaning it runs very similarly in AWS, Azure, Google, and even on your own hardware.

Virtual Machines

A virtual machine is an emulation of a physical computer, running an operating system and any code you choose. They are usually running on shared hardware in a data center. You have the ability to SSH into it, install software, mount volumes, and customize it to your needs. Your service provider should have a variety of options for memory, CPUs, GPUs, and SSDs. While most services offer the capability for automatic scaling, it's not the default setting and they cannot scale down to zero. You're billed for the time the machine is running, regardless of whether it's being actively used.

BYO / Bare Metal Hardware

This is any hardware that you own or rent directly. It may mean renting a raw computer in a rack configured with GPU and other hardware you choose. It may involve purchasing servers and shipping them to a data center where they get unpacked, assembled, installed, and turned on. It may also mean buying a high power machine for use by an in-house team, perhaps for machine learning or AI.

Factors to Consider When Choosing a Hosting Option

When choosing a hosting option, consider the following factors to ensure your decision aligns with your project's needs.

Scalability

Handing a variable load can be a challenge for any architecture. If your load varies, or if you're hoping to go viral, you may need to scale up quickly, hopefully automatically.

Latency

The time between the request and the response is important for user experience. If this part of your app is customer facing, results should return as fast as possible, ideally in single digit milliseconds.

Quality Assurance Process

Somehow, you need to make sure your code works. Ideally before pushing to production. Having a QA environment is key, and having preview environments can take your process to the next level.

Developer Experience

Software ends up higher quality and can be made much faster when it's easier to develop. If engineers need to jump through hoops to make changes, they will waste time and find workarounds.

Price

Choosing the wrong infrastructure may result in paying for unused scale or features.

Maintenance

This can be more expensive than the bills you get from your cloud provider.

Risk

If something goes wrong, whose problem is it? What's the likelihood of something going wrong at 3 am, and if it does, is it something you need the expertise to fix, or is it part of your providers SLA?

Pros and Cons of Each Hosting Option

Each hosting option has its own set of advantages and disadvantages. Understanding these can help you make an informed decision that best suits your project's needs.

Lambdas (Serverless Functions)

Advantages:

• Scalability: Automatically scales from zero to an upper bound you choose
• Cost-Effectiveness: You pay only for the compute time you consume, reducing co``sts for sporadic or low-traffic applications.

Disadvantages:

• Latency: Although they often have low latency, lambdas are the only option where your boss will mention latency. If a request comes in and there's not an instance already running, ready to take the request, it will take 1/2 second or so to spin up a new instance.
• Limited Environments: Very little control over the environment, a limited set of languages, and typically a maximum execution time.
• Developer Experience: Running lambdas requires an emulator and may require triggering jobs manually.
• Shared Resources: Because each lambda has its own runtime, it is impossible to pool database connections. Although there are tools such as AWS RDS that can temper the problem, be aware that the lambdas may scale up faster than your infrastructure can handle.

Ideal Use Cases:

• Event-driven applications, such as image or data processing triggered by user actions.
• Background jobs where latency is ok.
• Jobs with hardware requirements that are out of line with the rest of your app, for example if you have many lightweight JSON endpoints, but a single job needs a lot of memory and a powerful CPU to train AI models.

Poor Fit:

• Long-running jobs and processes
• Tasks requiring extensive customization of the execution environment.
• APIs, HTML, and anything the user will interact with directly.
• Microservices, as latency issues can cascade.
• Jobs that directly open a connection to a database, as those connections cannot be pooled.
• Jobs that may scale past their dependencies, such as the database or search infrastructure

Managed Containers

Advantages:

• Scalability: Managed container services will automatically scale from zero (or 1) to a limit you choose. Load balancing is automatic.
• Latency: Instances can handle simultaneous connections, making cold starts very rare.
• Developer Experience: developers can run the container on their local machine, letting them run the code in an environment identical to production. The devcontainer standard makes this easier.
• Preview Environments: It can be so cheap and easy to spin up your application that CI can make a new instance for every PR.
• Deployment: With containers prebuilt, deploying is as easy as telling your provider which container to use.

Disadvantages:

• Hardware: options are often limited.
• Mounted volumes: Very few managed container services allow you to mount a high speed volume in a managed container. Each provider has its own file system alternatives and APIs.

Ideal Use Cases:

• Server rendered front ends, such as NextJS, Ruby on Rails, or any other front end server.
• API servers and microservices that need low latency, scalability, and deployment speed.

Poor Fit:

• Background jobs, it's not clear how many instances will be running and they may tax user facing APIs
• Databases, because there is no direct disk access.

Kubernetes

Advantages:

• Orchestration: Deploys multiple containers together, keeping your API and front end aligned.
• Portability: Applications can run on your laptop or any cloud provider.
• Scalability: Scales up easily.
• Community and Ecosystem: A large and active community, with a vast ecosystem of tools and add-ons for enhanced functionality.

Disadvantages:

• Complexity: Can be complex to set up and manage, especially for newcomers or smaller teams without dedicated DevOps resources.
• Resource Intensive: Can be resource-intensive, requiring more infrastructure overhead than simpler deployment models.

Ideal Use Cases:

• Microservices architectures that require robust service discovery and autoscaling.
• Multi-cloud applications.
• Tightly coupled apps, such as an API and a front end that are under active development.

Poor Fit:

• Small applications or projects where the overhead outweighs the benefits.
• Teams without the capacity to invest in learning and managing Kubernetes effectively.

Virtual Machines (VMs)

Advantages:

• Flexibility: VM providers offer a wide range of instance types to match specific workload requirements, from general-purpose to compute-optimized instances.
• Control: Full control including the operating system, configuration, and all applications running.

Disadvantages:

• Maintenance: Requires more management and maintenance effort compared to serverless/managed options.
• Risk: If something goes wrong, it may be something you or your team needs to fix.

Ideal Use Cases:

• Legacy applications that require a specific OS environment or extensive customization.
• Databases or other use cases requiring high speed disk access.

Poor Fit:

• Lightweight applications that can benefit from the agility and efficiency of containers or serverless.
• Background jobs that may scale up or down to extremes.

Bring Your Own (BYO) Hardware

Advantages:

• Complete Control: Full control over the hardware and software environment.
• Performance: Potential for higher performance by optimizing the hardware for specific tasks.
• Security: Can be more secure, as the entire stack is managed internally.
• Monthly Cost: Once you have paid the capital expenditure, monthly costs can be less than 10% the price of managed solutions.

Disadvantages:

• Upfront Costs: Significant upfront investment in hardware.
• Scalability: Scaling requires additional hardware purchases and physical space.
• Maintenance: Requires significant expertise to manage, secure, and scale effectively.
• Risk: If something goes wrong, you or your team will be the only ones responsible.

Ideal Use Cases:

• High-performance computing tasks that require specialized hardware configurations.
• Organizations with strict regulatory requirements that necessitate complete control over data and infrastructure.
• In-house data science, machine learning, and 3D rendering teams.

Poor Fit:

• Startups and small to medium enterprises that lack the capital for 5-6 figure hardware investments.
• Organizations without full DevOps teams.
• Applications that require rapid scalability to respond to fluctuating demand.

Making the Right Choice

Choosing the right hosting option depends on your specific use case. Here are some common scenarios and our recommendations for each.

Do you need specific hardware?

VMs, build your own

If you need common hardware, such as a GPU, a cloud provider can probably handle this for you. If you need more exotic hardware, such as cryptocurrency mining or AI chips, you will probably need to buy your own hardware. Lambdas, container instances, and Kubernetes offer no access to hardware.

Is it holding your database? Does it need high speed disk access?

VMs, build your own, or managed database services, Kubernetes

High speed disk access is important to databases, making it a great use case for VMs and BYO. Very few managed container services will let you mount a disk, and none will let you mount a disk in a lambda.

Also consider using a managed database services, such as AWS Aurora and MongoDB. These have similar price/complexity tradeoffs to VMs vs. containers.

Kubernetes can be used to host your database, but be very cautious because it was not designed for stateful use cases. There are tools such as persistent volumes and StatefulSets, but if your app is not configured correctly you may lose your data.

Does it access a database or other non-scaling shared resource?

Managed containers, VMs, Kubernetes, BYO

When your app scales automatically, they may overwhelm other parts of your infrastructure. This is of particular concern with lambdas, because requests all run in separate runtimes they cannot pool resources. This problem can be eased with tools like AWS RDS, but these incur additional expense and complication.

Is it a CRUD/GraphQL API?

Kubernetes, Managed containers

If you're making an API, latency is likely to be important. Most APIs can also serve multiple requests simultaneously because the database and network  are usually the limiting factors. Autoscaling and concurrency make managed containers and Kubernetes great choices.

Lambdas are a poor choice because their cold starts cause latency. Because lambdas cannot handle concurrent connections, you'll be paying for the lambda to be active while it's waiting on the database and network.

Are you making microservices?

Kubernetes, Managed containers

Microservices and quite chatty, calling each other many times for every request. This provides many opportunities for latency, and many opportunities for cold starts. This makes lambdas a poor choice.

Additionally, if a microservice calls your database directly, additional services and workarounds are needed to pool database connections.

Is it a front end?

Kubernetes, Managed containers

Two factors make managed containers and Kubernetes great for this situation: latency and preview environments. Your users will appreciate the speed, and your QA and product teams will love that they can see each PR before it's merged.

Will it usually be bound by network connections or database calls?

Kubernetes, managed containers, VMs

If your app is bound by external factors, concurrency can save considerably on both latency and cloud bills.

Is it a data ingestion process?

Lambdas

Because you only pay for the time used, and because 1/2 second of latency is rarely a deal breaker, lambdas are ideal for data ingestion.

Are the hardware requirements out of line with the rest of your infrastructure?

Lambdas

If the majority of your app is shuffling JSON around, but one part needs to resample video or train an AI, that part should be considerably more powerful than the rest. This is a good time for lambdas because you will only be paying for the time the high cost operation is running.

Is it a background operation?

Lambdas

If this part of your app is doing something like sending emails or indexing modified documents, it is nice to keep your worries to a minimum. Background and batch operations are a great fit for lambdas because latency is not important. Scaling from zero to infinity is very nice because it's likely that the job is inactive most of the time, but it also may scale up quickly.

Will preview environments help your process?

Managed containers, Kubernetes

Managed containers can generally be spun up fast enough and cheaply enough to make a container for every PR. This can allow QA and other concerned parties to see PRs before going live.

Doing this for lambdas can be more complicated because you need a whole environment. Spinning up VMs is much more expensive and time consuming.

Do you have in-house DevOps expertise?

If you do: Kubernetes or ask your team, if not: Managed containers and/or lambdas

If you don't have DevOps expertise, the problems that may crop up from managing your own hardware are unlikely to be worth the savings. Do you know what happens when your server hits 100% CPU? What happens if it runs out of disk space? What happens if traffic goes up 10x organically? What if it's 1000x from a DDoS attack? If you don't know, leave these problems to your service provider.

If you do have in house expertise, ask your team what they think.

In Summary

In conclusion, choosing the right hosting option is crucial for the success of your project. Managed containers often provide the best balance of scalability and latency, making them an ideal choice for many applications. Lambdas are well-suited for background tasks, but be aware of their limitations. While Kubernetes and bare metal offer more control, they also require more resources and expertise. Always consider your specific needs and resources before making a decision.

Be aware that your application is not the only part of your architecture. Your database, file storage, and search and more will need to scale along with your application.

‍