Case Study

9.1.1 Interfaces

As we've seen, the user interface consists of two parts located on the user’s local machine.

Once Bubble is globally installed as an npm package, the user can then access the CLI to initialize an instance of Bubble in a repository directory. This adds necessary files to the local repo and updates files for the dashboard to use.

The dashboard is a locally-run web application that provides a user-friendly visual interface for viewing, managing, and destroying preview apps. It combines a back-end, which makes API calls to AWS, with a front-end for the user to view in their browser.

9.1.2 GitHub Actions

Bubble uses GitHub Actions to automate the build, deploy, and destroy process for preview apps. These actions execute workflow files added by Bubble which are triggered through initial pull requests, subsequent commits to those requests, or HTTP requests to GitHub servers sent via the Bubble CLI.

GitHub Actions run on their own servers, so GitHub takes care of spinning them up and down and executing the workflow code.

9.1.3 AWS Services

The final piece of our architecture uses several AWS services to deploy the infrastructure for all preview apps.

Each preview app consists of three AWS microservices:

1. SSR logic and API routes are deployed to Lambda@Edge functions.
2. S3 Buckets store static assets such as images and pre-built files.
3. CloudFront serves both static and dynamic content by associating the functions and S3 buckets to each route of each preview app.

Permissions management and preview app metadata storage are handled by two additional services:

1. All AWS commands executed by Bubble are carried out through IAM users endowed only with the permissions they need to create, manage and destroy Bubble preview apps. This allows for separation of access from other resources on the user’s AWS account.
2. Two DynamoDB tables are provisioned to hold metadata related to users’ preview apps.

10.1.1 What Is GitHub Actions?

GitHub Actions is an automation tool that allows users to execute code written in repository files called "workflows" using GitHub Runners, which are temporary virtual servers that GitHub spins up and down.⁸

The process—using a Bubble deployment as an example—works as follows:

1. A Bubble workflow file in the user’s repository is set to listen for pull requests.
2. When a pull request is opened, the workflow is triggered and GitHub automatically spins up a runner to execute the workflow.
3. Based on the contents of the workflow file, the GitHub Runner builds and deploys a preview app.
4. After the workflow file finishes execution, the GitHub Runner is automatically spun down.

Instead of having to provision and configure our own build server to run each build process—which would add unnecessary complexity and setup work to our architecture—GitHub Actions provides a simpler way to integrate with GitHub and carry out the build process directly within the context of a repository.

10.1.2 Benefits & Drawbacks of Github Actions

As with most engineering decisions, there are tradeoffs associated with using GitHub Actions.

One challenge with using GitHub Runners is that they are entirely managed by GitHub and thus cannot be accessed directly, making it difficult to extract outputs and other environmental artifacts. In addition, development is much more constrained than it would be if we were configuring our own build server since workflows can only be defined with YAML files.

However, the benefits of using GitHub Actions ultimately outweighed these drawbacks when it came to Bubble’s needs. GitHub Actions includes a free tier that we estimate provides enough compute to make building and deploying preview apps very inexpensive, if not free, for a typical mid-sized team using Bubble.

Relying on GitHub Runners is also much simpler than implementing and configuring a custom build server and thus abstracts complexity away from our architecture.

Finally, since each section of a workflow file can be run on a separate GitHub Runner, GitHub Actions allows Bubble to deploy and tear down preview apps in parallel, making this process more efficient.

11.1.1 Containers

One option would be to deploy the server-side logic and static assets together in a container.

A container encapsulates the application code along with all the runtime elements necessary to execute it together in one package.⁹

With containers, we wouldn’t need to set up and configure multiple nodes of infrastructure or implement routing logic to connect them; instead, we’d be able to deploy only one single entity for each preview app, as the entire application would be spun up in the container as a whole. Since there’s only one piece to consider, containers can easily be moved around and deployed.

However, containers are constantly running, and would therefore incur costs even while the application is not in use. They also have longer build times since the virtualized OS and runtime need to be built and packaged together on top of the preview app itself.

11.1.2 Serverless Computing

An alternative to containers would be deploying the server-side logic and static assets to separate services. Given our use case of providing temporary preview apps, we found serverless functions to be the most compelling option for deploying server-side logic.

Despite being called serverless, these functions do run on servers, but they abstract away the logistics of provisioning server resources.¹⁰ They have no ongoing expenses: developers are only charged for the server capacity that their application uses during the time it’s being executed.

Since they don’t need to include system dependencies, serverless functions are also very fast to deploy, typically going live within milliseconds.

One downside of serverless functions is the added complexity of the architecture for preview apps: as opposed to containers, using serverless functions requires multiple nodes of infrastructure as well as routing logic to direct traffic between them.

Another inherent drawback is what’s called “cold starts”: since serverless functions aren’t constantly running, they have to be spun up before beginning execution. This can sometimes take several seconds, which can lead to longer loading times for SSR pages.

Ultimately, we chose to use serverless functions rather than containers. Bubble’s preview apps do not need to be highly performant, so cold starts are acceptable. In terms of the architectural complexity, although serverless functions necessitate additional infrastructure, being able to use separate services for SSR and SSG allows us to fully optimize for a hybrid approach which combines them.

As we delved further into utilizing serverless functions for our preview apps, we considered three open-source frameworks with which to automate deployment.

11.2.1 Terraform

Terraform is the first tool we considered.

Terraform is an “infrastructure as code” framework used to manage the lifecycle of cloud components.¹¹ There are many existing modules of Terraform for deploying frontend applications that are free to utilize and actively maintained.

Compared to other solutions, one of the most promising modules we found uses a more complex architecture to deploy apps. The AWS services used include CloudFront to handle incoming traffic, a Lambda@Edge function that acts as a proxy handler to route requests, an S3 bucket for static content storage, Lambda functions for serving the dynamic SSR pages, and an API Gateway that balances traffic between the Lambda functions.

The cost of this heavy underlying infrastructure is lengthy deployment times. During our test runs using this module with multiple applications, we found that deploying a simple app took more than 15 minutes. We also encountered a few deployment failures, which were challenging to debug because of this complexity and ultimately led us to question the reliability of this option.

11.2.2 FAB

Frontend Application Bundle (also called FAB) is another open-source framework that enables users to deploy frontend applications through a CLI.¹²

FAB zips up entire applications and ships static assets and server-side logic separately. Thus, the user has the option to choose separate platforms for serving static and dynamic content. For example, they could deploy static assets to AWS S3 while deploying serverless functions with Cloudflare workers. Although it aims to be platform agnostic, for serverless functions, FAB is currently most streamlined for Cloudflare Workers and AWS Lambdas.

When using FAB to deploy a few applications, it took no more than 5 minutes to deploy each app, which was significantly faster than other tools we considered, especially Terraform.

However, unlike the other options, FAB has not been maintained for over a year and a half.

11.2.3 Serverless Framework

The final option we considered was Serverless Framework.¹³

Serverless Framework deploys apps using only three AWS services: CloudFront, Lambda@Edge, and S3. This is a simpler architecture in comparison to the Terraform module, which may be a contributing factor to its faster deployment time. While provisioning the CloudFront distribution does take a few minutes, it still took less than half the time to deploy compared to Terraform.

One downside of using Serverless Framework is that unlike the flexibility of FAB, it currently only supports deployment on AWS infrastructure. Moreover, server-side logic can only be deployed to Lambda@Edge functions. However, while FAB is no longer maintained, Serverless Framework has continued to be a reliable framework considering how many people use it. Its contributors are still active and its latest version was released just a few months ago in March 2022.

In the end, we decided to go with Serverless Framework to build and deploy Bubble’s preview apps, due to its reliability, relatively fast deployment time, and simplified architecture, which would best serve Bubble’s goal of automating lightweight and convenient preview apps.

13.1.1 Polling

One solution we considered was polling: repeatedly trying to delete the lambdas and checking whether or not the attempt was successful. This would have involved setting up a process often referred to as a cron job, which enables developers to automatically and repeatedly execute scripts at specified times.

Had we chosen this approach, Bubble would have set up a cron job to execute in the background after the user executed bubble destroy: this cron job would have attempted to delete the Lambdas at a specified time every day until none remained.

The downside to this approach is that cron jobs run on the operating system: accordingly, configurations would have differed across operating systems. Multiple implementations would have had to be developed for different systems, all of them including the cron job itself as well as the manual setup of logging in case of errors, the monitoring of those logs, and the additional logic needed to restart the job if the computer were rebooted.

Rather than relying on the user’s local machine, another possibility would be implementing a cron job on AWS; however, this would increase costs for the user and add another resource for the application to manage.

As a result, we decided that cron jobs would not be a good fit for this specific task.

13.1.2 Additional CLI Command

Instead of cron jobs, we decided to add a command named bubble teardown which users can run themselves later after running bubble destroy to remove any remaining Lambda@Edge functions.

To make this change, we had to adjust how we maintained the information necessary to tell AWS to remove these functions. We decided to add another table to the DynamoDB database to store metadata for each Lambda. This meant that we also had to alter the workflow for building and deploying preview apps so that the Lambda metadata would be saved during each deployment.

This new table is now used when the user runs bubble teardown so that Bubble can identify which functions to delete when issuing AWS CLI commands; once all functions are successfully deleted, the table is then removed.

Modifying our build workflow and adding this additional table slightly increased the complexity of our application, but we felt it was simpler than implementing multiple cron jobs and the logic it would take to monitor and eventually remove them from the user's system or AWS.

13.2.1 Accessing the Logs

As previously discussed, we used Github Actions to deploy and build preview apps. One downside of GitHub Actions is how opaque the runtime environment is.

GitHub Actions run on temporary virtual servers which are spun up when a workflow is executed and spun down as soon as execution is complete; this means that the environment for each build is transitory. Therefore, we had to find a way to begin the process of capturing the logs live rather than after the fact.

Our first attempts to capture the logs during the build process were unsuccessful because the Action run logs are only available via an API call after the entire execution has finished.

We ended up having to cache the run_id for each workflow on GitHub and then implement an entirely separate workflow which was triggered immediately after each build action finished. This allowed us to access the build logs in their entirety within the new workflow.

13.2.2 Parsing the Logs

The next hurdle after retrieving the logs was cleaning and parsing them.

As part of displaying workflow execution logs on their site, GitHub injects a variety of metadata and special character sequences. These sequences are used by GitHub when streaming the logs on the GitHub Actions console on their site, but they lead to numerous string parsing errors when handled within a workflow on a GitHub Runner.

Trying to parse these logs was a painstaking process of manually going through thousands of lines of logs and attempting to eliminate these inserted characters without removing meaningful log data. The number of edge cases that required special handling kept growing and growing with no end in sight.

13.2.3 Storing the Logs

In addition to the snowballing parsing challenges, we also had to decide how and where we would store the logs. Build logs can easily contain thousands of lines, so they are not trivial to store.

Since we were already utilizing AWS microservices for our architecture, we limited our options to their offerings. We began by considering DynamoDB, which we were already using in each Bubble project. However, we realized that a set of logs could easily exceed the single-item storage limit.

Another option we thought about was using additional S3 buckets to store the logs as objects. After the initial parsing, there would be no need to ever update the logs; we could simply retrieve them later, in one piece, as they were originally stored. This means that object storage would be a good fit for the logs.

Although we’d found a suitable way to store the logs, another major concern that grew increasingly apparent was the potential impact of cost for the user: as the logs stored piled up, additional AWS charges would accrue for our end users.

As we progressed further down this route and realized both our development and architectural complexity were increasing notably, we chose to streamline this feature and move in a different direction.

13.2.4 Solution: Link to Logs

Instead of processing the logs ourselves and making them available in full on the Bubble dashboard, we decided to add a link on the dashboard for each preview app’s build log. When followed, this link takes the user directly to the GitHub workflow run where they have the option to download the logs for that specific preview app.

Although this option requires a couple more clicks for the user, it significantly reduces the complexity of Bubble itself as well as the storage needs and costs of the user’s AWS account.