The Pulumi Book | Blog

The book has a new home

Thu, 04 Jun 2026 00:00:00 GMT

Welcome to the new home of The Pulumi Book! If you’ve been here before, things look a little different — and quite a bit more like a book.

What’s new

The site is now built with Astro Starlight, the same documentation framework used by a growing number of open-source projects. A few things come along for the ride:

The whole book is here. Every chapter, section, and subsection is navigable from the sidebar, with a table of contents on each page.
The examples are one click away. There’s a dedicated examples page listing every runnable Pulumi program from the book.
This blog. A place for me to post updates as the book grows — new chapters, reworked examples, errata, and the occasional aside.

Generated straight from the manuscript

Here’s the part I’m most pleased about: the entire book section of this site is generated automatically from the manuscript. I write the book in Markua for Leanpub, and a small converter turns those chapters into the pages you’re reading now — code listings, figures, cross-references, and all. When I push an update to the manuscript, the website rebuilds itself.

That means there’s no separate “web version” of the book to keep in sync. There’s just the book, and this is one of the ways you can read it.

What’s next

More chapters, for one. I’ll post here as new material lands, so check back — or grab the RSS feed if that’s more your speed.

Thanks for reading, and as always, you can find me on Twitter / X.

Happy Pulumifying!

Serverless on AWS #1: A serverless video converter

Sat, 19 Aug 2023 00:00:00 GMT

A year or so ago, with great enthusiasm, I ordered a copy of the second edition of Serverless Architectures on AWS — and when it finally arrived, I promptly set it aside, and almost forgot about it entirely.

Life, as it happens, has a way of getting in the way. Last week, though, I finally managed to pick it back up, and I’m excited to start working through it chapter by chapter. Serverless, once you manage to get your head around it, really does change the way you think about cloud computing in fundamental ways — and I happen to be one of those people who’s about as bought into it philosophically as one could be. Whenever possible, if I can solve a problem serverlessly, I’m going to give that a try first.

As of now, I’m only a couple of chapters into the book, but I’m enjoying it so far — and I love that it covers such a broad range of material. Flipping through the upcoming sections, though (each of which covers a particular serverless use case and architecture), it occurred to me that it might be fun to work through all of the book’s examples using Pulumi. (Most of the examples seem to use some combination of Serverless Framework, SAM, and the AWS CLI and/or console, but serverless architectures in general lend themselves surprisingly well to Pulumi — especially when your language of choice, like mine, happens to be JavaScript.)

So in this post, I’ll kick off this little series-to-be by jumping right into the first architecture covered in the book: a simple serverless video-conversion pipeline.

Setting the stage

The hypothetical scenario is simple: You’ve got a video file somewhere, maybe something you shot with your phone, but it’s big — several gigs, say, so much too large to just throw up onto the web or drop into a blog post. To make use of this giant raw video file, you need a way to compress and resize it into a web-friendly format — which, these days, probably means converting it into an H.264 encoded MP4.

Of course, you could easily do this with something like iMovie. But let’s assume that for whatever reason, you can’t; maybe you don’t have an iPhone or a Mac, or you do, but you actually need to transcode like a thousand videos for work, or something, or you’ve found yourself in some other such situation that truly requires a bit of cloud automation. How do you do it? And more importantly, how do you do it serverlessly on AWS?

Serverless Architectures to the rescue

Serverless Architectures on AWS offers a clear and ostensibly simple solution: Wire together a few high-level Amazon services and call it a day. The fundamental idea is that you should be able start with a large video file, throw it at AWS, and wait patiently for the transcoded version (or versions) to emerge through the magic of AWS (and perhaps a little coding on your part).

Turns out that’s totally doable, and there really isn’t all that much to it — architecturally at least. The book proposes the following simple setup:

Here, the user — e.g., you — begins by uploading a video to an S3 bucket. A Lambda function configured to respond to file uploads handles the event by parsing its metadata (for the name of the file, its location, etc.) and then contacting an AWS managed service called Elemental MediaConvert to request a new conversion job, passing along a few details like the desired file format, bit rate, and name of the bucket where the transcoded files ultimately should be stored. MediaConvert then does its thing, converting the file according to your specs and writing the new video to S3.

And that’s pretty much it. And of course, because it’s all serverless, it scales automatically (meaning you can throw as many videos at the pipeline as you like), and you only pay for the time it takes AWS to run your Lambdas and convert each video file.

But the approach the book takes to implement this design — as an example of infrastructure as code — is a surprisingly awkward and laborious one. Here’s how it’s done:

First, in the AWS console, navigate to the S3 dashboard and create a new bucket for your transcoded video files — the ones MediaConvert will eventually create for you. Be sure to come up with a unique name for this bucket because no two S3 buckets can ever share the same name anywhere in the known universe. And remember the name you choose. You’ll need it later.
Then, navigate to the AWS IAM dashboard and create a new role to grant AWS Lambda permission to make calls on the MediaConvert service. Make sure you copy the new role’s Amazon Resource Name (ARN) as well — you’ll need that later, too.
Next, make another new IAM role, this one to grant MediaConvert permission to write video files to S3. Copy this role’s ARN somewhere also.
Navigate to the MediaConvert dashboard and locate your AWS account’s assigned MediaConvert API endpoint. (AWS assigns these automatically.) As you do this, doublecheck that the currently selected region (as shown in the upper-right corner of the console) is the same one you’re planning to deploy into later, as MediaConvert API URLs are region-specific. Copy the endpoint’s unique URL. The Lambda function will need it to submit conversion requests.
Install the Serverless Framework CLI and generate a new aws-nodejs project.
Open the generated YAML file and paste these four strings — destination bucket, Lambda role ARN, MediaConvert role ARN, and MediaConvert API endpoint — into their proper positions in the file. Add a few lines to expose these values as environment variables (so the Lambda can use them at runtime), then fill out the generated JavaScript function to complete the Lambda itself.
Finally, run serverless deploy to provision the upload bucket and Lambda resources (because you’ve already created everything else by hand).

Easy, right?

So, here’s the thing. This totally works, and the architecture is fine; it’s definitely the right set of AWS resources to use for an app like this one. All of the ingredients — buckets, IAM roles, function code — are necessary; Lamba needs permission to talk to MediaConvert, MediaConvert needs permission to write to S3, and so on. As far as the architecture itself is concerned, given the various product offerings and technical constraints of AWS, what’s presented here makes total sense, design-wise.

But man — all this clicking around in the AWS console, hard-coding of thought-up bucket names, copy-pasting of URLs and ARNs — all of it makes this simple job way more complicated and cumbersome than it needs to be. To me, this is very much not what infrastructure as code is supposed to look like.

Infrastructure-as-code, to me, is all about hands-off automation. The moment you, as practitioner of IaC, find yourself filling out web forms and clicking buttons to make resources or hard-coding or copying and pasting anything should be the moment your brain tells you that something is wrong. All of this stuff should be expressible in code, not just a sliver of it — and all of it within the scope of a single program. On top of that, you — and perhaps more importantly, anyone else — should be able to look at your program code and easily grasp how it all hangs together; you shouldn’t have to toggle between your IDE and the AWS console to assemble a mental picture of how an application does its job — particularly an application as simple as this one. And finally, you should also be able to stand up and easily switch between multiple deployment environments (think dev, staging, production) to target whichever environment you need, without having to change anything (like hard-coded S3 bucket names or resource IDs) in the code itself.

How it’s done with Pulumi

Now let’s see how you might go about implementing this same architecture with Pulumi.

For starters, you’ll need a new project. So if you haven’t already (assuming you’re interested in actually doing this yourself), install Pulumi and configure it for AWS. You can read all about how to do both in the getting-started guide.

Create the project using a starter template. There are many such templates to choose from, but for applications like this one (and for most serverless apps in general), I usually start with the aws-typescript template, as it gives you access (by way of the Node.js SDK) to Pulumi’s built-in support for function serialization, which is incredibly handy for wiring up JavaScript Lambda functions (as you’ll see in a moment). Start with this:

pulumi new aws-typescript

The template will prompt you for a handful of properties like the project name, stack name, and the AWS region to deploy into. Step through them, then open the project in your editor of choice.

A quick look at what we’ll be building

The biggest differences between the program you’re about to build and the implementation described in Serverless Architectures are these:

In the Pulumi version, there’s no need to create anything manually
You can do it all within the scope of a single 100-line Node.js program

Every component required by the program can be created or obtained by program itself: buckets, globally unique bucket names, IAM roles, service endpoints, all of it. No visits to the AWS console (or hard-coding of strings) required.

Secondly, the whole application can be written in one language — TypeScript (or if you like, plain ol’ JavaScript) — and expressed in a single TypeScript file. Thanks to the magic of Pulumi’s Node.js SDK, you can code both application and infrastructure in the same program — a huge advantage that makes writing, debugging, testing, refactoring, reviewing, and most importantly understanding the code way easier than having it all managed in multiple places by hand.

The infrastructure requirements, however, are the same. You’ll still need:

A bucket for uploads
Another bucket for transcodes
A Lambda function to invoke MediaConvert when a video is uploaded
A couple of IAM roles to grant Lambda and MediaConvert the permissions they need to do the work they need to do

At some point, it might be nice to have AWS tell you when a transcoding job is complete — but we can figure that out later. For now, this is fine; we should have all we need in terms of requirements to get going.

Now let’s do some actual building.

Begin by declaring the source and destination buckets

Open index.ts and clear out the boilerplate code that was generated by the new-project wizard, replacing it with the following lines:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const inputBucket = new aws.s3.Bucket("input", { forceDestroy: true });
const outputBucket = new aws.s3.Bucket("output", { forceDestroy: true });

I’ll explain what each line does as we go.

Lines one and two just import the @pulumi/pulumi and @pulumi/aws libraries — nothing too exciting there. (The former contains general-purpose Pulumi APIs for things like configuration, outputs, and stacks; the latter for working with AWS resources specifically.) The two lines that follow declare the program’s input and output buckets. Notice that for both buckets, we include the optional forceDestroy argument, which tells Pulumi to delete the bucket even when it’s not empty. In real-world scenarios, you might not want to use this setting (it’s false by default), as it’s there to keep you from accidentally deleting your data, but for us, it’s better to use it, as it’ll make make things easier to clean up when we’re done. If we didn’t set this to true, pulumi destroy would fail loudly with a complaint about the buckets being non-empty. Since we want to be able to clean everything up (videos and all) in one step, forceDestroy is the way to go.

Notice as well that we didn’t have to think up any globally unique names for the S3 buckets. Pulumi’s auto-naming feature takes care of that for us. (As an aside, I must say that I’m a huge and unabashed fan of auto-naming; if you’re using Pulumi and not using auto-naming, you almost surely should be. Yes, I get that the random naming can look a bit strange at first — but the quality-of-life improvements you get in exchange for these quirky appearances is so much more than worth it, I promise.)

Grant MediaConvert permission to write to S3

With the buckets defined, create an IAM role to grant MediaConvert permission to write new videos to the output bucket. Add the following lines to the program:

// ...

const convertRole = new aws.iam.Role("convert-role", {
    assumeRolePolicy: {
        Version: "2012-10-17",
        Statement: [
            {
                Effect: "Allow",
                Action: "sts:AssumeRole",
                Principal: {
                    Service: "mediaconvert.amazonaws.com",
                },
            },
        ],
    },
    managedPolicyArns: [aws.iam.ManagedPolicy.AmazonS3FullAccess],
});

This aws.iam.Role gives full (i.e., write) S3 access to any MediaConvert job that assumes it. Next, you’ll see how to use this role to post a new transcoding job to the MediaConvert service from AWS Lambda.

Write the Lambda upload handler

Now it’s time to write the function that’ll be run when a file shows up in the input bucket. This function is essentially where your program logic lives: It’s where you codify this: When I upload a video, transcode it with these settings and put the result in this bucket.

There are a few things this function needs to do:

First, it needs to extract the name of the uploaded file from the Lambda’s event argument. (It needs this to tell MediaConvert which video file to convert.)
Once it has the name, it can use the AWS SDK for JavaScript (which Pulumi bundles with @pulumi/aws) to instantiate a MediaConvert object and call createJob() — but it’ll need a couple of values first. One is the region of the MediaConvert service to use. (Recall that MediaConvert APIs are all region-specific.) The other is the URL of the region’s MediaConvert endpoint. Both can be read by the function at runtime — the region by reading the value currently configured on the Pulumi stack (which you set a moment ago when you ran pulumi new) and the URL by using that region to look up the endpoint with a call to describeEndpoints().
In addition to those two values, the function also needs the ARN of the IAM role you just defined (the one MediaConvert will assume when it runs) and the name of the output bucket to write to. Both can be read similarly using the .get() method available on all Pulumi outputs.
It needs need to specify the settings to use for the transcoding job itself. These, as you might imagine, can get pretty complex, but thankfully AWS provides a few shortcuts for the most common ones. (See Supported Output Codecs and Containers and Working with Output Presets in the AWS docs for details.)
Finally, it needs to be granted permission to make calls on the MediaConvert service. You can do this with the preconfigured IAM policy AWSElementalMediaConvertFullAccess.

Here’s the whole block, with comments. Go ahead and add this to index.ts now:

// ...

const region = new pulumi.Config("aws").require("region");

inputBucket.onObjectCreated(
    "handler",
    new aws.lambda.CallbackFunction("handler", {
        policies: [
            aws.iam.ManagedPolicy.AWSLambdaExecute,
            "arn:aws:iam::aws:policy/AWSElementalMediaConvertFullAccess",
        ],
        callback: async (event: aws.s3.BucketEvent) => {
            // Get the name of the file that was uploaded.
            const key = event.Records![0].s3.object.key;

            // Look up the region-specific MediaConvert endpoint.
            const client = new aws.sdk.MediaConvert({ region });
            const endpoints = await client.describeEndpoints().promise();
            const endpoint = endpoints.Endpoints![0].Url;

            // Submit a new MediaConvert job request.
            const jobRequest = await new aws.sdk.MediaConvert({ endpoint })
                .createJob({
                    Role: convertRole.arn.get(),
                    Settings: {
                        Inputs: [
                            {
                                FileInput: `s3://${inputBucket.id.get()}/${key}`,
                                AudioSelectors: {
                                    "Audio Selector 1": {
                                        SelectorType: "TRACK",
                                        Tracks: [1],
                                    },
                                },
                            },
                        ],
                        OutputGroups: [
                            {
                                Name: "File Group",
                                Outputs: [
                                    {
                                        Extension: "mp4",
                                        Preset: "System-Generic_Hd_Mp4_Avc_Aac_16x9_1280x720p_24Hz_4.5Mbps",
                                    },
                                ],
                                OutputGroupSettings: {
                                    Type: "FILE_GROUP_SETTINGS",
                                    FileGroupSettings: {
                                        Destination: `s3://${outputBucket.id.get()}/${key}`,
                                    },
                                },
                            },
                        ],
                    },
                })
                .promise();

            // Log the request result.
            console.log({ jobRequest });
        },
    }),
);

A few things to note about what’s going on here:

The first line just reads the AWS region from the stack configuration, as mentioned above.
The next line uses a couple of Pulumi resources to create a Lambda event handler known as a magic function. The Pulumi docs explain how magic functions work in more detail, but the gist is that when they’re expressed in this way, Pulumi is able to package up Lambda functions for you (along with their dependencies and any closed-over variables) and create the S3 triggers to invoke the function in response to interesting events — e.g., uploads.
The IAM policies to apply to the Lambda are passed as an array of strings — one of which, sadly, lacks a TypeScript constant of its own. At deploy-time, Pulumi will use these policies to create an IAM role for the Lambda to assume when it runs in response to an upload event.
The event passed into the function is exposed as a typed aws.s3.BucketEvent — a useful thing for figuring out (with the help of your IDE’s built-in type hinting) how to unwrap that event to get at its most useful parts.
Because we’re working with Node.js, we can also use the syntactic sweetness of async/await to make working with the promise-based AWS SDK a little nicer.
The console.log() statement at the end writes an entry to Amazon CloudWatch logs for the function — a convenient way to debug runtime behavior when you need to. (In a moment you’ll see how convenient that can be.)

With the buckets, IAM role, and Lambda now written, you’re just about ready to deploy to AWS. All you need now are the names of your buckets-to-be.

Export the source and destination bucket names

Add the following two lines to finish the program:

// ...

export const inputBucketID = inputBucket.id;
export const outputBucketID = outputBucket.id;

These last two lines expose the input and output bucket names as Pulumi stack outputs. Technically, they aren’t programmatically necessary, but if we left them out, we’d have to do a little digging to find out what the generated (i.e., auto-named) input and output bucket names were in order to work with them. Exporting them as outputs will render them onscreen when the deployment completes — and also give us a handy way of referencing them (as you’ll see) without having to copy and paste anything on the command line.

Deploy!

Now it’s time to push this program out into the world and start using it. Here, for reference, is what you should now have for index.ts:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Read the AWS region from the currently selected stack.
const region = new pulumi.Config("aws").require("region");

// Provision two buckets: one for uploads, one for transcodes.
const inputBucket = new aws.s3.Bucket("input", { forceDestroy: true });
const outputBucket = new aws.s3.Bucket("output", { forceDestroy: true });

// Define a role that grants MediaConvert permission to write to S3.
const convertRole = new aws.iam.Role("convert-role", {
    assumeRolePolicy: {
        Version: "2012-10-17",
        Statement: [
            {
                Effect: "Allow",
                Action: "sts:AssumeRole",
                Principal: {
                    Service: "mediaconvert.amazonaws.com",
                },
            },
        ],
    },
    managedPolicyArns: [aws.iam.ManagedPolicy.AmazonS3FullAccess],
});

// Handle uploads by extracting the video filename and creating a new MediaConvert job.
inputBucket.onObjectCreated(
    "handler",
    new aws.lambda.CallbackFunction("handler", {
        policies: [
            aws.iam.ManagedPolicy.AWSLambdaExecute,
            "arn:aws:iam::aws:policy/AWSElementalMediaConvertFullAccess",
        ],
        callback: async (event: aws.s3.BucketEvent) => {
            // Get the name of the file that was uploaded.
            const key = event.Records![0].s3.object.key;

            // Look up the region-specific MediaConvert endpoint.
            const client = new aws.sdk.MediaConvert({ region });
            const endpoints = await client.describeEndpoints().promise();
            const endpoint = endpoints.Endpoints![0].Url;

            // Submit a new MediaConvert job request.
            const jobRequest = await new aws.sdk.MediaConvert({ endpoint })
                .createJob({
                    Role: convertRole.arn.get(),
                    Settings: {
                        Inputs: [
                            {
                                FileInput: `s3://${inputBucket.id.get()}/${key}`,
                                AudioSelectors: {
                                    "Audio Selector 1": {
                                        SelectorType: "TRACK",
                                        Tracks: [1],
                                    },
                                },
                            },
                        ],
                        OutputGroups: [
                            {
                                Name: "File Group",
                                Outputs: [
                                    {
                                        Extension: "mp4",
                                        Preset: "System-Generic_Hd_Mp4_Avc_Aac_16x9_1280x720p_24Hz_4.5Mbps",
                                    },
                                ],
                                OutputGroupSettings: {
                                    Type: "FILE_GROUP_SETTINGS",
                                    FileGroupSettings: {
                                        Destination: `s3://${outputBucket.id.get()}/${key}`,
                                    },
                                },
                            },
                        ],
                    },
                })
                .promise();

            // Log the request result.
            console.log({ jobRequest });
        },
    }),
);

// Export the input and output bucket IDs.
export const inputBucketID = inputBucket.id;
export const outputBucketID = outputBucket.id;

Deploy the program with a single pulumi up, checking the preview to make sure everything looks right:

$ pulumi up
...

Do you want to perform this update? yes
Updating (dev)

     Type                                  Name              Status              Info
 +   pulumi:pulumi:Stack                   mediaconvert-dev  created (3s)        4 messages
 +   ├─ aws:iam:Role                       handler           created (0.93s)
 +   ├─ aws:iam:Role                       convert-role      created (1s)
 +   ├─ aws:s3:Bucket                      output            created (1s)
 +   ├─ aws:s3:Bucket                      input             created (1s)
 +   │  ├─ aws:s3:BucketEventSubscription  handler           created (0.40s)
 +   │  │  └─ aws:lambda:Permission        handler           created (0.35s)
 +   │  └─ aws:s3:BucketNotification       handler           created (0.73s)
 +   ├─ aws:iam:RolePolicyAttachment       handler-aadec3c3  created (0.34s)
 +   ├─ aws:iam:RolePolicyAttachment       handler-2cc11edf  created (0.48s)
 +   └─ aws:lambda:Function                handler           created (13s)

Outputs:
    inputBucketID : "input-8a99139"
    outputBucketID: "output-01cb7a4"

Resources:
    + 11 created

Duration: 20s

When the deployment completes (which should take only a few seconds), you’ll be left with your two uniquely named buckets.

That’s it! Now let’s transcode some video.

Upload a video to convert

Open a new terminal tab and navigate to the directory containing the project, then run the following command:

$ pulumi stack output inputBucketID

In response, you should get the name of the auto-named input bucket. Here’s what I see when I run that command, for example:

input-8a99139

Now, back in the other terminal tab (the one where you just ran pulumi up), run pulumi logs to tail the CloudWatch logs for the Lambda you just created:

$ pulumi logs --follow

Collecting logs for stack dev since 2023-07-07T15:25:22.000-07:00.

Let that run, then switch over to the tab with the input bucket name and upload a video file from your computer with the AWS CLI. Use pulumi stack output to pass the name of the input bucket to aws s3 cp:

$ aws s3 cp ~/Desktop/wailua-falls.mov s3://$(pulumi stack output inputBucketID)/

upload: ../../../Desktop/wailua-falls.mov to s3://input-8a99139/wailua-falls.mov

In a few seconds, your still-running pulumi logs command should show you that the Lambda was invoked and the convert job created. Here’s what I see:

Collecting logs for stack dev since 2023-07-08T04:43:55.000-07:00.

2023-07-08T05:53:52.739-07:00[               handler-6a8088f] INIT_START Runtime Version: nodejs:16.v15        Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:ce158dcc19c42286fef86a8dfb67e1efd92d0de18828736a00f3698410aabcb3
 2023-07-08T05:53:52.857-07:00[               handler-6a8088f] START RequestId: ee791b7f-c245-41bd-8a5e-d180847cf65a Version: $LATEST
 2023-07-08T05:54:01.170-07:00[               handler-6a8088f] 2023-07-08T12:54:01.170Z ee791b7f-c245-41bd-8a5e-d180847cf65a    INFO    {
  jobRequest: {
    Job: {
      AccelerationSettings: [Object],
      AccelerationStatus: 'NOT_APPLICABLE',
      Arn: 'arn:aws:mediaconvert:us-west-2:616138583583:jobs/1688820840907-0s24me',
      ClientRequestToken: 'f477b33d-7913-4285-9ae4-23d36282601e',
      CreatedAt: 2023-07-08T12:54:00.000Z,
      Id: '1688820840907-0s24me',
      Messages: [Object],
      Priority: 0,
      Queue: 'arn:aws:mediaconvert:us-west-2:616138583583:queues/Default',
      Role: 'arn:aws:iam::616138583583:role/convert-role-b765342',
      Settings: [Object],
      Status: 'SUBMITTED',
      StatusUpdateInterval: 'SECONDS_60',
      Timing: [Object]
    }
  }
}
 2023-07-08T05:54:01.209-07:00[               handler-6a8088f] END RequestId: ee791b7f-c245-41bd-8a5e-d180847cf65a
 2023-07-08T05:54:01.209-07:00[               handler-6a8088f] REPORT RequestId: ee791b7f-c245-41bd-8a5e-d180847cf65a   Duration: 8352.45 msBilled Duration: 8353 ms Memory Size: 128 MB     Max Memory Used: 86 MB  Init Duration: 117.12 ms

Soon (depending on the length of the video you submitted), you should see a new transcoded video appear in the output bucket:

$ aws s3 ls s3://$(pulumi stack output outputBucketID)/
2023-07-08 05:54:10 7954412 wailua-falls.mov.mp4

Copy that file from the output bucket to your computer:

$ aws s3 cp s3://$(pulumi stack output outputBucketID)/wailua-falls.mov.mp4 .

download: s3://output-01cb7a4/wailua-falls.mov.mp4 to ./wailua-falls.mov.mp4

And behold — the delightfully web-friendly result:

0:00/0:131×

Feel free to submit a few others, fiddle with the conversion settings, etc., to get a feel for how everything works. There’s a ton you can with this service — way more than I could ever hope to cover in a post like this one. See the MediaConvert docs for some creative inspiration, and when you’ve had enough fun for today, read on to learn about how to clean everything up.

Tidying up

Just as you were able to stand up this whole stack with a single pulumi up, you can tear it all down with a single pulumi destroy. Doing so removes everything you created with the program in one go — the Lambda, IAM roles, and both buckets, along with their contents. As before, you’ll get a preview of the changes before Pulumi actually applies any of them:

$ pulumi destroy

Previewing destroy (dev)
...

     Type                                  Name              Plan
 -   pulumi:pulumi:Stack                   mediaconvert-dev  delete
 -   ├─ aws:iam:RolePolicyAttachment       handler-aadec3c3  delete
 -   ├─ aws:iam:RolePolicyAttachment       handler-2cc11edf  delete
 -   ├─ aws:lambda:Function                handler           delete
 -   ├─ aws:iam:Role                       convert-role      delete
 -   ├─ aws:s3:Bucket                      input             delete
 -   │  ├─ aws:s3:BucketNotification       handler           delete
 -   │  └─ aws:s3:BucketEventSubscription  handler           delete
 -   │     └─ aws:lambda:Permission        handler           delete
 -   ├─ aws:s3:Bucket                      output            delete
 -   └─ aws:iam:Role                       handler           delete

Outputs:
  - inputBucketID : "input-8a99139"
  - outputBucketID: "output-01cb7a4"

Resources:
    - 11 to delete

Do you want to perform this destroy?
> yes
  no
  details

Choose yes if it all looks right (which it should):

Do you want to perform this destroy? yes
Destroying (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/christian-pulumi-corp/mediaconvert/dev/updates/29

     Type                                  Name              Status
 -   pulumi:pulumi:Stack                   mediaconvert-dev  deleted
 -   ├─ aws:iam:RolePolicyAttachment       handler-aadec3c3  deleted (0.90s)
 -   ├─ aws:iam:RolePolicyAttachment       handler-2cc11edf  deleted (0.79s)
 -   ├─ aws:lambda:Function                handler           deleted (0.84s)
 -   ├─ aws:s3:Bucket                      input             deleted (0.81s)
 -   │  ├─ aws:s3:BucketNotification       handler           deleted (0.75s)
 -   │  └─ aws:s3:BucketEventSubscription  handler           deleted
 -   │     └─ aws:lambda:Permission        handler           deleted (0.45s)
 -   ├─ aws:s3:Bucket                      output            deleted (1s)
 -   ├─ aws:iam:Role                       handler           deleted (0.81s)
 -   └─ aws:iam:Role                       convert-role      deleted (1s)

Outputs:
  - inputBucketID : "input-8a99139"
  - outputBucketID: "output-01cb7a4"

Resources:
    - 11 deleted

Duration: 6s

And there you have it.

Wrapping up

I must admit, as an amateur photographer and filmmaker, I tend to get pretty excited about this sort of thing; I love how easy these tools make solving these kinds of real-world media-management problems. Dealing with stuff like this can be a huge pain sometimes — especially when you’re drowning in terabytes of HD video and all you’re trying to do is share a few clips of your kids playing soccer with their grandparents.

And as a developer, it kind of blows my mind how easily I can do this, and with so little operational overhead. It’s one thing to provision a virtual machine and throw files at it once a week while it sits there running (and costing you money) 24 hours a day, begging to be compromised. It’s quite another to deploy configuration like this that doesn’t really do anything at all until it’s asked, and then quietly shuts back down when it’s finished doing whatever it is you’ve asked it to do. Even if you aren’t quite as sold on serverless as I am (yet!), you’ve got to admit that it’s nice not having to think about — let alone pay for or manage — running infrastructure that you only need every once in a while.

So I guess that’s it for this episode. Keep an eye out for more in the weeks ahead, subscribe if you like (right here if RSS is your jam), and check out the repo below for the code. Comments, questions, reach out anytime.

Thanks for reading!

Sync a Folder to Any Cloud with Pulumi

Sat, 22 Oct 2022 00:00:00 GMT

Lately I’ve been hacking away on a handful of Pulumi templates to make it easier to deploy common cloud architectures --- static websites, serverless apps, containers, virtual machines, and the like. The idea with templates is that with a single run of pulumi new, you should be able to kick out a finished program, in your language of choice, that works out of the box and can handle the most common scenarios, while still leaving you plenty of room for expansion and customization. Templates, I think, are one of Pulumi’s most unsung (and unique) features; unlike components, they’re open rather than closed, so they tend to be great not just for bootstrapping new projects, but also as tools for learning, as they’re pretty much made to be tinkered with.

My first task was to build out a set of templates for static websites, which, as you probably know, are essentially just a bunch of HTML, CSS, and JavaScript files stuffed into a folder somewhere. This site, for example, is a static website built with Next.js, one of the more popular static-site generators these days, and as of today, it clocks in at a little over a thousand pages. At work, we build pulumi.com with another popular static-site generator called Hugo, and all up, that site comes in at a little over 20,000 pages. Static websites vary drsmatically in terms of size and complexity, but ultimately they all have that same thing in common: they’re all just a bunch of files in a folder on a computer connected to the internet --- a computer that likely belongs either to Amazon Web Service, Microsoft Azure, or Google Cloud Platform.

But while building out the first set of these templates, I ran into a problem: getting files into the cloud isn’t always as easy as you’d think, at least not with declarative infrastucture-as-code tools like Pulumi.

Here’s an example. Imagine you had a bunch of files in a folder on your computer that you wanted to get into the cloud. With Pulumi, the way to do that would be to begin by writing a program declaring some cloud storage, which, if you were using AWS, might look something like this:

import * as aws from "@pulumi/aws";

// Make a bucket.
const bucket = new aws.s3.Bucket("my-bucket");

Easy enough, bucket created. Now, let’s upload some files!

Here’s where things get tricky, though. Most of the examples you’ll find online (including several I’ve written myself) handle this by using the programming environment to fetch a list of files from the filesystem, then declare them, one by one, as individual cloud resources, like so:

import * as aws from "@pulumi/aws";
import * as fs from "fs";
import * as mime from "mime";
import * as glob from "glob";

// Make a bucket.
const bucket = new aws.s3.Bucket("my-bucket");

// List the files in `myFolder`, then for each one, declare an `s3.BucketObject`.
glob.sync(`${myFolder}/**/*`).forEach((path: string) => {
    if (!fs.lstatSync(path).isDirectory()) {
        let object = new aws.s3.BucketObject(path.replace(siteDir, ""), {
            bucket: this.resource,
            source: new pulumi.asset.FileAsset(path),
            contentType: mime.getType(path) || "text/plain",
        });
    }
});

And this works, to be sure. With Python, Go, C#, Java, things look a bit different of course, but in general, the approach is the same: make a bucket, then use the facilities of your chosen programming language to get the files into the bucket.

But while being able to break out of the declarative box and write a little code when you need to is awesome --- indeed, it’s one of the other things that makes Pulumi so special --- there’s something about having to write code like this I also find a bit irritating. All I need to do point a push a folder into the cloud. Shouldn’t I be able to just say that? Why do I have to dig into the filesystem, generate and manipulate a bunch of file paths, programmatically deduce MIME types, and so on, just to move a bunch of stuff verbatim from one folder to another? I mean it’s great that you can do that, sure. But it’d also be great if you didn’t have to.

And sometimes, you just can’t. Take YAML, for instance. I love our YAML support. It’s so tidy:

resources:

resources:
    # Make a bucket.
    my-bucket:
        type: aws:s3:Bucket

But there’s one little problem with YAML: It isn’t a programming language. Unlike with languages like TypeScript and others, you can’t break out of the box and reach into the filesystem with YAML --- and that’s by design. We built YAML to give our users the tersest, most statically declarative option we could, and we did: you literally can’t write code with Pulumi YAML, just expressions. Of course, you can always convert your program to a different language if you find YAML too limiting, and that’s great, too --- but it’s also a pretty big thing to have consider when all you need to do is push a few files into the cloud.

So I took a few days to figure this out, for all languages and all three major clouds. The result is a little Pulumi package called Synced Folder.

Introducing Synced Folder

Synced Folder is a multi-language Pulumi component that you can install and use with any Pulumi-supported language. It’s deliberately simple and focused, and it works --- or at least it’s my hope that it works --- exactly as you’d imagine: make a bucket, sync a folder to that bucket, and be done. Here’s how it looks in TypeScript, for example:

import * as aws from "@pulumi/aws";
import * as synced from "@pulumi/synced-folder";

// Make a bucket.
const bucket = new aws.s3.Bucket("my-bucket", {
    acl: aws.s3.PublicReadAcl,
});

// Sync the contents of `my-folder` to the bucket.
const folder = new synced.S3BucketFolder("synced-folder", {
    path: "./my-folder",
    bucketName: bucket.bucket,
    acl: aws.s3.PublicReadAcl,
});

Here’s an example in Python:

from pulumi_aws import s3
import pulumi_synced_folder

# Make a bucket.
bucket = s3.Bucket(
    "my-bucket",
    acl=s3.CannedAcl.PUBLIC_READ,
)

# Sync the contents of `my-folder` to the bucket.
folder = pulumi_synced_folder.S3BucketFolder(
    "synced-folder",
    path="./my-folder",
    bucket_name=bucket.bucket,
    acl=s3.CannedAcl.PUBLIC_READ,
)

And in Go:

package main

import (
  "github.com/pulumi/pulumi-aws/sdk/v5/go/aws/s3"
  synced "github.com/pulumi/pulumi-synced-folder/sdk/go/synced-folder"
  "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
  pulumi.Run(func(ctx *pulumi.Context) error {

        // Make a bucket.
    bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
      Acl: s3.CannedAclPublicRead,
    })
    if err != nil {
      return err
    }

        // Sync the contents of `my-folder` to the bucket.
    _, err = synced.NewS3BucketFolder(ctx, "synced-folder", &synced.S3BucketFolderArgs{
      Path:       pulumi.String("./my-folder"),
      BucketName: bucket.Bucket,
      Acl:        s3.CannedAclPublicRead,
    })
    if err != nil {
      return err
    }

    return nil
  })
}

In C#:

using Pulumi;
using Pulumi.Aws.S3;
using Pulumi.SyncedFolder;

return await Deployment.RunAsync(() =>
{
    // Make a bucket.
    var bucket = new Bucket("my-bucket", new BucketArgs {
        Acl = CannedAcl.PublicRead,
    });

    // Sync the contents of `my-folder` to the bucket.
    var folder = new S3BucketFolder("synced-folder", new S3BucketFolderArgs {
        Path = "./my-folder",
        BucketName = bucket.BucketName,
        Acl = (string)CannedAcl.PublicRead,
    });
});

And finally, in YAML:

resources:
    # Make a bucket.
    my-bucket:
        type: aws:s3:Bucket
        properties:
            acl: public-read

    # Sync the contents of `my-folder` to the bucket.
    synced-folder:
        type: synced-folder:index:S3BucketFolder
        properties:
            path: ./my-folder
            bucketName: ${my-bucket.bucket}
            acl: public-read

And if you’re targeting Microsoft Azure or Google Cloud, it’s just the same: make an Azure Blob Storage container or a Google Cloud Storage bucket, then drop in a reference to Synced Folder and call it a day. See the docs for examples with various languages and clouds.

To deploy the contents of the folder, just run pulumi up in the usual way:

$ pulumi up

When you make local changes to the files in your synced folder --- add new files, delete old ones, change existing ones, etc. --- simply deploy the changes again in the same way. (Changes, by the way, are synced one way only, folder to cloud, so deleting a file in the cloud will have no effect on the files on your local machine.)

To remove the cloud bucket and all of its contents, run pulumi destroy:

$ pulumi destroy

Pretty neat, IMHO --- and again, hopefully works just like you’d expect.

Bonus feature: Unmanaged file objects

One thing you’ll notice about the component’s default behavior is that it manages your files as individual cloud resources --- e.g., as discrete Amazon s3.BucketObjects or Azure storage.Blobs --- and not just as an abstract “folder” resource. For folders that contain only a few files, that’s probably fine, and indeed it’s sometimes nice to have Pulumi track changes to those files individually as well. But if your folder were to contain, say, a thousand files or more, you could end up waiting a long time for Pulumi to examine and reconcile the state of every last one of those file objects. Moreover, if you happen to be using Pulumi on a commercial plan, you might not love the idea of each one of those objects contributing to your monthly bill.

So to address these two unpleasantries, I added an option to instruct Pulumi to ignore individual files and instead, delegate the work of synchronization to the cloud provider’s official CLI. By setting the optional property managedObjects to false, you can have Pulumi invoke the AWS, Azure, or Google Cloud CLI during an update to sync your files that way instead:

import * as aws from "@pulumi/aws";
import * as synced from "@pulumi/synced-folder";

const bucket = new aws.s3.Bucket("my-bucket", {
    acl: aws.s3.PublicReadAcl,
});

const folder = new synced.S3BucketFolder("synced-folder", {
    path: "./my-folder",
    bucketName: bucket.bucket,
    acl: aws.s3.PublicReadAcl,

    // 👇 Set this property to false to manage files with `aws s3 sync`.
    managedObjects: false,
});

Synchronization-wise, the effect the same: files that exist locally but not remotely are uploaded, and those that exist remotely but not locally are deleted. But the upshot is that it’s much faster and less expensive than using Pulumi alone, so if you’re okay with Pulumi shelling out to the aws, az, or gcloud command-line tools, you might want to give it a try. Again, see the docs for details and examples.

Enjoy! And happy Pulumifying.

Imperatively Declarative: How Pulumi Is Different

Mon, 09 Aug 2021 00:00:00 GMT

In conversations about infrastructure as code, the debate over imperative versus declarative tools still comes up from time to time. Actually, there’s not much left to debate: declarative’s pretty much won. But somehow, the subject still manages to get people going, probably because what “declarative” means isn’t quite as clear as it used to be --- and that’s partly because of tools like Pulumi.

When Pulumi comes up in one of these conversations, it usually gets placed on the imperative end of the spectrum; it’s an easy mistake to make, considering Pulumi programs are written in imperative languages like JavaScript. But it’s a mistake nonetheless. Here’s an example of such an exchange from a while back, for instance:

I will be very specific using my previous example. JS is imperative. JSON is declarative.

--- Brian LeRoux (@brianleroux) July 14, 2021

It’s worth mentioning that Brian is the creator of arc.codes, a command-line tool that lets you write blocks of JSON or YAML to deploy serverless functions and other things on AWS. Arc is a perfect example of simple, declarative infrastructure as code that’s focused on making the easy things easy. Take a look at this terse little Arc file, for example:

app: "hello-world"
http:
    - get: "/thing1"
    - get: "/thing2"

In Arc, this bit of YAML states that at the end of an Arc run, two publicly accessible HTTP endpoints should exist in AWS Lambda (at a URL dynamically assigned by AWS) at the paths /thing1 and /thing2, and that both endpoints should be wired up to respond to HTTP GETs. When you run this file with the Arc CLI --- assuming you’ve stashed your AWS credentials in the right place, and put your JavaScript functions in a nearby subfolder --- that’ll indeed be the case: a minute or so later, those endpoints will exist, and all will be right with the world. Easy.

Moreover, if you were to run that code a second time (having made no changes to the YAML or JavaScript), nothing would happen, because the “desired state” you’d expressed in the arc.yaml file would already have been achieved: with those two endpoints deployed and running in the AWS cloud, Arc (by way of CloudFormation) would have nothing more to do for you. That’s declarative infrastructure-as-code (IaC) at work: you describe what you want --- two HTTP endpoints --- and the IaC tool determines the how, computing the work to be done and then making it happen for you.

Imperative IaC, on the other hand, is different. In imperative programming (e.g., in most JavaScript), the code that you write is all about control --- do this, then that; if this, then that. A good example of the difference between declarative and imperative programming would be to compare the experience of building a web page statically with hand-crafted HTML (which is about as declarative as you can get):

...
<section id="things">
    <ol>
        <li>Thing 1</li>
        <li>Thing 2</li>
        <li>Thing 3</li>
    </ol>
</section>
...

… to building one dynamically by scripting the DOM:

let ul = document.createElement("ol");

for (let i = 0; i < 3; i++>) {
    let li = document.createElement("li");
    li.textContent = `Thing ${i + 1}`;
    ul.appendChild(li)
}

document.querySelector("#things").appendChild(ul);

Both yield the same result --- a three-item list --- but in fundamentally different ways. In HTML, the author says what they want, up front, and lets the browser handle the rest. In JavaScript, however, the author tells the browser how to create that list, algorithmically, one element at a time before attaching it programmatically to the page at some point later.

IaC tools vary similarly. Classically declarative tools like Arc, CloudFormation, Terraform, and others have you type out what you want, usually in some sort of structured configuration, and handle the work of provisioning and updating for you. Imperative tools don’t do nearly as much; instead, they give you the APIs to tell them what to do and how to do it.

As an example, imagine you wanted to create a couple of storage buckets on Amazon S3. To do that imperatively, you might reach for Amazon’s SDK for JavaScript and tap out a small imperative program like this one:

const { S3Client, CreateBucketCommand, ListBucketsCommand } = require("@aws-sdk/client-s3");
const client = new S3Client({ region: "us-west-2" });

(async () => {
    // Name a couple of buckets.
    const desiredBuckets = ["bucket-1", "bucket-2"].map(
        bucket => `some-interestingly-named-${bucket}`,
    );

    // Imperatively create them, by calling the AWS S3 API directly.
    desiredBuckets.forEach(async bucket => {
        await client.send(new CreateBucketCommand({ Bucket: bucket }));
    });

    // Finally, list all buckets, including the two you just created.
    console.log((await client.send(new ListBucketsCommand({}))).Buckets);
})();

You could run this program with Node.js (again, assuming your AWS creds were stashed in their proper locations), and in a few moments, produce the following result:

$ node index.js
[
  {
    Name: 'some-interestingly-named-bucket-1',
    CreationDate: 2021-03-08T18:00:04.000Z
  },
  {
    Name: 'some-interestingly-named-bucket-2',
    CreationDate: 2021-03-08T18:00:04.000Z
  },
]

Nice, right? And easy enough --- assuming you’re comfortable with JavaScript.

However, unlike the Arc example I shared earlier, running the program a second time would fail:

$ node index.js
UnhandledPromiseRejectionWarning: BucketAlreadyOwnedByYou

… which is unfortunate, but makes sense, considering the buckets would already have been created. To keep repeated runs of the program from failing --- an important consideration, say, if the program were running as a part of an automated deployment process --- you’d have to write a bit more code to check for the existence of each bucket before attempting to create it:

// ...

(async () => {
    const desiredBuckets = ["bucket-1", "bucket-2"].map(
        bucket => `some-interestingly-named-${bucket}`,
    );

    // First, fetch a list of all buckets.
    const allBuckets = await client.send(new ListBucketsCommand({}));
    const allBucketNames = allBuckets.Buckets.map(b => b.Name);

    // Create the new buckets...
    desiredBuckets

        // ...but only if they haven't been created already.
        .filter(name => !allBucketNames.includes(name))

        .forEach(async bucket => {
            await client.send(new CreateBucketCommand({ Bucket: bucket }));
        });
    // ...
})();

And that’d certainly work.

But at the same time, all you really need is a couple of S3 buckets, here, and already you’ve begun to accumulate a good bit of code --- code that has to be debugged, tested, maintained, and all the rest. If you wanted to assemble something a little more complicated --- a couple of serverless endpoints, maybe, or the virtual infrastructure to run a typical web application --- you’d be looking at writing a lot more code, and this pattern of checking whether to do something before actually doing it (or doing something slightly different, maybe, under certain conditions) would continue to the point that it’d be hard for someone else (or even a future version of yourself) to look at the code and understand what was really going on --- certainly much harder than looking at a few lines of declarative YAML. Sometimes, of course, imperative code is just what you need. But for lots of reasons, declarative tools are usually the right way to go --- which is why, as I said, the debate’s pretty much over.

Where does that leave Pulumi, though? If Pulumi programs really are written in imperative languages like JavaScript, doesn’t that make Pulumi itself an imperative tool, too, by extension?

In a word, no --- but understanding why the answer is no takes a bit more explanation.

Breakfast as code

I haven’t always been a big breakfast person, but these days, I am, and for me, breakfast usually means an egg, some toast, and a bit of orange juice, with an occasional bunch of leafy-green things thrown in for good measure. Represented as JSON, my usual breakfast looks something like this:

{
    "breakfast": {
        "eggs": {
            "count": 1,
            "kind": "scrambled"
        },
        "toast": {
            "count": 1,
            "kind": "multi-grain"
        },
        "juice": {
            "count": 1,
            "kind": "orange"
        }
    }
}

It’s a fairly common choice, as breakfasts go --- so common that I could probably walk into any café, hand someone this snippet of JSON, and wait patiently for the result to show up on the table in front of me. In a way, this is declarative breakfast-as-code: I say what I want --- egg, toast, juice --- and a bunch of skilled humans conspire to make that happen for me.

And while I certainly know there’s an order in which these things tend to happen --- the eggs need scrambling, so the chef may prep them first; the toast goes quicker, so that’ll probably happen later, etc. --- that order isn’t important to me as a customer. In the end, all I care about is that when breakfast is ready, it’s hot, and on my plate. The JSON document just describes my desired breakfast; it doesn’t tell the chef or anyone else how to make it. That’s what makes it declarative.

Static text like JSON and YAML aren’t the only ways to declare a desired breakfast, though. Here’s a little JavaScript program that allocates a similar set of breakfast objects and relationships. Again, notice there isn’t any how going on, here --- we’re still firmly in what territory:

import { Breakfast, Eggs, Toast, Juice } from "some-menu-or-something";

const breakfast = new Breakfast({
    eggs: new Eggs(1, "scrambled"),
    toast: new Toast(1, "multi-grain"),
    juice: new Juice(1, "orange"),
});

Here, breakfast still consists of three things --- object instances of Eggs, Toast, and Juice --- just as it did in the JSON representation. Assuming the constructors of these objects weren’t doing anything fancy under the hood (just allocating local instance properties of their own, say), you’d expect that running this program with Node.js would produce, for a moment, a breakfast variable referring to an instance of the Breakfast class, and that the breakfast instance would itself contain references to instances of each of its ingredients before the program finally exited. Without a doubt, this is imperative JavaScript code --- but this particular expression is totally declarative; we’ve simply stated that breakfast depends on three ingredients, and left it up to the JavaScript engine to handle the dependent allocations and the order in which to perform them.

As it happens, this a lot like how Pulumi works, too. A call to a Pulumi resource constructor (like new aws.s3.Bucket(), for example) is just an object declaration like any other, an expression of your desire to have an S3 bucket exist --- not to create the S3 bucket in that moment, but to have it exist when the program completes. At runtime, the Pulumi SDK and engine conspire to gather up all of the object allocations in your program, figure out their relationships (which objects depend on which, what values they need from each other, and so on), assemble a JSON-serializable object graph representing the full picture, and then use that graph to call on the cloud provider directly to produce the appropriate result. Just like with Arc and other statically declarative tools, the code you write with Pulumi still says what, not how, and Pulumi takes care of delivering the outcome for you.

Here’s what it looks like to make a couple of S3 buckets with Pulumi and JavaScript, for example:

const aws = require("@pulumi/aws");

const bucket1 = new aws.s3.Bucket("bucket1");
const bucket2 = new aws.s3.Bucket("bucket2");

If you wanted, since you’re working with JavaScript, you could even get a bit fancier by declaring the buckets with Array#map:

[1, 2].map(i => new aws.s3.Bucket(`bucket${i}`));

Run the program once, and you get two buckets (along with a “stack,” if you didn’t already have one):

$ pulumi up

Updating (dev)

     Type                 Name         Status
 +   pulumi:pulumi:Stack  buckets-dev  created
 +   ├─ aws:s3:Bucket     bucket1      created
 +   └─ aws:s3:Bucket     bucket2      created

Resources:
    + 3 created

Run it again, you get nothing, because the buckets you declared already exist:

$ pulumi up

Updating (dev)

     Type                 Name
     pulumi:pulumi:Stack  buckets-dev

Resources:
    3 unchanged

You could even reverse the sort order and still get the same result (since ultimately, it’s up to Pulumi to determine what needs to be done, and how):

[1, 2].map(i => new aws.s3.Bucket(`bucket${i}`)).reverse();

$ pulumi up

Updating (dev)

     Type                 Name
     pulumi:pulumi:Stack  buckets-dev

Resources:
    3 unchanged

Again, that’s declarative (and idempotent!) infrastructure as code --- it just happens to have been written with an imperative programming language. You could modify this program to add a third bucket, remove a bucket, declare a JavaScript function to be invoked in response to a bucket event, whatever you want, it’s always the same: Pulumi launches your chosen language runtime, listens for object allocations (by way of the @pulumi/aws SDK, for example), registers those allocations with the engine, computes an in-memory graph of resources and relationships, and then calls on your cloud provider directly to issue the appropriate set of changes, in the right order.

Great --- so now you know how Pulumi works.

But it’s still worth asking: is all of this really necessary? What kinds of problems does Pulumi actually solve? What makes this “imperatively declarative” approach to infrastructure worth the additional layers of indirection --- the language, runtime, dependencies, and the rest? Wouldn’t it be easier just to write a few lines of YAML and be done than to have to contend with all of this extra stuff?

Sure --- for simple things, maybe. But software has a funny way of starting out simple and suddenly becoming annoyingly complex --- often a lot sooner than you think.

When breakfast gets complicated

For me, thanks to my basic breakfast needs, getting what I want is usually no big deal. That’s because most cafés are going to have eggs, bread, and orange juice on hand and ready to make --- and also because I’m not all that fussy about the details.

But for my family, it’s more complicated. I have three kids, for example, all of whom have mild food sensitivities, and a wife who rarely eats out because of how hard it is to find something she likes. None of them could walk into a diner with an order like mine, because they’d need to be able to ask certain questions first: Are the eggs made with milk? Are the waffles gluten-free? Each of these questions needs to be answered, for real and important reasons, before our collective order can be submitted and fulfilled.

It’d be impossible, in other words, to walk into a restaurant with a handwritten order for a family like ours expecting to have it accepted verbatim without some kind of interaction first. Oh, the waffles aren’t gluten-free? Okay --- we’ll take an omelet instead. It’s always something, and I imagine it’s probably like that for most of us: we know what we want, and we’re usually able to get it, but not without a little negotiation during the process. At a high level, we know want “breakfast”, which is easy. But in practice, we almost always end up having to apply some sort of algorithm, however simple, during that process.

In fact, that’s kind of how everything works, software included --- and infrastructure (especially the cloud-based kind) is nothing not fundamentally software. If all you need is a couple of storage buckets or Lambdas or VMs, sure, you can kick out that stuff with a few lines of YAML and get on with your day --- and that’s awesome, to be sure. But more often, what you’ll find is that you’ll eventually need something more, some tiny bit of customization or other that the simple tool can’t quite give you out of the box --- and that’s when the trouble begins.

When the problem is straightforward and well bounded, in other words, simple tools are great, and often more than enough to get the job done. But when the problem is even a little bit complicated, or when the problem space expands beyond what those simple tools were originally designed for, the tools themselves will tend to bend and crack in the places that weren’t really made with complexity in mind.

Take our two buckets, for example. If you knew how many buckets you wanted to create and how you wanted to name them, you could do that fairly easily with HCL, the config language of Terraform:

provider "aws" {
  region = "us-west-2"
}
variable "buckets" {
  type = list(string)
  default = ["1", "2", "3"]
}
resource "aws_s3_bucket" "bucket" {
  count = length(var.buckets)
  bucket = "some-interestingly-named-bucket-${var.buckets[count.index]}"
}

If you’re not familiar with HCL, you might need to squint to figure out what’s going on here, but it’s a lot like our first bucket-provisioning example from earlier: we just loop through a list of strings (“1”, “2”, and “3”), creating a bucket for each one:

$ terraform apply

aws_s3_bucket.bucket[1]: Creating...
aws_s3_bucket.bucket[2]: Creating...
aws_s3_bucket.bucket[0]: Creating...
aws_s3_bucket.bucket[0]: Creation complete after 3s [id=some-interestingly-named-bucket-1]
aws_s3_bucket.bucket[1]: Creation complete after 3s [id=some-interestingly-named-bucket-2]
aws_s3_bucket.bucket[2]: Creation complete after 3s [id=some-interestingly-named-bucket-3]

Again, this totally works --- assuming the names you’ve chosen are globally unique.

Now imagine you had to name those buckets in a slightly more complicated way --- using stringified date, perhaps. Naming a bucket dynamically with a format string like YYYY-MM-DD is maybe possible with Terraform (or if not, maybe using a bit of shell scripting with and an HCL variable), but you’d definitely be running into the limits of what HCL is able to do on its own. That’s not a knock against HCL, either: every special-purpose language runs the risk of hitting these kinds of limitations eventually.

With general-purpose languages like JavaScript, though, this kind of thing is trivially easy, either with the language alone or with the help of a third-party package to make things even easier --- one like Day.js, for example:

import * as aws from "@pulumi/aws";
import * as dayjs from "dayjs";

// Keep a bucket for each of the last 7 days.
for (let i = 0; i < 7; i++) {
    new aws.s3.Bucket(dayjs().subtract(i, "day").format("YYYY-MM-DD"));
}

$ pulumi up

Updating (dev)

     Type                 Name         Status
 +   pulumi:pulumi:Stack  buckets-dev  created
 +   ├─ aws:s3:Bucket     2021-03-24   created
 +   ├─ aws:s3:Bucket     2021-03-29   created
 +   ├─ aws:s3:Bucket     2021-03-28   created
 +   ├─ aws:s3:Bucket     2021-03-27   created
 +   ├─ aws:s3:Bucket     2021-03-25   created
 +   ├─ aws:s3:Bucket     2021-03-23   created
 +   └─ aws:s3:Bucket     2021-03-26   created

Resources:
    + 8 created

Duration: 9s

When you carve away the language, Pulumi and Terraform are doing a lot of the same things: both work to assemble graphs of resources and dependencies, both use those graphs to communicate with cloud providers directly, and both manage state in conceptually similar ways. It’s at the language layer --- and up --- that they really start to diverge.

Again, how much that matters is for you to decide. But as a developer, I’ll take a full programming language (especially one I know well) any day of the week, because it means I can do anything the language and its ecosystem can do, and that I probably won’t end up in tears in six months when I’m faced with a problem that my tools can’t handle. Just yesterday, for example, I found myself wrestling with Bash trying to move a few files between Git repositories. After a frustrating couple of hours of hacking and Googling, I realized I could just use Node.js instead --- and when I did, I was done in a matter of minutes. An expert shell programmer might’ve made light work of what I was trying to do --- but I’m not an expert shell programmer, and Bash isn’t JavaScript. All it took was a couple of Node.js built-ins and libraries:

$ yarn add glob micromatch

… and eight lines of JavaScript later, I was done.

For me, language --- and all that comes with it --- is ultimately what it’s all about.

Like React for infrastructure

All of this reminds me of the progression we’ve seen over the last two decades in web development.

Think of React. Why do we have it? Because HTML alone isn’t enough, and imperative DOM scripting leads to reams of unmaintainable code. We got React because we, as developers, wanted to think about, and compose, our front-end applications in declarative ways --- but we needed to retain the flexibility of the JavaScript language. So we got React --- and with it, an imperatively declarative programming model for the web:

// Imperative code...
const offices = ["Akron", "Nashua", "Rochester", "Scranton", "Syracuse", "Utica"];

export default function DunderMifflinBranchOffices() {
    // ... declaratively rendered...
    return (
        <ul>
            {offices.map(office => (
                <li>
                    <span>{office}</span>
                    {office === "Scranton" && <span>← The best one</span>}
                </li>
            ))}
        </ul>
    );
}

...
<html>
    <body>
        <aside>
            <nav>
                <!-- ... and composed. -->
                <DunderMifflinBranchOffices />
            </nav>
        </aside>
        <main>...</main>
    </body>
</html>

It’s the same thing with infrastructure: we want a declarative mental model, but we need the control and composabilty of general-purpose languages. Hence tools like Pulumi.

It’ll be interesting to see where things go from here; I’m certainly biased, but also a fascinated observer. The trajectory is what interests me most, though --- that, and being able to manage my own infrastructure in ways that feel comfortable to me as a developer. (Which reminds me, I should probably write about how I manage this site sometime…)

Enjoy this post? It’s reprinted from pulumibook.com, the companion site for the book I’m currently writing about using Pulumi with TypeScript and Amazon Web Services. Check it out, or grab yourself a copy of the in-progress book today on Leanpub! 📚

Live Coding with Pulumi, Mapbox, and Leaflet

Tue, 03 Nov 2020 00:00:00 GMT

This past weekend, I did a Twitch session for Manning in which I built a little serverless app with Pulumi and Mapbox and deployed it on Amazon Web Services. It was the first time I’d ever done such a thing --- I’ve done many demos, but never a public live-coding session --- and surprisingly (to me, at least), I actually enjoyed it. In a little over an hour, I wrote an app with Pulumi and TypeScript that renders GPS-tagged photos on a map with AWS S3, Lambda, DynamoDB and API Gateway, and aside from a few little bumps, and my crazy-long COVID hair, everything worked out pretty well.

The hardest part was definitely setting up OBS Studio, but now that that’s done, I plan to start doing more of these, as they’re (a) fun and (b) probably the best way to demonstrate how Pulumi really works and what makes it so awesome.

You’ll find the full recording here, and the annotated code of the finished product’s up on GitHub. Enjoy!