Add Compose

[Mcrich23]

There has been a lot of discussion around a docker-compose counterpart for Container. This moves that discussion a bit further by introducing a basic translation layer for compose.

This pr introduces a new command group into the cli: Compose. It has two subcommands: up and down.

The origin behind this pr being made comes from the desire to run Vapor on Apple Containers. Hopefully, efforts like this make it possible quite soon.

Note: This is an early implementation of compose and there are features missing in Container that makes it not up to complete feature parity. This is bound to change as features such as port customization are introduced.

[Mcrich23]

Referencing the discussions/issues about how best to support compose: #55 #194 #101

[stevenanthonyrevo]

I've reviewed as much as I can!

I'm fairly limited due to my current MacOS version as anything below 6.2.0 will just error out.

Although, thanks for your work on this feature request as I was really hopeful for such a feature!

So on Good Faith, I'm going to Approve! If anyone else wants to review, feel free!

[Mcrich23]

Thank you! I am also planning to add support for the newfound networking capabilities in the coming days.

Also, if anyone knows how to restructure compose to call these commands without invoking the command line, I would be very grateful to hear about it!

[Mcrich23]

To address these things before the merge, I am going to make this pull request into a draft. If you want to try the tool compiled, you can get it at https://github.com/mcrich23/Container-Compose.

[jglogan]

Regarding Compose types, have you looked into generating the Swift code from the spec itself using quicktype. If that worked then we could automate that (running quicktype under container since it could be a manual step that we perform only when updates are required).

This is a rough sketch of how one could create a standalone swift package that provides a reader using Compose types generated from the JSON schema specification (the one under compose-spec/compose-go seems to be the canonical copy).

swift-docker-compose-types.tgz

[Mcrich23]

@jglogan I totally agree that the ideal way to do this is by autogenerated codable structures via quicktype. My wonder is if totally valid yaml that slightly differs from the mainstream definition will throw errors. One example is:

depends_on: foo

versus

depends_on: 
  - foo

[jglogan]

totally valid yaml

This particular case doesn't seem totally valid.

% docker compose -f docker-compose-syntax.yaml up -d
validating docker-compose-syntax.yaml: services.server.depends_on must be a list
% echo $?
1

docker-compose-syntax.yaml:

% cat docker-compose-syntax.yaml 
name: compose-test
services:
  db:
    image: postgres:latest
    restart: always
    environment:
      POSTGRES_USER: todo
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: todos
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  server:
    image: server
    depends_on: db
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgres://todo:secret@compose-test-db.test:5432/todos

volumes:
  pgdata:

[Mcrich23]

I apologize, that is a bad example, but it is true for other fields.

[jglogan]

I apologize, that is a bad example, but it is true for other fields.

No apologies needed! I'm just curious to see cases where the parser for Docker Compose would admit cases that don't comply to the JSON schema.

This doc describes everything the Compose parser does to transform YAML text into bound types: https://github.com/compose-spec/compose-go/blob/main/parsing.md.

[Mcrich23]

@jglogan So sorry for the delay. A lot going on personally including preparing for college (whoohoo!). Anyway, this is one example that comes to mind.

I feel like we maybe start with QuickType and find a way to fix these one off issues as we go. The good news is that people won't have an issue as long as people follow typical syntax instead of shortcuts.

[Mcrich23]

I wonder if we can make a general decoding pattern that allows all string arrays to be a single string outside of an array, etc.

Maybe member attribute or accessory macros could auto fill it so that just gets prepended to structs and we can still use QuickType.

[TomasLudvik]

@Mcrich23 any progress here? I can see it has been approved last week. Can we move it forward? I am excited to see this feature in next release :)

[ManuW]

Looks like the compose tool is also available as external tool: container-compose.

An internal tool would be good, but it's not a must. “Docker Compose” also started out as an external tool.
I would close this PR for now and refer to the external version.

[Mcrich23]

This PR has remained open to help provide information to the community around building a tool. The other compose pr currently open is way more likely to be merged. But at the moment one of the biggest challenges overall is the desire to do small and incremental PRs for the plugin.

[atovstonog]

@Mcrich23 any updates here?

[Mcrich23]

@atovstonog If you look above, this PR will not be merged due to the size of the PR. It is staying open to push for others to see the conversation as to how it could be built.

In the meantime, I have made a 3rd party tool called Container-Compose. You can even install it via homebrew!

[adamcee]

Hi all. Appreciate the thoughtful discussion around this. I see there is a third party tool Container-Compose which looks great and which I will be using to get the compose feature with Apple Container.

If possible I'd prefer to have the tool not be third party simply from a dependency management perspective - I installed container as recommended, but currently need to use brew to install container-compose. Let me know if I'm over-thinking this.

But, in that vein, I am trying to read thru the various discussion threads, and it sounds like the issue is the PR size and maybe some elements of the general approach? That the feature set was not minimal enough?

Asking because I'd like to understand this well enough to see if I could take a crack at moving the ball forward on having compose not be a 3rd party lib, if that is still desirable. Either by contributing to this PR or using it as a reference.

If possible could someone summarize:

• Is there still desire for compose to be internal?
• At a high level what needs to be done (blockers, etc) to accomplish that as desired?

Or, point me to the thread where that is discussed. There are several related Issues and I was struggling to find the key info.

[jglogan]

Closing for now, as we're not considering compose for upstream at this time. Please develop this as an external plugin for now. If you have any questions regarding plugin development (we're still getting docs together) feel free to open a discussion topic!

[arch-fan]

What about making Compose a core feature? Or is that not in the plans? I’d be willing to work on an implementation.

[Mcrich23]

Closing for now, as we're not considering compose for upstream at this time. Please develop this as an external plugin for now. If you have any questions regarding plugin development (we're still getting docs together) feel free to open a discussion topic!

Sounds good @jglogan and I am still happy to write those docs. I just feel that it is a waste of time without a way to pass down option groups into subsequent actions.

[luisneto]

@jglogan : Would it be possible to help us/me understand the reason(s) you are not considering compose for upstream at this time?
Thanks in advance!

[luisneto]

In addition, I'm confused about what's going on here, actually. Your comment says "Closing for now", but this pull request is still open.
What you did close was pull request #1394.

Did you really mean to write that comment here, @jglogan?

[jglogan]

@luisneto It's discussed to some extent above, and in discussion #194.

Docker Compose is a free-standing project. Its source isn't mingled with Docker Engine, CLI, buildkit, or the SDKs because its concerns are separate from those of the other Docker/Moby ecosystem projects. There's no need to colocate them in one project.

The maintainers of container have limited bandwidth and are focused on the container core functionality at present, trying to refine it and the Containerization library to the point where we can better support plugins for ecosystem integration. Our focus, for now, is on this, and we welcome contributions in this area. We understand the community desire for a Compose-like feature and get that it's frustrating not to have such a key piece of the puzzle (we'd love it too), but upstream in apple/container is not the place for it. @Mcrich23, @mazdak, the socktainer folks, and others are all working hard at this and I'm sure could benefit from contributions as well.