Now you have N problems

Micro services are definitely a hot topic at the moment, and many organisations large and small are breaking up their monolithic apps into smaller and smaller components.

Many start out on that journey by breaking out obviously asynchronous tasks - e.g. sending an email - into a separate service that receives messages from other parts of the application via a message bus of some sort.

The instant you do this though, you have just required your developers to now check out two repositories instead of one. Then you break out another service. Three. Then another. Four. Then you get a taste for it. Five. Six. Seven. Fifty. One hundred.

This may sound insane, one hundred repositories? But it’s definitely true - once you get the basics in place that allow you to automatically deploy more than one service it then becomes the default to break out new functionality into new services and on it rolls.

This then creates a challenge though for your development teams. How do they manage to get up and running from a blank start, on their first day, when they have to get hold of over 100 repositories, plus all of the local infrastructure dependencies (MQ, DB, Cache).

There are some fantastic projects that I would highly recommend that can deal with the base level of this - like https://boxen.github.com/ and https://www.vagrantup.com/ - but they tend to focus on everything but the project your working on, e.g. just getting all of the dependencies up and running repeatably and reliably across a number of developer machines.

There are also some great build tools like http://gruntjs.com/ and http://gulpjs.com/ that allow your projects to be built in a common and repeatable way. But by design these tools and their plugin eco-system are very focused on building and managing a single project, not many.

So what is a CTO to do? Well, by very definition an engineer should build be comfortable building their own tools, so that’s what we did.

Bosco

Bosco is a tool to help a team manage all of the projects that get created as a byproduct of a micro service architecture. It doesn’t replace Boxen or Vagrant, it isn’t a build tool that is attempting to replace Grunt or Gulp, but works around them.

Think of the following scenario:

A new developer joins your team. They get a shiny new laptop, maybe you already have boxen on it so all the standard tools are on board.

Their buddy directs them to the team repository on github, which they check out.

They install and run bosco for the first time, prompted to put in the organization’s github name, and an auth key from their account that allows read access to their repositories.

Now, they are ready to go - what can they do with bosco?

Well, broadly, the functionality is split across three main areas:

1. Managing Multiple Projects

These commands are all about making your life a little easier. The ones you use the most:

bosco morning

This is a wrapper for a number of commands (which you can run individually), but is basically the command you run each morning (or across the day), to be caught up on the changes that your team has made across all projects.

It does the following across all repositories: clone, pull, npm install, show activity log.

bosco grep

Allows you to quickly git grep across all the projects.

bosco stash

Stashes any uncommitted changes across all the projects.

I suspect you’re starting to get the drift - there are a set of commands that basically allow you to run the ‘standard’ set of commands you would use within a single project across many projects.

If you don’t want to run the command across all projects, you can filter down by passing an extra parameter:

bosco stash -r name

Where you can specify part of the project name (this is expanded as a regex) and the command will only run against projects that match - this is used in all commands that work across projects.

There are a lot more commands, and you can add your own into your bosco project folder (under /commands).

https://github.com/tes/bosco/tree/master/commands

2. Running and Stopping Multiple Projects

Ok, so you’re a master at using Bosco to make some operations against many projects a little simpler. Now how do you get the project up and running?

First, a small diversion into what you need to do with your services to make Bosco aware of how they start and stop.

package.json and bosco-service.json

If your project is a Node project, then all you need to do is specify a start script in the package.json. Bosco will find this, and when you tell it to run your projects it will use PM2 to start the project via that command.

If you’re project isn’t a Node project (e.g. it may actually just be a Docker container definition for Redis). Then you can specifiy more detailed configuration by placing a bosco-service.json in the root of the project.

Two examples are below:

A Node Project

{
  "tags":["review"],
  "service":{
     "dependsOn":["infra-rabbitmq","infra-mongodb","infra-redis"]
  }
}

This simply adds a tag to the project (this allows you to start a subset based on the tag - e.g. bosco run -t review), as well as define some dependent services that should be started if this one is started.

The start command is in Package.json.

A Docker Service

{
   "service": {
     "type":"docker",
     "name":"infra-redis",
     "registry":"docker-registry.tescloud.com",
     "username": "tescloud",
     "version": "latest",
     "alwaysPull": true,
     "docker":{
       "HostConfig": {
       "PortBindings": {
         "6379/tcp": [{
          "HostIp": "0.0.0.0",
          "HostPort": "6379"
          }]
        }
      }
    }
  }
}

This configuration tells Bosco that this project contains the definition for a Docker container, that when built is published to the registry shown. Bosco will run it after pulling it from the registry (it doesn’t build and run the container from the folder). This is a very light wrapper around the Docker remote API.

Running Projects

Ok, so you’re services are configured - you can now get running.

bosco run

This will run everything.

bosco run -r project

This will run based on a regex match against the project name.

bosco run -t review

This will run based on tags defined in the bosco-service.json files.

bosco run -watch

This will run the project, but in watch mode (using PM2).

Seeing what is running

bosco ps

This will show everything that is running.

Stopping

bosco stop

This will stop everything (the -r and -t tags still work for stop).

3. Managing Static Assets across Multiple Projects

The final piece of the puzzle when dealing with micro-services, especially micro services that render HTML (huh? see this post: compoxure) is that you need a way of managing static assets across projects - css, javascript and images.

Bosco is the natural place to do this, as it already knows all of our projects, and having it take on this extra responsibility wasn’t a big stretch.

bosco-service.json

If your project has static assets, you need to tell Bosco about them.

{
  "build": {
    "command": "node_modules/.bin/gulp build",
    "watch": {
      "command": "node_modules/.bin/gulp build -watch",
      "ready": "Finished 'build'"
    }
  },
    "files": {
      "upload": {
        "basePath": "dist/",
        "js": [
          "js/moxie.js",
          "js/plupload.dev.js",
          "js/tsl-uploader.js"
        ],
        "css": [
          "css/tsl-uploader.css"
        ],
        "img": [
          "img/header-bg.png"
        ],
        "swf": [
          "js/Moxie.swf"
        ]
      },
      "cx": {
        "html": [
          "ui/cx-templates/backend.html"
        ]
      }
    }
}

This configuration tells Bosco that the project is built using Gulp (this is optional - if there is no build step you can just point Bosco straight at the files in the project), and that the output of Gulp is a set of files that are defined in the ‘upload’ file group.

You can have as many file group’s as you like, as these are simply collections of assets that are grouped together in the final output (e.g. the JS is concatenated and minified, the CSS is concatenated) into a single file and then deployed together to a CDN.

To make this magic work, there are two key commands.

bosco cdn

This command runs across all of your projects, grabs all of the assets, and starts a server locally on port 7334 (http://localhost:7334) that serves them up. In addition to serving them up, it serves Compoxure fragments that reference the static assets.

This allows you to include the assets from a file group into a given page. Note that you don’t have to use Compoxure, you could do this via another mechanism like ESI, SSI or even Ajax depending on how your app works.

You can also run:

bosco cdn minify

If you want Bosco to minify the assets, exactly as it would if you ran the command to push to a CDN, but locally. This can be useful if you have any issues caused by the minification process.

Finally, once you’re changes are ready you can push the static assets up to a CDN for use in non-local environments.

bosco -e development -b 76 s3push

This will push all of the fragments and assets up to S3, into a path prefixed with ‘development/76’.

You can see a real one here: https://duqxiy1o2cbw6.cloudfront.net/development/default/index.html

See some of this in action: https://asciinema.org/a/13142

Summary

Bosco does a lot to help your development flow with Microservices. It isn’t a build or deployment tool - you should use other things for that - but it does aim to make the life of a developer on a day to day basis less complex when dealing with a lot of services.

I’ll be talking more about this at Fullstack in London this week, so do ask questions or let me know of any areas I should flesh out in more detail in future posts.