Cloudy sunset above mountains

Hedge - Building a new cloud deployment tool with service abstraction


At the start of my Siili career as an apprentice, our group was introduced to a mysterious tool: Hedge. Our apprentice project was to build an internal tool using Hedge as a cloud platform abstraction library and a deployment tool. After our apprentice program had come to an end, I was offered an opportunity to continue the development of Hedge in an internal project. Because we had already used Hedge I had some experience as a framework user. The task I was offered was to implement basic Amazon Web Services(AWS) support into Hedge. 

What is this Hedge you are talking about? 

Hedge offers automated serverless function code generated from common ring compatible handler methods, libraries for various levels of abstractions, and a set of common commands to build handlers, create artifacts, and to deploy created artifacts. Hedge is an open source software and is available in our github

Hedge is a platform agnostic ClojureScript framework for deploying ring compatible handlers to various environments with focus on serverless deployments.

Back to my journey with Hedge development 

The scope of platform agnostic in Hedge was uncertain for me when I started the development. First, I implemented a simple feature parity for AWS code creation and deployment with Serverless Framework. Later the definition of the platform agnostic, roadmap, and context of Hedge became more clear. 

When I started working with Hedge I had little Clojure experience and almost no hands-on experience with AWS. For AWS deployment I had checked how other tools do deployment and had chats with my peers. These information sources gave good advice on how to handle deployment. 

Why Clojure(Script)? ClojureScript in backend?! 

Clojure(Script) is popular and it is in high demand in Finland and at Siili Solutions. Using ClojureScript gives access to ClojureScript and JavaScript libraries. The functional paradigm has steep learning curve, but after learning the basics it is easy to realize that immutable data fits well with serverless handlers. Multithread-safety adds more security if one of the supported clouds is re-using processes with multiple threads. 

One of the Clojure build tools, Boot, is also extensively used with Hedge. Boot tasks are great for creating and chaining commands. It is fast and easy to develop a set of tasks which for example build input files, create all artifacts, and finally deploy artifacts to cloud. Then those tasks can be combined into one large task which does all with one command or uses small tasks to store artifacts into disk and later deploys artifacts to cloud. 

Clouds

 

Why platform agnostic framework? 

The main goal of Hedge is platform agnosticism: once the code and configuration has been written for one cloud platform, it is re-deployable to another cloud provider by changing deployment command and a few configuration directives. This improves code re-usability and limits the risk of being locked with specific cloud provider. 

To make the implementation easier, Hedge supports only a small common subset of provided cloud features. Limiting number of supported features might be a risk and might lower acceptance rate amongst developers. It is unknown if developers will use a framework which allows using only a small fraction of cloud features. The small number of features definitely narrows down the creativity of developers and might also limit usage of Hedge for some projects. 

Our implementation 

Hedge adds abstraction layers for: 

Code 

  • Handler function signature 
  • Logging 
  • Payload creation and emit 
  • Function exit condition signalling 
  • Persistent storage and queues 

Infrastructure 

  • Deployment 

Hedge users must write function handlers as ring-like handlers. Hedge then will create cloud-specific wrapper code between a user supplied handler function and cloud’s native handler entry point. The wrapped code handles differences between cloud providers and in the future, there will be libraries for the rest of code abstraction. 

 

Hedge is currently under heavy development and some of the listed features are still on roadmap. 

Lessons learned 

During this project I had to learn how create Cloudformation templates and stacks from template. Cloudformation templates for lambda functions with API Gateway endpoints first seemed overwhelming, but AWS SAM simplified template creation. AWS SAM is still a young technology and I did not find any big projects using it. 

I had to find one important piece of information from StackOverflow. Cloudformations templates using SAM must be deployed using change sets. Luckily it was documented there, since otherwise I probably would have spent a lot of time debugging a feature which was not supposed to work. 

As the documentation of SAM is still vague and mostly in GitHub only, I personally can recommend using SAM for serverless Cloudformation templates. 

See also:
"Modern Application Development: Should you skip microservices and go directly to serverless?"

A Guidance Framework for Architecting Portable Cloud and Multicloud Applications

Multi-cloud, what are the options? - Low level abstraction libraries

Pros and Cons of a Multi Cloud approach

(<3 siili_ clojure) - hope you do too!