Sharing Usefulness – Easy serverless web services.

TL;DR: Red Sift now enables you to create, deploy and share your serverless web services fast and simply

Often, having written a particularly nifty bit of code you want to share it with the world. Github and Gitlab are great for getting the code out, but what if you want to have a running demo as well?

Serverless computing platforms are great for hosting code without per minute charging and the hassle of setting up virtual machines, but current platforms, like Amazon Lambda, are limited in their choice of programming environment and language. Red Sift is a new platform that allows you to program in a wide variety of languages and run complex execution graphs rather than just a single function.

In this blog post, we’ll show you how to create and deploy a simple web service using Red Sift in just a few minutes. When you’ve done that, we’ll give you some pointers to Red Sift’s advanced features like full stack browser integration, Slack and bot-framework APIs.

This post will showcase the RPC functionality and expose npm’s DNS module in just a few simple steps. We chose this module to create a simple online version of the known Linux dig utility to interrogate DNS servers. After that you can expose any module that you wrote or from npm’s directory behind an API!

Newbies section

If this is the first time you are discovering Red Sift and our platform have a look at the docs and our website for more info. We call our apps Sifts. Sifts are easy to build and have many ways to integrate with other systems.

Our technology is production ready and build for the real world. To prove it, OnDMARC, our award winning email security product, is built as a Sift!

If you are not yet familiar with our service and want to play around with Sifts have a look here on how to get the latest SDK and set up your environment or simply in a new terminal run:

$ curl -sSL https://static-sdk.redsift.io/install | bash

Creating your Sift, the template to happiness

It’s really easy to bootstrap Sift projects when you start with a template generated by the SDK. Templates create a whole working Sift based on the main types of input (emails, bots, webhooks, rpc, etc.), but there is nothing stopping you from switching between types of inputs or combining them together in any way, they are just a stepping stone to get you cracking. Assuming you have the SDK installed open a new console window, move to a directory you want to use as a workspace and run:

$ redsift create edig

Use your arrow keys and select the rpc-sift entry and then skip the rest of the questions by simply pressing Enter. Before going into any further details let’s run it: (this may take a couple of minutes the first time)

$ redsift run edig

So long as you are not using Docker, the SDK will open a new tab with the SDK UI in it, (you should be using the Chrome browser BTW). If everything went well you should see the following screen:

Now to see that RPC functionality in action open a new terminal window and run the following command:

$ curl http://localhost:8356/simple -H 'Authorization: Api-Key sdk'

If you get “Nice one!” as a response then you just completed your first RPC call in one of the nodes. Hurray!

Dive a little deeper?

To understand where the magic happens there are only two places you need to take a look at sift.json and server/simple.js.

Sift.json defines the DAG (Directed Acyclic Graph) of execution nodes that Sifts are built on. A node performs its task when it receives an input and then outputs the result, which may trigger a subsequent node(s).

Sift’s internal stores hold the inputs and outputs for nodes and thus the overall state of the entire Sift.

In an RPC Sift, we have a special, system provided store which is the _rpc bucket. If this still sounds like Greek to you just follow the links above to start reading the appropriate doc sections.

https://gist.github.com/demosifter/bf0c1b30ed94ce630c7dcf5dadb4fd1c.js

The second file to look at is the actual implementation of the node itself inside the server/simple.js file. This code is triggered when the node receives a request, then constructs a simple reply with a 200 status code and outputs it to the api_rpc store, (internal name for the special _rpc bucket).

You will notice that the node is passed an array of input values which it iterates over with a map method. This is because in some cases nodes may be passed the aggregated result of several input events, such as when more than one email is fetched at once. This is not the case here since our requests will be processed serially but it’s a good pattern to get in the habit of.

https://gist.github.com/demosifter/05569f3edde4b161673d9e35f6f0fdd6

Bring e-dig to life!

Having got our RPC boilerplate to run, let’s move on to add the DNS functionality. To create our online version of the dig tool we have to do a few simple changes to our current project.

• install pify to promisify functions with callbacks by running the following command in your Sift’s directory
  npm install pify --prefix server/ -S
• change your RPC endpoint URL path in sift.json from /simple to /edig/txt/* to be more specific about what we are going to serve

https://gist.github.com/demosifter/9371fb16c10fa1f9a3187e04625936e5

• update the implementation in simple.js to start asking DNS servers for text records

https://gist.github.com/demosifter/57089677bd57865fe22675dc269f6ba2

All we did here is require our dependencies at the top and after doing a bit of string formatting return the response from the call to the dns.resolve() method by wrapping it with a promise.

Once you have updated all the relevant sections in your Sift described above and you checked that Watch Mode is ON for your SDK.

On the terminal you run the previous curl command now run the one below. Same request but to the new endpoint.

$ curl http://localhost:8356/edig/txt/_dmarc.ondmarc.com -H 'Authorization: Api-Key sdk'

If you get a response like the one below it all works fine. Yay!!

_dmarc.ondmarc.com.  IN TXT 'v=DMARC1; p=reject; pct=100; rua=mailto:dmarc@redsift.io,mailto:6301a309@inbox.ondmarc.com; ruf=mailto:dmarc@redsift.io,mailto:6301a309@inbox.ondmarc.com;'

Here we are checking our domain for its DMARC record. Do you have one for you own domain? Try it out now by appending your own domain after _dmarc. in the above curl command and see. For more information you can always test your domains on our page at ondmarc.com

Extra Credits: T0 get the the pretty printed result above you need some magic from echo -e to parse escape characters and sed to remove double quotes, like this:

$ echo -e $(curl http://localhost:8356/edig/txt/_dmarc.ondmarc.com -s -H ‘Authorization: Api-Key sdk’ | sed -E ‘s/(^”|”$)//g’)

Now share it!

Good job reading this far! If it was a scrolling accident don’t worry, we have something for you too here. The last ingredient we are missing, is actually sharing it with the world. Before that we probably want to test it privately on our dashboard in the cloud. If you have a GMail account just head over there to instantly sign in, otherwise you will have to create one first.

We use Github directly to host and retrieve Sift repositories. Provided that you have an account, if you just upload the code from this tutorial in a new repo on Github you will be able to install and play with what we created from the workshop section of our dashboard. You can skip this step just by forking our repository. For your Sifts to appear in the workshop you first need to connect your Github account at the profile page of the dashboard.

Once the Sift appears in your workshop just press the run button and you should be up and running in seconds. Your endpoint now will be available under the rpc.redsift.cloud domain. Last thing we need, is to fetch an API key under the Sift API Keys section from the profile page to curl our resource. Just replace the sdk placeholder we were using before with the actual key like the example here. Ta-da!

curl https://rpc.redsift.cloud/edig/txt/_dmarc.ondmarc.com -H 'Authorization: Api-Key 825cf5b352be381818d36f5639e89568a2e4d6b6'

Moreover, from the profile page you have the ability to manage multiple keys and more granular permissions assigned per key is coming soon.

But that’s not all!

Compared to other providers, we support more languages and more advanced features but does that come at a performance cost? For the simple edig Sift we built above we run a benchmark of 1000 requests to measure the latency of our instance and plotted its distribution:

The average response time was ~70ms and 98% of requests returned in less than 200ms. The few outliers can be attributed to many reasons such as network delays, the DNS lookups we performed or our internal container setup time to name a few.

Any rate limits maybe? Our rate limits per hour are quite generous, as you can check from the response headers of each RPC request e.g. Redsift-Ratelimit-Limit: 1000000.

We are always looking for ways to improve and make our platform better. If you have some feedback or an idea you want to share with us please drop us a line at team [at] resift dot io.