Ruby vs. Javascript — JSON Serialization Quirks

At Wheel, we recently integrated with a third party service. That service had workflows that would send us webhooks when entities progressed through that workflow. All fine and good right?

The Problem

So how do you authenticate a webhook request is coming from the right system. Here’s a list of a couple of ways that I’ve seen, but it’s certainly not exhaustive:

  • A shared secret, usually given to you from the API provider, or you set it yourself
  • Static IP list
  • A hash where the hash key is some shared secret and the value being hashed is some agreed upon thing, usually the request body.

The first two are easy enough to implement, although the second one might be a little tougher if you’re in front of a proxy, and that proxy doesn’t send along X-Forward-For for whatever reason.

The third one is the one our new third party service chose. Here’s the code they gave in their docs:

Simple enough right? Take the request body, make a SHA-256 digest using the API Token as the secret and the body as the message, then compare with a header.

The backend service we were using to authenticate the request is written in Node, so we wrote the following code to authenticate the request:

Unfortunately, when testing, we quickly found that the hashes we were generating were different than what was expected.

Here’s an example:

Hash Received from 3rd party:

9a173f94ebe94f073f15e8[...]381adeab631b146cbbf

Our calculated hash:

6ee12bb6b12daa1797fc2da[...]2091854faff60a1ca0

The Investigation

So I had quite the task in front of me: figure out why these two hashes were different. Luckily the exact response body is given to us in their UI, so I grabbed that and got to work.

First step was attempting to reproduce the hash they were giving me in Ruby. I figured that if their API docs have a Ruby implementation, I should start there:

irb(main):1:0> hash = { //just pasted JSON body exactly }
irb(main):2:0> body = hash.to_json
irb(main):3:0> signature = OpenSSL::HMAC.hexdigest(OpenSSL::Digest::SHA256.new, SECRET, body)
"9a173f94ebe94f073f15e8[...]381adeab631b146cbbf"

Okay, well that’s good. I got the hash result I was hoping to get. Let’s try in the Node console next:

> body = { //just pasted JSON body exactly }
> crypto.createHmac('sha256', SECRET).update(JSON.stringify(body)).digest('hex')
'6ee12bb6b12daa1797fc2da[...]2091854faff60a1ca0'

Okay, so that at least proves that we’re getting a different hash. But why?

Grabbing both the Ruby and the Javascript inputs to the HMAC digest, I did a diff of the two. Can you spot the culprit?

This string field of the request body, which contained a URL, was different! In fact it was different in two places, where an & was replaced with \u0026. Okay, so back to the Node console, let’s see if we can replace the & there.

> verifySignature(header, body.replace("&", "\u0026"))
'6ee12bb6b12daa1797fc2da[...]2091854faff60a1ca0'

Hmm.. exactly the same hash output. In fact, if we look at the return value of just the replace, we get the exact same string, & and all.

That’s because Javascript strings are UTF-8, they take Unicode characters and evaluate them as unicode, rather than ASCII.

Okay, so we’re getting somewhere. Ruby’s Hash#to_json is encoding & as unicode characters, and the resultant hash is different because those unicode characters are being interpreted as ASCII.

Javascript however is not so lenient, it’s interpolating the unicode characters back into the character &. I tried a number of different ways to get Javascript to ignore the escape characters, but couldn’t figure it out. Please share in the comments if you have a better way.

The Root Cause

It does seem strange the Ruby has this behavior though. So I looked up the JSON library that most people use, and tracked down the JSON serialization implementation.

Interesting, it doesn’t seem to actually encode the & character at all, but I could take an educated guess at what was.

ActiveSupport has a JSON serializer that wraps around Fiori’s JSON gem, and sure enough, there it was.

Based on that code, there seems to be ways to not escape the &, but obviously our 3rd party service is not doing that. It makes sense that ActiveSupport would want to escape those HTML entities by default, since it deals with websites, but for the purpose of hashing, it seems like overkill. Especially if that behavior isn’t portable between languages.

The Solution

Unfortunately, we don’t have the luxury of waiting for our 3rd party service to fix this bug, especially since it’s a breaking change to all of their API customers.

So we’ll work around it.

Node’s crypto module is actually pretty robust. You can give HMAC a number of different options:

hmac.update(data[, inputEncoding])

Strings are out, since I can’t force the string to ignore unicode escaped characters, so I’ll turn to Buffers instead.

Creating a Buffer from a string is easy:

Buffer.from(string)

However, once created, a buffer cannot be grown or shrunk, and in this case we want to replace all & with \u0026 which will grow our buffer by 5 characters for each ampersand.

The easiest solution? Create a bunch of buffers and then concat them together!

Result:

9a173f94ebe94f073f15e8[...]381adeab631b146cbbf

Voila! We have a match!

Conclusion

Certainly not the most elegant solution, but it works (at least for those 3 characters), and can be expanded upon if the need arises.

We’ve already submitted a bug with the 3rd party provider, but if you have come across similar issues and fixed them in a different way, let me know!

At Wheel, we’re powering next generation virtual care. This post is just one example of how our engineering organization fully embraces our core values.

If you’re in Product or Engineering and looking for a place where you can build products that will help expand access to healthcare — check out our careers page.
We’re excited to talk to you.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ed Mitchell

Ed Mitchell

2 Followers

Accomplished web application and data engineer. Always ready to learn new things, meet new people, and tackle new challenges.