At work a small percentage of requests were failing with bizzare error messages. A quick fix was to roll back our deployment, but it took a year and a half to find a solution to the underlying bug.

The root cause: prototype pollution

Prototype pollution

By setting a property on Object.prototype, we modify the prototype of all objects in Javascript. This is prototype pollution.

In this example, we set the hello property on our Object prototype. Now all objects in our Javascript process will have the hello property.

const foo = {}
console.log("hello" in foo) // false
Object.prototype.hello = "world"
console.log("hello" in foo) // true
console.log("hello" in {}) // true

You normally wouldn’t modify Object.prototype, but you can do so accidentally with the __proto__ property.

We can rewrite the previous example to use __proto__.

const foo = {}
console.log("hello" in foo) // false
foo["__proto__"].hello = "world"
console.log("hello" in foo) // true
console.log("hello" in {}) // true

Suspicious Code

This is a simplified recreation of the production code.

Because the bug is triggered by user-generated data, the bug was able to exist in the codebase for over a year and half before triggering an error.

This bug was only discovered after many of my previous attempts came up empty.

const events = [
  {
    name: "click",
    properties: [
      "user_id",
      "name",
      // the problematic user-provided attribute
      "__proto__",
    ],
  },
  {
    name: "navigate",
    properties: ["user_id", "url"],
  },
]
const countsByProperty = {}

events.forEach((event) => {
  for (const property of event.properties) {
    // when "property" is "__proto__", we retrieve the prototype of countsByProperty.
    let entry = countsByProperty[property]
    if (!entry) {
      entry = { count: 0 }
    }
    // "entry" is the Object prototype.
    entry.count += 1 // set the "count" property on "Object.prototype"
  }
})

// all objects will have an inherited property of "count".
console.log("count" in {}) // true

This on it’s own isn’t enough to cause damage. By setting count on the prototype, we set an “inherited property” for future objects. Iteration done correctly ignores them.

const a = {}
a.__proto__.count = 1

for (const key of a) {
  console.log(a)
}
// no output

In our case, we used lodash.omitBy to removed undefined properties from an object.

lodash.omitBy iterates over both “own properties” and “inherited properties”, returning a new object with all properties set as “own properties”.

The previously invisible count property, now became an iterable property in the new object created by Lodash. This new attribute triggered errors in our service, causing requests to fail.

const omitBy = require("lodash/omitBy")
const a = {}
a.__proto__.count = 1
console.log(a.count === 1) // true
console.log(Object.keys(a).length) // 0

const b = omitBy(a, (x) => x != null)
console.log(Object.keys(b).length) // 1

Once the object prototype was modified in our running service, the only fix was to restart.

It took three years from when the bug was introduced to when it was fixed in our system.

Concise Example

Here’s a minimal version of the prototype pollution code.

const countsByProperty = {}
const property = "__proto__"

const entry = countsByProperty[property] || { count: 0 }
// entry is equal to "Object.prototype".
entry.count += 1

// all future objects will have an inherited property of "count".
console.log("count" in {}) // true

Solutions

Removing the __proto__ attribute from objects in Javascript is the most robust solution.

Node

We can remove __proto__ via node --disable-proto=delete.

Deno

__proto__ is removed by default.

Browsers

There’s no solution to remove __proto__. Use Map instead of plain objects for hash maps.

Instead of an object:

const properties = ["__proto__", "name", "title"]
// plain object vulnerable to prototype pollution in browser.
const propertyCounts = {}
properties.forEach((property) => {
  if (!(property in propertyCounts)) {
    propertyCounts[property] = { count: 0 }
  }
  // vulnerable to prototype pollution.
  propertyCounts[property].count += 1
})

Use a Map:

const properties = ["__proto__", "name", "title"]
// Map() is safe to use with user-provided keys.
const propertyCounts = new Map()
properties.forEach((property) => {
  if (!propertyCounts.has(property)) {
    propertyCounts.set(property, { count: 0 })
  }
  propertyCounts.get(property).count += 1
})