This past Tuesday, 18 December 2012, from approximately 6:00 PM PST to 6:45 PM PST, our font network experienced a partial outage for many users around the world.

Our font network uses a worldwide system of servers run by a major commercial content delivery network (or CDN). Each request for fonts is routed to the nearest server in order to return a response as quickly as possible.

Unfortunately, our CDN provider pushed a bad configuration on Tuesday evening, and a change that was meant for only one of their customers got applied to many customers instead, including Typekit. From our CDN provider’s post-mortem:

The outage occurred as a result of a change we made to a configuration in our core caching platform. It’s important to note that this was not a security issue, or even a bug, but a simple human error that was not flagged by our standard staging and testing procedures.

This mistake resulted in font requests failing, which meant that some visitors saw fallback fonts on sites using Typekit for a period of about 45 minutes. The configuration change rolled out slowly, which meant that more and more traffic was affected over time. Typekit Enterprise customers who use their own CDN environment weren’t affected by the outage.

We have external and internal monitoring to detect problems with our font serving network, but our monitoring didn’t pick up on the growing issue until approximately 6:18 PM PST. At that time, we quickly got in touch with both our CDN provider and our customers (via the status blog and Twitter). At approximately 6:45 PM PST, our CDN provider finished rolling back the bad configuration changes, and font serving was restored to normal.

Requests for fonts failed quickly during the outage, and the fallback fonts specified in each site’s CSS were shown instead. This illustrates one reason why designing and implementing good fallback fonts is important. Not only does it improve the appearance of your site during an unlikely outage, but it’s also important for older browsers that don’t support web fonts.

We know that our customers have high expectations for uptime on our font network and prompt communication about issues and downtime. We sincerely apologize for the impact that this outage had on our customers and their visitors, and our delay in detecting the problem. Our CDN provider has responded proactively to prevent this type of configuration error in the future. At Typekit we’re working to improve our monitoring so we can detect and respond to outages like this one more quickly.

If you have questions or concerns, please get in touch with us at support@typekit.com.