Checkly updates app.checklyhq.com

You can now select the region from which we run you "ad hoc" checks when building and debugging your checks. This helps when your corporate firewall blocks our default Frankfurt (eu-central-1) region, or you want to check the latency from a specific region.

Head over to the API check editor or browser check editor and click the small flag icon in the button. Enjoy!

We just shipped three bug fixes for the following issues:

1. Our Webhook alerter was not sending the correct ALERT_SSL value in the ALERT_TYPE variable when an SSL expiry alert was triggered. This is now fixed and a test is added to our code base.

2. The public API was not paginating the /v1/checks endpoint correctly. Not enough results were returned. Patched that one too and we monitor this now with Checkly!

3. Lastly, when configuring Basic Auth credentials in the UI, the credentials would not get stored correctly due to a UI bug. That is working again.

Thanks to all that reported these bugs!

We just shipped some updates:

• We now alert on failing setup scripts that prohibit a check from running.
• We expose the "recovery from degraded" alert type on our web hooks.
• We introduced a dedicated "degraded status" gauge in our Prometheus integration.

Read all about the details in this blog post

We just fixed some bugs and usability/reliability issues with Github deployment triggers.

1. Environment URLS containing dashes "-" were sometimes not parsed and replaced correctly in API check URL's.

2. Our queue worker behaved unreliably during maintenance. We moved our queue to a more reliable solution.

3. We fixed some layout issues in the Github "check" markdown you see in your PR's and commits. We now also report the environment URL used in the Github check.

We did a full rewrite and a slight redesign of the home dashboard to make it snappier, more useful and easier to update with new features in the future.

Here's what changed:

• You can now search! Click the search bar or hit / and start typing.
• Click tags to filter by those tags.
• Click the column names to sort.
• We added the 95th percentile value to the metrics.
• The three-dotted "more" menu on the right now allows running, editing, copying and deleting a check.
• Aggregated metrics and check results are lazy loaded, allowing a far snappier initial loading.

Soft limits

You can now set a soft limit on the response times for API checks. Using a handy slider, you can set when a check is "just" degraded and when a check should be considered failing due to extreme latency.

Learn more about soft limits in our docs

Note: You can still set a hard assertion on the response time if you want.

Alert toggling

Checkly now also sends out degraded alerts through email, Slack etc.

However, we initially turned the new degraded alerts off when rolling out this feature to not trigger any extra alerts for users whose checks might be in the degraded zone.

To enable degraded alerts, just head over to your alert channel settings and check the box! More info in the docs

Parts of Checkly were less or not available today from roughly 12:00 to 13:00 CET. Features affected were

• The web application
• Public API

Other features (monitoring, alerting, deployments, public dashboards, triggers) were not affected.

Root cause

The cause of the outage was a too small limit on API calls to our authentication provider Auth0.com. Auth0 has a 20 requests per second limit on their public JWT endoint (https://auth0.com/docs/policies/rate-limits).

The code on our side however already caches and rate limits so we don't hit the provider limit. This was set too low in the context of the growing Checkly user base.

Note: This issue was 100% a mistake on the Checkly side. Auth0 has no blame here.

Triage

Resolving the issue took a little bit longer than ideal. Our alerting was on-point and immediately showed the JWT authentication rate limit to be the issue.

JwksRateLimitError: Too many requests to the JWKS endpoint

However, it was unclear whether this was on our side or on the provider's side. Figuring this out and patching the configuration setting therefore took longer than expected. The fact that the rate limit is set in requests per minute also did not help:

{ 
  cache: true,
  rateLimit: true,
  jwksRequestsPerMinute: 20
}

Lessons learned

• This was avoidable. Capacity planning with regard to Checkly's growth should have caught this.
• Checkly's monitoring was not affected. This is good proof that the monitoring & alerting backend is robust and can withstand API outages.

Today we have to do some maintenance on our API and the underlying database.

This means the Checkly web application will not be usable. We expect the maintenance to be done under 5 minutes, probably even quicker.

Your checks and associated alerting will keep running, only the web application at app.checklyhq.com will be briefly offline.

We just updated our alert channels! You can now

• Manage all alert channel preferences from one page
• Create custom webhooks payloads
• Add multiple Pagerduty channels

Alert channels

Checks are now "subscribed" to a channel and you can tweak which check is subscribed to which channel on the alert settings screen.

Before this update, you could also create channels per check in addition to global channel. This is no longer possible.

Note: We have migrated your "per check channels". This might result in some duplicates if you had two or more checks, each reporting to the same email. Please have a quick look on your alert settings tab and remove any double channels.

Custom webhooks

This was a long time coming and it's finally here. You now have full control over the payload of your webhook.

You can now:

• Add tokens, keys and other secrets from your environment variables to the URL and payload.
• Create completely custom webhook payloads using environment variables and event specific variables like the check status and name.

Have a look at our docs for a full overview of the available variables and examples of delivering webhooks to Jira and Trello.

We just rolled out a patch for two issues:

1. Browser checks triggered by Github deployments would not function correctly if these browser checks used encrypted variables stored in the check. The issue was a bug in the code that stopped the environment variables from being decrypted correctly.

2. The events overview showed double deployments whenever one deployment was actually running multiple checks. We tweaked the query, looks good now.

Happy Friday!