NAV Navbar

Scout Help Docs

Welcome to the help site for Scout Application Monitoring. Don’t have an account? Get started - it’s free.

Browse through the sidebar, search, email us, or join our Slack community. We’re here to help.

Ruby Agent

View our docs for installing, configuring, and troubleshooting the Scout Ruby agent.

Elixir Agent

View our docs for installing, configuring, and troubleshooting the Scout Elixir agent.

Signup - it’s free

Don’t have an account? Signup - our free plan monitors apps both large and small.

Overview

Scout Application Monitoring is a lightweight, production-grade application monitoring service built for modern development teams. Just embed our agent in your application: we handle the rest.

Here’s an overview of the key functionality in our application monitoring service:

Agents

We support Ruby on Rails and Elixir apps.

Our agent is designed to run in production environments and has low overhead. Every minute, the agent transmits metrics to our service over SSL.

There’s nothing to install on your servers.

User Interface

A complete overview of the Scout UX is available in the features area of this help site.

Ruby Agent

Requirements

Our Ruby agent supports Ruby on Rails 2.2+ and Ruby 1.8.7+ and the following app servers/background job frameworks:

Memory Bloat detection and ScoutProf require Ruby 2.1+.

Installation

Tailored instructions are provided within our user interface. General instructions:

1

Your Gemfile:

gem 'scout_apm'

Shell:

bundle install
2

Download your customized config file, placing it at config/scout_apm.yml.

Your customized config file is available within your Scout account.

3

Deploy.

Troubleshooting

No Data

Not seeing any data?

1

Is there a log/scout_apm.log file?

Yes:

Examine the log file for error messages:

tail -n1000 log/scout_apm.log | grep "Starting monitoring" -A20

See something noteworthy? Proceed to to the last step. Otherwise, continue to step 2.

No:

The gem was never initialized by the application.

Ensure that the scout_apm gem is not restricted to a specific group in your Gemfile. For example, the configuration below would prevent scout_apm from loading in a staging environment:

group :production do
  gem 'unicorn'
  gem 'scout_apm'
end

Jump to the last step if scout_apm is correctly configured in your Gemfile.

2

Was the scout_apm gem deployed with your application?

bundle list scout_apm
3

Did you download the config file, placing it in config/scout_apm.yml?

4

Did you restart the app?

5

Are you sure the application has processed any requests?

tail -n1000 log/production.log | grep "Processing"
6

Using Unicorn?

Add the preload_app true directive to your Unicorn config file. Read more in the Unicorn docs.

7

Oops! Looks like messed up. Send us an email with the following:

  • The last 1000 lines of your log/scout_apm.log file, if the file exists:
    tail -n1000 log/scout_apm.log.
  • Your application’s gems bundle list.
  • Rails version
  • Application Server (examples: Passenger, Thin, etc.)
  • Web Server (examples: Apache, Nginx, etc.)

We typically respond within a couple of hours during the business day.

Significant time spent in “Controller” or “Job”

When viewing a transaction trace, you may see time spent in the “controller” or “job” layers. This is time that falls outside of Scout’s default instrumentation. There are two options for gathering additional instrumentation:

  1. Custom Instrumentation - use our API to instrument pieces of code that are potential bottlenecks.
  2. ScoutProf - install our BETA agent which adds ScoutProf. ScoutProf breaks down time spent within the controller layer. Note that ScoutProf does not instrument background jobs.

Missing memory metrics

Memory allocation metrics require the following:

If the above requirements are not met, Scout continues to function but does not report allocation-related metrics.

Updating to the Newest Version

The latest version of scout_apm is SEE CHANGELOG.

1

Ensure your Gemfile entry for Scout is: gem 'scout_apm'

2

Run bundle update scout_apm

3

Re-deploy your application.

The gem version changelog is available here.

Configuration Options

The Ruby agent can be configured via the config/scout_apm.yml Yaml file and/or environment variables. A config file with your organization key is available for download as part of the install instructions.

ERB evaluation

ERB is evaluated when loading the config file. For example, you can set the app name based on the hostname:

common: &defaults
  name: <%= "ProjectPlanner.io (#{Rails.env})" %>

Configuration Reference

The following configuration settings are available:

Setting Name Description Default Required
name Name of the application (ex: ‘Photos App’). Rails.application.class.to_s. sub(/::Application$/, '') Yes
key The organization API key. Yes
monitor Whether monitoring should be enabled. false No
log_level The logging level of the agent. INFO No
log_file_path The path to the scout_apm.log log file directory. Use stdout to log to STDOUT. Environment#root+log/ or STDOUT if running on Heroku. No
hostname The hostname the metrics should be aggregrated under. Socket.gethostname No
proxy Specify the proxy URL (ex: https://proxy) if a proxy is required. No
host The protocol + domain where the agent should report. https://apm.scoutapp.com No
uri_reporting By default Scout reports the URL and filtered query parameters with transaction traces. Sensitive parameters in the URL will be redacted. To exclude query params entirely, use path. filtered_params No
disabled_instruments An Array of instruments that Scout should not install. Each Array element should should be a string-ified, case-sensitive class name (ex: ['Elasticsearch','HttpClient']). The default installed instruments can be viewed in the agent source. [] No
ignore An Array of web endpoints that Scout should not instrument. Routes that match the prefixed path (ex: ['/health', '/status']) will be ignored by the agent. [] No
enable_background_jobs Indicates if background jobs should be monitored. true No
dev_trace Indicates if DevTrace, the Scout development profiler, should be enabled. Note this setting only applies to the development environment. false No
profile Indicates if ScoutProf, the Scout code profiler, should be enabled. true No
revision_sha The Git SHA that corresponds to the version of the app being deployed. See docs No
detailed_middleware When true, the time spent in each middleware is visible in transaction traces vs. an aggregrate across all middlewares. This adds additional overhead and is disabled by default as middleware is an uncommon bottleneck. false No

Environment Variables

You can also configure Scout APM via environment variables. Environment variables override settings provided in scout_apm.yml.

Environment variables have the same names as those in the Yaml config file, but are prefixed with SCOUT_. For example, to set the organization key via environment variables:

export SCOUT_KEY=YOURKEY

Deploy Tracking Config

Scout can track deploys, making it easier to correlate changes in your app to performance. To enable deploy tracking, first ensure you are on the latest version of scout_apm. See our upgrade instructions.

Scout identifies deploys via the following:

  1. If you are using Capistrano, no extra configuration is required. Scout reads the contents of the REVISION and/or revisions.log file and parses out the SHA of the most recent release.
  2. If you are using Heroku, enable Dyno Metadata. This adds a HEROKU_SLUG_COMMIT environment variable to your dynos, which Scout then associates with deploys.
  3. If you are deploying via a custom approach, set a SCOUT_REVISION_SHA environment variable equal to the SHA of your latest release.
  4. If the app resides in a Git repo, Scout parses the output of git rev-parse --short HEAD to determine the revision SHA.

Enabling DevTrace

To enable DevTrace, our in-browser profiler:

1

Ensure you are on the latest version of scout_apm. See our upgrade instructions.

2

Add dev_trace: true to the development section of your scout_apm.yml config file or start your local Rails server with: SCOUT_DEV_TRACE=true rails server.

3

Refresh your browser window and look for the speed badge.

Request Queuing

Scout can measure the time it takes a request to reach your Rails app from farther upstream (a load balancer or web server). This appears in Scout as “Request Queuing” and provides an indication of your application’s capacity. Large request queuing time is an indication that your app needs more capacity.

request queuing

To see this metric within Scout, you need to configure your upstream software, adding an HTTP header that our agent reads. This is typically a one-line change.

HTTP Header

The Scout agent depends on an HTTP request header set by an upstream load balancer (ex: HAProxy) or web server (ex: Apache, Ngnix).

Protip: We suggest adding the header as early as possible in your infrastructure. This ensures you won’t miss performance issues that appear before the header is set.

The agent will read any of the following headers as the start time of the request:

%w(X-Queue-Start X-Request-Start X-QUEUE-START X-REQUEST-START x-queue-start x-request-start)

Include a value in the format t=MICROSECONDS_SINCE_EPOCH where MICROSECONDS_SINCE_EPOCH is an integer value of the number of microseconds that have elapsed since the beginning of Unix epoch.

Nearly any front-end HTTP server or load balancer can be configured to add this header. Some examples are below.

Apache

Apache’s mod_headers module includes a %t variable that is formatted for Scout usage. To enable request queue reporting, add this code to your Apache config:

RequestHeader set X-Request-Start "%t"

HAProxy

HAProxy 1.5+ supports timestamped headers and can be set in the frontend or backend section. We suggest putting this in the frontend to get a more accurate number:

http-request set-header X-Request-Start t=%Ts

Nginx

Nginx 1.2.6+ supports the use of the #{msec} variable. This makes adding the request queuing header straightforward.

General Nginx usage:

proxy_set_header X-Request-Start "t=${msec}";

Passenger 5+:

passenger_set_header X-Request-Start "t=${msec}";

Older Passsenger versions:

passenger_set_cgi_param X-REQUEST-START "t=${msec}";

Note: The Nginx option is local to the location block, and isn’t inherited.

ActionController::Metal

Prior to agent version 2.1.26, an extra step was required to instrument ActionController::Metal and ActionController::Api controllers. After 2.1.26, this is automatic.

The previous instructions which had an explicit include are no longer needed, but if that code is still in your controller, it will not harm anything. It will be ignored by the agent and have no effect.

Rack

Rack instrumentation is more explicit than Rails instrumentation, since Rack applications can take nearly any form. After installing our agent, instrumenting Rack is a three step process:

  1. Configuring the agent
  2. Starting the agent
  3. Wrapping endpoints in tracing

Configuration

Rack apps are configured using the same approach as a Rails app: either via a config/scout_apm.yml config file or environment variables.

Starting the Agent

Add the ScoutApm::Rack.install! startup call as close to the spot you run your Rack application as possible. install! should be called after you require other gems (ActiveRecord, Mongo, etc) to install instrumentation for those libraries.

# config.ru

require 'scout_apm'
ScoutApm::Rack.install!

run MyApp

Adding endpoints

Wrap each endpoint in a call to ScoutApm::Rack#transaction(name, env).

This may be fairly application specific in details.

Example:

app = Proc.new do |env|
  ScoutApm::Rack.transaction("API User Listing", env) do
    User.all.to_json
    ['200', {'Content-Type' => 'application/json'}, [users]]
  end
end

If you run into any issues, or want advice on naming or wrapping endpoints, contact us at support@scoutapp.com for additional help.

Sinatra

Instrumenting a Sinatra application is similar to instrumenting a generic Rack application.

Configuration

The agent configuration (API key, app name, etc) follows the same process as the Rack application config.

Starting the agent

Add the ScoutApm::Rack.install! startup call as close to the spot you run your Sinatra application as possible. install! should be called after you require other gems (ActiveRecord, Mongo, etc).

require './main'

require 'scout_apm'
ScoutApm::Rack.install!

run Sinatra::Application

Adding endpoints

Wrap each endpoint in a call to ScoutApm::Rack#transaction(name, env). For example:

get '/' do
  ScoutApm::Rack.transaction("get /", request.env) do
    ActiveRecord::Base.connection.execute("SELECT * FROM pg_catalog.pg_tables")
    "Hello!"
  end
end

See our Rack docs for adding an endpoint for more details.

Custom Context

Context lets you see the forest from the trees. For example, you can add custom context to answer critical questions like:

It’s simple to add custom context to your app. There are two types of context:

User Context

For context used to identify users (ex: email, name):

ScoutApm::Context.add_user({})

Examples:

ScoutApm::Context.add_user(email: @user.email)
ScoutApm::Context.add_user(email: @user.email, location: @user.location.to_s)

General Context

ScoutApm::Context.add({})

Examples:

ScoutApm::Context.add(account: @account.name)
ScoutApm::Context.add(database_shard: @db.shard_id, monthly_spend: @account.monthly_spend)

Default Context

Scout reports the Request URI and the user’s remote IP Address by default.

Context Types

Context values can be any of the following types:

Context Field Name Restrictions

Custom context names may contain alphanumeric characters, dashes, and underscores. Spaces are not allowed.

Attempts to add invalid context will be ignored.

Example: adding the current user’s email as context

Add the following to your ApplicationController class:

before_filter :set_scout_context

Create the following method:

def set_scout_context
  ScoutApm::Context.add_user(email: current_user.email) if current_user.is_a?(User)
end

Example: adding the monthly spend as context

Add the following line to the ApplicationController#set_scout_context method defined above:

ScoutApm::Context.add(monthly_spend: current_org.monthly_spend) if current_org

Custom Instrumentation

Traces that allocate significant amount of time to Controller or Job are good candidates to add custom instrumentation. This indicates a significant amount of time is falling outside our default instrumentation.

Limits

We limit the number of metrics that can be instrumented. Tracking too many unique metrics can impact the performance of our UI. Do not dynamically generate metric types in your instrumentation (ie self.class.instrument("user_#{user.id}", "generate") { ... }) as this can quickly exceed our rate limits.

Instrumenting method calls

To instrument a method call, add the following to the class containing the method:

  class User
    include ScoutApm::Tracer

    def export_activity
      # Do export work
    end
    instrument_method :export_activity
  end

The call to instrument_method should be after the method definition.

Naming methods instrumented via instrument_method

In the example above, the metric will appear in traces as User#export_activity. On timeseries charts, the time will be allocated to a Custom type.

To modify the type:

instrument_method :export_activity, type: 'Exporting'

A new Exporting metric will now appear on charts. The trace item will be displayed as Exporting/User/export_activity.

To modify the name:

instrument_method :export_activity, type: 'Exporting', name: 'user_activity'

The trace item will now be displayed as Exporting/user_activity.

Instrumenting blocks of code

To instrument a method call, add the following:

  class User
    include ScoutApm::Tracer

    def generate_profile_pic
      self.class.instrument("User", "generate_profile_pic") do
        # Do work
      end
    end
  end

Naming methods instrumented via instrument(type, name)

In the example above, the metric appear in traces as User/generate_profile_pic. On timeseries charts, the time will be allocated to a User type. To modify the type or simply, simply change the instrument corresponding method arguments.

Testing instrumentation

Improper instrumentation can break your application. It’s important to test before deploying to production. The easiest way to validate your instrumentation is by running DevTrace and ensuring the new metric appears as desired.

After restarting your dev server with DevTrace enabled, refresh the browser page and view the trace output. The new metric should appear in the trace:

custom devtrace

Docker

Scout runs within Docker containers without any special configuration.

It’s common to configure Docker containers with environment variables. Scout can use environment variables instead of the scout_apm.yml config file.

Heroku

Scout runs on Heroku without any special configuration. When Scout detects that an app is being served via Heroku:

Configuration

Scout can be configured via environment variables. This means you can use heroku config:set to configure the agent. For example, you can set the application name that appears in the Scout UI with:

heroku config:set SCOUT_NAME='My Heroku App'

See the configuration section for more information on the available config settings and environment variable functionality.

Using the Scout Heroku Add-on

Scout is also available as a Heroku Add-on. The add-on automates setting the proper Heroku config variables during the provisioning process.

Cloud Foundry

Scout runs on Cloud Foundry without any special configuration.

We suggest a few configuration changes in the scout_apm.yml file to best take advantage of Cloud Foundry:

  1. Set log_file_path: STDOUT to send your the Scout APM log contents to the Loggregator.
  2. Use the application name configured via Cloud Foundry to identify the app.
  3. Override the hostname reported to Scout. Cloud Foundry hostnames are dynamically generated and don’t have any meaningful information. We suggest using a combination of the application name and the instance index.

A sample config for Cloud Foundry that implements the above suggestions:

common: &defaults
  key: YOUR_KEY
  monitor: true
  # use the configured application name to identify the app.
  name: <%= ENV['VCAP_APPLICATION'] ? JSON.parse(ENV['VCAP_APPLICATION'])['application_name'] : "YOUR APP NAME (#{Rails.env})" %>
  # make logs available to the Loggregator
  log_file_path: STDOUT
  # reports w/a more identifiable instance name using the application name and instance index. ex: rails-sample.0
  hostname: <%= ENV['VCAP_APPLICATION'] ? "#{JSON.parse(ENV['VCAP_APPLICATION'])['application_name']}.#{ENV['CF_INSTANCE_INDEX']}"  : Socket.gethostname %>

production:
  <<: *defaults

development:
  <<: *defaults
  monitor: false

test:
  <<: *defaults
  monitor: false

staging:
  <<: *defaults

Instrumented Libraries

The following libraries are currently instrumented:

Additionally, Scout can also instrument request queuing time.

You can instrument your own code or other libraries via custom instrumentation.

Environments

It typically makes sense to treat each environment (production, staging, etc) as a separate application within Scout and ignore the development and test environments. Configure a unique app name for each environment as Scout aggregates data by the app name.

There are 2 approaches:

1. Modifying your scout_apm.yml config file

Here’s an example scout_apm.yml configuration to achieve this:

common: &defaults
  name: <%= "YOUR_APP_NAME (#{Rails.env})" %>
  key: YOUR_KEY
  log_level: info
  monitor: true

production:
  <<: *defaults

development:
  <<: *defaults
  monitor: false

test:
  <<: *defaults
  monitor: false

staging:
  <<: *defaults

2. Setting the SCOUT_NAME environment variable

Setting the SCOUT_NAME and SCOUT_MONITOR environment variables will override settings settings your scout_apm.yml config file.

To isolate data for a staging environment: SCOUT_NAME="YOUR_APP_NAME (Staging)".

To disable monitoring: SCOUT_MONITOR=false.

See the full list of configuration options.

Disabling a Node

To disable Scout APM on any node in your environment, just set monitor: false in your scout_apm.yml configuration file on that server, and restart your app server. Example:

common: &defaults
  name: <%= "YOUR_APP_NAME (#{Rails.env})" %>
  key: YOUR_KEY
  log_level: info
  monitor: false

production:
  <<: *defaults

Since the YAML config file allows ERB evaluation, you can even programatically enable/disable nodes based on host name. This example enables Scout APM on web1 through web5:

common: &defaults
  name: <%= "YOUR_APP_NAME (#{Rails.env})" %>
  key: YOUR_KEY
  log_level: info
  monitor: <%= Socket.gethostname.match(/web[1..5]/) %>

production:
  <<: *defaults

Aft you’ve disabled a node in your configuration file and restarted your app server, the node show up as inactive in the UI after 10 minutes.

Ignoring transactions

There are a couple of approaches to ignore web requests and background jobs you don’t care to instrument. These approaches are listed below.

By the web endpoint path name

You can ignore requests to web endpoints that match specific paths (like /health_check). See the ignore setting in the configuration options.

In your code

To selectively ignore a web request or background job in your code, add the following within the transaction:

req = ScoutApm::RequestManager.lookup
req.ignore_request!

Ignoring all background jobs

You can ignore all background jobs by setting enable_background_jobs: false in your configuration file. See the configuration options.

Elixir Agent

Requirements

Our Elixir agent supports Phoenix 1.2.0+ and Elixir 1.4+.

Installation

Tailored instructions are provided within our user interface. General instructions:

AAdd the scout_apm dependency.

Your mix.exs file:

# mix.exs

 def application do
   [mod: {YourApp, []},
    applications: [..., :scout_apm]]
 end

 def deps do
   [{:phoenix, "~> 1.2.0"},
    ...
    {:scout_apm, "~> 0.0"}]
 end

Shell:

mix deps.get

B Download your customized config file, placing it at config/scout_apm.exs.

Your customized config file is available within your Scout account. Inside the file, replace "YourApp" with the app name you’d like to appear within Scout.

CIntegrate into your Phoenix app.

config/config.exs:

# config/config.exs
import_config "scout_apm.exs"

config :your_app, YourApp.Repo,
  loggers: [{Ecto.LogEntry, :log, []},
            {ScoutApm.Instruments.EctoLogger, :log, []}]

web/web.ex:

# web/web.ex
defmodule HeroReview.Web do
  def controller do
    quote do
      use Phoenix.Controller
      use ScoutApm.Instrumentation
      ...

DRestart your app.

mix phoenix.server

Troubleshooting

Not seeing data? Send us an email with the following:

We typically respond within a couple of hours during the business day.

Configuration

The Elixir agent can be configured via the config/scout_apm.exs file. A config file with your organization key is available for download as part of the install instructions. A application name and key is required:

config :scout_apm,
  name: "Your App", # The app name that will appear within the Scout UI
  key: "YOUR SCOUT KEY"

Alternately, you can also use environment variables of your choosing by formatting your configuration as a tuple with :system as the first value and the environment variable expected as the second.

config :scout_apm,
  name: { :system, "SCOUT_APP_NAME" },
  key: { :system, "SCOUT_APP_KEY" }

Configuration Reference

The following configuration settings are available:

Setting Name Description Default Required
name Name of the application (ex: ‘Photos App’). Yes
monitor Whether monitoring data should be reported. true No
key The organization API key. Yes
dev_trace Indicates if DevTrace, the Scout development profiler, should be enabled. false No
host The protocol + domain where the agent should report. https://apm.scoutapp.com No
log_level The logging level of the agent. Possible values: :debug, :info, :warn, and :error. :info No

Updating to the Newest Version

AEnsure your mix.exs dependency entry for scout_apm is: {:scout_apm, "~> 0.0"}`

Bmix deps.get scout_apm

CRecompile and deploy.

Deploy Tracking Config

The Elixir agent doesn’t support deploy tracking yet. Contact support@scoutapp.com to be notified of updates.

Auto-Instrumented Libraries

Our install instructions walk through instrumenting the following libraries:

See instrumenting common librariess for guides on instrumenting other Elixir libraries.

Instrumenting Common Libraries

We’ve collected best practices for instrumenting common transactions and timing functions below. If you have a suggestion, please share it. See our custom instrumentation quickstart for more details on adding instrumentation.

Phoenix Channels

Web or background transactions?

Naming channel transactions

Provide an identifiable name based on the message the handle_out/ or handle_in/ function receives.

An example:

defmodule FirestormWeb.Web.PostsChannel do
  use FirestormWeb.Web, :channel
  use ScoutApm.Tracing

  # Will appear under "Web" in the UI, named "PostsChannel.update"
  @transaction(type: "web", name: "PostsChannel.update")
  def handle_out("update", msg, socket) do
    push socket, "update", FetchView.render("index.json", msg)
  end

GenServer calls

It’s common to use GenServer to handle background work outside the web request flow. Suggestions:

An example:

defmodule Flight.Checker do
  use GenServer
  use ScoutApm.Tracing

  # Will appear under "Background Jobs" in the UI, named "Flight.handle_check".
  @transaction(type: "background", name: "check")
  def handle_call({:check, flight}, _from, state) do
    # Do work...
  end

Task.start

These execute asynchronously, so treat as a background transaction.

Task.start(fn ->
  # Will appear under "Background Jobs" in the UI, named "Crawler.crawl".
  ScoutApm.Tracing.transaction(:background,"Crawler.crawl") do
    Crawler.crawl(url)
  end
end)

Task.Supervisor.start_child

Like Task.start, these execute asynchronously, so treat as a background transaction.

Task.Supervisor.start_child(YourApp.TaskSupervisor, fn ->
  # Will appear under "Background Jobs" in the UI, named "Crawler.crawl".
  ScoutApm.Tracing.transaction(:background,"Crawler.crawl") do
    Crawler.crawl(url)
  end
end)

Exq

To instrument Exq background jobs, use ScoutApm.Tracing, and add a @transaction module attribute to each worker’s perform function:

defmodule MyWorker do
  use ScoutApm.Tracing

  # Will appear under "Background Jobs" in the UI, named "MyWorker.perform".
  @transaction(type: "background")
  def perform(arg1, arg2) do
    # do work
  end
end

Absinthe

Requests to the Absinthe plug can be grouped by the GraphQL operationName under the “Web” UI by adding this plug to your pipeline.

HTTPoison

Download this ScoutApm.HTTPoison module into your app’s /lib folder, then alias ScoutApm.HTTPoison when calling HTTPoison functions:

defmodule Demo.Web.PageController do
  use Demo.Web, :controller
  # Will route function calls to `HTTPoision` through `ScoutApm.HTTPoison`, which times the execution of the HTTP call.
  alias ScoutApm.HTTPoison

  def index(conn, _params) do
    # "HTTP" will appear on timeseries charts. "HTTP/get" and the url "https://cnn.com" will appear in traces.
    HTTPoison.get("https://cnn.com")
    render conn, "index.html"
  end
end

MongoDB Ecto

Download this example MongoDB Repo module to use inplace of your existing MongoDB Repo module.

Custom Instrumentation

You can extend Scout to record additional types of transactions (background jobs, for example) and time the execution of code that fall outside our custom instrumentation.

For full details on instrumentation functions, see our ScoutApm.Tracing Hex docs.

Transactions & Timing

Scout’s instrumentation is divided into 2 areas:

  1. Transactions: these wrap around a flow of work, like a web request or a GenServer call. The UI groups data under transactions. Use @transaction module attributes and the transaction/4 macro.
  2. Timing: these measure individual pieces of work, like an HTTP request to an outside service or an Ecto query. Use @timing module attributes and the timing/4 macro.

Instrumenting transactions

Via @transaction module attributes:

defmodule YourApp.Web.RoomChannel do
  use Phoenix.Channel
  use ScoutApm.Tracing

  # Will appear under "Web" in the UI, named "YourApp.Web.RoomChannel.join".
  @transaction(type: "web")
  def join("topic:html", _message, socket) do
    {:ok, socket}
  end

  # Will appear under "Background Jobs" in the UI, named "RoomChannel.ping".
  @transaction(type: "background", name: "RoomChannel.ping")
  def handle_in("ping", %{"body" => body}, socket) do
    broadcast! socket, "new_msg", %{body: body}
    {:noreply, socket}
  end

Via transaction/4:

import ScoutApm.Tracking

def do_async_work do
  Task.start(fn ->
    # Will appear under "Background Jobs" in the UI, named "Do Work".
    transaction(:background, "Do Work") do
      # Do work...
    end
  end)
end

See the ScoutApm.Tracing Hexdocs for details on instrumenting transactions.

Timing functions and blocks of code

Via @timing module attributes:

defmodule Searcher do
  use ScoutApm.Tracing

  # Time associated with this function will appear under "Hound" in timeseries charts.
  # The function will appear as `Hound/open_search` in transaction traces.
  @timing(category: "Hound")
  def open_search(url) do
    navigate_to(url)
  end

  # Time associated with this function will appear under "Hound" in timeseries charts.
  # The function will appear as `Hound/homepage` in transaction traces.
  @timing(category: "Hound", name: "homepage")
  def open_homepage(url) do
    navigate_to(url)
  end

Via timing/4:

defmodule PhoenixApp.PageController do
  use PhoenixApp.Web, :controller
  import ScoutApm.Tracing

  def index(conn, _params) do
    timing("Timer", "sleep") do
      :timer.sleep(3000)
    end
    render conn, "index.html"
  end

See the ScoutApm.Tracing Hexdocs for details on timing functions and blocks of code.

Limits on category arity

We limit the number of unique categories that can be instrumented. Tracking too many unique category can impact the performance of our UI. Do not dynamically generate categories in your instrumentation (ie timing("user_#{user.id}", "generate", do: do_work()) as this can quickly exceed our rate limits.

Adding a description

Call ScoutApm.Tracing.update_desc/1 to add relevant information to the instrumented item. This description is then viewable in traces. An example:

timing("HTTP", "GitHub_Avatar") do
  url = "https://github.com/#{user.id}.png"
  update_desc("GET #{url}")
  HTTPoison.get(url)
end

Tracking already executed time

Libraries like Ecto log details on executed queries. This includes timing information. To add a trace item for this, use ScoutApm.Tracing.track. An example:

defmodule YourApp.Mongo.Repo do
  use Ecto.Repo

  # Scout instrumentation of Mongo queries. These appear in traces as "Ecto/Read", "Ecto/Write", etc.
  def log(entry) do
    ScoutApm.Tracing.track(
      "Ecto",
      query_name(entry),
      entry.query_time,
      :microseconds
    )
    super entry
  end

end

In the example above, the metric will appear in traces as Ecto/#{query_time(entry)}. On timeseries charts, the time will be allocated to Ecto.

See the scout_apm hex docs for more information on track/.

Testing instrumentation

Improper instrumentation can break your application. It’s important to test before deploying to production. The easiest way to validate your instrumentation is by running DevTrace and ensuring the new metric appears as desired.

After restarting your app with DevTrace enabled, refresh the browser page and view the trace output. The new metric should appear in the trace.

Custom Context

Context lets you see the forest from the trees. For example, you can add custom context to answer critical questions like:

It’s simple to add custom context to your app. There are two types of context:

User Context

For context used to identify users (ex: email, name):

ScoutApm.add_user(key, value)

Examples:

ScoutApm.Context.add_user(:email, user.email)
ScoutApm.Context.add_user(:name, user.name)

General Context

ScoutApm.Context.add(key, value)

Examples:

ScoutApm.Context.add(:account, account.name)
ScoutApm.Context.add(:monthly_spend, account.monthly_spend)

Default Context

Scout reports the Request URI and the user’s remote IP Address by default.

Context Value Types

Context values can be any of the following types:

Context Key Restrictions

Context keys can be an Atom or String with only printable characters. Custom context keys may contain alphanumeric characters, dashes, and underscores. Spaces are not allowed.

Attempts to add invalid context will be ignored.

Environments

It typically makes sense to treat each environment (production, staging, etc) as a separate application within Scout. To do so, configure a unique app name for each environment. Scout aggregates data by the app name.

An example:

# config/staging.exs

config :scout_apm,
  name: "YOUR APP - Staging"

Enabling DevTrace

To enable DevTrace, our in-browser profiler:

1

Add dev_trace: true to the scout_apm section of your config/dev.exs file:

# config/dev.exs
config :scout_apm,
  dev_trace: true
2

Restart your app.

3

Refresh your browser window and look for the speed badge.

API

Introduction

The ScoutAPM API is currently fairly narrow in scope. It is intended to support 3rd party dashboards, exporting summary data from your applications.

If you have ideas on API enhancements, please contact us at support@scoutapp.com

Authorization

Obtaining a token

  1. Log into the ScoutAPM website
  2. Go to your organization’s settings
  3. Enter a name (for your use), and obtain a token

Sending Authorization

The key must be provided with every request, via one of several methods:

Response Format

Every endpoint returns a JSON object. The object will at the minimum, have a "header" field with embedded status and message fields. If the endpoint returned results, it will be under a results keys.

{
  "header": {
    "status": {
      "code": 200,
      "message": "OK"
    },
    "apiVersion": "0.1"
  },
  "results": { ... }
}

API Endpoints

Applications

Applications List

/api/v0/apps - returns a list of applications and their ids

Results:

"results": {
  "apps": [
    {
      "name": "MyApp Staging",
      "id": 100
    },
    {
      "name": "MyApp Production",
      "id": 101
    }
}

Application Detail

/api/v0/apps/:id - returns information about a specific application

Results:

"results": {
  "app": {
    "id": 101,
    "name": "MyApp Production"
  }
}

Metrics

Known Metric List

/api/v0/apps/:id/metrics returns a list of known metric types

Results:

"results": {
  availableMetrics": [
    "response_time",
    "errors",
    "throughput"
  ]
}

Metric Data

/api/v0/apps/:id/metrics/:metric_type - will return a time series dataset for the metric.

Parameters:

These two times must not be more than 2 weeks apart

Results:

"results": {
  series": {
    "response_time": [
      [
        "2016-05-16T22:00:00Z",
        90.33333333333333
      ],
      [
        "2016-05-16T22:01:00Z",
        86.87233333333333
      ]
    ]
  }
}

Features

Scout is Application Monitoring built for modern development teams. It’s built to provide the fastest path to a slow line-of-code. Signup for a free trial.

App Performance Overview

The overview page provides an at-a-glance, auto-refreshing view of your app’s performance and resource usage (mean response time by category, 95th percentile response time, throughput, error rate, and more). You can quickly dive into endpoint activity via click-and-drag (or pinch-and-expand with a mobile device) on the overview chart.

overview

Additionally, you can compare metrics in the overview chart and see how your app’s performance compares to different time periods.

Endpoint Details

You can view metrics for specific controller-action and background job workers. There is a similar chart interaction to the App Performance Overview page, with one difference: your selection will render an updated list of transaction traces that correspond to the selected time period: stream

You can sort traces by response time, object allocations, date, and more.

Transaction Traces

Scout collects detailed transactions across your endpoints automatically. The transaction traces provide a number of visual queues to direct you to hotspots. Dig into bottlenecks - down to the line-of-code, author, commit date, and deploy time - from this view.

transaction traces

Call Breakdown

Method calls are aggregated together and listed from most expensive to least expensive. The time displayed is the total time across all calls (not the time per-call).

stream show breakdown

SQL Queries

Scout captures a sanitized version of SQL queries. Click the “SQL” button next to a call to view details.

stream show sql

Don’t see the query next to an SQL call?

For performance reasons, Scout doesn’t attempt to sanitize large SQL queries. When this occurs, you won’t see an “SQL” button to view the raw query next to an SQL method call.

Code Backtraces

You’ll see “CODE” buttons next to method calls that are >= 500 ms. If you’ve enabled the GitHub integration, you can see the line-of-code, associated SQL or HTTP endpoint (if applicable), author, commit date, and deploy time for the relevant slow code.

stream show git

If you don’t enable the GitHub integration, you’ll see a backtrace.

ScoutProf

Every millisecond, ScoutProf captures a backtrace of what each thread in your application is currently running. Over many backtraces, when you combine them, it tells a story of what code paths are taking up the most time in your application.

Compared with our more traditional instrumentation of libraries like ActiveRecord, Net::HTTP and similar, ScoutProf works with your custom code. Now, when your application spends time processing your data in custom application code, or in libraries that Scout doesn’t yet instrument, instead of only being able to assign that to a large bucket of ActionController time, the time can be broken down to exactly what is taking up the most time.

Notice how the time in ActionController is broken down:

scoutprof

We still employ our traditional instrumentation, because it can give us deeper insights into common libraries, capturing the SQL being run, or the URL being fetched.

Enabling ScoutProf

ScoutProf is a BETA feature. To enable:

1

Modify your Gemfile entry for scout_apm, changing it to: gem 'scout_apm', '~> 3.0.x'

2

bundle update scout_apm

3

Deploy

A detailed ScoutProf FAQ is available in our reference area.

Database Monitoring

A database is shared resource: one expensive query can trigger a flood of slow queries that impact many web endpoints and background jobs. Scout’s database monitoring helps in three primary areas:

  1. Identifying the cause of database capacity issues.
  2. Understanding your query workload and which web endpoints and background jobs are the greatest users of database time.
  3. Highlighting changes in query workload and performance metrics vs. the norm.

Database metrics appear in two areas of the UI:

  1. Database Overview
  2. Web Endpoint & Background Job Breakdown

The video below shows the two areas in action. There is a detailed description of each area beneath the video.

Database Overview

database monitoring

Overview chart

The database monitoring overview chart helps answer important questions at-a-glance. For example:

The overview chart shows the top 4 most time-consuming queries and an aggregate of all other queries vs. overall query throughput.

Zooming

The primary interaction point of Scout’s database monitoring is the zoom functionality.

If you notice a spike in time spent by database queries, simply click and drag on the overview chart. The queries list will update, showing changes during the zoom window.

Annotations are added to the queries list when zooming:

Queries List

A list of queries, broken down by their ActiveRecord model, operation, and caller (either a web endpoint or background job) is listed below the overview chart.

By default, this list is truncated to display just the most time-consuming queries.

Database Events

Scout highlights significant events in database performance in the sidebar. For example, if time spent in database queries increases dramatically, you’ll find an insight here. Clicking on an insight jumps to the time window referenced by the insight.

Endpoint Database Breakdown

A breakdown of time spent per-transaction appears beneath the overview chart for both web endpoints and database metrics. Database queries are broken out in this breakdown:

database breakdown

Pricing

Database Monitoring is available as an addon. See your billing page for pricing information.

Database Monitoring Library Support

Scout currently monitors queries executed via ActiveRecord, which includes most relational databases (PostgreSQL, MySQL, etc).

Memory Bloat Detection

If a user triggers a request to your Rails application that results in a large number of object allocations (example: loading a large number of ActiveRecord objects), your app may require additional memory. The additional memory required to load the objects in memory is released back very slowly. Therefore, a single memory-hungry request will have a long-term impact on your Rails app’s memory usage.

There are 3 specific features of Scout to aid in fixing memory bloat.

Memory Bloat Insights

The Insights area of the dashboard identifies controller-actions and background jobs that have triggered significant memory increases. An overview of the object allocation breakdown by tier (ActiveRecord, ActionView, etc) is displayed on the dashboard.

memory insights

Memory Traces

When inspecting a transaction trace, you’ll see a “Memory Allocation Breakdown” section:

memory trace

For perspective, we display how this trace’s allocations compare to the norm.

Alerting

alerts_chart

Alerting keeps your team updated if your app’s performance degrades. Alerts can be configured on the app as a whole and on individual endpoints. Metrics include:

Alert conditions

Configure alert conditions via the “Alerts” pill in the UI:

alert_conditions

Notification groups

Alerts are sent to a notification group, which is composed of notification channels. You can configure these under your org’s settings menu:

alert notification groups

Deploy Tracking

deploy tracking

Correlate deploys with your app’s performance: Scout’s GitHub-enhanced deploy tracking makes it easy to identify the Git branch or tag running now and which team members contributed to every deploy.

Scout tracks your deploys without additional configuration if you are running Capistrano. If you aren’t using Capistrano or deploying your app to Heroku, see our deploy tracking configuration docs.

Sorting

You can sort by memory allocations throughout the UI: from the list of endpoints, to our pulldowns, to transaction traces.

memory sort

Git Integration

If your code is hosted at GitHub, you can see the relevant slow line-of-code within the Scout user interface when viewing a transaction trace. Additionally, you’ll also see the:

Git integration must be configured on the settings page for each app. Scout integrates with GitHub via OAuth. Pick the repository name and branch name used for your application.

git settings

Context

Context lets you see the forest from the trees. For example, you can add custom context to answer critical questions like:

Adding custom context is easy - learn how via Ruby or Elixir.

When viewing a transaction trace, click the “Context” section to see the context Scout has collected.

Endpoints Performance

Endpoints Overview

The endpoints area within Scout provides a sortable view of your app’s overall performance aggregated by endpoint name. Click on an endpoint to view details.

endpoints overview

Time Comparisons

You can easily compare the performance of your application between different time periods via the time selection on the top right corner of the UI.

time compare

DevTrace

DevTrace is our development profiler: it’s included with our Ruby and Elixir libraries. DevTrace can be used for free without signup. Enabling DevTrace adds a speed badge when navigating your app in development. Clicking the speed badge reveals a shareable transaction trace of the request.

devtrace

View our Ruby and Elixir instructions.

Chart Embeds

You can embed an app’s overview chart inside another web page (ex: an internal key metrics dashboard):

  1. Access the application dashboard within the Scout UI.
  2. Adjust the timeframe and metrics to those you’d like to include in the embedded chart.
  3. Click the embed icon and copy the relevant code.

chart_embed

Note that you’ll need to update the provided iframe url with a Scout API key.

When clicking on an embedded chart, you’ll be redirected to the relevant application.

Data Retention

Scout stores 30 days of metrics and seven days of transaction traces.

Service Status

We’re transparent about our uptime and service issues. If you appear to be experiencing issues with our service:

Contacting Support

Don’t hesitate to contact us at support@scoutapp.com any issues. We typically respond in a couple of hours during the business day.

Reference

How we collect metrics

Scout is engineered to monitor the performance of mission-critical production applications. Here’s a short overview of how this happens:

Performance Overhead

Our agent is designed to run in production environments and is extensively benchmarked to ensure it performs on high-traffic applications.

Our most recent benchmarks (lower is better):

overhead

We’ve open-sourced our benchmarks so you can test on our own. If your results differ, reach out to us at support@scoutapp.com.

Call Aggregation

During a transaction, the Scout agent records each database call, each external HTTP request, each rendering of a view, and several other instrumented libraries. While each individual pieces of this overall trace has a tiny memory footprint, large transactions can sometimes build up thousands and thousands of them.

To limit our agent’s memory usage, we stop recording the details of every instrument after a relatively high limit. Detailed metrics and backtraces are collected for all calls up to the limit and aggregated metrics are collected for calls over the limit.

Security

We take the security of your code metrics extremely seriously. Keeping your data secure is fundamental to our business. Scout is nearing a decade storing critical metrics with our server monitoring product and those same fundamentals are applied here:

Information sent to our servers

The following data is sent to our servers from the agent:

HIPAA compliance

While Scout itself is not HIPAA compliant, our agent can be installed safely in HIPAA compliant environments. To ensure user data is properly de-identified:

  1. Disable sending HTTP query params if these contain sensitive data via the uri_reporting config option.
  2. Do not add custom context (like reporting the current user in the session).

Email support@scoutapp.com with any questions on security compliance.

Git Integration

Scout only needs read-only access to your repository, but unfortunately, Github doesn’t currently allow this - they only offer read-write permissions through their OAuth API.

We have asked Github to offer read-only permissions, and they’ve said that the feature coming soon. In the mean time, we’re limited to the permissions structure Github offers. Our current Git security practices:

All that said, we suggest the following:

Workaround for read-only Github Access

With a few extra steps, you can grant Scout read-only access. Here’s how:

ScoutProf FAQ

Does ScoutProf work with Stackprof?

ScoutProf and StackProf are not guaranteed to operate at the same time. If you wish to use Stackprof, temporarily disable profiling in your config file (profile: false) or via environment variables (SCOUT_PROFILE=false). See the agent’s configuration options.

How is ScoutProf different than Stackprof?

Stackprof was the inspiration for ScoutProf. Although little original Stackprof code remains, we started with Stackprof’s core approach, and integrated it with our APM agent gem, changed it heavily to work with threaded applications, and implemented an easy to understand UI on our trace view.

What do sample counts mean ?

ScoutProf attempts to sample your application every millisecond, capturing a snapshot backtrace of what is running in each thread. For each successful backtrace captured, that is a sample. Later, when we process the raw backtraces, identical traces get combined and the sample count is how many of each unique backtrace were seen.

Why do sample counts vary?

Samples will be paused automatically for a few different reasons:

The specifics of exactly how often these scenarios happen depend on how and in what order your ruby code runs. Different sample counts can be expected, even for the same endpoint.

What are the ScoutProf requirements?

What’s supported during BETA?

During our BETA period, ScoutProf has a few limitations:

The ScoutProf-enabled version of scout_apm can be safely installed on all environments our agent supports: the limitations above only prevent ScoutProf from running.

Billing

Free Trial

We offer a no-risk, free 14-day trial. Delete your monitored apps before the trial concludes and you don’t pay.

Billing Date

Your first bill is 30 days after your signup date.

Subscription Style

You can choose the subscription style that makes sense for your organization. We offer two subscription styles:

Per-Server

This is the default approach. You are billed for the number of servers that are actively reporting on your billing date.

Per-Request

If you have a smaller application or have many smaller instances or Docker containers per-request billing may make more sense. Volume discounts are automatically applied as your application handles more throughput. Contact support@scoutapp.com for pricing options.

Replacing New Relic

Scout is an attractive alternative to New Relic for modern dev teams (frequent deploys, using Git, deploying to many micro instances & containers, using vendors for key infrastructure services like Amazon RDS, etc). We provide a laser-focus on getting to slow custom application code fast vs. wide breadth as debugging slow custom application code is typically the most time-intensive performance optimization work.

In many cases, Scout is able to replace New Relic as-is. However, there are cases where your app has specific needs we currently don’t provide. Don’t fret - here’s some of the more common scenarios and our suggestions for building a monitoring stack you’ll love:

Our Monitoring Stack

Curious about what a company that lives-and-breathes monitoring (us!) uses to monitor our apps? We shared our complete monitoring stack on our blog.

Talk to us about your monitoring stack

Don’t hesitate to email us if you need to talk through your monitoring stack. Monitoring is something we know and love.