O365beat Released

An open-source shipper for Office 365 logs

Summary

We’re happy to announce the production release of o365beat, an open source log shipper used to fetch Office 365 audit logs from the Office 365 Management Activity API and forward them with all the flexibility and capability provided by the beats platform (specifically, libbeat).

With business email compromise on the rise and more teams moving to the convenience of hosted productivity services, we want our friends to have a free, open source, and convenient tool to gather facts and help secure their Office 365 tenancies.

Introduction

We’re happy to announce the production release of o365beat, an open source log shipper used to fetch Office 365 audit logs from the Office 365 Management Activity API and forward them with all the flexibility and capability provided by the beats platform (specifically, libbeat).

The first non-alpha release is v1.2.0 - it has a detailed README, and questions are welcome. Please contact us, open an issue, or submit a pull request if you have any issues in testing or production.

Getting Started with o365beat

The easiest way to get started with o365beat is to use the pre-built binaries available in the latest release.

These pre-built packages include configuration files which contain all the necessary credential information to connect to the audit logs for your tenancy. The default configuration file (o365beat.yml) pulls this information from your environment, like so:

o365beat:
  # period Defines how often API is polled for new content blobs
  # 5 min default, as new content (probably) isn't published too often
  # period: 5m

  # pull secrets from environment (e.g, > set -a; . ./ENV_FILE; set +a;)
  # or hard-coded here:
  tenant_domain: ${O365BEAT_TENANT_DOMAIN:}
  client_secret: ${O365BEAT_CLIENT_SECRET:}
  client_id:     ${O365BEAT_CLIENT_ID:}     # aka application id (GUID)
  directory_id:  ${O365BEAT_DIRECTORY_ID:}  # aka tenant id (GUID)
  registry_file_path: ${O365BEAT_REGISTRY_PATH:./o365beat-registry.json}

  # the following content types will be pulled from the API
  # for available types, see https://docs.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-reference#working-with-the-office-365-management-activity-api
  content_types:
    - Audit.AzureActiveDirectory
    - Audit.Exchange
    - Audit.SharePoint
    - Audit.General

NOTE: If you decide to hard-code these values, be sure to replace the ${:} syntax, which pulls from the environment. For example, use tenant_domain: acme.onmicrosoft.com or ${O365BEAT_TENANT_DOMAIN:acme.onmicrosoft.com}, not tenant_domain: ${acme.onmicrosoft.com:}.

Prerequisites and Permissions

O365beat requires access to the Office 365 Management APIs for your tenancy. Instructions for getting this set up are available in the Microsoft documentation

The README has more detailed instructions, with pictures!

Run

To run o365beat with all debugging output enabled, run:

./o365beat --path.config . -c o365beat.yml -e -d "*" # add --strict.perms=false under WSL 1

Receive with Logstash or Graylog

If you’re receiving o365beat logs with logstash, use the input type beats:

input {
  beats {
    port => "5044"
  }
}

Or you can send them to a graylog beats input, just select “Beats” from the dropdown. There’s also a guide in the sidecar docs.

Schema

As of v1.2.0, o365beat includes a processor to map the raw API-provided events to Elastic Common Schema (ECS) fields. This allows this beat to work with standard Kibana dashboards, including capabilities in Elastic SIEM.

See the Office 365 Management API schema documentation for details on the raw events. The ECS mapping is as follows, see the README for more details:

# from: https://docs.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-schema
# to: https://www.elastic.co/guide/en/ecs/current/ecs-client.html

processors:
  - convert:
      fields:
        - {from: "Id", to: "event.id", type: string}                # ecs core
        - {from: "RecordType", to: "event.code", type: string}      # ecs extended
        # - {from: "CreationTime", to: "", type: ""}                # @timestamp
        - {from: "Operation", to: "event.action", type: string}     # ecs core
        - {from: "OrganizationId", to: "cloud.account.id", type: string} # ecs extended
        # - {from: "UserType", to: "", type: ""}                    # no ecs mapping
        # - {from: "UserKey", to: "", type: ""}                     # no ecs mapping
        - {from: "Workload", to: "event.category", type: string}    # ecs core
        - {from: "ResultStatus", to: "event.outcome", type: string} # ecs extended
        # - {from: "ObjectId", to: "", type: ""}                    # no ecs mapping
        - {from: "UserId", to: "user.id", type: string}             # ecs core
        - {from: "ClientIP", to: "client.ip", type: ip}             # ecs core
        # - {from: "Scope", to: "", type: ""}                       # no ecs mapping

Please open an issue or a pull request if you have suggested improvements to this approach.

If you’re adventurous, or would like to contribute, you can of course build the project from source, the instructions for which are on github.

Conclusion

Keeping good logs is critical to effective security. If you don’t have good facts, you can forget about dashboards and correlation and automation and investigation and much of anything else. Hopefully this project helps your team capture and retain the logs you need from your Office 365 tenancy. If you’d like assistance getting set up, or in taking the next step once you’ve started keeping these logs, please contact us!