Maester vs ScubaGear: From Microsoft 365 Findings to Fixes

CISA ScubaGear and Maester are two free tools that test a Microsoft 365 tenant's configuration against published security baselines and report what fails. ScubaGear, maintained by the US Cybersecurity and Infrastructure Security Agency, checks the tenant against CISA's Secure Cloud Business Applications (SCuBA) baselines and writes HTML, JSON and CSV reports. Maester is a community PowerShell framework that runs those SCuBA checks plus CIS, Entra ID and Exchange Online tests, more than 360 in all, and produces an interactive HTML report with remediation steps. Most small businesses benefit from running both, and the real work starts after the report: deciding which failures are real, fixing them in a safe order and rerunning the tool to prove it.

A note on the name before going further: a search for "scuba gear" returns diving equipment, so this post says CISA ScubaGear and Maester for Microsoft 365 throughout. It is written for the person who owns a tenant and wants to know what these tools test, what their output looks like and how NetSys uses them in a Microsoft 365 security review.

What is CISA ScubaGear?

ScubaGear is, in the words of its GitHub repository, "an assessment tool that verifies that a Microsoft 365 (M365) tenant's configuration conforms to the policies described in the Secure Cloud Business Applications (SCuBA) Secure Configuration Baseline documents." CISA's SCuBA project page says the baselines were written to secure federal civilian agencies' cloud environments, and that CISA has made the tool and the baselines available to private sector organizations as well. As of October 2026 the repository lists baselines for Microsoft Entra ID, the security suite (advanced threat protection settings), Exchange Online, Power BI, Power Platform, SharePoint and Teams.

Mechanically, ScubaGear is a PowerShell module installed from the PowerShell Gallery on a Windows computer. It collects tenant settings through Microsoft 365 APIs, then uses Open Policy Agent to compare them against policies written in Rego from the baselines. It is an assessment tool: it reads configuration and reports on it, and remediation is a separate, human step. The project notes that SCuBA controls are mapped to NIST SP 800-53 and the MITRE ATT&CK framework, which helps when an auditor asks which framework a fix satisfies, and that a YAML configuration file is required for submissions under CISA's Binding Operational Directive 25-01. The code is distributed under the Creative Commons Zero license.

What is Maester?

Maester describes itself as "your Microsoft Security test automation framework": a PowerShell module built on Pester, the PowerShell testing framework, and Microsoft Graph, that turns Microsoft security recommendations into runnable tests. Its library draws on several sources: the CISA SCuBA checks (70 or more), tests based on the CIS Microsoft 365 Foundations Benchmark v3.1.0 (60 or more), the Entra ID Security Config Analyzer (EIDSCA), ORCA for Exchange Online, and more than 170 community tests, for a total the project puts above 360 as of October 2026.

Maester's output is an interactive HTML report with drill-down into each test, remediation steps and links to the Microsoft admin portal where the setting lives. It can run on demand from a workstation or on a schedule in GitHub Actions, Azure DevOps or Azure Automation, and it supports Workload Identity Federation so a scheduled run does not need a stored secret. The core framework and its tests are free and open source, maintained by the Maester team; a paid hosted option, Maester Cloud, exists for teams that do not want to run it themselves.

Maester vs ScubaGear: what does each one test?

The two overlap on SCuBA, and that is useful rather than redundant: when both tools fail the same control you have two independent readings of the same setting, and when they disagree you have found either a test that needs an exception or a setting that changed between runs.

What do the reports look like, and how do you read them?

A ScubaGear run produces an interactive HTML compliance report, a JSON file with the same results in structured form for parsing and comparison between runs, and a CSV that opens in Excel for the owner who wants a flat list. Maester's HTML report groups tests by source and area, shows pass and fail counts, and lets you expand a failed test to read what was checked, why it matters and the remediation steps, with a link to the admin portal page. That is convenient on the first read and more useful on the tenth, because the same tests appear run after run and you can see which fixes stuck.

Reading either report well means resisting the urge to treat the fail count as the score. Three questions matter for each failure: is the setting actually wrong for this business, is there a documented exception, and what breaks if we change it? A failed check on legacy authentication is almost always a real finding. A failed check on external sharing may be a business decision that needs a written exception rather than a change.

How do findings become fixes?

1. Run with a dedicated identity. Both tools sign in to the tenant to read configuration, Maester through Microsoft Graph and ScubaGear through Microsoft 365 APIs. Use an account or app registration created for the assessment, record the permissions it was granted, and remove it when the engagement ends.
2. Freeze a baseline copy. Keep the first run's HTML and JSON untouched. Every later run is compared against it.
3. Triage by control, not by tool. Merge both reports into one list keyed by the setting (for example, the Conditional Access policy that blocks legacy authentication), so one fix closes every test that checked it.
4. Separate true failures from exceptions. Record exceptions in writing with the business reason, the owner and a review date. An undocumented exception is itself a finding.
5. Sequence for safety. Break-glass accounts and report-only Conditional Access first, then enforcement, then Defender for Office 365 policies, then consent, forwarding and sharing settings. The order avoids locking out the owner; the Conditional Access glossary entry explains report-only mode.
6. Get approval per change. Each fix names what changes, who approved it, the window and the rollback.
7. Rerun and attach the evidence. The rerun report with the test now passing is the proof that goes in the file, next to the before copy.
8. Schedule the next run. Microsoft changes defaults and admins change settings. A scheduled monthly Maester run catches drift before an auditor or an attacker does.

Which should a small business run?

Run both, in this order: ScubaGear once for a clean SCuBA alignment report, then Maester on a schedule for ongoing coverage. If you only have time for one, Maester's broader library and scheduled runs give you more day to day, and it includes the SCuBA tests. If a customer, insurer or federal contract asks specifically for SCuBA alignment, produce it from ScubaGear, the tool CISA wrote for that purpose. Neither replaces the human work above, and neither looks at endpoints, backups or staff behavior; for those, the wider cybersecurity risk assessment applies.

How does NetSys use Maester and ScubaGear in a Microsoft 365 review?

Both tools are part of NetSys Microsoft 365 security reviews. We run them with a dedicated, documented identity, merge the results into one findings list, and interpret them with help from Claude: the model groups hundreds of results by control, separates failures from accepted exceptions and drafts remediation text, and a NetSys engineer verifies each item in the tenant before it enters the report. NetSys participates in Anthropic's Cyber Verification Program with Defense Access. Findings are delivered as a prioritized list with risk, effort, fix and owner, remediation runs in the sequence above with the client's approval, and the rerun reports are attached as evidence. The evidence we collect alongside the tool output is listed in our Microsoft 365 security audit checklist, and the nine settings most small tenants need first are in our Microsoft 365 security best practices checklist.

Frequently asked questions

Is CISA ScubaGear free?

Yes. The repository on GitHub is distributed under the Creative Commons Zero license, and the module installs from the PowerShell Gallery in a PowerShell 5 terminal on a Windows computer. CISA's project page describes it as a no-cost assessment tool available to private sector organizations as well as federal agencies.

Does ScubaGear change any settings in my tenant?

It is an assessment tool. It collects tenant settings through Microsoft 365 APIs and compares them against the SCuBA baselines with Open Policy Agent; it reports, and a person remediates. That is also why a written approval per change belongs in the workflow.

Can Maester run the CISA SCuBA tests?

Yes. Maester lists CISA SCuBA among its test sources, with 70 or more SCuBA tests alongside its CIS, EIDSCA, ORCA and community tests, so one scheduled Maester run covers SCuBA alignment and the broader benchmarks together.

What permissions do the tools need?

Both read tenant configuration, Maester through Microsoft Graph and ScubaGear through Microsoft 365 APIs. Grant the access to a dedicated account or app registration created for the assessment, write down what was granted and by whom, and remove it when you finish. Maester's support for Workload Identity Federation means a scheduled run in GitHub Actions or Azure DevOps can avoid a stored secret altogether.

How often should you run them?

Once for the baseline, again after each round of remediation to prove the fixes, and then on a schedule. Maester is built for scheduled runs, so monthly is practical. Run ScubaGear again whenever someone needs a current SCuBA alignment report.

Do these tools cover endpoints, backups or phishing?

No. They test the Microsoft 365 tenant's configuration: identity, Conditional Access, Exchange Online, SharePoint, Teams and the Power Platform. Device security, backup restores and how staff respond to a convincing email are separate parts of a security assessment.

NetSys engineers run Maester and ScubaGear in Microsoft 365 security reviews and validate each finding before it reaches you. Request a Microsoft 365 security review with a NetSys engineer.