Skip to main content

Air-Gapped Mode: Client-Side Anonymization

For organizations with strict Data Loss Prevention (DLP) policies, CloudWise provides client-side anonymization scripts that replace sensitive AWS identifiers with deterministic hashes before data leaves your network.

Data Retention by Tier
TierRetentionCross-Upload Correlation
Free / Shield / Agentic7 days❌ No (new salt each upload)
Compliance365 days✅ Yes (reuse your own client-held salt)

For quarterly trend analysis and audit reports, consider upgrading to Compliance tier.

Security notice: --anonymize before export script 1.14.0​

File names were not anonymized before version 1.14.0

If you ran the export script with --anonymize using version 1.13.0 or earlier, many file names inside the ZIP were not anonymized. They contained raw S3 bucket names, Kinesis stream names and EC2 instance IDs, and the names or IDs of Firehose streams, Step Functions state machines, Elastic Beanstalk environments, Glue jobs, EMR clusters, AppSync APIs, Transfer Family servers, Global Accelerators and AWS Backup plans. For a Lambda function, DynamoDB table or other resource whose name contains an underscore, some metric file names kept the part of the name before the first underscore (for example billing from billing_worker). The ZIP's manifest.json listed the same file names. SNS topic names, SQS queue names and CloudWatch dashboard names were also left raw inside the files, and account_info.json included the first four characters of your salt.

Version 1.14.0 anonymizes all of these. Each file name carries the same pseudonym as its record, so CloudWise still matches each metric to its resource. Version 1.14.0 also stops --anonymize from corrupting sizes of about 1 TB or more, which dropped those S3 buckets' size-based checks. Download the script again, verify its checksum, and re-export.

Earlier uploads. Where copies of an upload are kept:

  • The uploaded ZIP is deleted 30 days after upload (7 to 9 days on Free, Shield and Agentic), and its stored copy 7 days after that.
  • On Compliance, the upload's session record keeps the ZIP's file name (for example cloudwise-export.zip, not the names inside it) for 365 days.
  • If an upload failed, its error message is kept in the compliance audit log for 7 years, or until your account is deleted.
  • CloudWise's processing logs are kept for 60 days in production. Uploads made with 1.13.0 --anonymize may have logged raw resource names and IDs.

To delete a specific earlier upload, email support@cloudcostwise.io with the upload's date, or call DELETE /api/v1/offline/upload/{upload_id}. The endpoint is API-only; the app has no button for it yet. It removes only that upload's own data:

  • The upload disappears from the app at once.
  • Its findings, and the cost rows it still owns, are then expired in the background, on a best-effort basis. Cost rows that a later upload of the same period has replaced belong to that upload and are kept. If the background cleanup does not complete, the rows expire at their original retention time instead.
  • Its uploaded files are deleted once the upload's record has been removed.

Security notice: --anonymize before export script 1.15.0​

Some fields were not anonymized before version 1.15.0

Version 1.14.0 and earlier uploaded these values as they were, even with --anonymize:

  • free-text descriptions and comments;
  • CloudFront aliases, and origin IDs and custom header values;
  • EC2 key pair names and instance-profile names;
  • RDS, Redshift and Lightsail master usernames and database names (DBName);
  • AWS Backup plan names;
  • WorkSpaces computer names;
  • ECS cluster and service names, unless the export also held a CPU metric file for the cluster;
  • tag keys and values returned as a map rather than a list (MSK, EKS and API Gateway tags);
  • RDS Custom engine version names;
  • Elastic Beanstalk application environment variable names.

Those versions also anonymized some AWS values that name no customer, such as an EC2 instance's state (running, stopped). Checks that depend on those values, such as stopped instances with attached EBS volumes, found nothing on an anonymized export.

Version 1.15.0 anonymizes these fields and keeps AWS state and type values as they are. Download the script again, verify its checksum, and re-export. The retention and deletion steps in the notice above apply to earlier uploads.

Overview​

With client-side anonymization:

  • AWS Account IDs (12-digit numbers) are replaced with acct_xxxxxxxxxxxxxxxx format
  • Resource IDs (EC2 instances, EBS volumes, etc.) keep their AWS prefix and get a pseudonym, in i-xxxxxxxxxxxxxxxx format
  • Tag Keys (custom tag names) are replaced with tag_xxxxxxxxxxxxxxxx format by the export script (cloudwise-export.sh --anonymize). The standalone CUR script (cloudwise-anonymize-cur.py) does not rename tag keys, because they are column headers; it replaces only the tag values in those columns
  • File names inside the ZIP carry the same pseudonyms as the records they belong to (export script 1.14.0 and later)
  • Named fields such as key pair names, master usernames, database names and CloudFront aliases are pseudonymized, and free-text descriptions and comments are removed (export script 1.15.0 and later)
  • Fields CloudWise doesn't read are dropped, by field name (export script 1.29.0 and later). A field AWS adds later is dropped too, unless its name is one CloudWise reads
  • Strings in the fields that stay are replaced with a keyed hash, unless they are on the list below (export script 1.29.0 and later)
  • AWS values that name no customer stay as they are: region codes, and states, types, engines and instance classes (for example stopped, gp3, postgres, db.m5.large)
  • ARNs keep their structure. The account ID and the resource names the anonymizer recognizes are replaced, and other parts of an ARN can stay raw
  • Cost analysis remains accurate because hashing is deterministic
  • Resource identifiers are replaced before data leaves your network, with the known limitations below
Known limitations

--anonymize is not complete. It works from a list of field names, and we tested that list with stubbed AWS answers, not a real account. Some fields stay raw, so review the output before you upload it. Values uploaded as they are include:

  • What AWS defines and CloudWise needs to find waste: states, types, engines and versions, instance and volume classes, regions and zones, sizes, dates and costs
  • Some IDs AWS assigns, such as a hosted zone's ID and a WorkSpaces bundle ID
  • The word all on an RDS snapshot shared publicly
  • The tag keys Name, Environment, Owner, Team and CostCenter (their values are replaced, though short values such as true or 12 stay raw)
  • A field AWS adds later whose name is one CloudWise keeps raw, such as Type or Status
  • Names of one character

A resource name that looks like an AWS region code stays raw in its ARNs.

A resource name that is 1-2 characters long, is a number, or is exactly a word AWS uses (for example service or cost) is replaced where the anonymizer knows it is a name, such as its own name field. It can stay raw inside an ARN, such as notebook-instance/db-nb.

One description is kept: an EBS snapshot's description written by AWS, Created by CreateImage(i-...) for ami-.... CloudWise needs it to find snapshots of deregistered AMIs. The IDs in it are anonymized.

Lambda environment variables​

The export script replaces the value of every Lambda environment variable with REDACTED before it writes lambda_functions.json, with or without --anonymize (script 1.29.5 and later). No check reads them. With --anonymize the variable names are hashed too.

Before 1.29.5 the values were redacted only with --anonymize. An export made with an older script and without --anonymize included them as AWS returned them, and the upload keeps them until its retention ends: the uploaded ZIP and its stored copy are deleted on the schedule in Earlier uploads above. Download the script again, verify its checksum, and re-export.

Quick Start​

Option 1: Use the Export Script with --anonymize​

The CloudWise export script supports anonymization directly:

# Download the script and verify its checksum
curl -sLO https://cloudcostwise.io/scripts/cloudwise-export.sh
curl -sL  https://cloudcostwise.io/scripts/SHA256SUMS | grep cloudwise-export.sh | (sha256sum -c 2>/dev/null || shasum -a 256 -c)

# Run with anonymization enabled
bash cloudwise-export.sh --anonymize

# Or with a custom salt for reproducible hashes
bash cloudwise-export.sh --anonymize --salt "your-secret-salt"

Option 2: Anonymize an Existing CUR File​

If you already have a CUR CSV file, use the standalone anonymization script:

# Download the anonymization script and verify its checksum
curl -sLO https://cloudcostwise.io/scripts/cloudwise-anonymize-cur.py
curl -sL  https://cloudcostwise.io/scripts/SHA256SUMS | grep cloudwise-anonymize-cur.py | (sha256sum -c 2>/dev/null || shasum -a 256 -c)

# Anonymize your CUR file (requires Python 3)
python3 cloudwise-anonymize-cur.py your-cur-file.csv --output anonymized-cur.csv

# (Optional) Generate a mapping file for internal reference
python3 cloudwise-anonymize-cur.py your-cur-file.csv --output anonymized-cur.csv --output-mapping

Anonymization Details​

What Gets Anonymized​

Original FormatAnonymized FormatExample
AWS Account ID (12-digit)acct_[16-hex-hash]123456789012 → acct_457343e8e55b0c5a
EC2 Instance IDi-[16-hex-hash]i-1234567890abcdef0 → i-9239c50bfc649611
EBS Volume IDvol-[16-hex-hash]vol-0abc123def456789 → vol-e6077f6d18f21da6
Security Group IDsg-[16-hex-hash]sg-0123456789abcdef0 → sg-71fc81e0c57d6991
Tag Keys (custom), export script onlytag_[16-hex-hash]Project-Manhattan → tag_99622b5c0e4e9e52
ARN Account Componentacct_[16-hex-hash]arn:aws:ec2:us-east-1:123456789012:... → arn:aws:ec2:us-east-1:acct_457343e8e55b0c5a:...
Pseudonym Length Change (export script 1.11.0)

As of export script 1.11.0, pseudonyms use 16 hex characters of a keyed hash (up from 12) to reduce collision risk at scale.

How Hashing Works​

Anonymization uses keyed HMAC-SHA256, with a user-provided (or auto-generated) salt as the key:

hash = HMAC-SHA256(key=salt, message=original_value)
anonymized_id = prefix + first_16_hex_characters_of_hash

Because hashing is deterministic:

  • The same account ID always produces the same hash (with the same salt)
  • Cross-referencing between uploads works correctly
  • Cost aggregation by account remains accurate

Salt Management​

The salt is a secret value that makes your hashes unique:

  • Auto-generated salt: If you don't provide a salt, one is generated automatically
  • Custom salt: Use --salt "your-secret" for reproducible results across multiple exports
  • Keep your salt secure: Anyone with your salt and hash could theoretically brute-force original values
caution

If you lose your salt, you won't be able to correlate new uploads with previous anonymized data.

Upload Validation​

When you upload files to CloudWise, the upload wizard automatically detects whether your data is anonymized:

Anonymization Status Badges​

BadgeMeaning
🛡️ Anonymized Data (green)The file is marked anonymized, or the check matched only anonymized IDs. It doesn't check every field and some fields stay raw, so review the files before upload.
⚠️ Raw Data Detected (amber)File contains raw AWS identifiers. Consider anonymizing first.
🔶 Mixed Data (orange)File contains both raw and anonymized identifiers.
❓ Unable to detect data format (gray)Could not determine anonymization status.

Validation Step​

Before uploading, you'll see a validation step that shows:

  1. Detected mode (raw, anonymized, mixed)
  2. Sample identifiers found in your files
  3. Warning if raw data is detected
  4. Option to proceed or go back and anonymize

Mapping Files​

When anonymizing data, you can optionally generate a mapping file with the --output-mapping flag:

# Export script
./cloudwise-export.sh --anonymize --output-mapping

# CUR script
python3 cloudwise-anonymize-cur.py input.csv --output output.csv --output-mapping

Both write cloudwise-id-mapping.json, which maps each pseudonym back to its original value (other fields omitted here):

{
  "mappings": {
    "acct_457343e8e55b0c5a": "123456789012",
    "i-9239c50bfc649611": "i-1234567890abcdef0"
  }
}
warning

Never upload the mapping file to CloudWise. Keep it secure in your internal systems for reference only. The export script's mapping file also holds your full salt.

Best Practices​

For DLP Reviews​

  1. Always use --anonymize when exporting data for CloudWise
  2. Store your salt in a secure location (password manager, secrets manager)
  3. Review the output before uploading: check that IDs carry pseudonyms (acct_, i-, res_ and similar), and look for names the anonymizer missed
  4. Delete original files after anonymizing if required by your DLP policy

For Consistent Analysis​

  1. Use the same salt across all exports to maintain identifier consistency
  2. Keep a local mapping file for internal incident investigation
  3. Document your anonymization process for audit purposes

Troubleshooting​

"Raw Data Detected" Warning​

If you see this warning during upload:

  1. Go back to the upload step
  2. Remove the current files
  3. Re-export using --anonymize flag or run the anonymization script
  4. Upload the anonymized files

Hash Inconsistency Between Uploads​

If the same account shows different hashes:

  1. Verify you're using the same salt value
  2. Check that the salt wasn't accidentally modified
  3. Consider re-anonymizing all historical data with a new consistent salt

Performance Issues with Large Files​

For very large CUR files (>1GB):

# The script processes in streaming mode, but you can optimize:
# 1. Split your CUR file by month if possible
# 2. Run on a machine with more memory
# 3. Use the script directly (not piped through bash)
python3 cloudwise-anonymize-cur.py large-file.csv --output output.csv

FAQ​

Q: Why does my data expire after 7 days?

Air-Gapped Mode is designed for evaluation purposes. On Free, Shield and Agentic, Air-Gapped uploads expire after 7 days because the mode on those plans is meant for evaluation. The Compliance plan keeps them for 365 days. For ongoing automated monitoring, connect your AWS account to use Connected Mode (Free, Shield and Agentic; Compliance connects none by design); how much cost history it keeps depends on your plan.

Q: Can I extend the 7-day retention period?

Yes, on the Compliance plan: it keeps Air-Gapped uploads for 365 days and allows a longer upload history. On the other plans the retention is fixed. Connected Mode, on Free, Shield and Agentic, keeps cost history for a window that depends on your plan. The Compliance plan connects no AWS account: it works from Air-Gapped uploads only.

Q: Can CloudWise reverse the anonymization?

No. HMAC-SHA256 is a one-way keyed hash. Without your salt and original data, the anonymized values cannot be reversed.

Q: Do I lose any analysis capabilities with anonymized data?

Sometimes. Cost analysis, waste detection and recommendations all run on anonymized data, and findings show pseudonyms. A check that reads a value the anonymizer replaced or removed can find less: for example, a non-production check can't read the hashed value of an Environment tag.

Q: What if I need to investigate a specific resource?

Use your local mapping file to translate the anonymized ID back to the original AWS resource ID.

Q: Is the salt stored by CloudWise?

No. The salt never leaves your system.

Q: Are tag keys also anonymized?

Yes, by the export script as of version 1.8.0. Custom tag keys (like Project-Manhattan or Client-Acme) become tag_xxxxxxxxxxxxxxxx. Standard tag keys like Name, Environment, Owner, Team, and CostCenter are kept for usability. The standalone CUR script does not change tag keys, only tag values.

See Also​