Digital Forensics / Intermediate: artifact-specific offline analysis / Lesson 7

Normalize logs without losing record accounting

Preserve raw records, parser outcomes, source identity, and event semantics while building a comparison table.

About 4 min readContent reviewed 2026-10-28

On this page

Lesson goals

Separate syntax decoding from field validation and semantic mapping.
Account for accepted, rejected, and unresolved rows explicitly.

Understand the source before mapping its fields

A login event, an application action, and a collector receipt may all contain time and user fields while describing different observations. Build a source dictionary with event meaning, field types, timezone basis, identifiers, retention context, and known gaps. Do not normalize unrelated actor fields into one person column without evidence. A log is generated by a process with its own errors and visibility, not an infallible record of everything that happened. The classroom fixtures are deliberately small and created for the lesson; they do not establish actual logging coverage or incident completeness.

Keep syntax failure separate from field rejection

A line can be valid JSON yet miss an event identifier or use the wrong field type. The first program labels malformed syntax and missing fields separately while preserving line positions. It never silently drops a row. This is a narrow parser for a declared three-line fixture, not a general log-ingestion system. Real inputs need reviewed byte and nesting bounds, encoding policy, duplicate-key handling, and source-specific schema rules. Record the raw object or a restricted reference alongside the normalized row so an analyst can inspect how the interpretation was produced.

python Classify every line of a synthetic log fixture

import json
lines = ['{"event":"e1","kind":"write"}', '{"kind":"write"}', '{broken']
results = []
for number, raw in enumerate(lines, 1):
    try:
        value = json.loads(raw)
    except json.JSONDecodeError:
        results.append((number, "INVALID_JSON"))
        continue
    results.append((number, "ACCEPTED" if isinstance(value, dict) and isinstance(value.get("event"), str) else "MISSING_EVENT"))
assert results == [(1,"ACCEPTED"),(2,"MISSING_EVENT"),(3,"INVALID_JSON")]
print("three input rows accounted for")

Expected output

three input rows accounted for

The complete program parses fixed strings and retains all three outcomes. It reads no log file or service and performs no acquisition. The deliberate malformed string is handled inside the program and does not execute any content.

Normalize with explicit mappings and retained originals

A normalized table can use stable column names while retaining source-specific meaning. The second program maps a known application record into event_id, action, and source, leaving the original untouched. It does not infer a human identity or convert time. A source field named user can represent an account, service principal, or asserted value; preserve that distinction. Keep parser and mapping versions in the derivation record. If an unknown event kind appears, retain it with a documented unresolved category rather than forcing it into the nearest familiar action.

python Map fields without inventing identity

raw = {"event":"e7", "operation":"record-write", "actor":"service-lab"}
normalized = {"event_id":raw["event"], "action":raw["operation"], "source":"lab-application"}
assert normalized == {"event_id":"e7", "action":"record-write", "source":"lab-application"}
assert raw["actor"] == "service-lab"
assert "person" not in normalized
print("explicit source mapping preserved")

Expected output

explicit source mapping preserved

The complete program creates a derivative dictionary from one invented source record. It demonstrates the selected mapping only, with no claim about actual application field semantics, log authenticity, or actor attribution.

Reconcile input and output populations

Before interpreting event counts, reconcile total input rows with accepted, rejected, duplicate, and unresolved rows under the documented policy. Deduplication can be legitimate when the same delivery appears twice, but two distinct events with identical text must not be collapsed accidentally. Use source and event identifiers with their stated uniqueness scope. Keep rejected rows available through an approved restricted reference, and describe their potential effect on the question. A summary that hides five percent of input as parser noise can materially change a timeline or the apparent frequency of an action.

Verify the table before making incident claims

Compare selected normalized rows with the exact source fixtures and independently expected values. Include valid, malformed, missing-field, unknown-kind, and duplicate-delivery cases where relevant. Document what the parser did not support. A successful run means the program completed its stated checks; it does not establish that all real records were available or that an event did not occur. Preserve transformation notes, row accounting, and privacy limits in the report. This makes later correlation reproducible without presenting a synthetic log exercise as a real investigation.

Common errors and fixes

Malformed rows disappear silently

Why it happens: Parser failure is treated as irrelevant input.
How to fix it: Retain failure categories and reconcile every input row.
Prevent it next time: Test row accounting alongside field extraction.

Different user fields become one person

Why it happens: Normalization invents an identity relationship.
How to fix it: Preserve source meaning and require independent identity evidence.
Prevent it next time: Use a documented field dictionary.

Identical text is always deduplicated

Why it happens: Event identity and delivery repetition are confused.
How to fix it: Apply the documented identifier scope and preserve distinct events.
Prevent it next time: Review counts before and after deduplication.

Check your understanding

1. One hundred input rows yield ninety normalized rows. What must be explained?

Show worked solution

Account for the ten remaining rows by explicit categories such as syntax failure, unsupported kind, or duplicate delivery under the chosen policy. Preserve their references and assess whether omission limits the question. Do not report ninety total events merely because ninety rows survived parsing.

2. Why retain the source name in a normalized record?