MAMBA (NetBox Automation Tool) Design

DevOps
1

MAMBA: Netbox ETL pipeline design notes

*this post assumes at least a bit of familiarity with the NetBox UI

First: local repo organization

  • Repo should currently contains two folders: scripts and tests

  • Once this repo exists locally create a new folder at root named “data”

  • Inside the “data” folder create folders for each site (matching its existing site slug in NetBox)

  • Inside each “site” folder create a folder per facility located there (matching the facility name written in its location description in NetBox)

  • Add the data files (json & txt, csv) to each “facility” folder they pertain to

  • “data” folder and everything inside must only exist locally for privacy purposes → DO NOT PUSH ANY FILES LOCATED IN THE “data” FOLDER (regardless of whether they were added or generated)

  • Please ONLY push if:

    • Edits made to any of the .py files in the “scripts” folder

    • Edits made to field_mappings.json

In short, the pipeline is three scripts, plus an extra standalone tool:

  1. merge.py (extract & transform)

    1. Takes a facility directory with a json model and 0+ txt/csv files and merges all of the information into a single “merged.json”

    2. Handles three source formats

      1. Colon style .txt where one txt file equals one asset; e.g. name: leaf\nserial: 12345\n…

      2. Pipe style .txt where one txt file holds a different asset per row e.g. name|serial|…\nleaf|12345|…\nleaf2|67890|…

      3. Standard csv style holding a different asset per row e.g. name,serial,…\nleaf,12345,…\nleaf2,67890,…

    3. Matches txt/csv records to json model’s resources by id

    4. Auto-merges non-conflicting fields when more than one source matches the same resource; flags conflicts if existing

    5. Txt/csv file/row with no match to an existing resource id gets appended to “resources” as new entry

    6. Before merging check:

      1. Whether each device is associated with a rack (slot number preferred as well)

        1. Data will still merge without, but resources without a specified rack will fail to be created in netbox (updates are fine)

      2. A resource’s rack unit defaults to 1U; if the resource takes up multiple slots or only a fraction of a slot please specify by adding height (RU) into the resource information (ex. height (RU): 0.5)

        1. NOTE: NetBox does not allow rack units to be split vertically, only horizontally, so rack elevations will not be perfectly accurate, but close

    7. Usage: python scripts/merge.py <facility_dir_path> <facility_dir_path/merged.json>

  2. validate.py

    1. Reads “merged.json” and checks whether each resource actually has enough data to create a NetBox Device (device type, device role, site (sourced from facility_dir.parent.name), status (defaults to “active” unless txt file specifies otherwise)

      1. Required for device type: manufacturer (sourced from json model’s resource[productInfo][Manufacturer]), model (sourced from json model’s resource[productInfo][Model]), slug (auto-generated by script), height (defaults to 1.0 unless txt/csv file specifies otherwise)

      2. Required for device role: name (sourced from json model’s resource[roles]), slug (auto-generated by script), color (script groups by role name to assign colors e.g. servers→black, switches→blue, etc.)

    2. Writes to “validation_report.json”

    3. Organizational accuracy validated only, no live ipmi validation

    4. Usage: python scripts/validate.py <facility_dir_path> <facility_dir_path/merged.json>

  3. Standalone tool to be run at this point: customize.py

    1. Scans a “merged.json” file for field paths not already decided (in load.py or field_mappings.json) and prompts user to label path as device/interface/module tag, custom field, or skip

      1. Device/interface/module inferred by script from resource path

      2. Choose tag if value is same across many resources

      3. Choose custom field for unique values across many resources

      4. NOTE: any field names containing words like “prv”, “password”, etc. prompts an extra warning about including

    2. Decisions written to field_mappings.json (which load.py reads automatically from then on)

    3. So fields will only need to be decided once and apply globally not per facility

    4. Usage: python scripts/customize.py <facility_dir_path/merged.json>

  4. load.py (load)

    1. Reads “merged.json” and “validation_report.json” and idempotently creates/updates everything in NetBox

    2. Three different modes

      1. preview: build and print every payload without connecting to NetBox

      2. dry-run: connects to NetBox read-only to look up what already exists

      3. live: actually creates/updates objects in NetBox
        1. Also prompts user to enter optional “changelog_message” for bulk creates and updates per run

      4. *the full logs for each of these runs get written to json files in the facility directory, but the terminal will echo lines containing words like “failed”, “warning”, etc.

    3. Requires pynetbox [pip install pynetbox] & NETBOX_URL and NETBOX_TOKEN variables for API call

    4. Usage: python scripts/load.py <facility_dir_path> <facility_dir_path/merged.json> <facility_dir_path/validation_report.json> --mode [preview | dry-run | live]

Additional design decisions worth noting:

  • API token: use NetBox v2 API token (nbt_. format) – do not use v1 token

  • Manual enter: regions, sites, locations, and racks must be manually entered before running load.py; custom fields must be created before running —mode live

  • Rack facing: script auto groups nodes/servers front facing and switches rear facing, but only sets when rack slot is also specified

  • Cables: reverse-scans every resource’s “NICs.ports[].connector.element/index” to find which interface references a given cable id; cables without names are auto-assigned a unique name-index in the script; cable description includes cable’s source id, manufacturer, model if exists