Writing Transforms in Python - Part 4

Full Series

Part 1 - The Default Transform & Working with Records
Part 2 - Reading from Records
Part 3 - Writing to Records
Part 4 - Removing Fields from a Record
Part 5 - Working with Strings in Python

Removing Fields from a Record

There are two main ways to remove a field from a Record… Or three, if we count simply clearing out the value.

Removing a Value using Indexing / Subscript notation

If we want to leave the field, but remove the value, we can simply overwrite it using square bracket syntax - The specific “empty” value we set it to may vary, depending on where you’ll eventually be using this data.

# Set AccountType's value to an empty string
record["AccountType"] = ""

# Set AccountType to None, i.e. NULL in terms of JSON, SQL Databases, etc... 
record["AccountType"] = None

Completely Removing a Field

If we want to completely remove a Field, along with the Value, we can use the del operator:

del record["AccountType"]

This will remove the “AccountType” key from the record, along with its value, which may be useful if we want to completely eliminate that field from the record as part of our Transform: If we don't want it going forward, or we're changing our record to match a new "schema" or format.

Removing a Field (while reading the value) using "pop"

Python also provides a “destructive read” method called pop , which can behave similarly to both square bracket syntax, and the “get” method, while also removing the key and value from a dictionary. This is especially convenient if you want to remove the value, but also use that value for something else in your Transform (for example, adding it back in under a different name...)

old_account_type = record.pop("AccountType")

It’s worth noting that with only a single argument, the field name, pop behaves like square bracket syntax - It will raise an Exception (causing an Error, and stopping your Transform) if the field is not available. This may be preferable, but if you would like to account for the field not being present, you can provide a default value as a second argument, similar to get . Let’s perform a simple “data migration” using all these features:

record["AccountNumber"] = record.pop("AccountType", "N/A")

This might be a little tricky to read at first: From left to right, we are doing an assignment to the "AccountNumber" field, but the value is actually being (destructively) read from the "AccountType" field, with a default value if it's not present. Let's put it all together, more in the order that it will actually happen in Python:

1. This code attempts to read the “AccountType” field from record
   1. But will default to “N/A”, instead of causing an Error, if “AccountType” is not presently available on our record...
2. It then writes this (possibly defaulted) value into a new “AccountNumber” field
   1. Creating it (or overwriting it, if it already exists!) on the record.

Again - You don't strictly need to use pop , and could accomplish the same thing using some of the techniques we've already covered, but it has some advantages:

• It's less code than reading, and then subsequently removing the value
  • (Less code for you to write, and maintain, and for others to read...)
• It makes your intention clearer: You are removing a key and value, with the implication that you're also reading it for later use...

Continue reading part 5 of this series here, where we dive into working with strings in Python.