Replace Line After Match with SED

In this post, I'm going to share with you a little bit of SED mastery that has come in extremely handy a number of times. If you need to replace a single instance of a line of text that is repeated throughout a file, this may be a solution for you! If the line before is unique, you can match on that and get sed to replace the correct line for you.

Just want the answer? Jump straight to it.

A couple of weeks ago I needed to solve a fairly simple configuration problem. As with things that seem simple on the surface, it was actually very hard to do in the way I wanted. I work as a DevOps Engineer so spend my day writing scripts and automating things. A lot of what I do uses terraform and often terraform written by vendors. My philosophy with this type of thing is to try not to modify vendor code unless absolutely necessary. The problem with modifying vendor code is that your changes might break when they make an update. Obviously changing configuration in vendor code is ok, but my preference is to script it as part of the deployment pipeline so changes can be applied to fresh, unmodified vendor code.

I've been working with a vendor for a number of months now to deploy a new solution using Kubernetes, but we've been running into a few issues where their baselines need to be tweaked for our environment. In most cases it's been fairly simple to target a particular value to be changed using YQ or JQ. If you've never used these tools, they are extremely useful and let you programmatically access and edit yaml and JSON. However, it gets a bit complicated when the value you need to change is very generic, used throughout the codebase and impossible to target using methods like yq. The particular changes that I needed to make were to some java parameters within a set value block for a helm chart inside terraform. The same terraform file had this generic line of java parameters in 6 places, none of which could easily be targatted using reliable methods. Unfortunately, it's impossible to target it with YQ and doing a simple find and replace with sed changes either the first instance it finds or all of them. No good here, I just need to change one instance of it. Luckily for me, the previous line on each of the matching lines were unique. After a wild afternoon of google-fu, I cobbled together an answer and had a little less hair.

Your mileage may vary depending on your OS and what commands your version of SED supports. This makes uses of the POSIX extension which is technically not valid SED. If your version of SED doesn't come with the POSIX extension you'll probably get an error. This does not work on my Mac, but a simple hack of mounting the file/folder into an alpine Linux container with docker gets round that. docker run -it -v $(pwd):/data alpine:latest /bin/sh

Let's take the following, simplified version of my problem:

set {
  name = "service-a"
  values = "--parameter1=true,--parameter2=false,--parameter3=1024mb"
}

set {
  name = "service-b"
  values = "--parameter1=true,--parameter2=false,--parameter3=1024mb"
}

set {
  name = "service-c"
  values = "--parameter1=true,--parameter2=false,--parameter3=1024mb"
}

How do you replace or update the parameters for service-b? Sed to the rescue. Sed is a "stream editor" that has some powerful text manipulation functionality. In the example above, the previous line for each instance of --parameter1=true,--parameter2=false,--parameter3=1024mb is unique, allowing us to use the previous line as a match.

Tip: Most SED examples pick the forward slash (/) as a delimiter. If your inputs contain this character, you can simply swap this out for another character you are not using. Not all characters are supported as delimiters but I'm sure you'll find something that works.

For this example we will assume the above snippet is stored in a file config.tf

sed -i -e "/name = \"service-b/{n;s/.*/values = \"replaced\"/}" config.tf

After running the above SED command the config.tf file should look like this:

set {
  name = "service-a"
  values = "--parameter1=true,--parameter2=false,--parameter3=1024mb"
}

set {
  name = "service-b"
  values = "replaced"
}

set {
  name = "service-c"
  values = "--parameter1=true,--parameter2=false,--parameter3=1024mb"
}

Let's take a look at what SED is doing.

The -i flag tells SED to do an inplace substitution on the file given to it.

The -e flag denotes an expression that SED should perform.

This is a fairly advanced looking expression, let's break it down into each part, seperated by delimiters:

In this example the / character is being used as the delimiter and separates each part of the expression.

Part one of the expression usually contains a command followed by a delimiter /. In the expression above, I have not included a command which tells SED just to perform a match.

Part two name = \"service-b is the pattern to match on and gives SED an "address" or location to perform the next part of the expression against. Notice that you need to escape certain characters such as double or single quotes as well as slashes.

Part three {n;s/.*/values = \"replaced\"/} is what to perform on the matched address from part two. This is surrounded by curly brackets { } which denotes a group. Using a group allows you to perform multiple sub-expressions against the matched address. Part three actually contains 2 sub-expressions, you separate different expressions within SED with a ;:

Sub-expression one: n this is the next command which ultimately means to go to the next line. This is a little more complicated than that and depends on whether you are using the -n flag or not. In this example we are not using -n so works as described.

Sub-expression two: s/.*/values = \"replaced\"/ consists of an entire expression with a command s which tells SED to perform a substitution. .* is the pattern to match and is a regular expression or regex. The . means to match any character and the * means 0 to many times, this will match the entire line. The third part of this expression values = \"replaced\" is the replacement value and will replace anything that was matched by the pattern (.*).

You can find the full SED manual here.

Hopefully this may give you some insight into some more advanced uses of SED and may just provide you with the answer I spent ages hunting for!