Creating Upgradable Templates

Creating Upgrade and Downgrade Paths for Switching between Versions of an iApp Template

Introduction

One of the most powerful features of iApp templates is the ability to copy them, and make new versions with new capabilities. An existing Application Service can be re-parented to a new iApp template to take advantage of new features (re-parenting is simply clicking the change button on the template row while in the Reconfigure menu). However, if there are too many changes in the new template, it can become incompatible with the old template, and problems arise during re-parenting. Re-parenting works when the section and input names remain the same between versions. If these names change, the data associated with the changed inputs are lost upon re-parenting. In cases like this, the user is forced to re-enter information that existed in the original template after re-parenting. If the changes were severe enough, re-parenting is not possible at all, and the deployment must be rebuilt from scratch using the new template. In the past, this issue has been the source of considerable difficulty, but the F5 Product Management Engineering (PME) team has developed a way to solve these sorts of problems.
Beginning in BIG-IP version 11.4.0, the iApp templates that ship with the BIG-IP system will be able to upgrade an iApp application service from the version of the template on which it was created, to the new version; despite the fact that the new templates in 11.4 are significantly different. This document explains how this new re-parenting/upgrade process works, and how to accomplish it within the iApp framework.
In version 11.4.0, there will be an upgrade and downgrade procedure available on the BIG-IP system that helps users with this process. The upgrade procedure (and the data fed into it) enables an existing iApp service to work with a new version of the template. The downgrade procedure allows a path back to the legacy template, in case users are not satisfied with the new one. If you want to build a template that is able to perform these upgrades and downgrades, and you need it to work on version prior to 11.4.0, you must include the procedures in Appendix A and Appendix B in the template.

How does it work?

When a user submits an iApp template, the answers to all of the questions that are visible at the time of submission are stored in variables in an iApp Application Service Object (ASO). These variables are presented to the implementation section, and are also used to re-populate the fields when reentering a template. The variables inside an ASO should be familiar to developers using them in the implementation section. These are the variables that take the form

<section name>__<input name>
. In order to make the ASO that was deployed with the old version of a template compatible with the new version, the variable names need to be changed to map to the new section and input names. In many cases the answers need to be transformed as well. This is accomplished by defining all of the required transformations in array-based data structures which can be passed to the upgrade and downgrade procedures.
The upgrade procedure takes two arrays as parameters: the upgrade array and the translations array. The upgrade array provides mappings between the input fields from the old template to those of the new template. The upgrade array allows moving inputs between sections, or otherwise renaming them, and still preserves the data from the old template into the new one. For example, if the old template contains an input named “Foo” in a section named “Intro”, the upgrade array can be used to map that value to a different section in the new template.
The translation array creates translations for input values. This allows the values in drop-down lists to be changed between versions. Without the translation array, if there was a choice input in the old template with possible values of ‘No’ and ‘Yes’ and the possible values for that input were changed to something more descriptive in the new template, such as ‘enabled’ and ‘disabled’, there would be a problem after re-parenting. The value would be moved forward correctly, but the value stored in the ASO would be out of step with the input and with what the back-end code is expecting. The translations array allows translating the values for the inputs to their new values, so an answer of ‘Yes’ can be changed to ‘enabled’ during the upgrade and continues to work in the new version.
The downgrade procedure takes a single array as a parameter. The downgrade array defines which tables in the new template correspond to which tables in the older template. All of the values for the old template, except for the tables, are preserved in the ASO. This means that as long as the tables are reproduced from their corresponding table in the new template during downgrade, it is possible to get back to the old template.

Workflow

Upgrading a template requires two steps. First, there is one round of submitting the template to perform the upgrade of all of the ASO’s values to their new values. And then the user has to reenter the template and submit it again order to run the implementation section with the new values and change the configuration accordingly.

Re-Parenting vs. In-Place upgrades

The PME team investigated two different approaches to doing upgrades, called the Separate Template method and the Converged Template method. They both have advantages (described below), but in the end, the Converged template method was chosen because it was the cleaner of the two, especially considering the issues associated with users upgrading their BIG-IP systems from a previous version to the new one. Separate Template method
With the Separate Template method, both the old version and the new version of the template exist on the same box with different names. The new version of the template contains the upgrade capabilities. When a user wants to do an upgrade, they enter the new template, select upgrade, and then the user is able to select an iApp service from a from a drop-down list. When the user submits the template, the values are pulled from the selected ASO, run through the upgrade procedure, and stored in the new ASO. When the user reenters the template, they see it populated with the values from the old template. They can submit again and get their new configuration.
This method is the most straightforward, and makes a lot of sense for certain situations, but there are some real drawbacks. The most difficult is the fact that the old configuration still exists. The old ASO isn’t replaced; its values are just copied and translated into a new ASO. This means that the old ASO must be deleted or the virtual server IP addresses on one of the deployments must be changed. Otherwise, there would be a virtual server IP address conflict on submission. This lack of flexibility was ultimately the largest factor in the choice to use the more involved Converged Template method.

Converged Template method

With the Converged Template method, both the old version and the new version of the template exist in the same template, but only one is visible and accessible at any point in time. If a user starts a new deployment with the new template, they only see the new version of the template. However, if a user migrates an ASO to this new template (or they upgrade their BIG-IP version and get a new template attached to their ASO that way) then they will see the old version of the template UI when opening the template. The user is shown a message saying that the version they just opened has been deprecated, and they are offered the option of performing an upgrade. If the user doesn’t want to upgrade, they can continue to work with the deprecated template. If they choose upgrade, the ASO is upgraded to work with the new version of the template. When the user next enters the template, they see the new version of the template with the values carried over. To complete the upgrade and update the configuration, the user simply submits the template. The option to downgrade back to the old version of the template is always available, in case the new version does not meet the needs of the user.
This method is more work and makes for large templates; all of the code for both versions exists in the template, along with the upgrade and downgrade arrays. Also, some special coding is required to make it all work, especially for the template to auto-detect and show the correct version. Even so, this method enables a much easier to use and more flexible solution, and as previously mentioned, was the choice of the PME team. This is the method that is described in the remainder of this document.

Implementation

In the following sections, we describe how to combine the old and new versions of the template into a single template, how to make the auto-detection work correctly, and how to build the upgrade, downgrade, and translation arrays. The exact way you go about doing this depends very much on the nature of the templates and how extensive the changes are between versions.
The PME team uses the following steps to create converged, upgradable templates:
1. Develop and test the new version of the template in its own template file.
2. Analyze how to converge the old and new templates into a single file and do so
3. Write the upgrade, translation, and downgrade arrays.
4. Test the upgrade and downgrade paths

Building an Upgrade Array

The upgrade array maps values from one input to another. If the array is built using array set notation and two columns of data, the left column represents the old template and the right column represents the new template. The syntax of the right side is setting values in one of two arrays, vx and tx. The tx array is used for table elements and the vx array is used for everything else. The following sample array provides the upgrade path for three inputs in a section named basic.
array set upgrade_var {
    ::basic__addr                       {[set vx(vs_pool__vs_addr)          ##]}
    ::basic__need_snatpool              {[set vx(net__snat_type)            ##]\
                                         [set vx(net__snatpool)             ##]}
    ::basic__snatpool_members           {[set tx(net__snatpool_members)     ##]}
}

What this says is that the values from addr input in the basic section of the old template are moving to the vs_addr input in the vs_pool section of the new template. The need_snatpool input value is split into two different inputs. Both the snat_type and snatpool inputs in the net section will receive the value that came from the need_snatpool input in the old template. All of the values in the snatpool_members table in the basic section of the old template are mapped into the snatpool_members table in the net section of the new template.
The value ## is a convenience value. It gets translated into the value stored in the input in the left column. This is the most common value to use, but another value or input could be substituted. For example, the following example could be used in the right column.
[set vx(basic__legacy_advanced) [expr { \
   (![iapp::is ::basic__snat_1 {No}] || \
    ![iapp::is ::basic__snat_2 {No}] ) &&    \
     [iapp::is ::server_pools__lb_method_choice default {least-connections-member}] \
       ?no:yes}]]

This sets the ASO variable basic__legacy_advanced is set not to whatever value ## would have resolved to, but instead to the result of the expression. Left side variables require the leading :: while on the right side, this should be omitted.

Building a Translations Array

The translation array defines any changes that need to happen to choice drop-down inputs. The translation array contains two different kinds of translations: universal and specific. Universal translations happen anywhere that value is found. Specific translations only apply to a particular input.
The following translation array definition contains four universal translation and specific translations for the input snat in the section basic.
array set upgrade_trans [subst {
     Yes                   yes
     No                    no
     enabled               yes
     disabled              no
     basic__snat {
         Yes    {/#default#}
         No     {/#do_not_use#}
     }
}]

What this array says is that all answers of ‘Yes’ should be changed to an all lowercase ‘yes’ for any choice element in the template. Similarly, ‘No’ gets translated to ‘no’, ‘enabled’ gets translated to ‘yes’, and ‘disabled gets translated to ‘no’. The exception to those rules is that for just the variable basic__snat (the choice element named snat in the basic section) the value ‘Yes’ is translated to ‘/#default#’ and ‘No’ gets translated to ‘/#do_not_use#’.
This can be made more complicated than just a simple value change. TCL code can be inserted into this array to make more complicated choices. The following entry makes a choice during translation.
server_pools__create_new_pool {
    {Create New Pool} {/#create_new#}
    {Use Pool...}     [expr { [info exists ::server_pools__reuse_pool_name] \
                      ? $::server_pools__reuse_pool_name : {/#create_new#} }]
}

This says that for the create_new_pool element in the server_pools section, translate the string ‘Create New Pool’ to the string ‘/#create_new#’ and if the value is ‘Use Pool…’ then check if the ASO variable ::server_pools__reuse_pool_name exists. If it does exist then use its value, otherwise use the string ‘/#create_new#’.
Universal translations are simple string-to-string translations. If there are spaces in the strings, then they need to be enclosed in curly braces.
Specific translations are specified with a variable name with no preceding :: on the left side and a compound value on the right side containing two columns defining the translations.

Building a Downgrade Array

The downgrade array is the simplest of the three. Since most of the old values are still preserved in the ASO, it is only necessary to map the tables that need to be mapped back to the old template. The following sample translation array illustrates its use.
array set downgrade_tbl {
    ::vs_pool__members         server_pools__servers
    ::net__snatpool_members    basic__snatpool_members
}

This says that members table from the vs_pool section of the new template gets mapped back to the servers table in the server_pools section of the old template and the snatpool_members input from the net section of the new template gets mapped to the snatpool_members input in the basic section of the old template. Leading :: are required on the left side and should be omitted on the right side.

Converging the Templates

Building the upgrade and translation arrays is relatively straightforward. Making the old and new versions coexist in the same template and making the active-version auto-detection work is trickier to describe. There are several things that must happen for this to work correctly. All of the code for both templates, both presentation section and implementation section, must exist simultaneously in the template and there needs to be a way to detect which version to display. Every input from the old template must still exist in a section by the same name. You can have new sections that are just for the new version of the template and new inputs, but you have to keep all of the old ones, otherwise the upgrade cannot operate.
How separate old and new content needs to be depends on how extensive the changes are. In some cases one might have two completely different templates with different section names and no real overlap. In other cases, there might be some sections that are displayed only for the older template and some that are displayed only for the newer template, and some that are shared between them. The shared sections would be ones that have not changed between versions. On the back end, the code for both implementation sections needs to be included, with a check right at the beginning that decides which code is run.
Putting all of the code together into a single file is the easy part. The more difficult and subtle part detects which version of the template to display and process. It is necessary to detect from the APL code if this is a new deployment, a reentrancy into an ASO that was built using the new-style template, or if the ASO is from the old-style template. This is accomplished by usurping an input from the old template and making it perform double duty. This variable is called the pivot variable. The input in question needs to be a choice drop-down. It is much easier to handle the logic if it has only two values. The input needs to be at the top of the top-most section that has real inputs. If it is at least in the right section, it can be moved within the section without changing the ASO variable name. Sections can also be reordered if necessary. Reordering doesn’t change the ASO variable name.
For example, we use an old template with a first section like the following, where … represents other lines of inputs that are not relevant to this explanation.
section basic {
    
    
    choice snat default "No" { "No", "Yes" }
    
    
    
}

In this example, we use the snat input. We move it to the top, hide it from view, change the default, and then add two extra options to it, so it looks like this:
section basic {
    optional ( "HIDE" == "THIS" ) {
        choice snat default "no_legacy" {
            "Yes",
            "No",
            "legacy",
            "no_legacy"
         }
    }
    
    
    
}

If this template is entered for the first time, the value will be ‘no_legacy’ because that is the default. However, if this ASO came from the old template, this value will be either ‘Yes’ or ‘No’. If this value is either ‘Yes’ or ‘No’, then we know we should be displaying the old template. In that case, we want to provide the user with the option to upgrade using the following code:
section basic {
    optional ( "HIDE" == "THIS" ) {
        choice snat default "no_legacy"
    }

    # For old-style template
    optional ( snat == "Yes"
            || snat == "No" ) {
        message deprecated

        choice upgrade default "No" display "xxlarge"<Content for old version of this section of the template>
        …
    }

    # For new-style template
    optional ( snat == "legacy"
            || snat == "no_legacy" ) {
        …
        <Content for new version of this section of the template>
        …
    }

}

If the user selects upgrade and submits, the upgrade process on the back-end changes the value of basic__snat to legacy so that the state is preserved. On reentrancy, it will be obvious that the new template should be displayed, but that this was once an old-style template, so a downgrade options should be offered.
We have succeeded in detecting in which mode the template should be displayed and run, but in the process, we have lost access to the original value that was contained in this field. In order to save this value, we create a couple of new inputs, one for each possible value in the old version of the input. This is why we want to use a drop-down with as few options as possible. In this case, we only need two. If the original input had five possible values, then we would need five new inputs. Add something like the following to the part of this section that only shows up when the template is in legacy, old-style mode.
optional ( snat == "Yes" ) {
    choice snat_1 display "xxlarge" default "Yes" { "No", "Yes" }
}
optional ( snat == "No" ) {
    choice snat_2 display "xxlarge" default "No" { "No", "Yes" }
}

We present one optional for each possible value. Inside each, we place a new version of the old input with its own name. The default for this version should be the same as the value checked against in the optional. That way, the one that is displayed will come up with the right value. On the back-end, you will have to change the code to get this value from the right input.
Finally, wrap the rest of the section in optional based on the pivot variable so that old-style sections and new-style sections are only displayed when that it appropriate.
# For old-style template
optional ( snat == "Yes"
        || snat == "No" ) {
    <old-style section>
    <old-style section>
    <old-style section>
    …
}

# For new-style template
 optional ( snat == "legacy"
         || snat == "no_legacy" ) {
     <new-style section>
     <new-style section>
     …
 }

 <shared section>

Managing the Pivot Variable

There are a few final things needed to make this all work. During upgrade, we need to change the value of our pivot variable to ‘legacy’ so that the new template pages will be displayed on reentrancy. We also need to store the original value of the pivot variable somewhere in the ASO so that we can get it back if the user ever chooses to downgrade. This is stored in a variable named offload_history. We accomplish this using the upgrade array. Here is a sample entry in the upgrade array that handles these two tasks for the pivot variable in our example.
::basic__snat { \
    [set vx(offload_history) ##] \
    [set vx(basic__snat) "legacy"] \
}

Appendix A: The Upgrade procedure

The upgrade procedure takes two arrays as inputs. The parameter upgrade_var is the upgrade array described above and upgrade_trans is the translation array described above. There is no return value.
proc upgrade_template { upgrade_var upgrade_trans } {
    upvar $upgrade_var   upgrade_var_arr
    upvar $upgrade_trans upgrade_trans_arr

    # create the new variables from the old
    foreach { var } [array names upgrade_var_arr] {

        # substitute old variable name for abbreviation "##"
        regsub -all {##} $upgrade_var_arr($var) \$$var map_cmd

        # run the mapping command from inside the array
        if { [catch { subst $map_cmd } err] } {
            if { [string first "no such variable" $err] == -1 } {
                puts "ERROR $err"
            }
        }
    }

    # move variables over and apply translations
    set var_mods ""
    set var_adds ""
    foreach var [array names vx] {

        # if the APL variable name is in the translation array,
        # then use the custom translation built for that variable.
        if { [info exists upgrade_trans_arr($var)] } {
            array set sub_arr [subst $upgrade_trans_arr($var)]
            if { [info exists sub_arr($vx($var))] } {
                set vx($var) $sub_arr($vx($var))
            }
            array unset sub_arr
        # else, if the APL variable value is in the translation array,
        # then use the generic translation of that value.
        } elseif { [info exists upgrade_trans_arr($vx($var))] } {
            set vx($var) [subst $upgrade_trans_arr($vx($var))]
        }

        # add to tmsh command string
        if { [info exists ::$var] } {
            append var_mods "\n $var \{ value \"$vx($var)\" \} "
        } else {
            append var_adds "\n $var \{ value \"$vx($var)\" \} "
        }
    }

    # move tables over
    set tbl_mods ""
    set tbl_adds ""
    foreach tbl [array names tx] {

        # convert table from APL format to TMSH format
        if { ![llength $tx($tbl)] } {
            set tbl_def "column-names none"
        } else {
            set rows_def ""
            foreach apl_row $tx($tbl) {
                array set row_arr [join $apl_row]
                append rows_def "\n  \{ row \{ "
                foreach apl_col [array names row_arr] {
                    append rows_def "$row_arr($apl_col) "
                }
                append rows_def "\}\}"
            }
            set tbl_def \
            "\n  column-names \{ [array names row_arr] \} rows \{ $rows_def \}"
            array unset row_arr
        }

        # add to tmsh command string
        if { [info exists ::$tbl] } {
            append tbl_mods "\n $tbl \{ $tbl_def \} "
        } else {
            append tbl_adds "\n $tbl \{ $tbl_def \} "
        }
    }

    # construct the "tmsh modify" command
    set cmd "sys application service $tmsh::app_name "
    if { [llength $var_mods] } {
        append cmd "\nvariables modify { $var_mods }"
    }
    if { [llength $var_adds] } {
        append cmd "\nvariables add { $var_adds }"
    }
    if { [llength $tbl_mods] } {
        append cmd "\ntables modify { $tbl_mods }"
    }
    if { [llength $tbl_adds] } {
        append cmd "\ntables add { $tbl_adds }"
    }

    # Execute with debug output. This conversion takes place within the
    # existing ASO, so tmsh modify is used instead of tmsh create.
    debug "TEMPLATE UPGRADE"
    iapp::tmsh_modify $cmd
    return
}

Appendix B: The Downgrade procedure

The downgrade array takes three parameters. The first two, pivot_var and upgrade_var, are the variable names for the pivot variable and the upgrade choice. In our example above, those would be basic__snat and basic__upgrade, respectively. The third parameter is the downgrade array, described above. There is no return value.
proc ::iapp::downgrade_template { pivot_var upgrade_var downgrade_table } {
    upvar $downgrade_table downgrade_tbl_arr

    # The ASO variable "offload_history" is used to recover the legacy
    # choice a user made about SSL offload. It should be present in all cases.
    # This conditional only handles the case where a user has deliberately
    # deleted it by manipulating the ASO directly from tmsh.
    if { ![info exists ::offload_history] } {
        set ::offload_history "No"
    }

    # BIG-IP erases table contents when the APL optional hides the table.
    # Since the prior data is not available, this downgrade must back-convert
    # existing table data. Unlike tables, variables remaintact from the
    # legacy ASO.
    set tbl_def ""
    foreach tbl [array names downgrade_tbl_arr] {
        append tbl_def "$downgrade_tbl_arr($tbl) \{ "
        if { [llength [subst $$tbl]] } {
            set rows_def ""
            foreach apl_row [subst $$tbl] {
                array set row_arr [join $apl_row]
                append rows_def "\n  \{ row \{ "
                foreach apl_col [array names row_arr] {
                    append rows_def "$row_arr($apl_col) "
                }
                append rows_def "\}\}"
            }
            append tbl_def \
            "column-names \{ [array names row_arr] \} rows \{ $rows_def \}"
            array unset row_arr
        } else {
            append tbl_def "rows none"
        }
        append tbl_def " \} "
    }
    regsub -all {\n} $tbl_def {} tbl_def
    set cmd "sys app ser $tmsh::app_name \
        variables modify \{ \
            $pivot_var \{ value $::offload_history \} \
            $upgrade_var \{ value No \} \
        \} \
        tables modify \{ $tbl_def \}"
    debug "TEMPLATE DOWNGRADE"
    iapp::tmsh_modify $cmd
    return
}

Appendix C: Sample Arrays

array set upgrade_var_arr {
    ::basic__persist_on_source_ip { \
        [set vx(offload_history) ##] \
        [set vx(basic__persist_on_source_ip) "legacy"] \
        [set vx(basic__legacy_advanced) [expr { \
              (![iapp::is ::basic__persist_on_source_ip_1 {No}] || \
               ![iapp::is ::basic__persist_on_source_ip_2 {No}] ) &&    \
              [iapp::is ::server_pools__lb_method_choice default {least-connections-member}] \
              ?no:yes}]]}
    ::basic__persist_on_source_ip_1    {[set vx(vs_pool__persistence)       ##]}
    ::basic__persist_on_source_ip_2    {[set vx(vs_pool__persistence)       ##]}

    ::basic__addr                       {[set vx(vs_pool__vs_addr)          ##]}
    ::basic__port                       {[set vx(vs_pool__vs_port)          ##]}
    ::basic__protocol                   {[set vx(vs_pool__protocol)         ##]}
    ::basic__timeout                    {[set vx(vs_pool__timeout)          ##]}

    ::server_pools__create_new_pool     {[set vx(vs_pool__pool_to_use)      ##]}
    ::server_pools__lb_method_choice    {[set vx(vs_pool__lb_method)        ##]}
    ::server_pools__create_new_monitor  {[set vx(app_health__monitor)       ##]}
    ::server_pools__servers             {[set tx(vs_pool__members)          ##]}
    ::server_pools__monitor_type        {[set vx(app_health__monitor_type)  ##]}
    ::server_pools__monitor_interval    {[set vx(app_health__frequency)     ##]}
    ::server_pools__monitor_tcp_send    {[set vx(app_health__tcp_send)      ##]}
    ::server_pools__monitor_tcp_recv    {[set vx(app_health__tcp_recv)      ##]}
    ::server_pools__monitor_udp_send    {[set vx(app_health__udp_send)      ##]}
    ::server_pools__monitor_udp_recv    {[set vx(app_health__udp_recv)      ##]}
}

array set upgrade_trans_arr [subst {
    {Use Default Profile}   /#default#
     Yes                   yes
     No                    no
     enabled               yes
     disabled              no
     offload_history {
         Yes  Yes
         No   No
     }
     basic__persist_on_source_ip_1 {
         Yes    {/#default#}
         No     {/#do_not_use#}
     }
     basic__persist_on_source_ip_2 {
         Yes    {/#default#}
         No     {/#do_not_use#}
     }
     server_pools__create_new_pool {
         {Create New Pool}       {/#create_new#}
         {Use Pool...}           [expr { [info exists ::server_pools__reuse_pool_name] \
                                  ? $::server_pools__reuse_pool_name : {/#create_new#} }]
     }
     server_pools__create_new_monitor {
         {Create New Monitor}    {/#create_new#}
         {Use Monitor...}        [expr { [info exists ::server_pools__reuse_monitor_name] \
                                  ? $::server_pools__reuse_monitor_name : {/#create_new#} }]
     }
 }]

array set downgrade_tbl_arr {
    ::vs_pool__members         server_pools__servers
}

The BIG-IP API Reference documentation contains community-contributed content. F5 does not monitor or control community code contributions. We make no guarantees or warranties regarding the available code, and it may contain errors, defects, bugs, inaccuracies, or security vulnerabilities. Your access to and use of any code available in the BIG-IP API reference guides is solely at your own risk.