Prestaging

Stage 1: Pre-Staging Checklist

These are the necessary activities to prepare for the upgrade and perform a backup of dependencies and other essential platform tasks.

1. Click the links to view the showcase webinars, demo videos, and overview slides to get an understanding of the newest features and product improvements for IAP feature releases.

   Feature Release
   2020.2
   2021.1

2. Click these links to review all breaking changes in the Itential software that can impact the upgrade timeline for your target release.

   Breaking Changes
   Target Release 2020.1.x
   Target Release 2020.2.x
   Target Release 2021.1.x

3. Audit (review) all deprecation tasks prior to initiating the upgrade.

   Deprecations
   Target Release 2020.1.x
   Target Release 2020.2.x
   Target Release 2021.1.x

4. Audit (confirm) all dependencies meet the required minimum version. The latest dependency versions can be found in the Release Notes of the Itential documentation site.

   • Backup and copy all existing Mongo, Redis and RabbitMQ configuration files prior to upgrading dependencies.

     Dependency	Minimum Version
     Node	12.22.1
     Mongo	4.2.12
     Redis	5.0.12
     RabbitMQ	3.8.14
     RabbitMQ Erlang	22.3.4.2
     Python	3.6

   • Upgrade the dependencies to align with the required pre-requisites for the target IAP version.

5. Itential assumes you are using different type of environments. Follow this sequence when upgrading to a newer IAP version: DEV-UAT-PROD.

   • Upgrade DEV to continue with new development. Upgrade UAT for sanity checks and regressions of Itential apps and custom apps. The DEV and UAT upgrade will serve as a source of truth database.
   • As a pre-requisite for PROD upgrade, allow the IAP platform to soak for 24-hours after upgrading dependencies. Be sure to monitor Mongo, Redis, and RabbitMQ behavior during that time.

6. Upgrade Mongo and enable the compatibility settings.

   db.adminCommand( { setFeatureCompatibilityVersion: <version> } )

   db.adminCommand( { setFeatureCompatibilityVersion: "4.2" } )

7. Check Mongo compatibility.

   db.version()
   db.getCollection.(“system.version”).({}) - Mongo Compatibility Version Check

8. Perform a platform healthcheck.

   • Run a LOE script (report) to audit the workflows for Schema Validation Errors.
   • Next, run a repair script to fix all errors identified in the LOE report.
   • Then audit the deprecated tasks. These can be found in the Deprecations of the Itential documentation site.

9. Validate the indexes for jobs, tasks, and workflows from the Mongo CLI.

   • Connect to Mongo.

     mongo > use pronghorn
     pronghorn.db>

   • Run the following commands.

     db.jobs.getIndexes()
     db.tasks.getIndexes()
     db.workflows.getinsdexes()

   • Expect, at a minimum, 14 Jobs, 8 Tasks, and 4 Workflows.

   • If you do not have the expected indexing, run the following command in app-workflow_engine.

     npm run index

10. ⚠ Stop all scheduled jobs and capture the input and groups data from all Automation Catalog jobs.

11. ⚠ Before any major version upgrade, all jobs should be paused, canceled or completed before starting the upgrade process. For any jobs started on IAP 2019.3, or any version prior to 2019.3, it is important to ensure no tasks are left in a primed state. Tasks in the primed state are not supported with the migration script, so it is imperative that all jobs are paused and no primed tasks exist prior to the upgrade. Jobs with primed tasks will hang and the primed tasks will never run. Any tasks that are left on the system in a primed state must be removed manually.

12. Confirm the TaskWorker settings in workFlowEngine and the properties.json file.

    workFlowEngine  Service Config properties - “activate”: false

    properties.json file [ “processTasksonStart” : false ]

    • Stop TaskWorker and wait 30-minutes to complete the current running jobs in progress.
    • Stop IAP.

13. Perform a MongoDump and backup the Mongo Collections.

    • Run the following command.

      mongodump --host <MONGO HOSTNAME> --port=<MONGO PORT> --db=<PRONGHORN DB> --archive=<BACKUP LOC>/mongoDump<date>.bson

      or

      mongodump --authenticationDatabase admin -u admin -p admin --db pronghorn -o /tmp/mongo-backup --excludeCollection jobs --excludeCollection tasks --gzip

    • Backup and copy all Mongo Collections (Worfklows, Transformations, ac_agenda_job, Jobs, Tasks, Forms and Templates).

    • Confirm there is sufficient disk space in the target directory (/tmp).

14. Backup the pronghorn database.

    tar -czvf <BACKUP LOC>/iapBackup.tar.gz /opt/pronghorn/itential-<P/E>-release

15. Perform a NSO backup.

    ncs-backup

16. Download the IAP target version bin file from the Nexus repository.

17. Complete the following to bounce (soft restart) Redis, Rabbit MQ, and Mongo, and then audit the dependency vitals. This is done to clean-up logs history and the logging buffer.

    • Run this vital check curl command to capture the dependencies current status.

      curl -s http://<server>:port/status

    • Run the following to audit the Mongo Cluster and Replica status.

      sudo systemctl mongod stop

    • Wait 5 minutes and then start Mongo.

      sudo systemctl mongod start

    • Run the monitoring checkpoints to audit the Mongo Cluster.

      rs.status()
      rs.printSlaveReplicationInfo()

    • Bounce Redis, and then audit Sentinel and the Cluster status.

      sudo systemctl stop redis-sentinal.service

      sudo systemctl stop redis.service

    • Wait 5 minutes and then start the services.

      sudo systemctl start redis-sentinal.service

      sudo systemctl start redis.service

    • Audit the Sentinel status.

      redis-cli -h redis-XX -p 26379 info|grep status

    • Run this command on all Redis servers.

      redis-cli -h redis-XX -p 6379 info|grep role

    • Bounce RabbitMQ, and then audit the Cluster status.

      sudo service rabbitmq-server stop

    • Wait 5 minutes and then start the RabbitMQ server.

      sudo service rabbitmq-server start

    • Audit the RabbitMQ Cluster.

      sudo rabbitmqctl cluster_status

    • Run this vital check curl command after bounce to capture the current status of dependencies.

      curl -s http://<server>:port/status

18. Clean-up the deprecated (old) applications and adapters (as listed below) to prepare for a clean IAP start.

    Deprecated Applications
    @itential/app-smart_template
    @itential/app-object/
    @itential/app-json/pronghorn.json
    @itential/app-tools/
    @itential/app-string
    @itential/app-number
    @itential/app-golden_services
    @itential/app-golden_config
    @itential/app-array/

    Deprecated Adapters
    adapter @itential/adapter-mongo

    
    

    • From the Mongo CLI, remove the deprecated apps/adapter from service_configs.

      db.getCollection('service_configs').remove({"name" : "<app-name>"})

    • From the Mongo CLI, remove the deprecated apps/adapter from iap_profiles - profile1.

      db.getCollection('iap_profiles -profile1').remove({"name" : "<app-name>"})

    • From IAP, remove the deprecated apps/adapter from the @itential scope in pronghorn.

      /opt/pronghorn/current/node_modules/@itential
      rm -rf <app-name>

19. Audit the platform vitals for rogue processes, diskspace, CPU utilities, and memory.

20. Stop the cron jobs.

21. Archive the log files.

22. Confirm DB backup is complete.

Next Step

Go to Stage 2: Staging to download the target IAP release for the upgrade.

➤ Home Page