Cognos TM1 + Control version using Git

vladimir meniko -

In this post you will find information how to control versions of your Cognos Tm1 system using GIT. Why this can be usefull for you? Here are some reasons for that:

  • You need to trace changes that was made to the rules and processes throw the time. Especially it be can useful, if you work in command
  • You have two servers, and need to have track of what was changed in one server and transfer changes to another one.

To complite steps from this article you need to know how to do common Cognos TM1 administation tasks (create TI processes, chores, etc.) and have a little experience with GIT, as this topic does not provide this information.

How it will works

Process is really easy and I think that you can do it by yourself, but this article can provide you with scripts to start with.
First of all, you will set up TI process and batch script that will make snapshots of the system and store it whenever you want. Then PHP script will use this snapshots to create git repository.

Batch script

First of all you need to configure TM1 server to make daily snapshots of your rules and processes from Date directory. Here you need one Batch script. It will get two parameters as input: model name and path to servers data folder.

Code of batch script (copyRules.bat):

@ECHO OFF

rem Get input parameters
set ModelName=%~1
set PathToModelDataFolder=%~2

echo %ModelName%
echo %PathToModelDataFolder%

rem Define folder where we will store snapshots of rules 
set PathTarget=D:\TM1\Backup\Git\

rem Format date
set DateFolder=%Date:~10,4%-%Date:~7,2%-%Date:~4,2%
echo %DateFolder%

rem Define folders
set "setPathSourceFull=%PathToModelDataFolder%"
set "setTargetFolderDate=%PathTarget%\%DateFolder%\"
set "setTargetFolderFull=%PathTarget%\%DateFolder%\%ModelName%\"
set "setTargetFolderFullRux=%PathTarget%\%DateFolder%\%ModelName%\RUX\"
set "setTargetFolderFullPro=%PathTarget%\%DateFolder%\%ModelName%\PRO\"

rem Create folders
if not exist %setTargetFolderDate% mkdir "%setTargetFolderDate%"
if not exist %setTargetFolderFull% mkdir "%setTargetFolderFull%"
if not exist %setTargetFolderFullRux% mkdir "%setTargetFolderFullRux%"
if not exist %setTargetFolderFullPro% mkdir "%setTargetFolderFullPro%"

rem Copy only files with extension .rux and .pro
xcopy "%setPathSourceFull%*.RUX" %setTargetFolderFullRux% /y
xcopy "%setPathSourceFull%*.pro" %setTargetFolderFullPro% /y

TI Process

Now you need to create TI process, that will call Batch script. Name parameter is created from directory of your Cognos TM1 server and path to server is parsed from logs directory.

TI Process code:

# Names of folders
sFolderDataName = 'data\';
sFolderLogName = 'logs\';
sFolderTempName = 'tempfiles\';

# Folders
sFolderLog = GetProcessErrorFileDirectory;
sFolderServer = Delet(sFolderLog, Scan(sFolderLogName, sFolderLog), Long(sFolderLogName));

sFolderData = sFolderServer|sFolderDataName;
sFolderTemp = sFolderServer|sFolderTempName;

# Create name for model from folder server path
modelName = sFolderServer;
WHILE(Scan('\', modelName) <> 0);
	nPos = Scan('\', modelName);
	modelName = Delet(modelName, nPos, 1);
	if(Long(modelName) + 1 <> nPos);
		modelName = Insrt('_', modelName, nPos);
	endif;
END;
WHILE(Scan(':', modelName) <> 0);
	nPos = Scan(':', modelName);
	modelName = Delet(modelName, nPos, 1);
END;

modelName = LOWER(modelName);
pathToModelDataFolder = sFolderServer | sFolderDataName;

cmd = 'C:\TM1\Backup\Git\copyRules.bat "'|modelName|'" "'|pathToModelDataFolder|'"';

# Call batch program
ExecuteCommand(cmd, 1);

Chore

Create new chore, that starts created earlier process every day. I usually set up execution time for 11:50 PM.

Create GIT Repository

Now you will have such folder structure with snapshots:

Create empty folder and organize folders and files like this:

Put created snapshots in Import data folder. Create empty file, as it will be needed to create started point of your GIT.

PHP script will do all remaining job, it will initialise new git repository and add step-by-step all versions of rules to it.

This script was written in PHP. Start it using this command line: php program.php.

PHP Script

<?php

// First read folder, to define dates
$dateFolders = array();
if ($handle = opendir('./Import data')) {
    while (false !== ($dateFolder = readdir($handle))) {
        if($dateFolder == '.' || $dateFolder == '..' || $dateFolder == '.DS_Store'){
        	continue;
        }
        array_push($dateFolders, $dateFolder);
    }
    closedir($handle);
}

// check date folders
foreach ($dateFolders as $i => $dateFolder) {
	if(!preg_match('/(\d{4,4})-(\d{2,2})-(\d{2,2})/', $dateFolder, $res)){
		echo "ERROR: wrong date folder '$dateFolder'\n";
		exit();
	}
}

//sort folders
function sortDatesFunc($a, $b){
	preg_match('/(\d{4,4})-(\d{2,2})-(\d{2,2})/', $a, $resA);
	preg_match('/(\d{4,4})-(\d{2,2})-(\d{2,2})/', $b, $resB);

	if($resA[1] != $resB[1]){ // year
		// echo "year\n";
		return ($resA[1] < $resB[1]) ? -1 : 1;
	} else if($resA[3] != $resB[3]){ // month
		// echo "month\n";
		return ($resA[3] != $resB[3]) ? -1 : 1;
	} else if($resA[2] != $resB[2]){ // day
		// echo "day\n";
		return ($resA[2] != $resB[2]) ? -1 : 1;
	} else {
		echo "ERROR: same date folders\n";
		exit;
	}
}
usort($dateFolders, "sortDatesFunc");

// Process every date, find models
$cmd = 'rm -rf .git';
exec($cmd);
echo "✓ Remove previous git\n";

$cmd = 'git init';
exec($cmd);
echo "✓ Init new git\n";

$cmd = "rm -rf ./Data/*";
exec($cmd);

$cmd = 'git add * >/dev/null 2>&1; git commit -m "start" >/dev/null 2>&1';
exec($cmd);
echo "✓ Create start point in Git\n";

$cmd = 'git rev-list --max-parents=0 HEAD';
exec($cmd, $hashStart);
$hashStart = $hashStart[0];
echo "Start hash is: $hashStart\n";

$models = array();
foreach($dateFolders as $i => $date){
	echo "-- Date: $date\n";
	if ($handle = opendir("./Import data/$date")) {
	    while (false !== ($model = readdir($handle))) {
	        if($model == '.' || $model == '..' || $model == '.DS_Store'){
	        	continue;
	        }
	        $model = str_replace(' ', '_', $model); 

	        echo "-- -- Model: $model\n";

			// check if Branch exists throw hash
			if(array_search($model, $models) !== FALSE){
				echo "-- -- -- Branch exists, switch: $model\n";
				$cmd = "git checkout $model >/dev/null 2>&1";	
			} else {
				array_push($models, $model);
				echo "-- -- -- Branch created: $model\n";
				$cmd = "git checkout $hashStart >/dev/null 2>&1; git checkout -b $model >/dev/null 2>&1";
			}
			exec($cmd);

	        $cmd = "rm -rf ./Data/*";
	        exec($cmd);
			
			$path = str_replace(' ', '\ ', "./Import data/$date/$model/*"); 
	        $cmd = "cp -R $path './Data/'";
			exec($cmd);

			$cmd = "git add * >/dev/null 2>&1; git commit -m '$date' >/dev/null 2>&1";
			exec($cmd);
	    }
	    closedir($handle);
	}
}

Also you need .gitignore file, that will exclude some folders and files. I exclude system ruxes and process. You can change it any time of you need.

.gitignore file:

/Import data/
*.pl
.gitignore
commands.txt
}*.pro
}*.rux
/Other files

Compare versions

Thats all! Just do what you need with your new GIT of your Cognos TM1 server: