Little preface

When I lived in Russia I very often met a lot of censorship in internet – you could not access opositional websites, news sites, could not use some social media platforms (yes, instagram, facebook and linkedIn is blocked there). Later when I move to other countries, I understand that not only my country cleans up internet, others do the same (of course they do it differ, but anyway). So I started searching for some solution that would work at most of the time. And I find it – it is V2Ray.

As it is writen on the official website V2Ray is part of the Project V toolkit, that allows you to create your own private network that will work over internet. It is hard to be discovered and to be blocked even with DPI systems, that goverments use today for censorship. V2Ray is popular even in China, where the scale of censor beats all limits.

One of the great posibilies of V2ray is ability to masqarade your traffic as usual HTTPs. In such case your ISP provider will think that you just access this site. I use such configuration and will disclose installation process for you here. Also there are different ways to configure V2ray, you can read about them here.

Explanation

How does it works?

Here is little schema that disclose some common parts of the configuration that we will implement.

1. V2Ray client on your personal computer wraps all your internet traffic into HTTPs and sends it to your server. ISP (and all censorship middleware) does not see anything strange in your traffic as it looks like normal HTTPs traffic;
2. All encrypted trafic arrives to Nginx server on your VPS server. Nginx decrypt it and send it forward to V2Ray server;
3. V2Ray server checks authentification of traffic using internal mechanisms and sends it to global internet if everything is good.

Why not to use classic VPN protocols?

Of course you can use classic VPN protocols if they work in your situation. But because of there popularity goverments with there friendly technical companies invest a lot of money to find solutions to block them. For example:

1. WireGuard, OpenVPN can be blocked with DPI (and is blocked in Iran, Egypt, Uzbekistan and in many other countries). Of course there a lot of different bypassing methods to make it work – for example use additional obfuscation methods and so on. But it can be difficult to make it work sometimes or have some techinal limitations;
2. A lot of good solutions, such as Outline VPN are using UDP protocol, that can be blocked by some companies (it's strange to hear, but I've experienced it). And you need to find some other solution.

When you understand that, you will find out that having one more solution is nice opportunity.

What will you need?

To make V2Ray work as it is explained in this article, you will need:

1. Dedicated or virtual private server in some region where there is no censorship or it is weak (Germany, Netherlands, Latvia and so on). Personally I use Digitalocean, I like there price politics (it starts from 4$ per month and it is enough for V2Ray to work) and excellent level of service, but you can choose other service provider
2. Valid internet domain, that you can control and for which you can obtain SSL serficate
3. Some skills in administation of linux servers
4. Time to deploy and configure everything
   In case if you have not got account in DigitalOcean, here are my referal link. If you will sign up with it you will get up to 200$ bonus for your first server and will support me as author. Thanks!

Installation guide

Step 1. Initial setups

First of all create VPS server and obtain SSH access to it. For better security I recommend to secure your service creating non-root user and enabling ufw firewall. You can view how to do it in this instruction. Just do not forget to open 443 port for our configuration of v2ray.

Step 2. Installing v2ray

3. After all initial setup, we need to load and install v2ray on our server

$ bash <(curl -L https://raw.githubusercontent.com/v2fly/fhs-install-v2ray/master/install-release.sh)

4. Now we can enable v2ray and check does it works or not (status of server must be active)

$ sudo systemctl enable v2ray
$ sudo systemctl start v2ray
$ sudo systemctl status v2ray
● v2ray.service - V2Ray Service
     Loaded: loaded (/etc/systemd/system/v2ray.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/v2ray.service.d
             └─10-donot_touch_single_conf.conf
     Active: active (running) since Tue 2022-11-29 09:32:50 UTC; 4s ago
       Docs: https://www.v2fly.org/
   Main PID: 2244 (v2ray)
      Tasks: 6 (limit: 512)
     Memory: 8.2M
        CPU: 29ms
     CGroup: /system.slice/v2ray.service
             └─2244 /usr/local/bin/v2ray run -config /usr/local/etc/v2ray/config.json

As you can see service was successfully started and is using configuration file that is located at /usr/local/etc/v2ray/config.json. We will return to this later.

Step 3. Setting up domain DNS records

5. Next step is to create A-type DNS record for your domain, so that our server can receive trafic that is going to it. I use third level domain for this bypass.meniko.ru (but it can be anything you like). After you do this, you need to wait until new DNS record would be global wide available, it can take from 30 minutes to hours, need to wait a bit.

Step 4. Creating SSL sertificate

6. When DNS records was successfully updated and we can ping our new domain, we can go further and setup HTTPs sertificate with LetsEncrypt and CertBot and setup our Nginx server on our VPS server. First install everything with this:

$ sudo apt-get install nginx -y
$ sudo apt-get install certbot python3-certbot-nginx

7. Now is the time to get SSL sertificate using CertBot, it will ask for your email address, agree with terms and tell it domain address that we created before

$ sudo certbot --nginx
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): ***@meniko.ru

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: n
Account registered.
Please enter the domain name(s) you would like on your certificate (comma and/or
space separated) (Enter 'c' to cancel): bypass.meniko.ru
Requesting a certificate for bypass.meniko.ru

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/bypass.meniko.ru/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/bypass.meniko.ru/privkey.pem
This certificate expires on 2023-02-27.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

Deploying certificate
Successfully deployed certificate for bypass.meniko.ru to /etc/nginx/sites-enabled/default
Congratulations! You have successfully enabled HTTPS on https://bypass.meniko.ru

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

8. We can go to our domain and check that connection is now secure (CertBot automatically updates Nginx config for our domain)

Step 5. Configure Nginx

10. Great lets configure our Nginx. All its configs are located in the folder /etc/nginx/sites-available. If you are making changes on new server, default config file for your new domain will be called default. In this case open this config file and replace it content with this lines. Do not forget to change my domain bypass.meniko.ru to yours.

server {
  listen 80;
  server_name bypass.meniko.ru;
  return 301 https://bypass.meniko.ru$request_uri;
}

server {
  listen 443 ssl;
  server_name bypass.meniko.ru;

  ssl_certificate /etc/letsencrypt/live/bypass.meniko.ru/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/bypass.meniko.ru/privkey.pem;

  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  location / {
    root /var/www/html;
    index index.html index.htm index.nginx-debian.html;
  }
  
  location /v2ray {
    if ($http_upgrade != "websocket") {
        return 404;
    }
    proxy_redirect off;
    proxy_pass http://127.0.0.1:10000;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    
    # Show real IP in v2ray access.log
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }    
}

10. Time to check is everything ok with our config and restart the server. Also check that status of nginx is active.

$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
$ sudo systemctl restart nginx
$ sudo systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: active (running) since Tue 2022-11-29 11:40:44 UTC; 4min 25s ago
       Docs: man:nginx(8)
    Process: 4190 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 4191 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
   Main PID: 4192 (nginx)
      Tasks: 2 (limit: 512)
     Memory: 3.3M
        CPU: 49ms
     CGroup: /system.slice/nginx.service
             ├─4192 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
             └─4193 "nginx: worker process"

Configure V2Ray server

11. Now we need to update config file for our V2ray server that we installed previously. Default V2ray config is located at /usr/local/etc/v2ray/config.json, open it and insert this lines.

    You need change id of client (you can use this service to generating new one). Also note path /v2ray, you can change it, but do not forget to change nginx and clients config respectively. For educational purposes I recommend to leave it as is.

    As you can see here we generate 1 inbound and 1 outbound routing. In these example we use vmess protocol for extra encryption of data. For comunication beetween client and server we will use websockets. As I said previously there are other options, you can read about them on official site

{
  "inbounds": [
    {
      "port": 10000,
      "listen":"127.0.0.1", 
      "protocol": "vmess",
      "settings": {
        "clients": [
          {
            "id": "95880cb0-0b23-4e87-9042-c8a1bc4e5c0f",  
            "alterId": 0
          }
        ]
      },
      "streamSettings": {
        "network": "ws",
        "wsSettings": {
        "path": "/v2ray"
        }
      }
    }
  ],
  "outbounds": [
    {
      "protocol": "freedom",
      "settings": {}
    }
  ]
}

12. Now we can restart V2ray service and check is it works (as usual status must be active):

$ sudo systemctl restart v2ray
$ sudo systemctl status v2ray
● v2ray.service - V2Ray Service
     Loaded: loaded (/etc/systemd/system/v2ray.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/v2ray.service.d
             └─10-donot_touch_single_conf.conf
     Active: active (running) since Tue 2022-11-29 11:57:13 UTC; 4s ago
       Docs: https://www.v2fly.org/
   Main PID: 4346 (v2ray)
      Tasks: 5 (limit: 512)
     Memory: 14.1M
        CPU: 29ms
     CGroup: /system.slice/v2ray.service
             └─4346 /usr/local/bin/v2ray run -config /usr/local/etc/v2ray/config.json

We successfully configured our server side, now time to setup our client.

Configure our V2Ray client

In this part of article I will show you how to configure clients on Windows (GUI client) and Mac OS (CLI client) machines respectively. Also I would like to note that there are a lot of different clients, you can choose from. You can find full list here at official site.

Before we begin, here is client configuration file that would be used in both configurations for Windows and Mac OS. You will need to save it with extension .json.

{
  "inbounds": [
    {
      "port": 1080,
      "listen": "127.0.0.1",
      "protocol": "socks",
      "sniffing": {
        "enabled": true,
        "destOverride": ["http", "tls"]
      },
      "settings": {
        "auth": "noauth",
        "udp": false
      }
    }
  ],
  "outbounds": [
    {
      "protocol": "vmess",
      "settings": {
        "vnext": [
          {
            "address": "bypass.meniko.ru",
            "port": 443,
            "users": [
              {
                "id": "95880cb0-0b23-4e87-9042-c8a1bc4e5c0f",
                "alterId": 0
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "ws",
        "security": "tls",
        "wsSettings": {
          "path": "/v2ray"
        }
      }
    }
  ]
}

Configuration of Qv2ray GUI client for Windows

As I said previously there are a lot of different options to choose as v2ray client for Windows. I found that Qv2ray is one of the most intresting solutions for this, as it is popular and have good localization. Here I will show you how to configure it, but you can choose any other that you like the most.

1. First of all you will need to download v2ray core from official Github account. https://github.com/v2ray/v2ray-core/releases

2. Unpack it and place whenever you want on your Windows computer;

3. Next download Qv2ray from this page. At the moment of writing this article there was 2.7.0 version and I download Qv2ray-v2.7.0-Windows-Installer.exe

4. Install Qv2Ray that was downloaded on previous step

5. Open installed Qv2Ray client

6. Now you need to define where your v2core is located (that we downloaded on first two steps). In Qv2Ray open Preferences menu and go to Kernel settings and define paths as shown on image

7. Now it’s time to add our client v2ray config to Qv2ray. Create v2ray.json file whenever you want and copy client config to it (you can find it little a bit above). Do not forget to change address and id to match your configurations). Lastly add it to Qv2ray via + Import → Advanced menu as shown on image

8. If you do everything correctly you can connect to your server and check does your IP address was changed to new one

Now you can use your v2ray and access any site that is available from network where your server is located. Congratulations!

Configuration for Mac OS with CLI

There are a lot of good GUI software for Mac OS as well. But here I would not go this way, we will use v2ray in CLI mode and configure everything manually.

1. To install V2Ray you can use brew package manager (if you have not use it before, it’s time for it; you can find instructions on official site)

$ brew install v2ray

2. If installation was successfull, this command should return version of V2ray:

$ v2ray -version
V2Ray 4.45.2 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.17.11 darwin/amd64)
A unified platform for anti-censorship.

3. Now we need to create configuration file for our client, create file config.json in any directory and insert lines of client configuration that was shown before (change address and id to match your configurations)

4. Now it’s time to start your v2ray client. Just call v2ray service and define path to config file with in -config parameter

$ v2ray -config config.json
V2Ray 4.45.2 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.17.11 darwin/amd64)
A unified platform for anti-censorship.
2022/12/01 11:28:24 [Info] main/jsonem: Reading config: config.json
2022/12/01 11:28:24 [Warning] V2Ray 4.45.2 started

5. V2ray is now working and has started Proxy5 server on 127.0.0.1:1080 to which you need send traffic of your computer

6. Define this proxy server in Mac OS settings. Go to Preferences → Network → Choose your active interface and go to advanced options → Open Proxies tab → SOCKS Proxy and add Socks Proxy server address and port as show on image

7. Now everything should work and you can go and check whenever your address was changed. We’re done!

I hope that this article was helpful for you and you succesfully configure your own v2ray server. This configuration must allow you to unlock internet even in most difficult and restricted situations – such as blocked ports on your local network, resticted UDP packages and many other.

For better security you also need to think about hiding your DNS requests using DNS over HTTPs or some other technics. But this is not topic of this acticle.

If you have any questions or comments leave them below.

In this post you will find two easy bash scripts that can help you in process of generation multiple certificates in WireGuard VPN service. I create it to reduce time of certificate generation for myself, but it can be useful for you, as I could not find such solution with Google.

To use these scripts you need to have configured WireGuard service and access to wg tool.

Next create folder and create bash file, give him executable rights and put this code in it. Change PublicKey property of your servers WireGuard config. Create folder serts in the same folder.

#!/bin/bash

file="publickey";
[ -e $file ] && rm $file;
file="privatekey";
[ -e $file ] && rm $file; 

wg genkey | tee privatekey | wg pubkey > publickey

privatekey=`cat privatekey`
publickey=`cat publickey`

echo Print name of sert with lowcases;
read name;

echo Print last ip number;
read iplast;

echo $name" "$iplast;

wg set wg0 peer $publickey allowed-ips 10.0.0.$iplast

#print to sert
sertname=$name".meniko.vpn.conf";
folder='./serts';

echo "[Interface]" >> $folder"/"$sertname
echo "PrivateKey = "$privatekey >> $folder"/"$sertname
echo "Address = 10.0.0."$iplast"/24" >> $folder"/"$sertname
echo "DNS = 1.1.1.1, 8.8.8.8" >> $folder"/"$sertname

echo "[Peer]" >> $folder"/"$sertname
echo "PublicKey = [public key of server]" >> $folder"/"$sertname
echo "AllowedIPs = 0.0.0.0/0" >> $folder"/"$sertname
echo "Endpoint = vpn.meniko.ru:51820" >> $folder"/"$sertname

When you executes this bash script, you will be prompted for name and ip address of your user. After that script will create certificate for you and register it in WireGuard. Execute it with admin rights.

Next script can do restoring work, if something goes wrong, it reads every cert and resets it in WireGuard.

#!/bin/bash

for file in ./serts/*
do
    echo "----"
    echo $file

    private=''
    public=''
    iplast=''


    while IFS= read -r line
    do
        # find private key
        pat="^PrivateKey = (.+)$"
        if [[ $line =~ $pat ]];
        then
            echo $line
            private="${BASH_REMATCH[1]}"
        fi

        # find ip adress
        pat2="^Address = 10\.0\.0\.(.+)\/24$"
        if [[ $line =~ $pat2 ]];
        then
            echo $line
            iplast="${BASH_REMATCH[1]}"
        fi
       echo $line
    done < "$file"

    echo $private
    public=`echo $private | wg pubkey`
    echo $public
    echo $iplast

    wg set wg0 peer $public allowed-ips 10.0.0.$iplast
done

In this post I would like to share all my favourite apps for Mac OS, which I use in my life and which I would like to recommend to Mac users.

Productivity

Notion (freemium model) – app that allows to keep notes, run projects, create knowledge databases in one place and share what you need with team or world. Previously I use Evernote, but Notion is much more productive focus ready app.

Audio Hijack (71$) – app that allows capture audio from any app on your mac, I use it to record audio from conferences.

Typora (free) – one of best markdown editors I have tried. I use it for writing all my posts and documents in MarkDown

Entertainment

Beamer (19,99$) – app that allows streaming your video files directly from your Mac to Apple TV or Chromecast. It supports all common video formats (mkv, mov and other).

IINA (open source) – one of the best video players for mac os with large support of different video formats.

Security

Little Snitch (47,25$) – app that allow to monitor apps network activity and helps to organise it.

Enpass (2,40$ per month) – password manager to keep your passwords organised and secure.

Programming

Postman (free for personal use) – one of best tools to test API for your projects.

Other

iStat Menus (11,99$) – app helps you to monitor your system health and has a lot of useful features.

VMWare Fusion (150$) – app for virtualisation operation systems on Mac OS. Previously I use VirtualBox, but it was buggy and slow. VMWare is much much better.

MTMR (open source) – app for customisation touch bar, you can use existing themes or create custom buttons and blocks.

BetterTouchTool (9$) – app to create custom shortcuts for keyboard, TrackPad and Apple MagicMouse and a lot more.

belenaEtcher (open source) – automatic tool to flash OS to external hard drives.

HandBrake (open source) – tool for converting videos from one format to another, I usually use it to compress videos from Adobe Premiere.

Karabiner-Elements (open source) – A powerful app for keyboard customising. I use it for customising keys on external keyboards.

OBS (open source) – software for capturing and steaming videos from mac os, gamers use it for streaming, but I usually use it for capturing video of mac os videos together with Audio Hijack (see above).

This article with describe how to convert PDF file to PDF consisting only from images. This operation permits future editing PDF using such converters as iLovePDF.

To do this you need to install imagemagick from there official site, or install via package managers. For Mac OS you can use brew for it:

brew install imagemagick

Next you can execute next command, that will convert PDF to PNG images

convert -density 1200 name_of_pdf.pdf -resample 300 -scale 4096x4096 page.png

Next you need convert PNG images from first command to PDF, using:

convert *.png -scale 4096x4096 output.pdf

This two commands can take a time to execute, depending from the size of your PDF file.

Also you can create alias in bash, adding these command to yours ~/.zshrc (or another shell)

pdftopng(){
   mkdir -p topng
   convert -density 1200 $1 -resample 300 -scale 4096x4096 topng/page.png
   convert topng/*.png -scale 4096x4096 comp.$1
}

Don't forget to do source ~/.zshrc.

This post describes method how to automate process of compressing PDF files in Mac or Linux environments using GhostScript library.

First of all you need to install GhostScript library on your computer. On mac os you can do it with brew:

brew install ghostscript

Or using apt on Linux systems

sudo apt update
sudo apt install ghostscript

Now you can do converting using gs command in terminal. There are a lot of options, but here I would give you my solution.

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/default \
   -dNOPAUSE -dQUIET -dBATCH -dDetectDuplicateImages \
   -dCompressFonts=true -r150 -sOutputFile=output.pdf input.pdf

If you would like to make shortcut in your terminal to start this commands quicker, you can add this code to your ~/.zshrc file.

nano ~/.zshrc

pdfcompress ()
{
   mkdir -p compressed
        gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/default \
                -dNOPAUSE -dQUIET -dBATCH -dDetectDuplicateImages \
                -dCompressFonts=true -r150 -sOutputFile=compressed/$1 $1
}
pdfcompressall (){
   for i in $(ls | grep .pdf); do pdfcompress "$i"; done
}

This will allow you call GhostScript command as pdfcompress name_of_pdf.pdf to compress one pdf file, or using pdfcompressall to compress all pdf files in current folder.

Some systems exports data in table format with multiheader structure, that looks something like this.

If you need to use this data in some of your applications you can manually merge headers, but if this file is updated frequently these can be tediously. In this case I can offer you JS script that will do this work for you.

At the end of itexecution you will get csv like this. As you can see all headers are grouped and merged into one level:

Or JSON like this:

Script uses two packages: XLSX, json2csv. To install packages use:

npm install --save json2csv xlsx

Function that makes all work is:

const XLSX = require('xlsx');
const fs = require('fs');
const util = require('util');
const {Parser} = require('json2csv');

const xlxsToCSVSync = (file, fileOutput, skipTopLines) => {
	const letterToNumber = (val) => {
		var base = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', i, j, result = 0;
		for (i = 0, j = val.length - 1; i < val.length; i += 1, j -= 1) {
			result += Math.pow(base.length, j) * (base.indexOf(val[i]) + 1);
		}
		return result;
	};

	const cellParsing = (cellName) => {
		let match = cellName.match(/([A-Za-z]+)(\d+)/);
		if(!match) return false;
		return {c: letterToNumber(match[1])-1, r: match[2]-1}
	}

	if(!fs.existsSync(file)){
		throw `File ${file} does not exists`
	}

	const xlsxData = fs.readFileSync(file);
	const workbook = XLSX.read(xlsxData, {type:'buffer'});
	
	if(Object.keys(workbook.Sheets).length > 1) throw `Excel contains more than one page`;
	let sheet = Object.keys(workbook.Sheets)[0];

	// define header height
	let headerHeight = 1
	let headerLeftTop, headerRightBottom;

	for(let cell in workbook.Sheets[sheet]){
		let coor = cellParsing(cell);
		if(!coor) continue;
		if(coor.r < skipTopLines) continue;
		if(coor.r >= (skipTopLines + headerHeight)) break;

		let findMerge = workbook.Sheets[sheet]['!merges'].find(x => x.s.c == coor.c && x.s.r == coor.r);
		if(findMerge){
			let width = findMerge.e.c - findMerge.s.c + 1
			let height = findMerge.e.r - findMerge.s.r + 1
			if(height > 1){
				headerHeight = coor.r - skipTopLines + height
			}
		}

		if(typeof headerLeftTop == 'undefined' || (coor.c < headerLeftTop.coor.c && coor.r < headerLeftTop.coor.r)) headerLeftTop = {cell, coor};
		if(typeof headerRightBottom == 'undefined' || (coor.c > headerLeftTop.coor.c && coor.r > headerLeftTop.coor.r)) headerRightBottom = {cell, coor};
	}

	// define header cells
	for(let cell in workbook.Sheets[sheet]){
		let coor = cellParsing(cell);
		if(!coor) continue;
		if(coor.r < skipTopLines) continue;
		if(coor.r >= (skipTopLines + headerHeight)) break;

		let findMerge = workbook.Sheets[sheet]['!merges'].find(x => x.s.c == coor.c && x.s.r == coor.r);
		if(findMerge){
			let width = findMerge.e.c - findMerge.s.c + 1
			let height = findMerge.e.r - findMerge.s.r + 1
		}
	}

	console.log(`Header height: ${headerHeight}`);

	// header define
	let headers = []
	const headerParser = (topLevel, line, leftOffset, widthParent) => {
		if(typeof leftOffset == 'undefined') leftOffset = 0;
		if(typeof widthParent == 'undefined') widthParent = 0;
		// let childLeftOffset = leftOffset;

		for (let cell in workbook.Sheets[sheet]){
			let coor = cellParsing(cell);
			if(!coor) continue;

			// row filter
			if(coor.r < skipTopLines) continue;
			if((coor.r + 1) != (skipTopLines + topLevel)) continue;
			if(coor.r >= (skipTopLines + headerHeight)) break;

			// column filter
			if(coor.c < leftOffset) continue;
			if(widthParent > 0 && (coor.c + 1) > (leftOffset + widthParent)) break;

			let height, width;
			let findMerge = workbook.Sheets[sheet]['!merges'].find(x => x.s.c == coor.c && x.s.r == coor.r);
			if(findMerge){
				width = findMerge.e.c - findMerge.s.c + 1
				height = findMerge.e.r - findMerge.s.r + 1
			} else {
				height = 1
				width = 1
			}

			let cellValue = workbook.Sheets[sheet][cell].v;
			if(!cellValue) continue;

			let curLine = line
			if(line != '') curLine += '.';
			curLine += cellValue;

			if((height + topLevel - 1) == headerHeight){
				headers.push({
					name: curLine,
					column: coor.c, 
					width
				})
				// headers.push(curLine)
			} else if((height + topLevel - 1) < headerHeight){
				headerParser((topLevel + height), curLine, coor.c, width);
			}
		}
	}
	headerParser(1, '');

	console.log(`Number of data columns: ${headers.length}`)

	// default row object
	let defaultRow = {}
	let headersList = [] 
	headers.forEach(h => {
		defaultRow[h.name] = null;
		headersList.push(h.name);
	})

	// define header cells
	let data = []

	for(let cell in workbook.Sheets[sheet]){
		let coor = cellParsing(cell);
		if(coor.r < (skipTopLines + headerHeight)) continue;
		if(!coor) continue;

		let rowNumber = coor.r - headerHeight - skipTopLines;
		if(rowNumber < 0){
			console.log(`${cell} is lower 0 (${rowNumber})`)
			process.exit();
		}
		if(typeof data[rowNumber] == 'undefined'){
			data[rowNumber] = Object.assign({}, defaultRow);
		}
		
		let cellValue = workbook.Sheets[sheet][cell].v;
		if(!cellValue) continue;		

		let column = headers.find(x => x.column == coor.c);
		if(column){
			data[rowNumber][column.name] = cellValue;
		}
	}

	const json2csvParser = new Parser({ headersList });
	const csv = json2csvParser.parse(data);

	console.log(`Data rows: ${data.length}`);

	fs.writeFileSync(fileOutput, csv)
	return data;
}

module.exports = xlxsToCSVSync;

To execute it use next script. Make sure that previous function is located in the same directory and is called excelToCSV.js or make appropriate changes.

Also you need to change path to file on 11 line. Additionally you can specify number of rows that should be skipped in source xlsx file.

const fs = require('fs')
const path = require('path');

const xlxsToCSVSync = require(path.join(__dirname,'excelToCSV.js'));

console.log(`To convert XLSX file to CSV and JSON formats change variables 'fileName' and 'skipLines' in script.\n` + 
			`  'skipLines' variable defines number of lines that will be skipped at the top of xlxs file\n`+
			`XLSX file must contain only one page sheet to be procedeed\n`);

// -------------- CHANGED VARIABLES --------------
const fileName = './table.xlsx'
const skipLines = 0
// -----------------------------------------------

const filePath = path.join(__dirname, fileName);
const parse = path.parse(filePath);

console.log(`Try to convert file '${fileName}' (${filePath})`)

if(!fs.existsSync(filePath)){
	console.log(`File '${filePath}' does not exists`);
	process.exit();
}

(async () => {
	let data = xlxsToCSVSync(filePath, path.join(parse.dir, `${parse.base}.csv`), skipLines);
	fs.writeFileSync(path.join(parse.dir, `${parse.base}.json`), JSON.stringify(data))
})()

In my case files structure of folder with scripts looks like:

├── excelToCSV.js
├── exeScript.js
├── node_modules
│   ├── ...
├── package-lock.json
├── package.json
└── table.xlsx
├── table.xlsx.csv
└── table.xlsx.json

In this article you will find two lightweight Node.js scripts that will allow you to send very large files from one computer to another peer-to-peer (P2P). There is a lot of different scenarious when they can be usefull: for example if your file is too big to be send via RDP protocol (as it has limit at 2Gb per file).

Prerequisites:

What you need:

• You need Node.JS interpreter to be installed on both computers (you can download it from here): server (from which you will load file) and client (from where you will load it);
• Ability to open ports on server machine;
• Ability to reach server from client (firewall should no break off connection).

How does it works?

There is two scripts:

• Server side script – reads specified file from computer and sends it to client, when client requests it;
• Client side – sends requests to server and downloads file to current folder;

Server.js side script

To execute server side script open terminal (command line), go to folder where you saved file and write: node node server.js --path pathToFile, change pathToFile with path to file that you would like to send to client. Path to file can be absolute or relative.

By default script will use 8000 port, if you would like to use another one, change it in the script (4 line).

const fs = require('fs');
const path = require('path');

const port = 8000;

;(() => {
	const filePathI = process.argv.indexOf('--path');
	if(filePathI == -1){
      console.log('Please call script with --path arg.');
      process.exit();
   }

	const filepath = process.argv[filePathI + 1];

	if(!fs.existsSync(filepath)){ console.log(`File does not exists!`); process.exit(); }
	const stats = fs.statSync(filepath);

	if(stats.isDirectory()){
      console.log(`File is Directory. You could not transfer directories with this script`);
      process.exit();
   }

	console.log(`You are about to transfer file: '${filepath}'`);
	console.log(`File size is ${Math.round(stats.size/1024/1024*100)/100} Mb`)

	const server = require('http').createServer();
	server.on('request', (req, res) => {
		if(req.url == '/info'){
			console.log(`Send information on file`);
			res.end(JSON.stringify({path: filepath, pathDesc: path.parse(filepath), stats}));
		} else if(req.url == '/file'){
			console.log(`Start tranfering of file`);

			const src = fs.createReadStream(filepath);
			src.pipe(res);
			src.on('end', () => {
				console.log("Transfer successfully");
				process.exit();
			});
		}
	});

	server.on('error', (e) => {
      console.error(e);
      process.exit();
   })
	server.listen(port);
})()

Client.js side script

Now is the time to load file to client machine:

1. Save client.js file to some place on machine;
2. Open file and change server variable, put yours machine address and port that server.js is using (by default 8000 port will be used);
3. Open terminal and navigate to the folder from step 1;
4. Execute node client.js. Script will start loading process. After loading process you should see message Loading is finished, if so everything is OK and you successfully load file. If You see some other message something wrong happened.

 const fs = require('fs');
 const http = require('http');
 const url = require('url');
 const path = require('path');
 
 var server = 'http://serverAddress:8000/'
 
 const getInfo = ()=>{
   return new Promise((resolve, reject) => {
     console.log(`-- Request information about the file from server`)
     http.get(url.resolve(server, '/info'), (res) => {
       let rawData = '';
       res.on('data', (chunk) => { rawData += chunk; });
       res.on('end', () => {
         try {
           return resolve(JSON.parse(rawData))
         } catch (e) {
             return reject(e);
         }
       });
     }).on("error", (err) => {
       return reject(err);
     });
   })
 }
 
 function processBar(fullsize, loaded) {
   var bars = 20;
   var loadBars = Math.round(bars * (loaded / fullsize));
   var remainBars = bars - loadBars;
 
   console.log(`[${'='.repeat(loadBars)}>${'.'.repeat(remainBars)}] ${Math.round(loaded/1024/1024*100)/100}/${Math.round(fullsize/1024/1024*100)/100}Mb`);
 }
 
 const loadFile = (info) => {
   return new Promise((resolve, reject) => {
     console.log(`-- Start file loading`)
     var pathParsed = path.parse(info.path)
     console.log(`File name is '${info.pathDesc.base}'`)
 
     if(fs.existsSync(info.pathDesc.base)){
       return reject(`Such file already exists in current folder`);
     }
 
     console.log(`File size is ${Math.round(info.stats.size/1024/1024*100)/100} Mb`)
 
     processBar(info.stats.size, 0)
 
     http.get(url.resolve(server, '/file'), (resp) => {
       var file = fs.createWriteStream('./'+info.pathDesc.base)
       var getSize = 0;
       resp.pipe(file);
       resp.on('data', chunk => {
         getSize += chunk.length
         processBar(info.stats.size, getSize)
       })
       resp.on('end', () => {
         console.log("\nLoading is finished");
         return resolve();
       });
     }).on("error", (err) => {
       return reject(err.message);
     });
   })
 }
 
 getInfo().then((info) => loadFile(info))
 .catch(e => {
   console.log(`Error in processing, stop of script execution:`)
   console.error(e)
 })

Good luck!

In this article I will disclose information about how Cognos TM1 stores information about cubes views in .vue files. For Cognos TM1 developers this information can be usefull in many different situations, for example:

1. For automatic updates of views – changing used subsets, chosen elements of context dimensions;
2. Finding what views uses some dimensions subset and etc.

There are 2 more simular articles about Cognos TM1 system codes in files:

1. Definition of system lines in Cognos TM1 Subset files (.sub)
2. Definition of system lines in Cognos TM1 Process files.

In the image above you can see the view and below you can see the code of it from .vue file.

There are multiple ways how developer can define appearance of some dimension in the view: with predefined subset, as custom list of elements and as list of all elements. To make the description of file more clear, I replace this information and explain it at the end of this article, defining all available options. Places in the code are replaced with lines: <-- Block of code that defines dimension, look below -->.

Code description

Defining list of elements or subset for some dimension after 7 code

Use subset as dimensions list

Use custom list of elements as dimensions list

When you specify custom list of element for dimension in view this syntax is used. For more information look article about codes in .sub files.

Use all elements as dimensions list

When you press All button in Subset Editor Cognos uses special system All subset, that contains all elements of dimension.

In this article I will disclose information about how Cognos TM1 stores information about dimensions subsets in .sub files. This information can be usefull if you are Cognos TM1 developer and need to automatically change many subsets in some way, for example: turn on/off expand above in multiple subsets, find what subsets of dimension have some element. Of cource there are a lot more ways how you can use this information.

There are 2 more simular articles about Cognos TM1 system codes in files:

1. Definition of system lines in Cognos TM1 Views files (.vue);
2. Definition of system lines in Cognos TM1 Process files.

Cognos TM1 subsets can be created in three different ways:

• as list of all elements;
• as custom list of elements;
• using MDX expression.

Consequently files for these three types of subsets have some difference, thus I will divide description in three sections.

All elements

283,100 – encoding of file, 100 for utf8;
284,"SubD1" – name of subset;
11,20190730094532 – time of last edit in format (YYYYMMDDHHMMSS);
274,AliasName — name of alias that is used for this subset or empty if alias does not used;
18,0 – in my models it usually equals to 0 and I don't know it purpose;
275, – used when created with MDX or empty otherwise;
278,1 – 1 if subset contains all elements of dimension, 0 otherwise;
281,0 – 1 if subset is expanded above, 0 otherwise.

Custom list of elements

Distinguished codes from "All elements" syntax: 281 and 270.

283,100 – encoding of file, 100 for utf8;
284,"SubD1" – name of subset;
11,20190730094532 – time of last edit in format (YYYYMMDDHHMMSS);
274,AliasName — name of alias that is used for this subset or empty if alias does not used;
18,0 – in my models it usually equals to 0 and I don't know it purpose;
275, – used when created with MDX or empty otherwise;
278,0 – 1 if subset contains all elements of dimension, 0 otherwise;
281,0 - 1 if subset is expanded above, 0 otherwise.
270,4 - after this code list of custom chosen elements is located;
Total
D1.EL1
D1.EL2
D1.EL3

Created with MDS

Distinguished codes from "All elements" syntax: 275, 281, 270.

283,100 – encoding of file, 100 for utf8;
284,"SubD1" – name of subset;
11,20190730094532 – time of last edit in format (YYYYMMDDHHMMSS);
274,AliasName — name of alias that is used for this subset or empty if alias does not used;
18,0 – in my models it usually equals to 0 and I don't know it purpose;
275,14 – If number is presented after this code it tells Cognos that MDX is used for creating this subset. 14 is length of MDX string.
{[D1].members}
278,0 – 1 if subset contains all elements of dimension, 0 otherwise;
281,0 – 1 if subset is expanded above, 0 otherwise.
270,0 – as subset created as MDX this code have 0 custom choosen elements;

In these article I would like to disclose some extra information about installation and work of Cognos BigFix and ILMT systems.

Problem with obtaining evaluation license key

While installing of BigFix Server you can encounter some problem with obtaining the key from IBM with error related to timeout: "Unable to submit license request HTTP: Error 28: Timeout was reached". I did not find straight way to fix it, but after a lot of tries I see success page.

How to install ILMT

1. Follow instructions on https://www.ibm.com/support/knowledgecenter/SS8JFY_9.2.0/com.ibm.lmt.doc/Inventory/planinconf/t_enabling_fixlet_internet_win.html
   1. Open BigFix Console, on the left bottom corner click BigFix Management
   2. Choose License Overview in left top window
   3. Read and accept license
   4. Find "IBM License Reporting" in list and click "Enable"
2. Then follow this instruction: https://www.ibm.com/support/knowledgecenter/en/SS8JFY_9.2.0/com.ibm.lmt.doc/Inventory/planinconf/t_installing_server_interactive_win.html
   • If you don't see needed task in "IBM License Reporting (ILMT)" group – click "Refresh Console"
   • Wait until BigFix downloades ILMT. You can view status on the summary tab of the task

1. When task finish executing you will find file In folder "C:\Program Files (x86)\BigFix Enterprise\BES Installers\ILMT_installer" find zip file, unarchive it and execute "setup-server-windows-x86_64.bat" with Admin rights;
2. When installation completes, choose "Open browser and complete installation", follow installation steps.

Setup ILMT

• Create new user in SQL server with sysadmin rights.

• If you create new user that will connect to MSSQL, then check whevever auth with login and password is permitted in MS SQL Server. You can check this by connecting to MS Server with new login. If you see such error, you need to fix it.

  A connection was successfully established with the server, but then an error occurred
  during the login process. (provider: Shared Memory Provider, error: 0 - No process is
  on the other end of the pipe.)
  
  (Microsoft SQL Server, Error: 233) 
  

  Do following:

  To solve this, connect to SQL Management Studio using Windows Authentication, then right-click on server node → "Properties" → "Security" and enable SQL Server and Windows Authentication mode.

On the next step specify databases users (by default this will look like this, only change SQL user name.

How to install client of BigFix

1. Copy folder "C:\Program Files (x86)\BigFix Enterprise\BES Installers\Client" from machine where BigFix server is installed to client (not only setup.exe, but also actionsite.afxm file)
2. Execute "setup.exe" to install BigFix client on client machine;
3. After some time (1-5 minutes), you will see new machine in BigFix Console.

How to execute machine scanner for ILMT

1. Open "Systems Lifecycle" in BigFix Console
2. Execute these fixlets (to execute find them in "Fixlets and Tasks" list, choose one, press Take action and then OK). List of fixlets to complete machine scanning:

• Install or Update Scanner (ошибка на этом этапе в Windows 2008 R2)
• Initiate Software Scan
• Upload Software Scan Results
• Run Capacity Scan and Upload Results

Find new machine in IBM License Metric Tool page

1. Go to "IBM License Metric Tool" page (https://[name of machine]:9081/)
2. Open "Management" → "Data Imports"
3. Press "Import Now".

Add information about VMs to make calculations of PVUs more accurate

1. Go to "IBM License Metric Tool" page (https://[name of machine]:9081/)
2. Open "Management" → "VM Managers"
3. Add information about VM
4. Give BigFix time to gather information about VMs to update reports (in my work information updates after 12 AM of the next day).

Case: You have a lot of excels created with Cafe tool. But at once you move your servers to another address or may be there is a need to create copy of excel to connect to dev/test server. Cognos TM1 does not give us easy way to do this and the most obvious way is to recreate these files. In this article I will show how to do it another (more geek) way.

First of all – excel file is just a zip archive that contains a lot of different files inside. There are also files that are generated by Cafe extension, which one we will change to get the desired results.

I will show two ways how to do this: first one – manually using text editor Sublime and second one using small NodeJS script.

Manual change with Sublime 3

1. Change extension of .xlsx or .xls or .xlsm file to .zip;
2. Unzip archive;
3. Open Sublime;
4. Open find and replace panel Shift+Cmd+F (on Mac OS) or Shift+Ctrl+F (on Windows);
5. In Find field insert current server name which is used in Excel file (for example http://oldserver:9510;
6. In Where field you need to insert path to folder which contains unzipped excel;
7. In Replace field insert new name of server (for example http://newserver:9510);
8. Press Replace button. Sublime will replace old server name to new one in all files;
9. Save and close all tabs that Sublime opens. These are files where Sublime finds the name of old server;
10. Create new .zip archive with all excels content. Make sure that _rels, docProps, xl and other files/folders are the top level objects in .zip

11. Rename back .zip to .xlxs, .xls or .xlxm.

Thats all, now your new Excel is ready to be used with new server.

Automatic change using NodeJS script

Also there are another way to do this changes. Here is the NodeJS async function that you can use to do all staff. Just call cafeServerChanger function with 3 parameters:

1. Path to the excel file;
2. Previous server name (http://oldserver:9510);
3. Name of server that you would like to use in new file (http://newserver:9510)

Do not forget to install 4 npm packages: extract-zip, moment, archiver, rimraf that are used in script.

const fs = require('fs');
const path = require('path');
const util = require('util')

const extract = require('extract-zip');
const moment = require('moment');
const archiver = require('archiver');
const rimraf = require("rimraf")

const walk = (dir, done) => {
  var results = [];
  fs.readdir(dir, function(err, list) {
    if (err) return done(err);
    var i = 0;
    (function next() {
      var file = list[i++];
      if (!file) return done(null, results);
      file = dir + '/' + file;
      fs.stat(file, function(err, stat) {
        if (stat && stat.isDirectory()) {
          walk(file, function(err, res) {
            results = results.concat(res);
            next();
          });
        } else {
          results.push(file);
          next();
        }
      });
    })();
  });
}

const escapeRegExp = (string) => {
  return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
}

const cafeServerChanger = async (file, serverFrom, serverTo) => {
  try {
    if(!fs.existsSync(file)) throw `Could not find file '${file}'`

    const fileParse = path.parse(file);
    
    const tempName = 'temp_' + moment().format('YYYYMMDDHHmmssSSS') + '_' + Math.round((Math.random() * (10000 - 1) + 1));
    const tempFolder = path.join(fileParse.dir, tempName);
    fs.mkdirSync(tempFolder)

    // unzip excel to temp folder
    await new Promise((resolve, reject) => {
      extract(file, { dir: tempFolder }, function (err) {
        if(err) return reject(err);
        console.log('Unzip successfully');
        return resolve();
      })
    })
    
    // create bin buffer for server line
    const nullHEX = 0x00;
    let lineLetterBuffArray = [];
    for(let i = 0; i < serverTo.length; i++){
      let l = serverTo[i];
      let buffer = Buffer.from([l.charCodeAt(0), nullHEX]);
      lineLetterBuffArray.push(buffer)
    }
    const finalBuffer = new Buffer.concat(lineLetterBuffArray);

    // replace all findings
    await new Promise((resolve, reject) => {
      let numberOfChanged = 0;
      walk(tempFolder, (err, zipFiles) => {
        if(err) return reject(err);
  
        for(let i = 0; i < zipFiles.length; i++){
          let file = zipFiles[i];
          let fileParse = path.parse(file);
          let content = fs.readFileSync(file).toString()
          let regExp = new RegExp(serverFrom, 'ig');

          if(fileParse.base.match(/^customProperty\d+.bin$/)){
            content = content.replace(/\0/g, '');
            if(content.match(regExp)){
              fs.writeFileSync(file, finalBuffer);
              numberOfChanged++;
            }
          } else {
            if(content.match(regExp)){
              content = content.replace(new RegExp(escapeRegExp(serverFrom), 'ig'), serverTo)
              fs.writeFileSync(file, content);
              numberOfChanged++;
            }
          }
        }
        console.log(`Change ${numberOfChanged} files`);
        return resolve(zipFiles)
      });
    })
    
    // Zip to new file
    const filePathResult = path.join(fileParse.dir, `${fileParse.name}.changed${fileParse.ext}`);
    if(fs.existsSync(filePathResult)) fs.unlinkSync(filePathResult);

    console.log(`Start compression`)
    return await new Promise((resolve, reject) => {
      var output = fs.createWriteStream(filePathResult);
      var archive = archiver('zip', {
        zlib: { level: 5 } // Sets the compression level.
      });

      archive.on('warning', (err) => {
        if (err.code !== 'ENOENT'){ 
          return reject(err);
        }
      });
      archive.on('error', (err) => reject(err));
      output.on('close', () => {
        console.log(archive.pointer() + ' total bytes');
        console.log(`New file is '${filePathResult}'`)
        console.log(`Delete temp folder`)
        rimraf.sync(tempFolder);
        return resolve(filePathResult);
      });

      output.on('end', () => console.log('Data has been drained'));

      archive.directory(tempFolder, false);
      archive.pipe(output);
      archive.finalize();
    })
  } catch(e){
    return Promise.reject(e);
  }
}

module.exports = cafeServerChanger;

You can use and modify these code in any way. Good luck!

Case: You have large folder and some process that makes changes in it. As there are too many nested folders and files there, manual comparison can take too much time, so you would like to find quicker and easier way.

In this post I would like to share with you my light Nodejs script, that can snapshot folders state. So you can take one snapshot before and one after some action, compare them and find changed files.

const fs = require('fs');
const path = require('path');

var walk = function(dir, done) {
  var results = [];
  fs.readdir(dir, function(err, list) {
    if (err) return done(err);
    var i = 0;
    (function next() {
      var file = list[i++];
      if (!file) return done(null, results);
      file = dir + '/' + file;
      fs.stat(file, function(err, stat) {
        if (stat && stat.isDirectory()) {
          walk(file, function(err, res) {
            results = results.concat(res);
            next();
          });
        } else {
          file = file.replace(folder, '')
          results.push({file, changetime: stat.ctime, size: stat.size});
          next();
        }
      });
    })();
  });
}

if(typeof process.argv[2] == 'undefined'){
  console.log(`Please provide path to director as the first argument, like:\n./script.js /some/path/`);
  process.exit()
}

const folder = process.argv[2];
if(!fs.existsSync(folder)){
  console.log(`File does not exists!`);
  process.exit()
}

const stats = fs.statSync(folder);
if(!stats.isDirectory()){
  console.log(`File is not directory`);
  process.exit()
}

const pathDesc = path.parse(folder);
const folderName = pathDesc.name;

walk(folder, function(err, results) {
  if (err) throw err;

  const fileName = `${folderName}.${Math.floor(Date.now()/1000)}.json`;
  fs.writeFileSync(`./${fileName}`, JSON.stringify(results, undefined, '  '));
  console.log(`Done. Write folder '${folder}' snapshot to file '${fileName}'`);
});

To execute this script provide path to folder as the first argument like this:

node script.js /way/to/folder

After this, script will recursivly walk through all nested folders and prepare list of files with information about there current size and time of last change.This information would be written to the file in JSON format.

[
  {
    "file": "/1 - create currency dimension.pro",
    "changetime": "2019-04-17T11:46:55.381Z",
    "size": 1887
  },
  {
    "file": "/1 - create margin range dimension.pro",
    "changetime": "2019-04-17T11:46:55.397Z",
    "size": 1899
  },...
]

Later you can compare two such files and find difference. Personally, for this purpose, I use DiffTabs package for sublime text editor:

TI process can be terminated using one of these functions:

• ProcessBreak;
• ProcessError;
• ProcessQuit;

Two of them ProcessError and ProcessQuit will terminate process with error message, that will look like this:

As you can see, this message does not give any concrete information to the end user, especially if you have multiple conditions that can generate the error in TI code.

In this tutorial I will show you how to customize this message and show popup that looks like this:

First of all you need to create new eror code in }tp_process_errors_localization dimension. To do this use this TI template or add element to dimension manually:

sErrorCode = 'CustomErrorCode';
sErrorCodeCaption = 'Custom Error Header';
sErrorType = 'Error';
cDim = '}tp_process_errors_localization';

DimensionElementInsertDirect(cDim, '', sErrorCode, 'N');
AttrPutS('en', cDim, sErrorCode, 'CaptionDefaultLocale');
AttrPutS(sErrorCodeCaption, cDim, sErrorCode, 'Caption');
AttrPutS(sErrorType, cDim, sErrorCode, 'ErrorType');

Change:

• sErrorCode - principal name of the error code, that will be used later, when you will execute process to show message;
• sErrorCodeCaption - user friendly name of the message, that will be displayed at the popup before semicolon;
• sErrorType - could be 'error' or 'warning'. Choose any.

After this step restart Cognos Application service, as sometimes it will not recognise new error codes and will show you message, that will contain only error code, without description:

Now you can modify process that is assigned to Applications Commit or Submit actions. Add this code before ProcessQuit or ProcessError functions. Write your message to pErrorDetails parameter.

ExecuteProcess('}tp_error_update_error_cube', 
  'pGuid', pExecutionId, 
  'pProcess', GetProcessName(),
  'pErrorCode', 'CustomErrorCode',
  'pErrorDetails', 'Custom message, that will help user to correct data',
  'pControl', 'Y');

Now Cognos must show you error message with yours description.

If you need to delete your custom error code, delete it with DimensionElementDeleteDirect and restart Cognos Application service.

sErrorCode = 'CustomErrorCode';
DimensionElementDeleteDirect('}tp_process_errors_localization', sErrorCode);

Also you can read about this Cognos feature in official IBM documentation.

Case

You have some Cognos TM1 Application that works well. But for some reason you had need to recreate some cube (for example to delete or add a dimension) that has view in this Application. You did all work, republish Application with the new view (or do nothing, as Cognos by default can see it and add automatically). Later when you try to access this view in Cognos Application you see such error message:

Now you are confused and don't know how to fix this!

Reason

This error could have many reasons, for example:

• Corrupted user view
• Wrong security settings
• Not deleted .blb files. In this post I will write about this one.

Such .blb files has following syntax and are located in root of data folder:

CubeName.ViewName_ApplicationID_\[}tp_tasks}ApplicationID].\[HierarchyLevel].CAMID(CamidOfUser).blb

Where CubeName, ViewName, ApplicationID, HierarchyLevel, CamidOfUser are variables.

Cognos creates them when user opens some application, make changes in the view and closes the window. Unlike other .blb files, these Cognos does not delete by default (this is correct for Cognos TM1 10.2 Pack 6) when you destroy cube.

So to fix the problem you need to find all wrong .blb files that are connected with certain cube, view and application and delete them.

Remove files with Powershell

I prefer to do this with Powershell:

1. Move to data folder (change dataFilesFolder with yours):

cd datafilesFolder

2. Allocate all .blb files that are connected with some cube and view (change CubeName and ViewName with yours) in the folder. Important: _ is needed to exclude .blb file for cubes view (they are alright).

Get-ChildItem 'CubeName.ViewName_*.blb'

3. When you make sure, that listed files is what you was looking for, you can delete them. To do that – add code -File | Foreach-Object {Remove-Item -LiteralPath $_.FullName} to previous command and execute it.

Get-ChildItem 'CubeName.ViewName_*.blb' -File | Foreach-Object {Remove-Item -LiteralPath $_.FullName}

Thats all. This should solve the problem, and you should have access to view in Application. If not try to log off from Cognos Tm1 Applications.

If you still does not have access – chech security cubes, may be you forget to grant some access to user.

I hope this helps to solve your problem!

If you tied from absence of code completion for Cognos TM1, I have good news for you. Today I publish TM1SublimeCompletion package for Sublime 3 that can solve this problem.

Description

Package contains completions for all TI, Rules, Worksheet and MDX function, global and local variables of IBM Cognos TM1 system. Here are the main features of package:

• Covers TM1 versions from 10.1.0 up to 10.3.0 (Cognos Analitycs);
• If function is valid in multiple places (for example in TI and Rule), but has different syntax, package will show you both variances.

This is just first version of it, so if you will find some mistakes – please let me know.

Installation

To install packages:

1. Install some package for highlighting Cognos TM1 code in Sublime, I prefer to use these one hermie64/tm1-sublime
2. Download Tm1SublimeCodeCompletion.sublime-completions file from git repository and put it in any place in package folder of Sublime
3. You are ready, have fun!

More information you can find on GitHub page of this project.