Nash!Com Run Special Tasks V3.0

The "nshrun" utility has been designed to run routine Domino database maintenance tasks with multiple threads in a coordinated way. The result is, the time taken to perform database maintenance will be dramatically reduced.

On large AIX and Solaris machines, a single instance of the compact task, doesn't take full advantage of multiple CPUs, large memory and fast I/O throughput. Conversely, nshrun can fully utilize the available resources. Even smaller W32 and Linux systems, can draw upon the major benefit of running multiple threads. Once started, each "worker" thread will perform the requested operations whilst, the "control" thread assigns work to each of those threads.

The majority on Domino Administrators create database maintenance "Program Documents", starting with a fixup, followed by a compact and concluded with updall. With nshrun, only a single document is necessary to cover any combination, or all of these tasks.
An existing nshrun customer confirmed their weekend database maintenance, has been reduced from 44 to 11 hours. They have been impressed with the results and are running nshrun as a nightly operation.

Finally, if you are planning to migrate to a new Domino release which introduces a new ODS, nshrun can help to reduce the time needed preparing the databases.
(e.g. nshrun -D discards view indexes, -u to rebuild views in new ODS format, -X to run a full fixup on all Notes databases).
In addition to this the solution can capture the list of currently used views and rebuild those views after a compact -D (stored in the database and can be done in multiple steps).


The following options are currently supported

Select Parameters:

To select databases you can specify either a single database or a directory with a file pattern. By default only this directory is processed. If you want to include subdirectories you have to specify the -r switch which will cause all subdirectories to be processed. You should always specify a file pattern, like shown in the example.
Note: On Unix you have to specify the path with quotes because depending on the shell you use the parameter list might get expanded thru the shell.
Example: "mail\*.nsf"

-r Process directories recursively (without this option only the specified directory will be processed)


Performance Parameters

-t<threads> Number of threads [default 2]
By default nshrun uses two threads. You can change the number of threads depending on the power of your machine and how busy you want your Domino server to be during the operations.
Generally you should not start more than 4-6 threads at the same time because they could block each other in the I/O area and this could lead to a longer run-time.
Specify the number of threads with the -t option.
Example: -t 4


Fixup Options:

The following options allow to fixup databases very similar to what Domino is doing with the fixup task. The code uses the same C-API calls to perform the fixup.
Below you find all currently supported options for fixup

-x Fixup database
-X Fixup database with full note scan
-h Fixup try hard
-n don't delete corrupt notes on fixup
-J Also fixup transloged databases


Index and FT-Index Update Options

The following options allow to update Indexes and FT-Indexes in Domino. This part works very similar to what updall does but on a more granular basis. Specially for FT indexes you have the choice to rebuild indexes and also skip FT indexes with attachments.
Indexing databases with corrupt attachments can cause a server crash. Specially in larger migrations this is a potential problem in the migration process. This nshrun provides an option skips full-text indexing for those databases. Each skipped database is logged. If you only specify the -m option, all databases containing attachment full-text index options are reported.
You can run this option before you make any modifications to your server to get a full list of databases with full-text index attachment options.
And after most part of the migration has been performed in one step you could watch the server more closely when it rebuilds the FT indexes for those databases having attachment indexing enabled.


View Folder Index

-u update standard folders/views (*see list of views/folders below)
-U update all existing indexes
-R rebuild all existing indexes
-Z create and update ALL indexes

($Inbox)
($Calendar)
$Drafts
$Sent
($Alarms)
$ToDo

Note: By default the All Documents view is not opened. In many customer environments the All Documents view is not used by may users. Not creating the index by default does reduce the size of a mail database. In case the user decides to use the view the first open of the view will trigger an index to be created.

-u<filename>

rebuild views from reference file (you need a copy-style compact -C to apply the settings on all data - but LZ1 cannot re-compress docs)


FT-Index

-f update full-text index
-F delete full-text index
( delete and update can be combined)

-m skip full-text index update for database where indexing of attachments is enabled.
-fm update full-text index only for databases WITHOUT indexed attachments
-fx update full-text index only for databases WITH indexed attachments
-Fm delete full-text index only for databases WITHOUT indexed attachments
-Fx delete full-text index only for databases WITH indexed attachments

-Fd

delete & disable FT Index

-f<options>

create new FT Index with options -- existing FT index with different options will be re-created
without options only existing FT indexes will be updated

c=case sensitive
s=steam index
p=index paragraphs etc
e=encrypted fields
a=index attachments
f=user filters for attachment index
z=create index with auto options


Set Database options from Reference Server

-g <reference server>,<action flags>

set database options from reference replica

v = build views from replica
q = set quotas from replica
f = create FT index from replica

note: you can also use a reference database instead of a server

Set Database Flags

-c <flags>

set/reset database flags

d enable DAOS
D disable DAOS
n enable Design Compression
N disable Design Compression
v enable Data Compression
V disable Data Compression
z enable LZ1
Z disable LZ1


Capturing Open View/Folder List and rebuilding Views/Folders from List

Another issue you might face during a migration is that you need to do a Compact -D on all databases to get rid of old indexes on a lower level. Running a Compact -D on all databases is a best practice and recommendation specially when moving to a different ODS or moving to a different platform. Compared to a updall -R all indexes are completely discarded and the indexer does not know which indexes have been open before. nshrun does allow to capture the names of the open indexes and can re-open them after a compact -D operation.
Those those operations could be either done on one run or in multiple steps (for example you are moving the server to a different OS or box).
The following two options allow to capture the current active views in a database and build views based on a text list stored in the database icon note after a compact -D operation. This is extremely helpful when migrating from one Domino release to another or when moving from operating system to another. The information about the currently build views and folders is stored inside the notes database and is transferred to the new environment.
-l Build List of active views
-p Build Views from List


Compact Options:

nshrun does also support the commonly used Compact options. In addition to those standard options it can be used with advanced selection options in case a database has higher or lower unused space.

-C Copy style compaction
-b Recover space without reducing file size (inplace-style)
-B Recover space and reduce file size (inplace-style)
-D Discard view indexes (copy-style)
-i Ignore errors (for copy-style only)

-s<n>[M K] compact if free space is above n %,MB or KB

Log Options:

-v Verbose logging
-w Show counter for processing DBs


Special Options:

-o<ods-ver> skip databases which are already in specified ODS
-O<ods-ver> only run compact on databases with specified ODS - other operations still apply");

Those two options help to reduce the time needed to re-process databases after the nshrun operation has been interrupted during a migration process. You can specify a target ODS version example: -o43 to skip all databases which are already in ODS43 (D6).


Special Features:

nshrun is designed to run standard operations in the same way standard tasks like updall or compact perform. In addition to the better scalability and performance nshrun provides detailed logging and diagnostic. Each worker-thread logs the operation it is working on with the number of the worker-thread prefixed. And using the log option -w you see the number of processed databases.
In addition the status line of nshrun provides counter information.

As a special feature for troubleshooting nshrun does call a special subroutine in the code passing the instance number of the thread. This allows to identify the thread-number from the call-stack of the process involved in the crash. Those kind of problems mostly occur thru database corruptions or attachment corruptions. The log in combination with the thread number from the call-stack allows to identify the database very easy.


nshrun Use Cases

The following abstract describes use cases for nshrun.


Usecase: Server Migration

During a server migration to a different OS or when the ODS changes you should do the following

- fixup on all databases (normally fixup -f -J)
- compact discarding all view indexes (normally compact -D)

nshrun dramatically lowers the time because of parallel operations on multiple databases and also combining multiple operations on the same database into one set of operations per database.
Below you find a detailed lists of operations that you should perform.

Preparation

- Shutdown the Domino server
- Physically delete all ft index directories
- NOTE: Ensure that NSF_BUFFER_POOL_SIZE_MB=512 is explicitly set becausw if the server is down the current Domino code will assume the process is a client process and only allocate 8MB of buffer pool! This can cause dramatical database corruption specially when fixup options are used.


First Step when server is down

nnshrun -t4 -l -D -X -r -w "*.nsf"

- Use 4 threads (-t4)
- Build list of open views (-l)
- Fixup Full Note Scan (-X)
- Discard all View Indexes via Compact (-D) and bring database to new ODS
- Run on all nsf files and include sub-directories (-r "*.nsf")
- Display the number of processed databases via counter (-w)

*) In case you did not disable translog you have to specify -J to fixup transaction logged databases

Second Step after server restart

lo nshrun -t4 -p -f -r -w -m "*.nsf"

- Use 4 threads (-t4)
- Rebuild all views that have been open before compact (-p)
- Update all FT Indexes (-f)
- Skip FT index update on databases with attachment indexing enabled (-m)
- Run on all nsf files and show counter (see previous step)

Third step update remaining FT indexes

lo nshrun -t4 -fx -w -r "*.nsf"

- Use 4 threads (-t4)
- Update FT index update on databases with attachment indexing enabled (-fx)
- Run on all nsf files and show counter (see first step)

Usecase: Nightly and Weekend Operations

Some customers run nightly fixup and udpall -R operations for pro-active server maintenance.
The recommendation is to only run a fixup and view rebuild when an error occurs.

The only operations you should run are

"nshrun / compact -b" to recover free space in the database

"nshrun compact -B" to free space to the file system

Depending on your backup solution you have to be very carefully about the -B option.
All copy-style compact operations and also the -B option assigning a new DBIID to your database.
In case you run Domino transaction logging in archive mode a new DBIID will cause the current backup of the database to be invalid.
A new backup of the database needs to be taken after such an operation.

Usually the best option in this case is to have compact -B operations only over the weekend before a full backup of all databases to avoid extra full backup copies of databases during the week.

During the week you might want to run a compact -b -s 10 to recover free space from the database.

However in some customer situations where quotas are used on the mail-file this approach might be problematic because the database file will not shrink until the weekend.
In case you have no archive style transaction logging enabled we recommend to use a nshrun/compact -B -s10 to avoid those kind of end-user issues.

nshrun provides additional options in such a case. Normally you might want to only recover space in a database but not always free it directly to the file-system by freeing the resources in the database to avoid file-system fragmentation (because the space will be allocated later anyway by new documents).

In such a case you might want different operations depending on the free size in the database.
For smaller amount of free space you should run with the -b option with larger percentage of free space you might want to recover the complete space via -B.

nshrun provides an option to allow the -b/-B options to be used in combination.

If you specify a lower case -s this will trigger -b if the database has more free space.
If you specify a upper case -S this will trigger -B if the database has more free space.

Both parameters also work in combination and will do the right compact option for each of the database is needed in one run.