QStar Archive Manager Troubleshooting Q&A



Download 37.02 Kb.
Date conversion28.01.2017
Size37.02 Kb.

QStar Archive Manager Troubleshooting Q&A



Change History

Date

Author

Comments

June 2, 2012

Vladas

Linux

troubleshooting initial version















































Contents


QStar Archive Manager Troubleshooting Q&A 1

How QStar Integral Volumes and CleverSafe Vaults relate? 2

Can I use same vaults form different computers? 2

Are QStar distributions compatible for different Linux versions? 3

In which directory I should install QStar Software? 3

How I extract QStar distribution? 3

How I install QStar software? 3

How do I license QStar software? 4

How do I configure Cleversafe QSSD device? 4

How I define access to the vaults? 4

How do I create Integral Volume, add vaults and mount? 5

What libraries I need to install on Linux? 5

How to check that portmapper is installed? 6

How do I mount Integral Volume set ? 6

How do I unmount IV set? 6

How do I export QStar file system through NFS? 6

How I move QStar software to other computer? 7

What QStar processes should be running in UNIX systems? 8

How do I know that set is mounted and healthy? 9

What is Covered error? 9

How do I know that something is wrong? 9

How do I perform mounted set cleanup? 9

How do I kill everything? 9

How do I know that data is communicated to Cleversafe ? 10

What files I need to send to QStar in case of error? 10

How do I perform data base backup? 10




How QStar Integral Volumes and CleverSafe Vaults relate?


The QStar Integral Volume (IV) presents a single file system which can be mounted, un-mounted or otherwise treated as a single namespace of the files with hierarchical structure. As mentioned before one IV is created on two CleverSafe Vaults – Metadata Vault and Data Vault.

The files and file metadata of such file system are created and maintained on the cache disk and subsequently migrated to the CleverSafe storage in the form of objects. The objects are of 4-16 MB size and may contain data from a single file or from multiple small files.


Can I use same vaults form different computers?


The data sharing shall be performed on the IV set basis. The IV set mounted on some computer can be shared through NFS or CIFS to other computers.

There is protection built in which allows to prevent CleverSafe vault pair sharing by some other IV on other computer. Technically this is possible to access vaults from any computer which can provide proper vault, user names and passwords. Unfortunately there is no tools to synchronize cache or vault modifications in consistent way. The protection processes attempts to mount the same vaults in the following way:



Node 1

Node 2

Create IV set



Node 2

Mount IV set



IV set is created but not mounted

Yes

Can be mounted in read-only mode

“Clean cache”, Restore DB mount



IV set is created and mounted

Yes

Can be mounted in read-only mode

“Clean cache”, Restore DB mount



Node is down (destroyed, failed, not accessible)

Yes

Can be mounted in read/write mode with “force” option.

“Clean cache”, Restore DB mount



The “Clean cache”, Restore DB mount allows to recreate status of the system (possibly modified by the Node 1) at some point in time. During clean cache, Restore DB mount (which is essentially is the method to recover state of the file system after computer crash) the QStar will restore data base from the CleverSafe Vaults, replay objects after data base backup and repopulate CFS data base.

Are QStar distributions compatible for different Linux versions?


No.

You need to request from QStar distribution build for compatible version of the Linux.

RedHat builds usually are not compatible with Ubuntu, SuSe or other distributions. That is usually determined by libc and other libraries incompatibilities. 32 or 64 bit distribution flavors needs to be specified.


In which directory I should install QStar Software?


All UNIX distributions should be installed in directory /opt/QStar. Please note that Q and S are capital letters. The UNIX distribution images are linked explicitly for this directory.

Symbolic links are supported in all platforms except AIX.


How I extract QStar distribution?


The distribution is delivered in the form of zipped tar file. You need to extract the file into directory /opt/QStar.

mkdir –p /opt/QStar

cd /opt/QStar

tar xfz /tmp/linux_2.6_x86_64_v6.0.1.26xx.tar.gz


How I install QStar software?


The /opt/QStar/INSTALL script performs mandatory steps to install the software.

cd /opt/QStar

./INSTALL

Confirm license agreement.

Select option 1.

Select option 5.

Select option 9 to exit.

How do I license QStar software?


You need to send to QStar host ID of your computer.

/opt/QStar/license/installkey –H

e58327eb

Here is standard way to request for license.

http://www.qstar.com/support/license-key-request-form/

You need to place received license file (names QStarLicenseFile.e58327eb) in /opt/QStar/license directory.


How do I configure Cleversafe QSSD device?

To make Cleversafe known to the QStar software it needs to be defined.



qssddefine –T cs [-H hostname] –i iplistCluster –u usernameCluster –p passwordCluster qssdname

This operation verifies that Cleversafe manager is accessible for the QStar software



qssdrefresh qssdname

This operation lists available Vaults. If vault access is configured they will be refreshed.



qssdlist –av qssdname

It list is empty please verify IP, username and password for the manager in qssddefine command


How I define access to the vaults?


Each vault which will be used in this installation needs to be configured separately.

qssdmanage –T cs –i iplistVault -u username -p Password [-x proxy:port] qssdname VaultName

Where:
-i iplistVault - Comma separated list of the IP addresses or names of the vault.

- u username – username for a Vault

-p password – password for a Vault

-x proxy:port - optional parameter if QStar server is behind the proxy server

qssdname - Name of the CleverSafe QSSD device

VaultName – name of the vault for which configuration is performed.



qssdrefresh qssdname

qssdlist –v qssdname

You should see Vault parameters in the list and log/syslog shall not contain errors about access to the vault.


How do I create Integral Volume, add vaults and mount?


To create IV user following commands:

vlcrset –T coud –u cs clever

To add media to the IV set (assuming that metadata vault is called Meta-01 and data vault is called Data-01 and QSSD device is called qssdname)



vladdtoset clever qssdname Meta-01 Data-01

To assign cache to the IV set ( assuming that cache will be located in /cache/clever directory):



mkdir /cache/clever

vlcache –c /cache/clever –p 10G clever
Prepare mount point and mount
mkdir /mnt/clever

mountiv clever

What libraries I need to install on Linux?


QStar needs “portmapper”. For Ubuntu install:

apt-get install nfs-kernel-server

RedHat usually comes with portmapper.

Ubuntu 112.04/ 11.10 require following library to be installed:

apt-get install libicu-dev

apt-get install uuid-dev

How to check that portmapper is installed?


netstat -an |grep 111

tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN netstat –al |grep 111


How do I mount Integral Volume set ?


mountiv setname is the command to use. Here setname is IV set name. In this document the setname will be defined as “clever”.

There are following main options to use:



mountiv –n clever - mounts set and cleans cache. The data will be retrieved from the Cleversafe if requested.

Mountiv –Inf clever – recovers data base from Cleversafe and mounts with clean cache. There are several uses for this mount option. Testing is one. Clean everything is another. Moving IV set to other computer is third.

Should be used when moving system to other computer. May take some time. Better if backup was forced before moving.


How do I unmount IV set?


umountiv clever

In some cases (when acesser is not responding to TCP/IP requests) “forced” unmount may be needed:



umountiv –f clever

How do I export QStar file system through NFS?


To export QStar file system (mounted as /mnt/clever) you need to create entry in /etc/exports file.

The standard Linux syntax of the /etc/exports apply but there is one mandatory parameter to be added: fsid=xxxxxx.

The fsid parameters value is not that important but every QStar file system (in fact every file system using FUSE driver) should have different value.

For example:



/mnt/clever *(rw,sync,insecure,no_subtree_check,no_root_squash,fsid=34)

In this case value 34 was chosen.

To activate new entry in /etc/fstab:

exportfs –av or

exportfs –r

Please note that once you exported file system you will not be able to unmount QStar file system. It will return “busy” status until you un-export the file system from NFS.

To un-export file system:

export fs –u *:/mnt/clever

How I move QStar software to other computer?


To make moving smoothly the following actions needs to be performed on original installation:

  1. Force migration of all Primary data to Cleversafe (using command mmarc –e –w clever or use GUI button “Start Migration”). If that is not done the Primary data residing in the cache will be lost. Obviously, if original installation is lost or damaged the data in Primary pages will be lost.

  2. Perform backup of the data base. This is not essential step because data will be recovered , but having full data base backup will save recovery time since there will be no objects to replay.

    1. emgcmd backup full clever

  3. Unmount set on original computer.

umountiv clever

If original computer just died you may expect some data loss if data was remaining on the cache.

The sequence of events shall be as following:


  1. Install QStar software on other computer.

  2. License software

  3. Create QSSD device

  4. Configure vaults and refresh them.

  5. Create IV set and add Vaults to the set

  6. Mount set with options

mountiv –Inf clever
Here: -I – recover data base for Cleversafe storage

-n – mount with clean cache

-f - tell Cleversafe (properties object) that now this computer will be managing those vaults.

What QStar processes should be running in UNIX systems?


If IV set “clever” is mounted the following processes should be running.

ps -ef |grep _qsetmount

root 31823 31790 1 15:34 ? 00:00:00 qsetmount clever


ps –ef |grep _ufsimount

root 31866 1 0 15:34 ? 00:00:00 _ufsimount /mnt/clever /cache/clever clever CleverSafe 65536 8200 5968162150190827164 2 8 0


Following QStar servers should be running:
ps –e |grep server

31445 ? 00:00:00 _qscsiserver

31447 ? 00:00:00 _jbserver

31451 ? 00:00:00 _mmserver

31453 ? 00:00:02 _vlserver

31465 ? 00:00:00 _fsi_server


How do I know that set is mounted and healthy?


If set is mounted and accessible the “mmparam clever” should show line Error condition No.

If “Covered error” status is displayed the syslog shall be analyzed to understand error source.


What is Covered error?


Since QStar file system is represented by at least 2 software layers, from the user point of view the migrator software is “Covered” – i.e. not directly exposed – to the user.

How do I know that something is wrong?


  1. Mmparam clever returns error:

mmparam clever

mmparam: common_socket.c.588: connect: Connection refused

The ufsimount process or qsetmount process is down

Perform mounted set cleanup.


How do I perform mounted set cleanup?


If something goes wrong the cleanup procesure involves following steps:

  1. Unmount set from Linux:

    1. umount –f /mnt/clever

  2. Unmount set from QStar servers

    1. umountiv –f clever

  3. make sure corresponding _qsetmount and _ufsimount processes are not running anymore:

    1. kill –KILL _ufsimountpid _qsetmountpid

      1. where _ufsimountpid _qsetmountpid needs to be retrieved from “ps –ef” command

How do I kill everything?


If normal start of the QStar servers is not working they can be killed explicitly:

killall –KILL _qscsiserver _jbserver _mmserver _vlserver _fsi_server _ufsimount _qsetmount

If at this time some file systems were mounted you need to cleanup linux mount points.

After that QStar servers may be restarted using “/opt/QStar/install/rc start” and sets remounted.


How do I know that data is communicated to Cleversafe ?


mmparam clever will show changing numbers on lines:

Archived since mount:

Replicated since mount:
For especially curious following command sequence enables for 10 seconds and disables trace log:
emgcmd config log trace clever ; sleep 10 ; emgcmd config log none clever
The log file will be in /opt/QStar/log/emg_debug_log file. Messages such as

Wrote [9099abe9dba9d3ec0687c4e4b182b7180000] 2138 bytes in 21254714 us

Indicates that writes are working.

clever: Object (MI102611V1:86f671a1c51e97de68feecc43a11e9a90000) was read, size is 4184488

Indicates successful reads.



Please note that trace log contains lot of other information not targeted for the end user. Prolonged run of the trace may slow down all operations. Nevertheless in some cases QStar service may request for such trace to analyze error situations.

What files I need to send to QStar in case of error?


In order to perform error analysis following information is required:

  • Content of the /opt/QStar/log directory.

  • Core files if any (/core* and /opt/QStar/cores/*)

  • Test scenario.

How do I perform data base backup?


The data base backup is performed according to the file /opt/QStar/emg/files/clever_rc.xml.

If there is a need to perform data base backup on demand the following command should be used:



emgcmd backup full clever

The data base backup is initiated asynchronously and it is finished when syslog contains following line:



05/28/2012 13:00:32.811275 INFO clever: Full DB backup finished: size is 112256 bytes

Forced backup is recommended when IV set is moved to another computer or for any other good reasons.


The database is protected by copyright ©ininet.org 2016
send message

    Main page