Difference between revisions of "SQWareProduction:hbase"

Revision as of 20:20, 16 June 2014

Presentation

Usage

It's the module locally operating of hbase.
It is entirely written in ksh and sql.
It allows to manage the operation for hbase.
Example uses of the tool:

Backup
Recovery
Compute statistics
Indicators collection to store in referential database SQWareRepository
Rotate logs
....

This module is deployed locally on the machine hosting the authorities to manage, in order to avoid any points of weakness related to an NFS mount or whatever. The deployment is done through a simple command SQWareCentral module using inventory located in SQWareRepository !
Installation guide for SQWareProduction for hbase

Explanation of directories

Directories located in $gvsqw_HbsBin:

bin : Directory containing standard scripts
bin_cust : Directory containing customs scripts for your environnement
etc : Directory containing the standard files of configuration
etc_cust : Directory containing the customs files of configuration for your environnement (overload standards)
help : Directory containing the standard files of help
help_cust : Directory containing the customs files of help for your environnement
lib : Directory containing the generic shell function librairies
lib_cust : Directory containing the generic shell function librairies (overload standards)
spm : Contain sql query toolkit
tools : Directory containing the standard scripts used on an ad hoc
tools_cust : Directory containing the customs scripts used on an ad hoc for your environnement
menu : Directory containing the standard menus
menu_cust : Directory containing the custom menus for your environnement

Warning:
Do not change the standard scripts and configuration files of dbSQWare.

Use directories *_cust for customizations. This will facilitate version upgrades.

Naming convention of files

The majority of files are named with suffixes:

*.ksh : for scripts
*.cfg : for configuration files
*.lib : for function librairies
*.hlp : for help files

Warning:
Do not change the standard scripts and configuration files of dbSQWare.

Use directories *_cust for customizations. This will facilitate version upgrades.

Naming convention in scripts

In general, the following naming convention is used in all scripts of SQWareProduction:

gvsqw_ {} : global variable initialized by the environment and / or generic library
lvsqw_ {} : local variable initialized by the script and / or a specific library
gfsqw_ {} : function defined by a generic library
lfsqw_ {} : function defined by the script and / or a specific library

bin

sqwhbs_GatherIndicators.ksh

Script $gvsqw_HbsBin/bin/sqwhbs_GatherIndicators.ksh:

See the detailed help for sqwhbs_GatherIndicators.ksh

sqwhbs_NetBackup.ksh

Script $gvsqw_HbsBin/bin/sqwhbs_NetBackup.ksh:

backup with NetBackup dump/backup files See the detailed help for sqwhbs_NetBackup.ksh

sqwhbs_ParallelRun.ksh

Script $gvsqw_HbsBin/bin/sqwhbs_ParallelRun.ksh:

Manages the execution of unix commands in parallel mode
Takes in parameter file with unix command to execute
You can choose:

The parallel degree
Aggressiveness of run (number of seconds between two runs)
Aggressiveness of check end (number of seconds between two checks of end)

See the detailed help for sqwhbs_ParallelRun.ksh

sqwhbs_RunJob.ksh

Script $gvsqw_HbsBin/bin/sqwhbs_RunJob.ksh:

Run HBase scripts and generate log and error file
based on $gvsqw_RootCfg/sqwhbs_Jobs.cfg configuration file
compress and purge oldest log files See the detailed help for sqwhbs_RunJob.ksh

etc

sqwhbs_CreInst_CrontabRef.cfg

Configuration file $gvsqw_HbsBin/../etc/sqwhbs_CreInst_CrontabRef.cfg:

See the detailed help for sqwhbs_CreInst_CrontabRef.cfg

sqwhbs_CreInst_Jobs.cfg

Configuration file $gvsqw_HbsBin/../etc/sqwhbs_CreInst_Jobs.cfg:

See the detailed help for sqwhbs_CreInst_Jobs.cfg

sqwhbs_GlobalVar.cfg

Configuration file $gvsqw_HbsBin/../etc/sqwhbs_GlobalVar.cfg:
Global var for HBase
It contains the operating variables of SQWareProduction.

Note:
You can overload variables contain in $gvsqw_HbsBin/../etc/sqwhbs_GlobalVar.cfg

by the same variables in $gvsqw_HbsBin/../etc_cust/sqwhbs_GlobalVar.cfg for all your environnement
or in $HOME/sqwConfig/sqwhbs_GlobalVar.cfg specifically for this user.

See the detailed help for sqwhbs_GlobalVar.cfg

.profile

Sample file for .profile $gvsqw_HbsBin/../etc/.profile:
Sample of lines to add to .profile of hbase users.
See the detailed help for .profile

.profile_confort

File to put aliases $gvsqw_HbsBin/../etc/.profile_confort:
Contains confort aliases of SQWareProduction for RDBMS hbase.

Note:
You can overload confort environnement $gvsqw_HbsBin/../etc/.profile_confort

with $gvsqw_HbsBin/../etc_cust/.profile_confort for all your environnement
or with $HOME/edtConfig/.profile_confort specifically for this user .

See the detailed help for .profile_confort

lib

sqwhbs_GatherIndicators.lib

shell functions library $gvsqw_HbsBin/../lib/sqwhbs_GatherIndicators.lib:
lib for sqwhbs_GatherIndicators.ksh

Note:
You can overload functions contained in $gvsqw_HbsBin/../lib/sqwhbs_GatherIndicators.lib

by function with same name in $gvsqw_HbsBin/../lib_cust/sqwhbs_GatherIndicators.lib for all your environnement
or by $HOME/sqwConfig/sqwhbs_GatherIndicators.lib specifically for this user .

See the detailed help for sqwhbs_GatherIndicators.lib

sqwhbs_GatherIndicators_Overload.lib

shell functions library $gvsqw_HbsBin/../lib/sqwhbs_GatherIndicators_Overload.lib:
lib for overLoad sqwhbs_GatherIndicators.ksh

Note:
You can overload functions contained in $gvsqw_HbsBin/../lib/sqwhbs_GatherIndicators_Overload.lib

by function with same name in $gvsqw_HbsBin/../lib_cust/sqwhbs_GatherIndicators_Overload.lib for all your environnement
or by $HOME/sqwConfig/sqwhbs_GatherIndicators_Overload.lib specifically for this user .

See the detailed help for sqwhbs_GatherIndicators_Overload.lib

sqwhbs_GenerateCreateInstance.lib

shell functions library $gvsqw_HbsBin/../lib/sqwhbs_GenerateCreateInstance.lib:
lib for generate create instance

Note:
You can overload functions contained in $gvsqw_HbsBin/../lib/sqwhbs_GenerateCreateInstance.lib

by function with same name in $gvsqw_HbsBin/../lib_cust/sqwhbs_GenerateCreateInstance.lib for all your environnement
or by $HOME/sqwConfig/sqwhbs_GenerateCreateInstance.lib specifically for this user .

See the detailed help for sqwhbs_GenerateCreateInstance.lib

sqwhbs_Global.lib

shell functions library $gvsqw_HbsBin/../lib/sqwhbs_Global.lib:
central lib of generic fonctions for hbase

Note:
You can overload functions contained in $gvsqw_HbsBin/../lib/sqwhbs_Global.lib

by function with same name in $gvsqw_HbsBin/../lib_cust/sqwhbs_Global.lib for all your environnement
or by $HOME/sqwConfig/sqwhbs_Global.lib specifically for this user .

See the detailed help for sqwhbs_Global.lib

sqwhbs_NetBackup.lib

shell functions library $gvsqw_HbsBin/../lib/sqwhbs_NetBackup.lib:
lib for sqwgen_NetBackup.ksh

Note:
You can overload functions contained in $gvsqw_HbsBin/../lib/sqwhbs_NetBackup.lib

by function with same name in $gvsqw_HbsBin/../lib_cust/sqwhbs_NetBackup.lib for all your environnement
or by $HOME/sqwConfig/sqwhbs_NetBackup.lib specifically for this user .

See the detailed help for sqwhbs_NetBackup.lib

sqwhbs_SetEnv.lib

shell functions library $gvsqw_HbsBin/../lib/sqwhbs_SetEnv.lib:
fonction to set env for $HBS_SID

Note:
You can overload functions contained in $gvsqw_HbsBin/../lib/sqwhbs_SetEnv.lib

by function with same name in $gvsqw_HbsBin/../lib_cust/sqwhbs_SetEnv.lib for all your environnement
or by $HOME/sqwConfig/sqwhbs_SetEnv.lib specifically for this user .

See the detailed help for sqwhbs_SetEnv.lib

tools

sqwhbs_GenerateCreateInstance.ksh

Script $gvsqw_HbsBin/../tools/sqwhbs_GenerateCreateInstance.ksh:
Generate create instance files See the detailed help for sqwhbs_GenerateCreateInstance.ksh

sqwhbs_SetEnv.ksh

Script $gvsqw_HbsBin/../tools/sqwhbs_SetEnv.ksh:
set env for $HBS_SID

Note:
You can replace the file in $gvsqw_HbsBin/../tools/sqwhbs_SetEnv.ksh

by $gvsqw_HbsBin/../tools_cust/sqwhbs_SetEnv.ksh for all your environnement
or by $HOME/edtConfig/sqwhbs_SetEnv.ksh specifically for this user .

See the detailed help for sqwhbs_SetEnv.ksh

help

Contains help files for -s option of bin scripts.

spm

SQL query toolkit
See the detailed help for spm

@@ Line 1: / Line 1: @@
-{{SQWareProductionPresSgbd|hbase|Hba|hba}}
+{{SQWareProductionPresSgbd|hbase|Hbs|hbs}}
 == bin ==
 === sqwhbs_GatherIndicators.ksh ===
-Script $gvsqw_HbaBin/bin/sqwhbs_GatherIndicators.ksh:<br>
+Script $gvsqw_HbsBin/bin/sqwhbs_GatherIndicators.ksh:<br>
 {{SQWareProduction_sqwhbs_GatherIndicators.ksh_presentation}}
 [[SQWareProduction:hbase:bin:sqwhbs_GatherIndicators.ksh|See the detailed help for sqwhbs_GatherIndicators.ksh]]<br><br>
 === sqwhbs_NetBackup.ksh ===
-Script $gvsqw_HbaBin/bin/sqwhbs_NetBackup.ksh:<br>
+Script $gvsqw_HbsBin/bin/sqwhbs_NetBackup.ksh:<br>
 {{SQWareProduction_sqwhbs_NetBackup.ksh_presentation}}
 [[SQWareProduction:hbase:bin:sqwhbs_NetBackup.ksh|See the detailed help for sqwhbs_NetBackup.ksh]]<br><br>
 === sqwhbs_ParallelRun.ksh ===
-Script $gvsqw_HbaBin/bin/sqwhbs_ParallelRun.ksh:<br>
+Script $gvsqw_HbsBin/bin/sqwhbs_ParallelRun.ksh:<br>
 {{SQWareProduction_sqwhbs_ParallelRun.ksh_presentation}}
 [[SQWareProduction:hbase:bin:sqwhbs_ParallelRun.ksh|See the detailed help for sqwhbs_ParallelRun.ksh]]<br><br>
 === sqwhbs_RunJob.ksh ===
-Script $gvsqw_HbaBin/bin/sqwhbs_RunJob.ksh:<br>
+Script $gvsqw_HbsBin/bin/sqwhbs_RunJob.ksh:<br>
 {{SQWareProduction_sqwhbs_RunJob.ksh_presentation}}
 [[SQWareProduction:hbase:bin:sqwhbs_RunJob.ksh|See the detailed help for sqwhbs_RunJob.ksh]]<br><br>
@@ Line 26: / Line 26: @@
 == etc ==
 === sqwhbs_CreInst_CrontabRef.cfg ===
-Configuration file $gvsqw_HbaBin/../etc/sqwhbs_CreInst_CrontabRef.cfg:<br>
+Configuration file $gvsqw_HbsBin/../etc/sqwhbs_CreInst_CrontabRef.cfg:<br>
 {{SQWareProduction_sqwhbs_CreInst_CrontabRef.cfg_presentation}}
 <br>
@@ Line 32: / Line 32: @@
 === sqwhbs_CreInst_Jobs.cfg ===
-Configuration file $gvsqw_HbaBin/../etc/sqwhbs_CreInst_Jobs.cfg:<br>
+Configuration file $gvsqw_HbsBin/../etc/sqwhbs_CreInst_Jobs.cfg:<br>
 {{SQWareProduction_sqwhbs_CreInst_Jobs.cfg_presentation}}
 <br>
@@ Line 38: / Line 38: @@
 === sqwhbs_GlobalVar.cfg ===
-Configuration file $gvsqw_HbaBin/../etc/sqwhbs_GlobalVar.cfg:<br>
+Configuration file $gvsqw_HbsBin/../etc/sqwhbs_GlobalVar.cfg:<br>
 {{SQWareProduction_sqwhbs_GlobalVar.cfg_presentation}}
 <br>
-{{SQWareProductionSurchargeVar|hbase|Hba|sqwhbs_GlobalVar.cfg}}
+{{SQWareProductionSurchargeVar|hbase|Hbs|sqwhbs_GlobalVar.cfg}}
 [[SQWareProduction:hbase:etc:sqwhbs_GlobalVar.cfg|See the detailed help for sqwhbs_GlobalVar.cfg]]<br><br>
 === .profile ===
-Sample file for .profile $gvsqw_HbaBin/../etc/.profile:<br>
+Sample file for .profile $gvsqw_HbsBin/../etc/.profile:<br>
 Sample of lines to add to .profile of hbase users.<br>
 [[SQWareProduction:hbase:etc:.profile|See the detailed help for .profile]]<br>
 === .profile_confort ===
-File to put aliases $gvsqw_HbaBin/../etc/.profile_confort:<br>
+File to put aliases $gvsqw_HbsBin/../etc/.profile_confort:<br>
 Contains confort aliases of SQWareProduction for RDBMS hbase.<br>
-{{SQWareProductionSurchargeConfort|hbase|Hba}}
+{{SQWareProductionSurchargeConfort|hbase|Hbs}}
 <br>
 [[SQWareProduction:hbase:etc:.profile_confort|See the detailed help for .profile_confort]]<br><br>
@@ Line 60: / Line 60: @@
 == lib ==
 === sqwhbs_GatherIndicators.lib ===
-shell functions library $gvsqw_HbaBin/../lib/sqwhbs_GatherIndicators.lib:<br>
+shell functions library $gvsqw_HbsBin/../lib/sqwhbs_GatherIndicators.lib:<br>
 {{SQWareProduction_sqwhbs_GatherIndicators.lib_presentation}}
-{{SQWareProductionSurchargeLib|hbase|Hba|sqwhbs_GatherIndicators.lib}}
+{{SQWareProductionSurchargeLib|hbase|Hbs|sqwhbs_GatherIndicators.lib}}
 <br>
 [[SQWareProduction:hbase:lib:sqwhbs_GatherIndicators.lib|See the detailed help for sqwhbs_GatherIndicators.lib]]<br><br>
 === sqwhbs_GatherIndicators_Overload.lib ===
-shell functions library $gvsqw_HbaBin/../lib/sqwhbs_GatherIndicators_Overload.lib:<br>
+shell functions library $gvsqw_HbsBin/../lib/sqwhbs_GatherIndicators_Overload.lib:<br>
 {{SQWareProduction_sqwhbs_GatherIndicators_Overload.lib_presentation}}
-{{SQWareProductionSurchargeLib|hbase|Hba|sqwhbs_GatherIndicators_Overload.lib}}
+{{SQWareProductionSurchargeLib|hbase|Hbs|sqwhbs_GatherIndicators_Overload.lib}}
 <br>
 [[SQWareProduction:hbase:lib:sqwhbs_GatherIndicators_Overload.lib|See the detailed help for sqwhbs_GatherIndicators_Overload.lib]]<br><br>
 === sqwhbs_GenerateCreateInstance.lib ===
-shell functions library $gvsqw_HbaBin/../lib/sqwhbs_GenerateCreateInstance.lib:<br>
+shell functions library $gvsqw_HbsBin/../lib/sqwhbs_GenerateCreateInstance.lib:<br>
 {{SQWareProduction_sqwhbs_GenerateCreateInstance.lib_presentation}}
-{{SQWareProductionSurchargeLib|hbase|Hba|sqwhbs_GenerateCreateInstance.lib}}
+{{SQWareProductionSurchargeLib|hbase|Hbs|sqwhbs_GenerateCreateInstance.lib}}
 <br>
 [[SQWareProduction:hbase:lib:sqwhbs_GenerateCreateInstance.lib|See the detailed help for sqwhbs_GenerateCreateInstance.lib]]<br><br>
 === sqwhbs_Global.lib ===
-shell functions library $gvsqw_HbaBin/../lib/sqwhbs_Global.lib:<br>
+shell functions library $gvsqw_HbsBin/../lib/sqwhbs_Global.lib:<br>
 {{SQWareProduction_sqwhbs_Global.lib_presentation}}
-{{SQWareProductionSurchargeLib|hbase|Hba|sqwhbs_Global.lib}}
+{{SQWareProductionSurchargeLib|hbase|Hbs|sqwhbs_Global.lib}}
 <br>
 [[SQWareProduction:hbase:lib:sqwhbs_Global.lib|See the detailed help for sqwhbs_Global.lib]]<br><br>
 === sqwhbs_NetBackup.lib ===
-shell functions library $gvsqw_HbaBin/../lib/sqwhbs_NetBackup.lib:<br>
+shell functions library $gvsqw_HbsBin/../lib/sqwhbs_NetBackup.lib:<br>
 {{SQWareProduction_sqwhbs_NetBackup.lib_presentation}}
-{{SQWareProductionSurchargeLib|hbase|Hba|sqwhbs_NetBackup.lib}}
+{{SQWareProductionSurchargeLib|hbase|Hbs|sqwhbs_NetBackup.lib}}
 <br>
 [[SQWareProduction:hbase:lib:sqwhbs_NetBackup.lib|See the detailed help for sqwhbs_NetBackup.lib]]<br><br>
 === sqwhbs_SetEnv.lib ===
-shell functions library $gvsqw_HbaBin/../lib/sqwhbs_SetEnv.lib:<br>
+shell functions library $gvsqw_HbsBin/../lib/sqwhbs_SetEnv.lib:<br>
 {{SQWareProduction_sqwhbs_SetEnv.lib_presentation}}
-{{SQWareProductionSurchargeLib|hbase|Hba|sqwhbs_SetEnv.lib}}
+{{SQWareProductionSurchargeLib|hbase|Hbs|sqwhbs_SetEnv.lib}}
 <br>
 [[SQWareProduction:hbase:lib:sqwhbs_SetEnv.lib|See the detailed help for sqwhbs_SetEnv.lib]]<br><br>
@@ Line 108: / Line 108: @@
 == tools ==
 === sqwhbs_GenerateCreateInstance.ksh ===
-Script $gvsqw_HbaBin/../tools/sqwhbs_GenerateCreateInstance.ksh:<br>
+Script $gvsqw_HbsBin/../tools/sqwhbs_GenerateCreateInstance.ksh:<br>
 {{SQWareProduction_sqwhbs_GenerateCreateInstance.ksh_presentation}}
 [[SQWareProduction:hbase:tools:sqwhbs_GenerateCreateInstance.ksh|See the detailed help for sqwhbs_GenerateCreateInstance.ksh]]<br><br>
 === sqwhbs_SetEnv.ksh ===
-Script $gvsqw_HbaBin/../tools/sqwhbs_SetEnv.ksh:<br>
+Script $gvsqw_HbsBin/../tools/sqwhbs_SetEnv.ksh:<br>
 {{SQWareProduction_sqwhbs_SetEnv.ksh_presentation}}
 <br>
-{{SQWareProductionSurchargeToo|hbase|Hba|sqwhbs_SetEnv.ksh}}
+{{SQWareProductionSurchargeToo|hbase|Hbs|sqwhbs_SetEnv.ksh}}
 [[SQWareProduction:hbase:tools:sqwhbs_SetEnv.ksh|See the detailed help for sqwhbs_SetEnv.ksh]]<br><br>