General:Concepts

From Wiki_dbSQWare
Revision as of 11:34, 28 April 2025 by Dtavernier (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Generalties

Limitations of this section

This section does not claim to handle all possible configuration cases of dbSQWare but will allow you to understand the general structure of the tool, interconnection between modules, the principles of personalization, …

For any kind of information, web site

(Visitez le wiki dbSQWare français, WikiFr)

Before running into the installation, please read the section « Base Installation », this will allow you to make a standard installation of dbSQWare.

dbSQWare, what is it ?

dbSQWare allows you to unite the use of databases Oracle, Sybase, SqlServer, MySQL, DB2, PostgreSQL, MongoDB, Cassandra, … thanks to a common and homogeneous base. The design of this platform provides great flexibility of use, of personalization and a unified approach to the exploitation and rendering of indicators on all types of DBMS managed by the tool.
It is neither an administration tool nor a monitoring tool (not a monitoring tool but a complement to it).

The product is intended (for its scripting part) for environments Unix/Linux only because it is essentially written in shell ksh and sql (MsSql too is fully managed from depuis unix thanks to a FreeTds connection, for more information http://www.freetds.org/). The web part, for its part, is written in PHP and javascript (jQuery).
For DBMS other than MsSql and installed on a Windows host, many functionalities (but not all) are accessible with a distant SQL connection (for the scripts supporting that).

List DBMs supported at this time :

  • Oracle
  • Sybase (ASE and RS)
  • MySQL
  • MsSql
  • MongoDB
  • DB2
  • PostgreSQL
  • Teradata
  • Cassandra
  • Adabas
  • Ingres


A search for “Homogeneity”

In use :

  • Consistent use, whatever the DBMS
  • Easy adaptation using configuration files and/or passing arguments
  • A single version of the tool for the entire fleet (synchronization by rsync)
  • The scripts adapt to the version of the DBMS processed (a single script for an action. Example : the script for oracle indicators supports from v7 to 19c multitenant architecture)
  • Launch without arguments
  • Online help (arguments and examples)
  • Dry run mode for validate the syntax (flag -Exec for the execution)


In the development and evolutions :

  • Homogeneous design for all DBMS
  • Code standardization (names, functions, structures, parsing of arguments, online help, …)
  • Generic multi-engine libraries
  • Modification of behavior by overloading libraries
  • Set of standard shell libraries that can be integrated into custom scripts


General structure of the tool

dbSQWare is composed of four complementary modules.

SQWareProduction is the local exploitation module (or remotely for 80% of the functionalities) of the DBMs. It makes possible to manage the operations in the broad sense of DBMS :

  • Backup
  • Restorations
  • Statistics
  • Alerts reporting
  • Job launch encapsulation
  • Running unix commands in parallel
  • ...

This module gather also a certain number of indicators which are uploaded to the repository of databases SQWareRepository. this module is made of a part for each managed DBMS and a core part including a set of settings and functions generic to all DBMS. the scripts are all written according to the same development standard (parsing of arguments, online help, mail on error, ascent of indicators into SQWareRepository, …).

SQWareRepository is the module for the management of the repository and the indicators in the database :
It allows you to manage the repository as well as the indicators stored in databases.
It's a MySQL database (>= 5.6) or MariaDB (>= 10.1), with generic tables as well as specific tables depending on the DBMS processed.

SQWareCentral is the central module of the tool. It allows all database servers to be managed from this single central point :

  • Centralized indicator collection,
  • Deployment of SQWareProduction via rsync,
  • Full-text search in repositories,
  • Simplified SSH connection to different instances in the repositories,
  • CMDB file generation,
  • Centralized indicator check.


This module is based, among other things, on the repository, SQWareRepository (dynamic generation of lists of instances to be processed, …).
It is composed of a core part, common to all DBMSs used and a specific module for each RDBMS (Oracle, Sybase, MsSql, MySQL, DB2, PostgreSQL, MongoDB, Cassandra, …).
Typically, the central point installation is done on a Rocky Linux VM, currently on release 9.5, 64 bits (2 vCPU and 4 Go RAM), or RHEL 9.5.

SQWareWeb is the web graphic rendering module for indicators :
It works with apache 2.x and is written in PHP (supported from 7.4 to 8.x), javascript (jQuery).
It allows the presentation of indicators and capacity planning in multiples forms :

  • Graphics (javascript)
  • Tables (with sorting, filtering and formatting locally on the browser)
  • Exports Excel, ...
  • ...


It is based entirely on the data contained in the database repository SQWareRepository.
No connection to client databases, the interface is only used for restoring indicators and configuring the repository.
The restitutions are presented in roughly the same way regardless of the RDBMS (except for its specificities), which makes navigation more pleasant and easier.
It is composed of a core part, common to all DBMSs used (template engine, graphic display, tables, ...) and a specific module for each RDBMS.

Standardization

One of the bases of “homogeneity” is standardization and genericity.

Naming convention

Trees

General tree structures (on SQWareCentral) : 

1 …/dbSQWare/SQWareProduction/… 	=> SQWareProduction module tree.
2 …/dbSQWare/SQWareRepository/… 	=> SQWareRepository module tree.
3 …/dbSQWare/SQWareCentral/… 	=> SQWareCentral module tree.
4 …/dbSQWare/SQWareWeb/… 		=> SQWareWeb module tree.


Then for the level below the modules there is a directory « generic » containing everything that is common to all RDBMS and one directory for each supported RDBMS.
This is what it looks like : 

 1 cassandra	=> specific for Cassandra
 2 db2		=> specific for DB2
 3 generic 	=> generic to all engines
 4 ingres		=> specific for Ingres
 5 mongodb 	=> specific for MongoDB
 6 mssql		=> specific for MsSql
 7 mysql		=> specific for MySQL
 8 oracle		=> specific for Oracle
 9 postgres	=> specific for PostgreSQL
10 sybase		=> specific for Sybase ASE
11 sybrep		=> specific for Sybase RS
12 teradata	=> specific for teradata
13 adabas          => specific for adabas


Then, for trees containing scripts, you will find directories like the following ; The directories of form *_cust are dedicated to customizing your environment, as much as possible, only touch the scripts and configuration files of these trees, This will make it easier for you to update the tool (unless there is a specific bug, there is no reason to touch the standard trees if you follow the customization recommendations).
Contents of directories : 

 1 bin		=> standard scripts 
 2 bin_cust	=> custom scripts for your environment
 3 etc		=> standard configuration files (global variables)
 4 etc_cust	=> custom configuration files (overload the standards)
 5 help		=> standard help files
 6 help_cust	=> custom help files for your environment
 7 lib		=> standard shell function libraries
 8 lib_cust	=> custom shell function libraries (overload the standards)
 9 tools		=> standard scripts used occasionally
10 tools_cust	=> standard scripts used occasionally for your environment
11 menu		=> standard menus (shell)
12 menu_cust	=> custom menus for your environment


So here, for example, is what it looks like for SQWareProduction Oracle and Generic: 

 1 SQWareProduction/oracle
 2 SQWareProduction/oracle/bin
 3 SQWareProduction/oracle/bin_cust
 4 SQWareProduction/oracle/etc
 5 SQWareProduction/oracle/etc_cust
 6 SQWareProduction/oracle/help
 7 SQWareProduction/oracle/help_cust
 8 SQWareProduction/oracle/lib
 9 SQWareProduction/oracle/lib_cust
10 SQWareProduction/oracle/menu
11 SQWareProduction/oracle/menu_cust
12 SQWareProduction/oracle/tools
13 SQWareProduction/oracle/tools_cust
14 SQWareProduction/generic
15 SQWareProduction/generic/bin
16 SQWareProduction/generic/bin_cust
17 SQWareProduction/generic/etc
18 SQWareProduction/generic/etc_cust
19 SQWareProduction/generic/lib
20 SQWareProduction/generic/lib_cust
21 SQWareProduction/generic/tools
22 SQWareProduction/generic/tools_cust


File (scripts's tree)

Most files are named with a prefix of the form : 

 1 sqwora_*	=> for Oracle
 2 sqwsyb_*	=> for Sybase ASE
 3 sqwrs_* 	=> for Sybase RS
 4 sqwmys_*	=> for Mysql
 5 sqwmsq_*	=> for Mssql
 6 sqwdb2_*	=> for DB2
 7 sqwpg_* 	=> for PostgreSQL
 8 sqwter_*	=> for Teradata
 9 sqwcas_*	=> for Cassandra
10 sqwada_*	=> for Adabas
11 sqwing_*	=> for Ingres
12 sqwgen_*	=> for generics one
13 sqwctl_*	=> for those from the module SQWareCentral


Most files are named with a suffix of the form : 

1 *.ksh	=> for scripts shell
2 *.cfg	=> for configuration files (global variables)
3 *.lib	=> for shell function libraries
4 *.hlp	=> for help files


File (web's tree)

Most files are named with a suffix of the form : 

1 *.php	=> For the PHP scripts
2 *.js	=> For the javascript scripts
3 *.chart	=> For the graphics configuration files
4 *.table	=> For the tables configuration files.


Contents of shell scripts

Generally, the convention followed in shell scripts is as follows : 

1 gvsqw_*{}	=> global variable initialized by the environment and/or a generic library
2 lvsqw_*{}	=> local variable initialized by the script and/or a specific library
3 gfsqw_*{}	=> function defined by a generic library
4 lfsqw_*{}	=> function defined by the script and/or a specific library


Database objects (SQWareRepository)

Objects prefixed by tsqw_% or isqw_% are generic to all engines. Objects prefixed by tsqwXXX_% or isqwXXX_% are specific for a particular DBMS (example : tsqwcas_% or isqwcas_% for Cassandra).

Names of generic objects : 

1 tsqw_%  	=> for tables
2 isqw_%		=> for indexes
3 isqw_%_u	=> for unique indexes 
4 isqw_%_pk	=> for primary keys


Names of specific object : 

1 tsqwXXX_%	=> for tables
2 isqwXXX_%	=> for indexes
3 isqwXXX_%_u	=> for unique indexes 
4 isqwXXX_%_pk	=> for primary keys


Specific example for Cassandra : 

1 tsqwcas_%	=> for  tables
2 isqwcas_%	=> for  indexes
3 isqwcas_%_u	=> for  indexes uniques
4 isqwcas_%_pk	=> for  primary keys


Overload principle

Please note, this section is an essential part of configuring dbSQWare without affecting future patches/upgrades. As explained before, you should not touch the files in the standard tree structures, but use the *_cust type directories to make any of your customizations.
When patching/upgrading, the dbSQWare_full_latest.tgz archive is extracted over the installed tree, which overwrites the standard files with the new version but keeps your customizations!
The basic principle of overloading is to create a file with the same name as in the standard (XXX) tree in the (XXX_cust) tree and redeclare the necessary variable(s)/library(s) there. You have a example in the next section.
Declare only what is strictly necessary for operation in your environment (no need to declare everything as for a database configuration).

Variables customizations

The file that is mainly updated for variables is sqwgen_GlobalVar.cfg, so we will take that as an example.
The following principle is based on $gvsqw_GenPath which represents the path of the executed script and on ${gvsqw_RdbmsRoot} which represents the trigram (in lowercase) of the DBMS on which the script runs and $gvsqw_RdbmsType, the DBMS-specific directory. See the paragraph on naming rules for actual file names.

General principle of cascading source (only if the files exist, 6 levels) : 

 1 # Generic standard files => for all DBMS
 2 $gvsqw_GenPath/../../generic/etc/sqwgen_GlobalVar.cfg
 3 
 4 # Generic custom files => for all DBMS
 5 $gvsqw_GenPath/../../generic/etc_cust/sqwgen_GlobalVar.cfg
 6 
 7 # For a non-standardized machine => specific only for this machine (all DBMS)
 8 $HOME/sqwConfig/sqwgen_GlobalVar.cfg
 9 
10 # Specific standard DBMS file => for the entire fleet
11 $gvsqw_GenPath/../../$gvsqw_RdbmsType/etc/sqw${gvsqw_RdbmsRoot}_GlobalVar.cfg
12 
13 # Specific custom DBMS file => for the entire fleet
14 $gvsqw_GenPath/../../$gvsqw_RdbmsType/etc_cust/sqw${gvsqw_RdbmsRoot}_GlobalVar.cfg
15 
16 # Specific DBMS file for a non-standardized machine => machine-specific
17 $HOME/sqwConfig/sqw${gvsqw_RdbmsRoot}_GlobalVar.cfg


Example for SQWareProduction Oracle installed in $HOME : 

 1 # Generic standard files => for the entire fleet
 2 $HOME/SQWareProduction/../../generic/etc/sqwgen_GlobalVar.cfg
 3 
 4 # Generic custom files => for all DBMS
 5 $HOME/SQWareProduction/../../generic/etc_cust/sqwgen_GlobalVar.cfg
 6 
 7 # For a non-standardized machine => machine-specific
 8 $HOME/sqwConfig/sqwgen_GlobalVar.cfg
 9 
10 # Specific standard DBMS file => for the entire fleet of this DBMS
11 $HOME/SQWareProduction/../../oracle/etc/sqwora_GlobalVar.cfg
12 
13 # Specific custom DBMS file => for the entire fleet of this DBMS
14 $HOME/SQWareProduction/../../oracle/etc_cust/sqwora_GlobalVar.cfg
15 
16 # Specific DBMS file for a non-standardized machine => machine-specific
17 $HOME/sqwConfig/sqwora_GlobalVar.cfg


Nuvola apps information.png
Note:
To see if all your custom files are supported by a script, before running it, do “export gvsqw_Debug=true” on your shell session and the script will log all loaded files (*.cfg and *.lib).


Customizing Shell Functions

In principle, except for advanced use cases of dbSQWare, you do not need to customize the functions, overloading variables or passing options is sufficient in the vast majority of cases (more than 99%).
Please note that customizing a function requires a minimum of shell skills and an impact analysis on the future operation of the scripts. We recommend that you seek assistance from support, at least for the first time.

Most tree scripts …/bin/ source (thanks to the function gfsqw_SourceOverLoadLibs) libraries with the same name as the script, replacing .ksh with .lib (example : yyy.ksh will source yyy.lib).

In the following explanation of cascading library sources, here is what the variables correspond to : 

1 gvsqw_GenPath => executed script's path
2 lvsqw_Lib     => library that we want to load (example sqwora_Global.lib)
3 lvsqw_LibGen  => generic name, we replace in lvsqw_Lib the DBMS-specific trigram by gen (example sqwgen_Global.lib)


General principle of cascading source (if files exist, 10 levels) : 

 1 # Standard generic library => for the entire park
 2 $gvsqw_GenPath/../../generic/lib/$lvsqw_LibGen
 3 
 4 # Custom generic library => for the entire park
 5 $gvsqw_GenPath/../../generic/lib_cust/$lvsqw_LibGen
 6 
 7 # Specific standard DBMS library => for the entire fleet
 8 $gvsqw_GenPath/../lib/$lvsqw_LibGen
 9 
10 # Specific custom DBMS library => for the entire fleet
11 $gvsqw_GenPath/../lib_cust/$lvsqw_LibGen
12 
13 # For a non-standardized machine => machine-specific
14 $HOME/sqwConfig/$lvsqw_LibGen
15 
16 # Standard generic library => for the entire park
17 $gvsqw_GenPath/../../generic/lib/$lvsqw_Lib
18 
19 # Custom generic library => for the entire park
20 $gvsqw_GenPath/../../generic/lib_cust/$lvsqw_Lib
21 
22 # Specific standard DBMS library => for the entire fleet
23 $gvsqw_GenPath/../lib/$lvsqw_Lib
24 
25 # Specific custom DBMS library => for the entire fleet
26 $gvsqw_GenPath/../lib_cust/$lvsqw_Lib
27 
28 # For a non-standardized machine => machine-specific
29 $HOME/sqwConfig/$lvsqw_Lib


Example for sqwora_ParallelRun.ksh from SQWareProduction Oracle installed in $HOME : 

 1 # Standard generic library => for the entire park
 2 $HOME/SQWareProduction/../../generic/lib/sqwgen_ParallelRun.lib
 3 
 4 # Custom generic library => for the entire park
 5 $HOME/SQWareProduction/../../generic/lib_cust/sqwgen_ParallelRun.lib
 6 
 7 # Specific standard DBMS library => for the entire fleet
 8 $HOME/SQWareProduction/../lib/sqwgen_ParallelRun.lib
 9 
10 # Specific custom DBMS library => for the entire fleet
11 $HOME/SQWareProduction/../lib_cust/sqwgen_ParallelRun.lib
12 
13 # For a non-standardized machine => machine-specific
14 $HOME/sqwConfig/sqwgen_ParallelRun.lib
15 
16 # Standard generic library => for the entire park
17 $HOME/SQWareProduction/../../generic/lib/sqwora_ParallelRun.lib
18 
19 # Custom generic library => for the entire park
20 $HOME/SQWareProduction/../../generic/lib_cust/sqwora_ParallelRun.lib
21 
22 # Specific standard DBMS library => for the entire fleet
23 $HOME/SQWareProduction/../lib/sqwora_ParallelRun.lib
24 
25 # Specific custom DBMS library => for the entire fleet
26 $HOME/SQWareProduction/../lib_cust/sqwora_ParallelRun.lib
27 
28 # For a non-standardized machine => machine-specific
29 $HOME/sqwConfig/sqwora_ParallelRun.lib


Nuvola apps information.png
Note:
To see if all your custom files are supported by a script, before running it, do “export gvsqw_Debug=true” on your shell session and the script will log all loaded files (cfg and lib).


Interoperability of modules

Communications between modules

From SQWareProduction

SQWareProduction is the only module that connects directly to client databases.
It then allows in particular to upload a certain number of indicators in SQWareRepository (via a local buffer).
It is SQWareCentral which comes to fetch the data in the SQWareProduction buffer by rsync.

From SQWareRepository

SQWareRepository is a MariaDB database, no flow is at his initiative.

From SQWareCentral

SQWareCentral connects to SQWareRepository to generate the lists of instances to be processed.
It allows you to launch SQWareProduction commands through an ssh connection.
It also allows the deployment of SQWareProduction scripts as well as the recovery of SQWareProduction traces by rsync.
Prerequisites :

  • Rsync package on central point and SQWareProduction clients
  • Ssh stream to SQWareProduction clients (TCP port 22 to SQWareProduction clients)
  • Flow to the MariaDB repository (usually both modules are on the same machine)


FromSQWareWeb

SQWareWeb connects to SQWareRepository which contains the indicators (no connection to client databases).
Prerequisites :

  • Packages httpd php php-pdo php-mysql
  • Flow to the MariaDB repository (usually both modules are on the same machine)


Flow Matrix

This section does not cover all possible cases, but should allow you to open the necessary streams in the presence of a firewall.

Type Number Source Destination Port (current) Commentary
always x SQWareCentral SQWareProduction 22 Rsync sources and remote execution
Specific Oracle x SQWareProduction
on CentralHost		 OracleHost 1521 When you want to collect AWR reports from a central location.
As many times as there are Oracle databases.
Specific SQL Server x SQWareProduction
on CentralHost		 MsSqlHost 1433 When you want to manage all MsSql from the central point.
Can also be installed on a different machine than the central bridge.
... x CentralHost all DBMS 1521, 1433, 5432, ... When you want to access instances from the central point in SQL.
This requirement is purely DBA and not necessary for dbSQWare.
Distribued 1 SQWareCentral SQWareRepository 3306 When the MariaDB repository is not on the central point (rare and not recommended)
Distribued 1 SQWareWeb SQWareRepository 3306 When the repo's MariaDB database is not on the central point
or SQWareWeb is not on the central point (both rare and not recommended)

Useful links

Here is some useful links for dbSQWare:


Join the dbSQWare group on Linkedin:
http://www.linkedin.com/groups?gid=3683269

Retrieved from "https://wiki.dbsqware.com/index.php?title=General:Concepts&oldid=32666"