FAQ:NewInstancePostgreSQL: Difference between revisions

From Wiki_dbSQWare
Jump to navigation Jump to search
← Older edit
mNo edit summary
mNo edit summary
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Work In Progress
==Generalties==
==Generalties==
<br>
<br>
Line 11: Line 9:
Use the portions that apply to your environment, and choose either the automatic or manual method as desired<br>
Use the portions that apply to your environment, and choose either the automatic or manual method as desired<br>


===General operating procedure for integrating a new PostgreSQL instance===
===General operating procedure for integrating a new PostgreSQL instance===
There will potentially be one or two connection cases to handle.
There will potentially be one or two connection cases to handle.
* "local" for Unix machines (or those with Cygwin) where you'll be running PostgreSQL on the machine hosting the instance (SSH connection from dbsqware@sqwarebox).
* "local" for Unix machines (or those with Cygwin) where you'll be running PostgreSQL on the machine hosting the instance (SSH connection from dbsqware@sqwarebox).
* "remote" for cases where you don't have SSH access to the machine (Windows, RDS, etc.).
* "remote" for cases where you don't have SSH access to the machine (Windows, RDS, etc.).


Line 28: Line 26:
You will see that these steps can be handled one by one or in batches!
You will see that these steps can be handled one by one or in batches!


==Integration of an PostgreSQLinstance==
==Integration of an PostgreSQL  instance==


===Setting up the environment===
===Setting up the environment===
Line 36: Line 34:
=====Explanation of "DbAlias" (the unique dbSQWare key)=====
=====Explanation of "DbAlias" (the unique dbSQWare key)=====
The dbSQWare key must be unique and consistent with what has been configured for the gvsqw_DbAlias variable in sqwpg_GlobalVar.cfg!<br/>
The dbSQWare key must be unique and consistent with what has been configured for the gvsqw_DbAlias variable in sqwpg_GlobalVar.cfg!<br/>
We'll use a variable, POSTGRES_SID, which has no value for PostgreSQL but will identify the instance in dbSQWare.<br/>
We'll use a variable, PG_SID, which has no value for PostgreSQL but will identify the instance in dbSQWare.<br/>
By convention, we often use MYS "_" APPLI "_" ENV.<br/>
By convention, we often use MYS "_" APPLI "_" ENV.<br/>
Example: PG_THISAPPS_PRD or PG_THISOTHERAPPS_DEV ...<br/>
Example: PG_THISAPPS_PRD or PG_THISOTHERAPPS_DEV ...<br/>
But nothing is imposed; you choose what "suits you"! You just need to set the POSTGRES_SID variable in the PostgreSQL account profile with the desired value.<br/>
But nothing is imposed; you choose what "suits you"! You just need to set the PG_SID variable in the MySQL account profile with the desired value.<br/>


=====Explanation of statuses=====
=====Explanation of statuses=====
*An "open" instance will be handled in "local" mode, meaning we will connect with mysql locally on the machine hosting the instance (ssh access from dbsqware@sqwarebox), status "ON".<br/>
*An "open" instance will be handled in "local" mode, meaning we will connect with psql locally on the machine hosting the instance (ssh access from dbsqware@sqwarebox), status "ON".<br/>
*An instance where we don’t have local unix access will be managed in "remote" mode (like Windows, RDS, ...), status "DIST".<br/>
*An instance where we don’t have local unix access will be managed in "remote" mode (like Windows, RDS, ...), status "DIST".<br/>
*The "OFF" status makes the instance "disappear" from the web view.<br/>
*The "OFF" status makes the instance "disappear" from the web view.<br/>
Line 90: Line 88:
|-
|-
| Db Alias
| Db Alias
| Unique key that identifies the instance in dbSQWare (no value for PostgreSQL).
| Unique key that identifies the instance in dbSQWare (no value for PostgreSQL ).
|-
|-
| Dbms Name
| Dbms Name
Line 185: Line 183:
# From : root@my_postgres_host
# From : root@my_postgres_host
mkdir -p /home/postgres
mkdir -p /home/postgres
chown mysql:mysql /home/postgres
chown postgres:postgres /home/postgres
chmod 750 /home/postgres
chmod 750 /home/postgres
</syntaxhighlight>
</syntaxhighlight>
<syntaxhighlight lang="sh" line>
##Stop instance (service mysql/mariadb stop -- or -- systemctl stop mysql/mariadb)
usermod -d /home/postgres mysql -s /bin/bash
##Start instance (service mysql/mariadb start -- or -- systemctl start mysql/mariadb)


#Or directly vi /etc/passwd !
#Or directly vi /etc/passwd !
example : postgres:x:5432:5432::/home/postgres:/bin/bash
example : postgres:x:1004:989::/home/postgres:/bin/bash
</syntaxhighlight>
</syntaxhighlight>


Assign a passwd to the postgres user (otherwise the key exchange connection will not work!):<syntaxhighlight lang="sh" line>
Assign a passwd to the postgres user (otherwise the key exchange connection will not work!):<syntaxhighlight lang="sh" line>
# From : root@my_postgres_host
# From : root@my_postgres_host
passwd mysql
passwd postgres
</syntaxhighlight>
</syntaxhighlight>


Line 249: Line 242:
</syntaxhighlight>
</syntaxhighlight>


For "unique" user, it is the "standard" user. By default, we set it to "mysql", and it can be modified in SQWareCentral.
For "unique" user, it is the "standard" user. By default, we set it to "postgres", and it can be modified in SQWareCentral.


====Adding the dbSQWare environment to ~/.profile or ~/.bash_profile====
====Adding the dbSQWare environment to ~/.profile or ~/.bash_profile====
Line 269: Line 262:


#dbSQWare
#dbSQWare
export gvsqw_MysBin=$HOME/SQWareProduction/mysql/bin
export gvsqw_MysBin=$HOME/SQWareProduction/postgres/bin
export gvsqw_Env='PRD'
export gvsqw_Env='PRD'
lvsqw_IsTerminal=$(tty 2>&1 >/dev/null;echo $?)
lvsqw_IsTerminal=$(tty 2>&1 >/dev/null;echo $?)
Line 278: Line 271:
</syntaxhighlight>
</syntaxhighlight>


=====Adding the variable POSTGRES_SID=====
=====Adding the variable PG_SID=====
Add the POSTGRES_SID variable in the ~/.profile or ~/.bash_profile
Add the PG_SID variable in the ~/.profile or ~/.bash_profile
<syntaxhighlight lang="sh" line>
<syntaxhighlight lang="sh" line>
# From : postgres@my_postgres_host
# From : postgres@my_postgres_host
export POSTGRES_SID=MYS_INST_PRD
export PG_SID=PG_INST_PRD
export PGDATA=/pgdata/$PG_SID/data
</syntaxhighlight>
</syntaxhighlight>


Line 290: Line 284:
</syntaxhighlight>
</syntaxhighlight>


===="Manual" connection test to the PostgreSQL instance ("local" connection)====
===="Manual" connection test to the PostgreSQL instance ("local" connection)====
/!\ Only instances that will be in "ON" status!
/!\ Only instances that will be in "ON" status!
<syntaxhighlight lang="sh" line># From : dbsqware@sqwarebox
<syntaxhighlight lang="sh" line># From : dbsqware@sqwarebox
Line 298: Line 292:
===Specific Procedure for the "Remote" Port (DIST Status)===
===Specific Procedure for the "Remote" Port (DIST Status)===


====Creating a Specific PostgreSQL Account (dbSQWare) (For Remote Connections)====
====Creating a Specific PostgreSQL Account (dbSQWare) (For Remote Connections)====
Example for a "Remote" Connection
Adapt to your needs depending on what you want to do...
 
Example with superuser role
psql
<syntaxhighlight lang="sql" line>
create user dbsdba password 'DoNotSetThisPassword!';
alter user dbsdba with superuser;
</syntaxhighlight>
 
Example with just "read"
#>= Version 10, 11, ...
psql
<syntaxhighlight lang="sql" line>
create user dbsdba password 'DoNotSetThisPassword!';
grant pg_monitor, pg_read_all_settings, pg_read_all_stats to dbsdba;
</syntaxhighlight>
 
Example for an “RDS” env
<syntaxhighlight lang="sql" line>
<syntaxhighlight lang="sql" line>
grant all privileges on *.* to 'DBSDBA'@'%' IDENTIFIED BY 'DoNotSetThisPassword!' with grant option;
create user dbsdba password 'Don'tPutThisPassword!';
grant rds_superuser to dbsdba;
</syntaxhighlight>


/* or */
Added the line to enable remote connection (PostgreSQL ACL, enter the created user and the IP address of the central point "sqwarebox")
create user 'DBSDBA'@'%' IDENTIFIED BY 'DoNotSetThisPassword!';
<syntaxhighlight lang="sh" line>
grant all privileges on *.* to 'DBSDBA'@'%' with grant option;
postgres@srvdbpg01:/pgdata/PG_120/data (PG_120) $ vi pg_hba.conf
flush privileges;
host all dbsdba 192.168.230.20/24 md5
</syntaxhighlight>
</syntaxhighlight>


You can enter any username and password you like!<br/>
Reloading the conf after modifying pg_hba.conf
This account must be registered under the Unix account mysql@sqwarebox...<br/>
<syntaxhighlight lang="sh" line>
<syntaxhighlight lang="sh" line>
vi /home/postgres/.passwd_mysql
#Reloading the pg_ctl conf (if $PGDATA is set)
DbAlias;UserName;Passwd;HostName;Port
pg_ctl reload
MYS_INST_PRD;DBSDBA;DoNotSetThisPassword!;srvdbmys01;3306
or
pg_ctl reload -D $PGDATA
 
##Reloading the conf with a psql connection
psql
select pg_reload_conf();
</syntaxhighlight>
</syntaxhighlight>


====Test "remote" connection====
You can enter any username and password you like!<br/>
This account must be registered under the Unix account postgres@sqwarebox ...<br/>
<syntaxhighlight lang="sh" line>
<syntaxhighlight lang="sh" line>
# From : dbsqware@sqwarebox
postgres@sqwarebox:/home/postgres () $ cat .passwd_postgres
menu_pg TestInstConnectionDistNoMail GenLstInstanceNew
DbAlias;UserName;Passwd;HostName;Port
PG_120;dbsdba;myPassword;srvdbpg01;5432
</syntaxhighlight>
</syntaxhighlight>


Line 332: Line 351:


===Update SQWare Repository===
===Update SQWare Repository===
In the dbSQWare repository via the administration console, update the status of the new PostgreSQL instance (currently NEW) to the desired value (ON, DIST).
In the dbSQWare repository via the administration console, update the status of the new PostgreSQL instance (currently NEW) to the desired value (ON, DIST).


===Testing the "Recovery" of Indicators===
===Testing the "Recovery" of Indicators===
Line 349: Line 368:
You can check with the next chapter.<br/>
You can check with the next chapter.<br/>
<br/>
<br/>
Connect to the machine that has the “PostgreSQL” instance and type the following commands: <br/>
Connect to the machine that has the “PostgreSQL ” instance and type the following commands: <br/>
<syntaxhighlight lang="sh" line>
<syntaxhighlight lang="sh" line>
# From dbsqware@vmyswarebox
# From dbsqware@vmyswarebox
c MYS_INST_PRD
c PG_INST_PRD
</syntaxhighlight>
</syntaxhighlight>


<syntaxhighlight lang="sh" line>
<syntaxhighlight lang="sh" line>
#Source de l'env si multi-instance (from postgres@my_postgres_host)
#Source de l'env si multi-instance (from postgres@my_postgres_host)
e MYS_INST_PRD
e PG_INST_PRD
</syntaxhighlight>
</syntaxhighlight>


Line 368: Line 387:


====Explanations====
====Explanations====
The previous steps created some default files. ($HOME/MYS_INST_PRD/sqwConfig)
The previous steps created some default files. ($HOME/PG_INST_PRD/sqwConfig)
<syntaxhighlight lang="sh" line>
<syntaxhighlight lang="sh" line>
#Enter the instance name
#Enter the instance name
e MYS_INST_PRD
e PG_INST_PRD
cfg
cfg
cat sqwpg_Jobs.cfg
cat sqwpg_Jobs.cfg
Line 379: Line 398:
#############################################
#############################################
#IndicDba: Gather DBAs indicators
#IndicDba: Gather DBAs indicators
IndicDba:$gvsqw_MysBin/sqwpg_GatherIndicators.ksh -I MYS_INST_PRD
IndicDba:$gvsqw_MysBin/sqwpg_GatherIndicators.ksh -I PG_INST_PRD


#Dump: backup all databases with mysqlDump
#Dump: backup all databases with postgresDump
Dump:$gvsqw_MysBin/sqwpg_DumpAllDatabases.ksh -I MYS_INST_PRD -Exec
Dump:$gvsqw_MysBin/sqwpg_DumpAllDatabases.ksh -I PG_INST_PRD -Exec


#Optimize : optimize tables for all databases
#Optimize : optimize tables for all databases
Optimize:$gvsqw_MysBin/sqwpg_OptimizeAllDatabases.ksh -I MYS_INST_PRD -Exec
Optimize:$gvsqw_MysBin/sqwpg_OptimizeAllDatabases.ksh -I PG_INST_PRD -Exec


#Analyse : analyze tables for all databases
#Analyse : analyze tables for all databases
Analyze:$gvsqw_MysBin/sqwpg_AnalyzeAllDatabases.ksh -I MYS_INST_PRD -Exec
Analyze:$gvsqw_MysBin/sqwpg_AnalyzeAllDatabases.ksh -I PG_INST_PRD -Exec


#XtraFull: Backup full with XtraBackup
#XtraFull: Backup full with XtraBackup
XtraFull:$gvsqw_MysBin/sqwpg_XtraBackup.ksh -I MYS_INST_PRD -Type full -Exec
XtraFull:$gvsqw_MysBin/sqwpg_XtraBackup.ksh -I PG_INST_PRD -Type full -Exec


#XtraInc: Backup incremental with XtraBackup
#XtraInc: Backup incremental with XtraBackup
XtraInc:$gvsqw_MysBin/sqwpg_XtraBackup.ksh -I MYS_INST_PRD -Type inc -Exec
XtraInc:$gvsqw_MysBin/sqwpg_XtraBackup.ksh -I PG_INST_PRD -Type inc -Exec


#MariaFull: Backup full with MariaBackup
#MariaFull: Backup full with MariaBackup
MariaFull:$gvsqw_MysBin/sqwpg_MariaBackup.ksh -I MYS_INST_PRD -Type full -Exec
MariaFull:$gvsqw_MysBin/sqwpg_MariaBackup.ksh -I PG_INST_PRD -Type full -Exec


#MariaInc: Backup incremental with MariaBackup
#MariaInc: Backup incremental with MariaBackup
MariaInc:$gvsqw_MysBin/sqwpg_MariaBackup.ksh -I MYS_INST_PRD -Type inc -Exec
MariaInc:$gvsqw_MysBin/sqwpg_MariaBackup.ksh -I PG_INST_PRD -Type inc -Exec


#PurgeAlert: Rotate error log
#PurgeAlert: Rotate error log
PurgeAlert:$gvsqw_MysBin/sqwpg_PurgeAlert.ksh -I MYS_INST_PRD
PurgeAlert:$gvsqw_MysBin/sqwpg_PurgeAlert.ksh -I PG_INST_PRD
</syntaxhighlight>
</syntaxhighlight>


This is a configuration file used by the sqwpg_RunJob.ksh script to execute actions.<br/>
This is a configuration file used by the sqwpg_RunJob.ksh script to execute actions.<br/>
<br/>
<br/>
File for a default cron job $HOME/CrontabRef_MYS_INST_PRD:
File for a default cron job $HOME/CrontabRef_PG_INST_PRD:
<syntaxhighlight lang="sh" line>
<syntaxhighlight lang="sh" line>
## mm(0-59) hh(0-23) dd(1-31) MM(0-12) DAY(0-sunday, 1-monday, ...) command
## mm(0-59) hh(0-23) dd(1-31) MM(0-12) DAY(0-sunday, 1-monday, ...) command
## MYS_INST_PRD
## PG_INST_PRD
#Purge alert
#Purge alert
00 08 * * 0 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I MYS_INST_PRD -A PurgeAlert > $HOME/tmp/PurgeAlert_MYS_INST_PRD.log 2>&1'
00 08 * * 0 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A PurgeAlert > $HOME/tmp/PurgeAlert_PG_INST_PRD.log 2>&1'
# Dump
# Dump
00 20 * * * bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I MYS_INST_PRD -A Dump > $HOME/tmp/Dump_MYS_INST_PRD.log 2>&1'
00 20 * * * bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A Dump > $HOME/tmp/Dump_PG_INST_PRD.log 2>&1'
# Optimize
# Optimize
00 04 * * 6 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I MYS_INST_PRD -A Optimize > $HOME/tmp/Optimize_MYS_INST_PRD.log 2>&1'
00 04 * * 6 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A Optimize > $HOME/tmp/Optimize_PG_INST_PRD.log 2>&1'
# Analyze
# Analyze
00 05 * * 6 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I MYS_INST_PRD -A Analyze > $HOME/tmp/Analyze_MYS_INST_PRD.log 2>&1'
00 05 * * 6 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A Analyze > $HOME/tmp/Analyze_PG_INST_PRD.log 2>&1'
# XtraBackup
# XtraBackup
#00 21 * * 6  bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I MYS_INST_PRD -A XtraFull > $HOME/tmp/XtraFull_MYS_INST_PRD.log 2>&1'
#00 21 * * 6  bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A XtraFull > $HOME/tmp/XtraFull_PG_INST_PRD.log 2>&1'
#00 21 * * 1-5 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I MYS_INST_PRD -A XtraInc > $HOME/tmp/XtraInc_MYS_INST_PRD.log 2>&1'
#00 21 * * 1-5 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A XtraInc > $HOME/tmp/XtraInc_PG_INST_PRD.log 2>&1'
# MariaBackup
# MariaBackup
#00 21 * * 6  bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I MYS_INST_PRD -A MariaFull > $HOME/tmp/MariaFull_MYS_INST_PRD.log 2>&1'
#00 21 * * 6  bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A MariaFull > $HOME/tmp/MariaFull_PG_INST_PRD.log 2>&1'
#00 21 * * 1-5 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I MYS_INST_PRD -A MariaInc > $HOME/tmp/MariaInc_MYS_INST_PRD.log 2>&1'
#00 21 * * 1-5 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A MariaInc > $HOME/tmp/MariaInc_PG_INST_PRD.log 2>&1'
</syntaxhighlight>
</syntaxhighlight>


Line 431: Line 450:
Setting up cron:
Setting up cron:
<syntaxhighlight lang="sh" line>
<syntaxhighlight lang="sh" line>
mv $HOME/CrontabRef_MYS_INST_PRD $HOME/CrontabRef
mv $HOME/CrontabRef_PG_INST_PRD $HOME/CrontabRef
vi $HOME/CrontabRef
vi $HOME/CrontabRef
crontab $HOME/CrontabRef
crontab $HOME/CrontabRef
Line 457: Line 476:


Examples :
Examples :
  postgres@my_postgres_host:$HOME/admin/MYS_INST_PRD/logs/DumpAllDatabases (MYS_DBA_PRD) $ bin
  postgres@my_postgres_host:$HOME/admin/PG_INST_PRD/logs/DumpAllDatabases (MYS_DBA_PRD) $ bin
  total 100
  total 100
  drwxr-x---  2 dbsqware dba 4096 Sep  6 14:09 .
  drwxr-x---  2 dbsqware dba 4096 Sep  6 14:09 .
Line 487: Line 506:
  -rwxr-x---  1 dbsqware dba 3025 Aug 16 15:50 sqwpg_XtraBackup.ksh
  -rwxr-x---  1 dbsqware dba 3025 Aug 16 15:50 sqwpg_XtraBackup.ksh


  postgres@my_postgres_host:$HOME/SQWareProduction/mysql/bin (MYS_INST_PRD) $  
  postgres@my_postgres_host:$HOME/SQWareProduction/postgres/bin (PG_INST_PRD) $  
  ./sqwpg_DumpAllDatabases.ksh -h
  ./sqwpg_DumpAllDatabases.ksh -h
   
   
  Sourcing sqwpg_Global.lib v2023.06 SQWareProduction for PostgreSQL (dbSQWare) ...
  Sourcing sqwpg_Global.lib v2023.06 SQWareProduction for MySQL (dbSQWare) ...
   
   
  Usage: sqwpg_DumpAllDatabases.ksh [-h] -I instance [+ options]
  Usage: sqwpg_DumpAllDatabases.ksh [-h] -I instance [+ options]
Line 501: Line 520:
   
   
  PARAMETERS
  PARAMETERS
       -I  instance      : Target instance to mysqldump.
       -I  instance      : Target instance to postgresdump.
  OPTIONS
  OPTIONS
       -h  help          : Display the full usage.
       -h  help          : Display the full usage.
       -s                : Display samples of usage.
       -s                : Display samples of usage.
       -P  Nb threads    : Number of threads in parallel (by default 2).
       -P  Nb threads    : Number of threads in parallel (by default 2).
       -RD directory      : Directory to write dump file (default /backups/mysql/$POSTGRES_SID).
       -RD directory      : Directory to write dump file (default /backups/postgres/$PG_SID).
       -ID listdb        : List of databases to be dumped (by default all).
       -ID listdb        : List of databases to be dumped (by default all).
       -ED listdb        : List of databases to exclude from dump.
       -ED listdb        : List of databases to exclude from dump.
Line 516: Line 535:
       -AGE  Nb sec      : Nb of seconds between two checks of end (by default 10s).
       -AGE  Nb sec      : Nb of seconds between two checks of end (by default 10s).
       -FRT  return_code  : Force return code value on error.
       -FRT  return_code  : Force return code value on error.
       -Opt  option      : Additional options for mysqldump.
       -Opt  option      : Additional options for postgresdump.
       -Dist              : For distant connection to database (change $gvsqw_ConnectString to $gvsqw_DistConnectString ).
       -Dist              : For distant connection to database (change $gvsqw_ConnectString to $gvsqw_DistConnectString ).
       -AddMail email    : Email address to add at 'xxx'.
       -AddMail email    : Email address to add at 'xxx'.

Latest revision as of 13:59, 28 April 2025

Generalties


Warning:
Before anything, don't go further if you don't have read the first section "dbSQWare Concepts " !.

In this chapter, we will consider that SQWareCentral has been installed on dbsqware@sqwarebox.

Limits of this section

This chapter only deals with adding a new instance to a base that is already configured and functional.
Use the portions that apply to your environment, and choose either the automatic or manual method as desired

General operating procedure for integrating a new PostgreSQL instance

There will potentially be one or two connection cases to handle.

  • "local" for Unix machines (or those with Cygwin) where you'll be running PostgreSQL on the machine hosting the instance (SSH connection from dbsqware@sqwarebox).
  • "remote" for cases where you don't have SSH access to the machine (Windows, RDS, etc.).

Classic steps:

  • Setting up the environment
  • Checking system prerequisites for the "local" part (bash + rsync)
  • Manual tests
  • Deployment of the instance

You will see that these steps can be handled one by one or in batches!

Integration of an PostgreSQL instance

Setting up the environment

General

Explanation of "DbAlias" (the unique dbSQWare key)

The dbSQWare key must be unique and consistent with what has been configured for the gvsqw_DbAlias variable in sqwpg_GlobalVar.cfg!
We'll use a variable, PG_SID, which has no value for PostgreSQL but will identify the instance in dbSQWare.
By convention, we often use MYS "_" APPLI "_" ENV.
Example: PG_THISAPPS_PRD or PG_THISOTHERAPPS_DEV ...
But nothing is imposed; you choose what "suits you"! You just need to set the PG_SID variable in the MySQL account profile with the desired value.

Explanation of statuses
  • An "open" instance will be handled in "local" mode, meaning we will connect with psql locally on the machine hosting the instance (ssh access from dbsqware@sqwarebox), status "ON".
  • An instance where we don’t have local unix access will be managed in "remote" mode (like Windows, RDS, ...), status "DIST".
  • The "OFF" status makes the instance "disappear" from the web view.
  • The "NEW" status is used as an intermediate status between the beginning and the end of setting up a new instance.
  • All other statuses allow the instance to be displayed in the repository (especially "KEEP") but will not be taken into account by the automatic processes.


Resume of statuses :

Status Description
ON locale under Unix (the most common)
DIST remote-only management
KEEP temporary status, during maintenance for example or before complete deletion
NEW temporary status, during configuration
OFF visual removal of the instance (without deleting its configuration)
XXX not supported...

Instance declaration in SQWareRepository with SQWareWeb

We will declare the new instance from the SQWareWeb administration interface.
Admin dbSQWare


Declare the new instance with the status "NEW" (Add or Duplicate).
Ajout d'une instance Ajout par duplication d'une instance
Add a new instance by duplicate an other one. Explanation of the fields :

Fields Description
Db Alias Unique key that identifies the instance in dbSQWare (no value for PostgreSQL ).
Dbms Name Type of SGBD
Virtual Host Virtual Host (same as Host if not clustered)

.

Host Name Hostname of the instance
User Name User associated with the instance
Port Listening port of the instance.
Comments A brief description of what the instance hosts
Status Instance status (see above for explanations).
Contact A contact if needed
Environnement Instance environment (PRD, PPR, REC, DEV, TST, ...)
Client Used only for filtering (enter a client name and/or department and/or service...)
GlobalHost Free field in which the hypervisor host is often entered, for example.
Custom1 Free field 1
Custom2 Free field 2
Regenerate the reference files of SQWareCentral.

Type the following command which will generate the reference files: 

# From : dbsqware@sqwarebox
gen_all

Specific procedure for the "local" part (status ON)

/!\ Only instances that will be in the "ON" status !

Verification of system prerequisites for the "local" part (bash + rsync)

Since SQWareProduction is mainly written in bash shell and synchronized from SQWareCentral using rsync, we therefore need "bash" and "rsync" to be installed !

Verification

Adapt to your username and machine name.

bash: 

# From : postgres@my_postgres_host
type bash
#or
which bash

rsync: 

# From : postgres@my_postgres_host
type rsync
#or
which rsync
Installation

Adapt to your machine type (use sudo if you are not root).

RedHat / CentOS / ... : 

# From : root@my_postgres_host
yum install -y bash rsync
#or
dnf install -y bash rsync

Ubuntu / Debian / ... : 

# From : root@my_postgres_host
apt install -y bash rsync

Unix user modification

Adapt to your machine type (sudo if you're not root). Modify the homedir (not mandatory, but avoids ending up in /var/lib/postgresql): 

# From : root@my_postgres_host
mkdir -p /home/postgres
chown postgres:postgres /home/postgres
chmod 750 /home/postgres
  1. Or directly vi /etc/passwd !

example : postgres:x:1004:989::/home/postgres:/bin/bash </syntaxhighlight>

Assign a passwd to the postgres user (otherwise the key exchange connection will not work!):

# From : root@my_postgres_host
passwd postgres

Deployment of SSH key(s) from SQWareCentral to the target machine(s)

There are two ways to proceed:

  • An automatic one (from dbsqware@sqwarebox, but it requires you to know the password of the postgres Unix account)
  • A manual one that you apply on each postgres Unix account

/!\ The password of your postgres Unix account must have been initialized, otherwise key-based authentication will not work!

Deployment of SSH key(s): batch method
# From : dbsqware@sqwarebox
menu_pg GenDeplSshKeys_SQWareCentral GenLstInstanceNew

Verify that this is indeed the list you want to deploy, then choose option 1 ...
=> Enter the Unix password when prompted !

Deployment of SSH key(s): manual method
# From : postgres@my_postgres_host
if [ ! -r $HOME/.ssh/id_rsa ]
then
ssh-keygen -t rsa -N ''''''' -f $HOME/.ssh/id_rsa
fi
chmod go-w $HOME
echo "ssh-rsa AAAAB......XSPpdV11 dbsqware@sqwarebox" >>$HOME/.ssh/authorized_keys
chmod 700 $HOME/.ssh
chmod 600 $HOME/.ssh/authorized_keys

=> Put the correct key in the "echo" (the one from dbsqware@sqwarebox)

SSH connection test from SQWareCentral

# From : dbsqware@sqwarebox
menu_pg TestSshConnection GenLstInstanceNew

Test of prerequisites on the target host (bash+rsync) via SSH from SQWareCentral

# From : dbsqware@sqwarebox
menu_pg TestSysPrerequisites GenLstInstanceNew

Deployment of SQWareProduction

# From : dbsqware@sqwarebox

#If deployment on "unique" user
menu_pg DeplScripts GenLstUniqueNew

#If deployment on "instance" user, the one specified in SQWareRepository
menu_pg DeplScripts GenLstInstanceNew

For "unique" user, it is the "standard" user. By default, we set it to "postgres", and it can be modified in SQWareCentral.

Adding the dbSQWare environment to ~/.profile or ~/.bash_profile

Once again, there are two ways to proceed:

  • An automatic one (from dbsqware@sqwarebox)
  • A manual one that you apply on each postgres Unix account
Adding the dbSQWare environment: batch method
# From : dbsqware@sqwarebox
menu_pg AdddbSQWareProfile GenLstInstanceNew

Check that this is indeed the list you want to deploy, then choose 1 ...
=> In ~/.profile or ~/.bash_profile, change the following variable with the appropriate value: gvsqw_Env='XXX'

Adding the dbSQWare environment: manual method

Add the following lines to ~/.profile or ~/.bash_profile 

# From : postgres@my_postgres_host

#dbSQWare
export gvsqw_MysBin=$HOME/SQWareProduction/postgres/bin
export gvsqw_Env='PRD'
lvsqw_IsTerminal=$(tty 2>&1 >/dev/null;echo $?)
if [ "$lvsqw_IsTerminal" = "0" ] && [ -r $gvsqw_MysBin/../etc/.profile_confort ]
then
. $gvsqw_MysBin/../etc/.profile_confort
fi
Adding the variable PG_SID

Add the PG_SID variable in the ~/.profile or ~/.bash_profile 

# From : postgres@my_postgres_host
export PG_SID=PG_INST_PRD
export PGDATA=/pgdata/$PG_SID/data

Test sendmail (Non mandatory)

# From : dbsqware@sqwarebox
menu_pg TestSendmail GenLstInstanceNew

"Manual" connection test to the PostgreSQL instance ("local" connection)

/!\ Only instances that will be in "ON" status! 

# From : dbsqware@sqwarebox
menu_pg TestInstConnectionOnNoMail GenLstInstanceNew

Specific Procedure for the "Remote" Port (DIST Status)

Creating a Specific PostgreSQL Account (dbSQWare) (For Remote Connections)

Adapt to your needs depending on what you want to do...

Example with superuser role psql 

create user dbsdba password 'DoNotSetThisPassword!';
alter user dbsdba with superuser;

Example with just "read"

  1. >= Version 10, 11, ...

psql 

create user dbsdba password 'DoNotSetThisPassword!';
grant pg_monitor, pg_read_all_settings, pg_read_all_stats to dbsdba;

Example for an “RDS” env 

create user dbsdba password 'Don'tPutThisPassword!';
grant rds_superuser to dbsdba;

Added the line to enable remote connection (PostgreSQL ACL, enter the created user and the IP address of the central point "sqwarebox") 

postgres@srvdbpg01:/pgdata/PG_120/data (PG_120) $ vi pg_hba.conf
host all dbsdba 192.168.230.20/24 md5

Reloading the conf after modifying pg_hba.conf 

#Reloading the pg_ctl conf (if $PGDATA is set)
pg_ctl reload
or
pg_ctl reload -D $PGDATA

##Reloading the conf with a psql connection
psql
select pg_reload_conf();

You can enter any username and password you like!
This account must be registered under the Unix account postgres@sqwarebox ...

postgres@sqwarebox:/home/postgres () $ cat .passwd_postgres
DbAlias;UserName;Passwd;HostName;Port
PG_120;dbsdba;myPassword;srvdbpg01;5432

Starting the Instance

Generating the Default SQWareProduction Configuration

/!\ Only instances that will be in "ON" status! 

# From : dbsqware@sqwarebox
menu_pg GenDefConf_SQWareProduction GenLstInstanceNew

Update SQWare Repository

In the dbSQWare repository via the administration console, update the status of the new PostgreSQL instance (currently NEW) to the desired value (ON, DIST).

Testing the "Recovery" of Indicators

At this point, you can test the recovery of missing indicators (in principle, only the instances you have just integrated!). 

# From : dbsqware@sqwarebox
## for those with status "ON"
repind_mys
## for those with status "DIST"
repind_mys_dist

Operation (stats, backups, etc.) for "ON" instances

Generating SQWareProduction configs

If you followed the instance insertion procedure, this part is already done by step "menu_pg GenDefConf_SQWareProduction GenLstInstanceNew".
You can check with the next chapter.

Connect to the machine that has the “PostgreSQL ” instance and type the following commands: 

# From dbsqware@vmyswarebox
c PG_INST_PRD

#Source de l'env si multi-instance (from postgres@my_postgres_host)
e PG_INST_PRD

too
./sqwpg_GenerateCreateInstance.ksh -dbsOnly

Vérifier que ce qui est proposé est bon et tapez « y » si c’est le cas.
A ce moment, vous avez généré les fichiers de configuration par défaut de SQWareProduction.

Explanations

The previous steps created some default files. ($HOME/PG_INST_PRD/sqwConfig) 

#Enter the instance name
e PG_INST_PRD
cfg
cat sqwpg_Jobs.cfg

#############################################
#IndicDba: Gather DBAs indicators
IndicDba:$gvsqw_MysBin/sqwpg_GatherIndicators.ksh -I PG_INST_PRD

#Dump: backup all databases with postgresDump
Dump:$gvsqw_MysBin/sqwpg_DumpAllDatabases.ksh -I PG_INST_PRD -Exec

#Optimize : optimize tables for all databases
Optimize:$gvsqw_MysBin/sqwpg_OptimizeAllDatabases.ksh -I PG_INST_PRD -Exec

#Analyse : analyze tables for all databases
Analyze:$gvsqw_MysBin/sqwpg_AnalyzeAllDatabases.ksh -I PG_INST_PRD -Exec

#XtraFull: Backup full with XtraBackup
XtraFull:$gvsqw_MysBin/sqwpg_XtraBackup.ksh -I PG_INST_PRD -Type full -Exec

#XtraInc: Backup incremental with XtraBackup
XtraInc:$gvsqw_MysBin/sqwpg_XtraBackup.ksh -I PG_INST_PRD -Type inc -Exec

#MariaFull: Backup full with MariaBackup
MariaFull:$gvsqw_MysBin/sqwpg_MariaBackup.ksh -I PG_INST_PRD -Type full -Exec

#MariaInc: Backup incremental with MariaBackup
MariaInc:$gvsqw_MysBin/sqwpg_MariaBackup.ksh -I PG_INST_PRD -Type inc -Exec

#PurgeAlert: Rotate error log
PurgeAlert:$gvsqw_MysBin/sqwpg_PurgeAlert.ksh -I PG_INST_PRD

This is a configuration file used by the sqwpg_RunJob.ksh script to execute actions.

File for a default cron job $HOME/CrontabRef_PG_INST_PRD: 

## mm(0-59) hh(0-23) dd(1-31) MM(0-12) DAY(0-sunday, 1-monday, ...) command
## PG_INST_PRD
#Purge alert
00 08 * * 0 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A PurgeAlert > $HOME/tmp/PurgeAlert_PG_INST_PRD.log 2>&1'
# Dump
00 20 * * * bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A Dump > $HOME/tmp/Dump_PG_INST_PRD.log 2>&1'
# Optimize
00 04 * * 6 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A Optimize > $HOME/tmp/Optimize_PG_INST_PRD.log 2>&1'
# Analyze
00 05 * * 6 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A Analyze > $HOME/tmp/Analyze_PG_INST_PRD.log 2>&1'
# XtraBackup
#00 21 * * 6   bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A XtraFull > $HOME/tmp/XtraFull_PG_INST_PRD.log 2>&1'
#00 21 * * 1-5 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A XtraInc > $HOME/tmp/XtraInc_PG_INST_PRD.log 2>&1'
# MariaBackup
#00 21 * * 6   bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A MariaFull > $HOME/tmp/MariaFull_PG_INST_PRD.log 2>&1'
#00 21 * * 1-5 bash -c '. $HOME/.profile PG_INST_PRD;$gvsqw_MysBin/sqwpg_RunJob.ksh -I PG_INST_PRD -A MariaInc > $HOME/tmp/MariaInc_PG_INST_PRD.log 2>&1'

Startup

Setting up cron: 

mv $HOME/CrontabRef_PG_INST_PRD $HOME/CrontabRef
vi $HOME/CrontabRef
crontab $HOME/CrontabRef
crontab -l

Update the file CrontabRef  : 

crontab -l > $HOME/CrontabRef
cat $HOME/CrontabRef

Useful commands

alias : e DBALIAS => source env for the instance bin, etc, too ... => to go to the standard SQWareProduction tree cbin, cetc, ctoo ... => to go to the custom SQWareProduction tree log => SQWareProduction logs dmp => go to the backup paths

Some script options: -h => online help -s => online examples -Exec => when present; if not set, only outputs commands without executing them.

Examples : 

postgres@my_postgres_host:$HOME/admin/PG_INST_PRD/logs/DumpAllDatabases (MYS_DBA_PRD) $ bin
total 100
drwxr-x---  2 dbsqware dba 4096 Sep  6 14:09 .
drwxr-x--- 15 dbsqware dba  187 Aug 16 15:50 ..
-rwxr-x---  1 dbsqware dba 2957 Aug 16 15:50 sqwpg_AnalyzeAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 2514 Aug 16 15:50 sqwpg_AnalyzeDatabase.ksh
-rwxr-x---  1 dbsqware dba 2608 Aug 16 15:50 sqwpg_CheckAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 2257 Aug 16 15:50 sqwpg_CheckDatabase.ksh
-rwxr-x---  1 dbsqware dba 5418 Aug 16 15:50 sqwpg_CopyDumpAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 5324 Aug 16 15:50 sqwpg_CopyDumpDatabase.ksh
-rwxr-x---  1 dbsqware dba 5610 Sep  6 10:45 sqwpg_CopyMariaBackup.ksh
-rwxr-x---  1 dbsqware dba 5594 Sep  6 14:08 sqwpg_CopyXtraBackup.ksh
-rwxr-x---  1 dbsqware dba 3248 Aug 16 15:50 sqwpg_DumpAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 3541 Aug 16 15:50 sqwpg_DumpDatabase.ksh
lrwxrwxrwx  1 dbsqware dba   36 Jan  2  2022 sqwpg_ExecSQL.ksh -> ../../generic/bin/sqwgen_ExecSQL.ksh
-rwxr-x---  1 dbsqware dba 2455 Aug 16 15:50 sqwpg_GatherIndicators.ksh
-rwxr-x---  1 dbsqware dba 2454 Aug 16 15:50 sqwpg_GatherStructure.ksh
-rwxr-x---  1 dbsqware dba 3043 Aug 16 15:50 sqwpg_MariaBackup.ksh
lrwxrwxrwx  1 dbsqware dba   38 Jan  2  2022 sqwpg_NetBackup.ksh -> ../../generic/bin/sqwgen_NetBackup.ksh
-rwxr-x---  1 dbsqware dba 2959 Aug 16 15:50 sqwpg_OptimizeAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 2522 Aug 16 15:50 sqwpg_OptimizeDatabase.ksh
lrwxrwxrwx  1 dbsqware dba   40 Jan  2  2022 sqwpg_ParallelRun.ksh -> ../../generic/bin/sqwgen_ParallelRun.ksh
-rwxr-x---  1 dbsqware dba 2252 Aug 16 15:50 sqwpg_PurgeAlert.ksh
-rwxr-x---  1 dbsqware dba 3855 Aug 16 15:50 sqwpg_RestoreAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 3463 Aug 16 15:50 sqwpg_RestoreDatabase.ksh
-rwxr-x---  1 dbsqware dba 3825 Aug 16 15:50 sqwpg_RotateLogSlowQuery.ksh
lrwxrwxrwx  1 dbsqware dba   40 Jan  2  2022 sqwpg_RsyncBackup.ksh -> ../../generic/bin/sqwgen_RsyncBackup.ksh
lrwxrwxrwx  1 dbsqware dba   35 Jan  2  2022 sqwpg_RunJob.ksh -> ../../generic/bin/sqwgen_RunJob.ksh
-rwxr-x---  1 dbsqware dba 3025 Aug 16 15:50 sqwpg_XtraBackup.ksh

postgres@my_postgres_host:$HOME/SQWareProduction/postgres/bin (PG_INST_PRD) $ 
./sqwpg_DumpAllDatabases.ksh -h

Sourcing sqwpg_Global.lib v2023.06 SQWareProduction for MySQL (dbSQWare) ...

Usage: sqwpg_DumpAllDatabases.ksh [-h] -I instance [+ options]

DESCRIPTION
   sqwpg_DumpAllDatabases.ksh dump databases in parallel
SUPPORT
   MySql   supported versions: 5.0 <= v <= 8
   MariaDB supported versions: 5.5 <= v <= 11.6

PARAMETERS
     -I  instance       : Target instance to postgresdump.
OPTIONS
     -h  help           : Display the full usage.
     -s                 : Display samples of usage.
     -P  Nb threads     : Number of threads in parallel (by default 2).
     -RD directory      : Directory to write dump file (default /backups/postgres/$PG_SID).
     -ID listdb         : List of databases to be dumped (by default all).
     -ED listdb         : List of databases to exclude from dump.
     -IL likeclause     : Like clause to generate databases list (example: 'db%').
     -EL likeclause     : Like clause to exclude databases (example: 'nodb%').
     -HI Nb_Generations : Number of generations to keep (by default 2).
     -WCD  whereclause  : Where clause to generate databases list.
     -AGR  Nb sec       : Nb of seconds between two runs (by default 1s).
     -AGE  Nb sec       : Nb of seconds between two checks of end (by default 10s).
     -FRT  return_code  : Force return code value on error.
     -Opt  option       : Additional options for postgresdump.
     -Dist              : For distant connection to database (change $gvsqw_ConnectString to $gvsqw_DistConnectString ).
     -AddMail email     : Email address to add at 'xxx'.
     -SendReport        : Send execution log report.
     -NoMail            : Deactivate sendmail on error xxx (by default, send on error).
     -Locale   locale   : Force Locale for help display (fr,en).
     -Consistent        : Lock for consistent mode (by default, no lock).
     -Exec              : Execute commands (default, display generated commands)


Enjoy !

Retrieved from "https://wiki.dbsqware.com/index.php?title=FAQ:NewInstancePostgreSQL&oldid=32689"