Difference between revisions of "FAQ:NewInstanceMySQL"

From Wiki_dbSQWare
Jump to: navigation, search
m (→‎Mise en exploitation de l'instance)
m (→‎Explanations)
Line 389: Line 389:
 
A ce moment, vous avez généré les fichiers de configuration par défaut de SQWareProduction.
 
A ce moment, vous avez généré les fichiers de configuration par défaut de SQWareProduction.
  
−
====Explications====
+
====Explanations====
−
Les étapes précédentes ont créé certains fichiers par défaut. ($HOME/MYS_INST_PRD/sqwConfig)
+
The previous steps created some default files. ($HOME/MYS_INST_PRD/sqwConfig)
 
<syntaxhighlight lang="sh" line>
 
<syntaxhighlight lang="sh" line>
−
#Entrez le nom de l’instance
+
#Enter the instance name
 
e MYS_INST_PRD
 
e MYS_INST_PRD
 
cfg
 
cfg

Revision as of 10:38, 28 April 2025

Work In Progress

Contents

Generalties


Nuvola apps important.png
Warning:
Before anything, don't go further if you don't have read the first section "dbSQWare Concepts " !.

In this chapter, we will consider that SQWareCentral has been installed on dbsqware@sqwarebox.

Limits of this section

This chapter only deals with adding a new instance to a base that is already configured and functional.
Use the portions that apply to your environment, and choose either the automatic or manual method as desired

General operating procedure for integrating a new MySQL/MariaDB instance

There will potentially be one or two connection cases to handle.

  • "local" for Unix machines (or those with Cygwin) where you'll be running MySQL on the machine hosting the instance (SSH connection from dbsqware@sqwarebox).
  • "remote" for cases where you don't have SSH access to the machine (Windows, RDS, etc.).

Classic steps:

  • Setting up the environment
  • Checking system prerequisites for the "local" part (bash + rsync)
  • Manual tests
  • Deployment of the instance

You will see that these steps can be handled one by one or in batches!

Integration of an MySQL/MariaDBinstance

Setting up the environment

General

Explanation of "DbAlias" (the unique dbSQWare key)

The dbSQWare key must be unique and consistent with what has been configured for the gvsqw_DbAlias variable in sqwora_GlobalVar.cfg!
We'll use a variable, MYSQL_SID, which has no value for MySQL/MariaDB but will identify the instance in dbSQWare.
By convention, we often use MYS "_" APPLI "_" ENV.
Example: MYS_THISAPPS_PRD or MYS_THISOTHERAPPS_DEV ...
But nothing is imposed; you choose what "suits you"! You just need to set the MYSQL_SID variable in the MySQL account profile with the desired value.

Explanation of statuses
  • An "open" instance will be handled in "local" mode, meaning we will connect with mysql locally on the machine hosting the instance (ssh access from dbsqware@sqwarebox), status "ON".
  • An instance where we don’t have local unix access will be managed in "remote" mode (like Windows, RDS, ...), status "DIST".
  • The "OFF" status makes the instance "disappear" from the web view.
  • The "NEW" status is used as an intermediate status between the beginning and the end of setting up a new instance.
  • All other statuses allow the instance to be displayed in the repository (especially "KEEP") but will not be taken into account by the automatic processes.


Resume of statuses :

Status Description
ON locale under Unix (the most common)
DIST remote-only management
KEEP temporary status, during maintenance for example or before complete deletion
NEW temporary status, during configuration
OFF visual removal of the instance (without deleting its configuration)
XXX not supported...

Instance declaration in SQWareRepository with SQWareWeb

We will declare the new instance from the SQWareWeb administration interface.
Admin dbSQWare


Declare the new instance with the status "NEW" (Add or Duplicate).
Ajout d'une instance Ajout par duplication d'une instance
Add a new instance by duplicate an other one. Explanation of the fields :

Fields Description
Db Alias Unique key that identifies the instance in dbSQWare (no value for MySQL/MariaDB).
Dbms Name Type of SGBD
Virtual Host Virtual Host (same as Host if not clustered)

.

Host Name Hostname of the instance
User Name User associated with the instance
Port Listening port of the instance.
Comments A brief description of what the instance hosts
Status Instance status (see above for explanations).
Contact A contact if needed
Environnement Instance environment (PRD, PPR, REC, DEV, TST, ...)
Client Used only for filtering (enter a client name and/or department and/or service...)
GlobalHost Free field in which the hypervisor host is often entered, for example.
Custom1 Free field 1
Custom2 Free field 2
Regenerate the reference files of SQWareCentral.

Type the following command which will generate the reference files: 

1 # From : dbsqware@sqwarebox
2 gen_all

Specific procedure for the "local" part (status ON)

/!\ Only instances that will be in the "ON" status !

Verification of system prerequisites for the "local" part (bash + rsync)

Since SQWareProduction is mainly written in bash shell and synchronized from SQWareCentral using rsync, we therefore need "bash" and "rsync" to be installed !

Verification

Adapt to your username and machine name.

bash: 

1 # From : mysql@my_mysql_host
2 type bash
3 #ou
4 which bash

rsync: 

1 # From : mysql@my_mysql_host
2 type rsync
3 #ou
4 which rsync
Installation

Adapt to your machine type (use sudo if you are not root).

RedHat / CentOS / ... : 

1 # From : root@my_mysql_host
2 yum install -y bash rsync
3 #ou
4 dnf install -y bash rsync

Ubuntu / Debian / ... : 

1 # From : root@my_mysql_host
2 apt install -y bash rsync


Unix user modification

Adapt to your machine type (sudo if you're not root). Modify the homedir (not mandatory, but avoids ending up in /var/lib/mysql): 

1 # From : root@my_mysql_host
2 mkdir -p /home/mysql
3 chown mysql:mysql /home/mysql
4 chmod 750 /home/mysql

1 ##Stop instance (service mysql/mariadb stop -- or -- systemctl stop mysql/mariadb)
2 usermod -d /home/mysql mysql -s /bin/bash
3 ##Start instance (service mysql/mariadb start -- or -- systemctl start mysql/mariadb)
4 
5 #Or directly vi /etc/passwd !
6 example : mysql:x:1004:989::/home/mysql:/bin/bash

Assign a passwd to the mysql user (otherwise the key exchange connection will not work!):

1 # From : root@my_mysql_host
2 passwd mysql

Creating a specific MySQL/MariaDB account for dbSQWare (for local connection)

Adapt to your needs depending on what you want to do...

Example for local connection : 

1 grant all privileges on *.* to 'DBSDBA'@'localhost' IDENTIFIED BY 'DoNotSetThisPassword!' with grant option;
2 
3 /* or */
4 create user 'DBSDBA'@'localhost' IDENTIFIED BY 'DoNotSetThisPassword!';
5 grant all privileges on *.* to 'DBSDBA'@'localhost' with grant option;

1 # From : mysql@my_mysql_host
2 #Then, create the .my.cnf file to allow connection without using passwd
3 echo "[client]
4 password = DoNotSetThisPassword!
5 " >> $HOME/.my.cnf
6 chmod 600 $HOME/.my.cnf

Deployment of SSH key(s) from SQWareCentral to the target machine(s)

There are two ways to proceed:

  • An automatic one (from dbsqware@sqwarebox, but it requires you to know the password of the oracle Unix account)
  • A manual one that you apply on each oracle Unix account

/!\ The password of your oracle Unix account must have been initialized, otherwise key-based authentication will not work!

Deployment of SSH key(s): batch method
1 # From : dbsqware@sqwarebox
2 menu_mys GenDeplSshKeys_SQWareCentral GenLstInstanceNew

Verify that this is indeed the list you want to deploy, then choose option 1 ...
=> Enter the Unix password when prompted !

Deployment of SSH key(s): manual method
1 # From : mysql@my_mysql_host
2 if [ ! -r $HOME/.ssh/id_rsa ]
3 then
4 ssh-keygen -t rsa -N ''''''' -f $HOME/.ssh/id_rsa
5 fi
6 chmod go-w $HOME
7 echo "ssh-rsa AAAAB......XSPpdV11 dbsqware@sqwarebox" >>$HOME/.ssh/authorized_keys
8 chmod 700 $HOME/.ssh
9 chmod 600 $HOME/.ssh/authorized_keys

=> Put the correct key in the "echo" (the one from dbsqware@sqwarebox)

SSH connection test from SQWareCentral

1 # From : dbsqware@sqwarebox
2 menu_mys TestSshConnection GenLstInstanceNew

Test of prerequisites on the target host (bash+rsync) via SSH from SQWareCentral

1 # From : dbsqware@sqwarebox
2 menu_mys TestSysPrerequisites GenLstInstanceNew

Deployment of SQWareProduction

1 # From : dbsqware@sqwarebox
2 
3 #If deployment on "unique" user
4 menu_mys DeplScripts GenLstUniqueNew
5 
6 #If deployment on "instance" user, the one specified in SQWareRepository
7 menu_mys DeplScripts GenLstInstanceNew

For "unique" user, it is the "standard" user. By default, we set it to "mysql", and it can be modified in SQWareCentral.

Adding the dbSQWare environment to ~/.profile or ~/.bash_profile

Once again, there are two ways to proceed:

  • An automatic one (from dbsqware@sqwarebox)
  • A manual one that you apply on each mysql Unix account
Adding the dbSQWare environment: batch method
1 # From : dbsqware@sqwarebox
2 menu_mys AdddbSQWareProfile GenLstInstanceNew

Check that this is indeed the list you want to deploy, then choose 1 ...
=> In ~/.profile or ~/.bash_profile, change the following variable with the appropriate value: gvsqw_Env='XXX'

Adding the dbSQWare environment: manual method

Add the following lines to ~/.profile or ~/.bash_profile 

 1 # From : mysql@my_mysql_host
 2 
 3 #dbSQWare
 4 export gvsqw_MysBin=$HOME/SQWareProduction/mysql/bin
 5 export gvsqw_Env='PRD'
 6 lvsqw_IsTerminal=$(tty 2>&1 >/dev/null;echo $?)
 7 if [ "$lvsqw_IsTerminal" = "0" ] && [ -r $gvsqw_MysBin/../etc/.profile_confort ]
 8 then
 9 . $gvsqw_MysBin/../etc/.profile_confort
10 fi
Adding the variable MYSQL_SID

Add the MYSQL_SID variable in the ~/.profile or ~/.bash_profile 

1 # From : mysql@my_mysql_host
2 export MYSQL_SID=MYS_INST_PRD

Test sendmail (Non mandatory)

1 # From : dbsqware@sqwarebox
2 menu_mys TestSendmail GenLstInstanceNew

"Manual" connection test to the MySQL/MariaDB instance ("local" connection)

/!\ Only instances that will be in "ON" status! 

1 # From : dbsqware@sqwarebox
2 menu_mys TestInstConnectionOnNoMail GenLstInstanceNew

Specific Procedure for the "Remote" Port (DIST Status)

Creating a Specific MySQL/MariaDB Account (dbSQWare) (For Remote Connections)

Example for a "Remote" Connection 

1 grant all privileges on *.* to 'DBSDBA'@'%' IDENTIFIED BY 'DoNotSetThisPassword!' with grant option;
2 
3 /* or */
4 create user 'DBSDBA'@'%' IDENTIFIED BY 'DoNotSetThisPassword!';
5 grant all privileges on *.* to 'DBSDBA'@'%' with grant option;
6 flush privileges;

You can enter any username and password you like!
This account must be registered under the Unix account mysql@sqwarebox...

1  vi /home/mysql/.passwd_mysql
2  DbAlias;UserName;Passwd;HostName;Port
3  MYS_INST_PRD;DBSDBA;DoNotSetThisPassword!;srvdbmys01;3306

Test "remote" connection

1 # From : dbsqware@sqwarebox
2 menu_mys TestInstConnectionDistNoMail GenLstInstanceNew

Starting the Instance

Generating the Default SQWareProduction Configuration

/!\ Only instances that will be in "ON" status! 

1 # From : dbsqware@sqwarebox
2 menu_mys GenDefConf_SQWareProduction GenLstInstanceNew

Update SQWare Repository

In the dbSQWare repository via the administration console, update the status of the new MySQL/MariaDB instance (currently NEW) to the desired value (ON, DIST).

Testing the "Recovery" of Indicators

At this point, you can test the recovery of missing indicators (in principle, only the instances you have just integrated!). 

1 # From : dbsqware@sqwarebox
2 ## for those with status "ON"
3 repind_mys
4 ## for those with status "DIST"
5 repind_mys_dist

Operation (stats, backups, etc.) for "ON" instances

Generating SQWareProduction configs

If you followed the instance insertion procedure, this part is already done by step "menu_mys GenDefConf_SQWareProduction GenLstInstanceNew".
You can check with the next chapter.

Connect to the machine that has the “MySQL/MariaDB” instance and type the following commands: 

1 # From dbsqware@vmyswarebox
2 c MYS_INST_PRD

1 #Source de l'env si multi-instance (from mysql@my_mysql_host)
2 e MYS_INST_PRD

1 too
2 ./sqwmys_GenerateCreateInstance.ksh -dbsOnly

Vérifier que ce qui est proposé est bon et tapez « y » si c’est le cas.
A ce moment, vous avez généré les fichiers de configuration par défaut de SQWareProduction.

Explanations

The previous steps created some default files. ($HOME/MYS_INST_PRD/sqwConfig) 

1 #Enter the instance name
2 e MYS_INST_PRD
3 cfg
4 cat sqwmys_Jobs.cfg

 1 #############################################
 2 #IndicDba: Gather DBAs indicators
 3 IndicDba:$gvsqw_MysBin/sqwmys_GatherIndicators.ksh -I MYS_INST_PRD
 4 
 5 #Dump: backup all databases with mysqlDump
 6 Dump:$gvsqw_MysBin/sqwmys_DumpAllDatabases.ksh -I MYS_INST_PRD -Exec
 7 
 8 #Optimize : optimize tables for all databases
 9 Optimize:$gvsqw_MysBin/sqwmys_OptimizeAllDatabases.ksh -I MYS_INST_PRD -Exec
10 
11 #Analyse : analyze tables for all databases
12 Analyze:$gvsqw_MysBin/sqwmys_AnalyzeAllDatabases.ksh -I MYS_INST_PRD -Exec
13 
14 #XtraFull: Backup full with XtraBackup
15 XtraFull:$gvsqw_MysBin/sqwmys_XtraBackup.ksh -I MYS_INST_PRD -Type full -Exec
16 
17 #XtraInc: Backup incremental with XtraBackup
18 XtraInc:$gvsqw_MysBin/sqwmys_XtraBackup.ksh -I MYS_INST_PRD -Type inc -Exec
19 
20 #MariaFull: Backup full with MariaBackup
21 MariaFull:$gvsqw_MysBin/sqwmys_MariaBackup.ksh -I MYS_INST_PRD -Type full -Exec
22 
23 #MariaInc: Backup incremental with MariaBackup
24 MariaInc:$gvsqw_MysBin/sqwmys_MariaBackup.ksh -I MYS_INST_PRD -Type inc -Exec
25 
26 #PurgeAlert: Rotate error log
27 PurgeAlert:$gvsqw_MysBin/sqwmys_PurgeAlert.ksh -I MYS_INST_PRD

C’est un fichier de paramétrage qui est utilisé par le script sqwmys_RunJob.ksh pour exécuter des actions.

Fichier pour une cron par défaut $HOME/CrontabRef_MYS_INST_PRD : 

 1 ## mm(0-59) hh(0-23) dd(1-31) MM(0-12) DAY(0-sunday, 1-monday, ...) command
 2 ## MYS_INST_PRD
 3 #Purge alert
 4 00 08 * * 0 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwmys_RunJob.ksh -I MYS_INST_PRD -A PurgeAlert > $HOME/tmp/PurgeAlert_MYS_INST_PRD.log 2>&1'
 5 # Dump
 6 00 20 * * * bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwmys_RunJob.ksh -I MYS_INST_PRD -A Dump > $HOME/tmp/Dump_MYS_INST_PRD.log 2>&1'
 7 # Optimize
 8 00 04 * * 6 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwmys_RunJob.ksh -I MYS_INST_PRD -A Optimize > $HOME/tmp/Optimize_MYS_INST_PRD.log 2>&1'
 9 # Analyze
10 00 05 * * 6 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwmys_RunJob.ksh -I MYS_INST_PRD -A Analyze > $HOME/tmp/Analyze_MYS_INST_PRD.log 2>&1'
11 # XtraBackup
12 #00 21 * * 6   bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwmys_RunJob.ksh -I MYS_INST_PRD -A XtraFull > $HOME/tmp/XtraFull_MYS_INST_PRD.log 2>&1'
13 #00 21 * * 1-5 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwmys_RunJob.ksh -I MYS_INST_PRD -A XtraInc > $HOME/tmp/XtraInc_MYS_INST_PRD.log 2>&1'
14 # MariaBackup
15 #00 21 * * 6   bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwmys_RunJob.ksh -I MYS_INST_PRD -A MariaFull > $HOME/tmp/MariaFull_MYS_INST_PRD.log 2>&1'
16 #00 21 * * 1-5 bash -c '. $HOME/.profile MYS_INST_PRD;$gvsqw_MysBin/sqwmys_RunJob.ksh -I MYS_INST_PRD -A MariaInc > $HOME/tmp/MariaInc_MYS_INST_PRD.log 2>&1'

Mise en exploitation

Mise en place de la cron : 

1 mv $HOME/CrontabRef_MYS_INST_PRD $HOME/CrontabRef
2 vi $HOME/CrontabRef
3 crontab $HOME/CrontabRef
4 crontab -l

Mise à jour du fichier CrontabRef  : 

1 crontab -l > $HOME/CrontabRef
2 cat $HOME/CrontabRef

Commandes utiles

alias intéressants : 

e DBALIAS => source l'env pour l'instance
bin, etc, too ... => pour aller dans arbo SQWareProduction standard
cbin, cetc, ctoo ... => pour aller dans arbo SQWareProduction custom
log => logs SQWareProduction
dmp => on va dans les chemins de backup

Quelques options des scripts : 

-h => aide en ligne
-s => exemples en ligne
-Exec => lorsque elle est présente, si non settée ne sort que les commandes sans les exécuter

Exemples : 

mysql@my_mysql_host:$HOME/admin/MYS_INST_PRD/logs/DumpAllDatabases (MYS_DBA_PRD) $ bin
total 100
drwxr-x---  2 dbsqware dba 4096 Sep  6 14:09 .
drwxr-x--- 15 dbsqware dba  187 Aug 16 15:50 ..
-rwxr-x---  1 dbsqware dba 2957 Aug 16 15:50 sqwmys_AnalyzeAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 2514 Aug 16 15:50 sqwmys_AnalyzeDatabase.ksh
-rwxr-x---  1 dbsqware dba 2608 Aug 16 15:50 sqwmys_CheckAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 2257 Aug 16 15:50 sqwmys_CheckDatabase.ksh
-rwxr-x---  1 dbsqware dba 5418 Aug 16 15:50 sqwmys_CopyDumpAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 5324 Aug 16 15:50 sqwmys_CopyDumpDatabase.ksh
-rwxr-x---  1 dbsqware dba 5610 Sep  6 10:45 sqwmys_CopyMariaBackup.ksh
-rwxr-x---  1 dbsqware dba 5594 Sep  6 14:08 sqwmys_CopyXtraBackup.ksh
-rwxr-x---  1 dbsqware dba 3248 Aug 16 15:50 sqwmys_DumpAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 3541 Aug 16 15:50 sqwmys_DumpDatabase.ksh
lrwxrwxrwx  1 dbsqware dba   36 Jan  2  2022 sqwmys_ExecSQL.ksh -> ../../generic/bin/sqwgen_ExecSQL.ksh
-rwxr-x---  1 dbsqware dba 2455 Aug 16 15:50 sqwmys_GatherIndicators.ksh
-rwxr-x---  1 dbsqware dba 2454 Aug 16 15:50 sqwmys_GatherStructure.ksh
-rwxr-x---  1 dbsqware dba 3043 Aug 16 15:50 sqwmys_MariaBackup.ksh
lrwxrwxrwx  1 dbsqware dba   38 Jan  2  2022 sqwmys_NetBackup.ksh -> ../../generic/bin/sqwgen_NetBackup.ksh
-rwxr-x---  1 dbsqware dba 2959 Aug 16 15:50 sqwmys_OptimizeAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 2522 Aug 16 15:50 sqwmys_OptimizeDatabase.ksh
lrwxrwxrwx  1 dbsqware dba   40 Jan  2  2022 sqwmys_ParallelRun.ksh -> ../../generic/bin/sqwgen_ParallelRun.ksh
-rwxr-x---  1 dbsqware dba 2252 Aug 16 15:50 sqwmys_PurgeAlert.ksh
-rwxr-x---  1 dbsqware dba 3855 Aug 16 15:50 sqwmys_RestoreAllDatabases.ksh
-rwxr-x---  1 dbsqware dba 3463 Aug 16 15:50 sqwmys_RestoreDatabase.ksh
-rwxr-x---  1 dbsqware dba 3825 Aug 16 15:50 sqwmys_RotateLogSlowQuery.ksh
lrwxrwxrwx  1 dbsqware dba   40 Jan  2  2022 sqwmys_RsyncBackup.ksh -> ../../generic/bin/sqwgen_RsyncBackup.ksh
lrwxrwxrwx  1 dbsqware dba   35 Jan  2  2022 sqwmys_RunJob.ksh -> ../../generic/bin/sqwgen_RunJob.ksh
-rwxr-x---  1 dbsqware dba 3025 Aug 16 15:50 sqwmys_XtraBackup.ksh

mysql@my_mysql_host:$HOME/SQWareProduction/mysql/bin (MYS_INST_PRD) $ 
./sqwmys_DumpAllDatabases.ksh -h

Sourcing sqwmys_Global.lib v2023.06 SQWareProduction for MySQL (dbSQWare) ...

Usage: sqwmys_DumpAllDatabases.ksh [-h] -I instance [+ options]

DESCRIPTION
   sqwmys_DumpAllDatabases.ksh dump databases in parallel
SUPPORT
   MySql   supported versions: 5.0 <= v <= 8
   MariaDB supported versions: 5.5 <= v <= 11.6

PARAMETERS
     -I  instance       : Target instance to mysqldump.
OPTIONS
     -h  help           : Display the full usage.
     -s                 : Display samples of usage.
     -P  Nb threads     : Number of threads in parallel (by default 2).
     -RD directory      : Directory to write dump file (default /backups/mysql/$MYSQL_SID).
     -ID listdb         : List of databases to be dumped (by default all).
     -ED listdb         : List of databases to exclude from dump.
     -IL likeclause     : Like clause to generate databases list (example: 'db%').
     -EL likeclause     : Like clause to exclude databases (example: 'nodb%').
     -HI Nb_Generations : Number of generations to keep (by default 2).
     -WCD  whereclause  : Where clause to generate databases list.
     -AGR  Nb sec       : Nb of seconds between two runs (by default 1s).
     -AGE  Nb sec       : Nb of seconds between two checks of end (by default 10s).
     -FRT  return_code  : Force return code value on error.
     -Opt  option       : Additional options for mysqldump.
     -Dist              : For distant connection to database (change $gvsqw_ConnectString to $gvsqw_DistConnectString ).
     -AddMail email     : Email address to add at 'xxx'.
     -SendReport        : Send execution log report.
     -NoMail            : Deactivate sendmail on error xxx (by default, send on error).
     -Locale   locale   : Force Locale for help display (fr,en).
     -Consistent        : Lock for consistent mode (by default, no lock).
     -Exec              : Execute commands (default, display generated commands)


Enjoy !

Retrieved from "https://wiki.dbsqware.com/index.php?title=FAQ:NewInstanceMySQL&oldid=32662"