LDAP

LDAP - Lightweight Directory Access Protocol

Runs one layer above TCP/IP stack. 

Based on client server model. 

Facilitates centralized user management. Eliminates user creation on each and every server. 

The users and usre profiles can be created on the LDAP server and accessed from all the LDAP clients.

The directory is similar to a database, but tends contain descriptive, attribute based information. 

The info in directory is read more often than it is written, hence, directory is tuned to give quick response to high volume lookup/search operations.

The data organization in LDAP is k/a LDAP Directory tree.

LDAP server daemon is k/a slapd.

slapd supports variety of backends - BDB (preferred) or LDBM.

BDB - utilises Berkley DB

LDBM - utilises Berkley DB or GDBM.

To import/export directory information between LDAP based directory servers, or to describe set of changes to be applied to the directory, file format k/a LDIF (LDAP Data Interchange Format) is used.

LDIF stored info in object hierarchies. 

An LDIF entry looks like - 

dn: o=TUDelft, c=NL

o: TUDelft

objectclass: organization

dn: cn=Luiz Malere, o=TUDelft, c=NL

cn: Luiz Malere

sn: Malere

mail: malere@yahoo.com

objectclass: person

Each entry is uniquely identified by a distinguished name. DN consists of name of entry plus the path of names tracing the entry back to the top of the directory hierarchy. 

objectclass defines collection of attributes that can be used to define an entry. LDAP provides following basic object classes - 

- Groups 

- Locations

- Organizations

- People

An entry can belong to more than one objectclass. 

Directory data is represented as attribute-value pairs. Any information has an associated descriptive attribute. Eg. Common Name attribute is used to store name of person - cn: Luiz Malere

Linux OS initialization

Init

System V Init daemon is the first process (pid=1) to start on linux and continues running until the system shuts down. It is the eventual parent of any process you start on linux.

System V was the first commercial Unix OS.

The primary function of init daemon is to launch the userspace, in other words, initialise the processes for linux startup by reading the information from the /etc/inittab file.

The file defines 3 important items for init process -

1. system's default run level
2. what processes to start, monitor and restart if they terminate
3. what actions to take when the system enters a new run level

each entry in the /etc/inittab has following fields -
   id:rstate:action:process

   id - unique identifier of entry
   rstate - list of run levels to which this entry applies. eg. 1 or 13 or 234
   action - how the process is to be run. possible values - initdefault, sysinit, 

                              boot, boot wait, wait, respawn

 process - command/script to run 

Run Levels - 

 0 - Power Down State

 1 - Single User State

 2 - Local Multiuser without remote network (eg NFS)

 3 - Full multiuser with network

 4 - Not used

 5 - Full multiuser with network and GUI 

 6 - System reboot

To check current run level - 

who -r

Once initd does it's job on initialising the userspace, add-on service managers like service and chkconfig are used to make the service management easier.

init diligently serves the master but is constrained in that it starts the processes serially, performs dependency checks on each one and waits for daemons to start so more daemons can start and so on, which leads to a slower than can be boot time.

Unix Domain Sockets

Unix Domain Socket is a communication endpoint for exchanging data between processes executing on the same host OS. Valid socket types on Unix are - SOCK_STREAM (kinda TCP), SOCK_DGRAM (kinda UDP) and SOCK_SEQPACKET (kinda SCTP).

API of Unix Domain sockets is similar to that of Internet socket, but rather than using underlying network protocol, all communication happens inside OS kernel. Processes reference sockets as FS inodes, so two processes can communicate by opening the same socket. 

$ netstat -a --protocol=unix

What the Swap (space) ?

If the system needs more memory and the RAM is full, inactive pages are moved from memory to swap space (rather than being discarded).

Swap space is portion of hard drive used for virtual memory.

Usually this is a dedicated partition aka swap partition.
It can also be a special file and is used as a swap FS.

Sizing swap space -
 2GB RAM - 4GB swap space
 Say for 250 GB RAM, 10-16GB of swap space is fine.

Create Swap File with dd -

# dd if=/dev/zero of=/swapfile bs=1024 count=5120
# chmod 0600 /swapfile
# mkswap /swapfile
# swapon /swapfile

add entry to /etc/fstab to enable loading the swap file on boot

# vi /etc/fstab
~ /swapfile             swap swap defaults 0 0

# swapon -a

Check status of swap FS -

# swapon -s
# cat /proc/swaps
# free -m

Disable Swap -

# swapoff /swapfile
# swapon -s

Create Swap FS using a partition -

get a partition using disk/parted, say, /dev/sdb

# partprobe /dev/sdb
# mkswap /dev/sdb2
# swapon /dev/sdb2 

# free -m

vm_swappiness - 

Swappiness is property of linux kernel that changes the balance between swapping out runtime memory as opposed to dropping pages from the system page cache.

Can be set to values between 0-100.

Low value means kernel will try to avoid swapping as much as possible, thereby improving application responsiveness.

Higher value will make kernel swap as aggressively as possible, leaving more memory free for caches.

DNS Service (SRV) Records

Service record type is basically a service location record.
Points to the location of a key network resource.

Wiki -
A service record is a specification of data in DNS, defining the location i.e. the hostname and port number, of servers for the specified services.
Generalized service location record, used for newer protocols instead of creating protocol-specific records such as MX.

SRV records are required for AD Domain Controllers.
  SRV records for Domain controllers are created as a part of process of promoting a server to the domain controller.

Other services that require SRV records -

- Kerberos
- LDAP

created in _msdcs.<domainname>zone

_kerberos record has a protocol of TCP, port 88
_ldap record uses port 389.

SRV records are created when a DNS server is created/installed
MS exchange also uses SRV records.

MS exchange SRV records allow Outlook and other clients to automatically discover the Exchange server.

Oracle Hang Management

Hang Manager (HM) has existed since 10.2.0.1. The main goal of Hang Manager is to reliably detect, and if hang resolution is enabled, resolve hangs in a timely manner. Over various releases, Hang Manager has been enhanced along with the wait event infrastructure on which it relies. However, it is only in 11.2.0.2 that Hang Manager actually resolves any hangs by terminating sessions and/or processes. This is the default operation in 11.2.0.2. Hang Manager will not terminate an instance unless the resolution scope, which is controlled by the initialization parameter _HANG_RESOLUTION_SCOPE, is set to INSTANCE. By default, this parameter is set to PROCESS.

Hang Manager is only active when RAC is enabled, that is, CLUSTER_DATABASE = TRUE. Also, the parameters _HANG_DETECTION_ENABLED and _HANG_RESOLUTION_SCOPE control hang detection and resolution respectively. Currently, Hang Manager does not operate on a non-RAC enabled instance.

DIA0 process is responsible for hang detection and deadlock resolution.

NOT TO BE CONFUSED WITH 

DIAG process: diagnosability process, performs diagnostic dumps and executes oradebug commands

When DIA0 process kills off a process for hang management,

ORA 32701 is thrown, and this indicates that "Hang Management" is in use.

Hang can occur due to variety of reasons:

 - application related

 - resource related

 - internal blockage in Oracle kernel

 - other reason

'Resolvable Hangs' int the incident trace file indicate one or more hangs that were found and identify the final blocking session and instance on which they occurred. If the current hang resolution state is 'PROCESS',

any hangs requiring session or process termination are automatically resolved by DIA0 process

 >11.2.0.2, Hang Management will try to resolve the hang by terminating the ultimate blocker

If 32701 occured, it is important to determine the cause of the hang.

In the example below, the DIA0 terminated session sid:3768 with serial # 19475 (ospid:73677)

 for terminating blocker process (ospid: 73677 sid: 3768 ser#: 19475) of hang with

-----------------------------------------------------------------------------------------

ORA-32701: Possible hangs up to hang ID=16 detected

Incident details in: /ade/b/3518352270/oracle/log/diag/rdbms/mainrac/mainrac1/incident/incdir_81/mainrac1_dia0_20207_i81.trc

DIA0 requesting termination of session sid:44 with serial # 1 (ospid:5964) on instance 2

  due to a GLOBAL, HIGH confidence hang with ID=16.

  Hang Resolution Reason: Although hangs of this root type are typically

  self-resolving, the previously ignored hang was resolved.

DIA0 could not resolve a GLOBAL, HIGH confidence hang with ID=16.

-----------------------------------------------------------------------------------------

-----------------------------------------------------------------------------------------

Errors in file /u01/app/oracle/diag/rdbms/dm02cdb1/orcl2/trace/orcl2_dia0_87406.trc  (incident=31458) (PDBNAME=CDB$ROOT):

ORA-32701: Possible hangs up to hang ID=86 detected

Incident details in: /u01/app/oracle/diag/rdbms/dm02cdb1/orcl2/incident/incdir_31458/orcl2_dia0_87406_i31458.trc

Thu Dec 31 21:59:43 2015

Sweep [inc][31458]: completed

Sweep [inc2][31458]: completed

Thu Dec 31 21:59:44 2015

DIA0 terminating blocker (ospid: 73677 sid: 3768 ser#: 19475) of hang with ID = 86

    requested by master DIA0 process on instance 2

    Hang Resolution Reason: Previously ignored hang - hang did not self-resolve

   as was expected.  Automatic hang resolution will be performed.

    by terminating session sid:3768 with serial # 19475 (ospid:73677)

DIA0 successfully terminated session sid:3768 with serial # 19475 (ospid:73677) with status 0.

DIA0 successfully resolved a GLOBAL, HIGH confidence hang with ID=86.

-----------------------------------------------------------------------------------------

*** 2015-12-31 21:59:40.213

Resolvable Hangs in the System

                     Root       Chain Total               Hang              

  Hang Hang          Inst Root  #hung #hung  Hang   Hang  Resolution        

    ID Type Status   Num  Sess   Sess  Sess  Conf   Span  Action            

 ----- ---- -------- ---- ----- ----- ----- ------ ------ -------------------

    86 HANG RSLNPEND    2  3768     3     3   HIGH GLOBAL Terminate Process 

 Hang Resolution Reason: Previously ignored hang - hang did not self-resolve

    as was expected.  Automatic hang resolution will be performed.

  Inst  Sess   Ser             Proc  Wait    Wait

   Num    ID    Num      OSPID  Name Time(s) Event

  ----- ------ ----- --------- ----- ------- -----

        PDBID PDBNm          

        ----- ---------------

      1   4528 13156     28552    FG     704 library cache lock

           11 MODERNIZATION_DEV_12

      1   2770 52684     22975    FG      50 library cache pin

           11 MODERNIZATION_DEV_12

      2   3768 19475     73677    FG    1183 not in wait

           11 MODERNIZATION_DEV_12

Dumping process info of pid[65.73677] (sid:3768, ser#:19475)

    requested by master DIA0 process on instance 2.

-----------------------------------------------------------------------------------------

*** 2015-12-31 21:59:40.213

[TOC00004]

Process diagnostic dump for oracle@dm02dbadm02.ttiinc.com, OS id=73677,

pid: 65, proc_ser: 187, sid: 3768, sess_ser: 19475

-------------------------------------------------------------------------------

os thread scheduling delay history: (sampling every 1.000000 secs)

  0.000000 secs at [ 22:59:39 ]

    NOTE: scheduling delay has not been sampled for 0.476107 secs

  0.000000 secs from [ 22:59:35 - 22:59:40 ], 5 sec avg

  0.000000 secs from [ 22:58:40 - 22:59:40 ], 1 min avg

  0.000000 secs from [ 22:54:40 - 22:59:40 ], 5 min avg

*** 2015-12-31 21:59:41.679

loadavg : 2.31 2.47 2.23

System user time: 0.03 sys time: 0.00 context switch: 17233

Memory (Avail / Total) = 75526.99M / 257965.57M

Swap (Avail / Total) = 23089.19M /  24576.00M

F S UID         PID   PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD

0 R oracle    73677      1 99  80   0 - 19898980 ?   21:39 ?        00:19:45 oracleorcl2 (LOCAL=NO)

ORA 00445 Background Process 'xxx' did not start after 120 seconds

ORA 00445 Background Process 'xxx' did not start after 120 seconds

When an Oracle background process was unable to spawn a child process after a stipulated time of 2 minutes, therefore startup for this child process was aborted.

The Parent process that starts this child process does it in synchronous manner (the code would be a sequence probly with nothing to handle the case where child is not able to start), and hence cannot continue with any of the tasks, until child process spawning is completed. Sometimes this may lead to instance wide hang.

Where to find details:

  - Alert log

  - incident trace file

     the incident trace file name would be mentioned in the alert log itself

       shows the instance/database wide waits that the child process encountered when coming up

       "PROCESS STATE" -> "Current Wait Stack"

  - traditional trace file generated at the time of issue

      shows details of the load on machine

              * load average

              * memory consumption

              * output of PS (process state)

              * output of GDB (to view function stack trace) basically stack trace

Need to review both the traditional and the incident trace files.

Stages of process startup

 - Queued

 - Forking

 - Execution

 - initialization

Forking and Execution phases are directly linked to the load on the system/resources.

Traditional trace file would contain information about what phase the process was in.

"waited for process"

    .. "to be spawned" - Forking/Queued

    .. "to initialize" - Execute/initialize

RCA:

Root cause falls in any of following 2 categories

 - Contention among processes

 - OS and network level issues OS resource issues/Network storage issues

Known issues/potential solutions:

1. Lack of OS resources and incorrect config

             memory or swap space may be insufficient to spawn a new process

            - check OS error lof to check for the time when the error occurred

                       AIX - errpt -a command

            - run HCVE report

                 verifies that the OS resources are set as recommended by oracle ( RDA - Health Check / Validation Engine Guide (Doc ID 250262.1) )

            - run OSWatcher

            - check the define ulimit settings on Unix

                    On UNIX systems, the ulimit command controls the limits on system resource, such as process data size, process virtual memory, and process file size.

         

         Solution:

                 - review the output of HCVE and allocate resource as recommended and feasible

                 - reset ulimit settings if not apt

    2. ASLR linux feat is being used

           it is designed to load shared memory resources in random addressed. In Oracle, multiple processes map a shared memory object at same address across processes.

           When ASLR is turned on, Oracle cannot guarantee the address of these shared objects and hence this error.

         

         mainly reported on RHEL5 and Oracle 11202.

         To verify ASLR:

                 - /sbin/sysctl -a | grep randomize

                 kernel.randomize_va_space = 1

              

                If parameter is set to anything other than 0, then ASLR is in use.

             Solution:

                      -disable ASLR, modify the following parameters as below in /etc/sysctl.conf

                                kernel.randomize_va_space = 0

                                kernel.exec-shield = 0

                           

Reboot is required for kernel.exec-shield to take effect

    3.   Incorrect database settings

                - PGA_AGGREGATE_TARGET to TRUE/FALSE

                        this should be a numeric value

               - PRE_PAGE_SGA is set to true

                            instructs Oracle to read entire SGA into memory at instance startup

                            OS page entries are then prebuilt for each SGA page. This can increase the time for instance startup but decreases the amt of time reqd for Oracle to reach full

                            performance capacity after startup

                        Also can increase process startup duration because every process that starts must access every page in SGA, can cause PMON process to start late and exceed timeout

               - check o/p of SQL> select * from v$RESOURCE_LIMIT;

                   provides details of resources like sessions, processes, locks etc.

                   it has initialization limits for resources, maximum values reached after last db startup and current utilization of resource.

               Solution:

                      - set PGA_AGGREGATE_TARGET to apt numeric value

                      - set PRE_PAGE_SGA to FALSE

                      - check if the limits were reached and accordingly increase the limits

4. Other causes/known issues

      

            - too much activity on mc

            - NFS latency issues

            - disk latency issues (IO latency)

            - network latency

Oracle Data Pump Architecture 11g

Understanding Data Pump Architecture will let you take advantage of its speed and features. Also, knowing the intricacies of a tool does lend you the powers to identify the use cases when that tool would be a better/worse fit than the alternatives.

So without further ado, here we go:

The MASTER TABLE

  This is a table created in the schema of the user who runs the data pump.

  It contains following details about the job:

-        current state of every object being imported/exported

-        locations of objects in the dumpfile set

-        status of every worker process

-        current set of dumpfiles

-        restart information etc

  During export, the master table is built as the first step of export job and written to the dump set as the last step.

  During import, the master table is loaded first into the schema from the dumpset

The PROCESSES

·         Client Process

The expdp and impdp sessions invoked by users are the client processes

A client is not required once a job is underway.

Multiple clients may attach/detach from a job as necessary for monitoring and control 

·         Shadow Process

A standard shadow (foreground) process is created when any client logs into Oracle database. Same is true for the Data Pump client processes.

It services Data Pump API requests.

In other words :

o   Upon receipt of DBMS_DATAPUMP.OPEN request, shadow process creates a job, which consists of creating Master table, Advanced Queuing (AQ) queues for communication among processes and master control process.

o   Once Data Pump job is running, main task of the shadow process is to service GET_STATUS requests from the client.

If client process detaches, shadow process also goes away.

·         Master Control Process (MCP)

This is the Captain of the Ship, and as there is only one captain for a ship, there is only one MCP for each Data Pump job.

It takes care of following:

o   job state

o   job description

o   restart information in master table

o   dumpfile info in master table

o   starting/stopping worker processes

Job is divided into various phases of metadata and data loading/unloading and each phase has associated worker processes.

MCP hands out work requests to worker processes appropriate for current phase. Majority of MCP processing is spent in this work dispatch loop.

MCP also has file management duties, maintaining active dumpfile list, handing out file pieces as requested by process unloading/loading data.

MCP process is of form: <instance>_DMnn_<pid>               

·         Worker Process

These are the petty officers and subordinates that actully do the work on Master's direction. (cruel world eh?)

Upon receipt of START_JOB request, MCP creates worker processes based on the PARALLEL parameter. Worker process performs tasks assigned by MCP (primarily loading/unloading metadata and data) and maintain object rows that make up bulk of master table.

As object rows are loaded/unloaded, status is updated in the master table, pending/completed/failed etc.

Workers also maintain TYPE COMPLETION ROWS which track the type of objects being worked on: tables/views/indexes etc. These rows are used during restart.

A worker process has name of form : *DWnn*

·         Parallel Query Process -

If External table data access method is chosen for loading/unloading table/partition, worker process creates parallel query processes which carry out actual load/unload. Worker process act as query coordinators. These are standard parallel query execution slaves.

In RAC, to take advantage of load sharing opportunity, the PQ processes may be created on a different instance and all other processes may operate on the initial instance.

Now let's move a little beyond the meta information and look at the bird's eye view of the actual work

The DATA MOVEMENT

There are four data movement methods

·         Data File Copying (Transportable Tablespace)

·         Direct Path Load

·         External Tables

·         Conventional Path

Data file copying has some limitations as certain data types cannot be transported.

Also, Character sets must be same at both the systems.

Some types of data, some types of tables can't be moved. e.g. encrypted data cannot be moved.

              ___________________________________________________

              EXPDP Conventional/Direct path loads write the dump

              in a binary stream that is at least 15% smaller

              than original EXP representation

              ___________________________________________________

The METADATA MOVEMENT

DBMS_METADATA package is used by worker processes.

EXP utility used to store object definitions like SQL DDL.

EXPDP on the other hand writes object definitions to dump files as XML docs.

XML might take more space but allows for more features (object's ownership, storage characteristics, tablespace etc. can be changed during import). COMPRESSED parameter might be added to reduce size.

This also allows more flexibility in remapping the data into different schemas, changing the object names during import etc.

The INTERPROCESS COMMUNICATION

AQ mechanism is used by Data Pump processes to communicate.

There are 2 queues:

1.       Command and Control Queue:

All processes except clients subscribe to this queue. All API commands, work requests and responses, file requests and log messages are processed by this queue.

2.       Status Queue:

Only MCP writes to this queue and only shadow process subscribes to this. Recieves work in progress and error message info.

The SECURITY

All the IO related to the data pump job is handled by the Oracle background server process.

Hence, the OS user doing the IO is 'oracle'.

This is a security risk as 'oracle' is a privileged account.

Hence, all directory specifications are made using Oracle directory objects, with the directory read/write grants established by the DBA.

The INITIALIZATION PARAMETERS

DISK_ASYNCH_IO should be set to TRUE to allow for asynchronous IO

DB_BLOCK_CHECKSUM should ideally be FALSE but impact is minimal (5%) otherwise

STREAMS_POOL_SIZE should be set sufficiently