 IBM Datastage Tutorial | A Complete Beginners Guide - (2023)

DataStage Tutorial

datastage-tutorials

DataStage Interview Questions and Answers

datastage-interview-questions-answers

Introduction to DataStage | DataStage Overview - Updated 2023

Introduction to DataStage

introduction-datastage

- - &gt;&nbsp;ETL project phases There are 3 phases Phase &ndash; I
<h3>Data profiling</h3>
&nbsp;
<ul>
<li>Source System Analyses is done in this phase</li>
<li>There are 5 Types of Analysis</li>
</ul>
<ul>
<li>Ca - - &gt;&nbsp;&nbsp;Column Analysis</li>
<li>Pa - - &gt;&nbsp;&nbsp;&nbsp;primary key Analysis</li>
<li>Fa - - &gt;&nbsp;&nbsp;foreign key Analysis</li>
<li>Bl - - &gt;&nbsp;&nbsp;&nbsp;base line Analysis</li>
<li>Cd - - &gt;&nbsp;&nbsp;&nbsp;cross domain Analysis</li>
</ul>
<ul>
<li>The output we get is whether the data is Directly or Not.</li>
<li>If the data is dirty proceed to next phase</li>
</ul>
Phase &ndash; II
<h3>Data Quality or cleansing</h3>
<ul>
<li>There are 5 Stages</li>
</ul>
<ol>
<li>Parsing</li>
<li>Cording</li>
<li>Standardize</li>
<li>Matching</li>
<li>Consolidate</li>
</ol>
Golden Copy is sent to the next phase Phase &ndash; III&nbsp;
<h3>Data Transformation</h3>
ETL Process&nbsp; Example If Hutch wants to introduce 10 /- recharge price. Then &ndash; the top level manager needs Some information &nbsp; <img class="alignnone wp-image-6923" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_63-460x197.png" alt="Screenshot_6" width="462" height="197" /> We have ETL Tools and ETL Programming Tools ETL Tools Extract source data from heterogeneous source (i.e from different source) ETL Programming Tools Extract data from only one External source. Characters&nbsp; o f Data Work House
<ul>
<li>Subject (that is &nbsp;w . r. t to customer or sales etc)</li>
<li>Integrate</li>
<li>Non &ndash; volatile (only read)</li>
<li>Historical Data&nbsp;</li>
</ul>
&nbsp; Active Data Base: (Historical data)&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; OLTP &nbsp;(Time sensitive /30- 90 days) <img class="alignnone wp-image-6924" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_72-460x210.png" alt="Screenshot_7" width="468" height="212" /> &nbsp; The data that is collected from different sources can will be of 30 &ndash; 90 days. Later On it is stored in Achieve Data Base which is an historical data.
<ul>
<li>ETL is a multilayer process.</li>
<li>Data Ware house is an data base that collects data from heterogeneous source as per</li>
</ul>
Business requirements required by an Toped level Manager.
<ul>
<li>Data Warehousing is an process Which has the Combination of ETL activities and</li>
</ul>
BI (Business Intelligence) activities . &nbsp; <img class="alignnone wp-image-6925" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_82-460x163.png" alt="Screenshot_8" width="444" height="157" /> &nbsp;
<table style="width: 655px; height: 189px;" border="1" cellspacing="1" cellpadding="1">
<tbody>
<tr>
<td>OLTP</td>
<td>OLAP</td>
</tr>
<tr>
<td>Transition</td>
<td>Analysis</td>
</tr>
<tr>
<td>Multi user</td>
<td>Less user</td>
</tr>
<tr>
<td>Less in size</td>
<td>Large in size</td>
</tr>
<tr>
<td>Volatile(Read/Write)</td>
<td>Non volatile(Read)</td>
</tr>
</tbody>
</table>
<img class="alignnone wp-image-6926" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_92-460x459.png" alt="Screenshot_9" width="422" height="422" />
<pre style="text-align: center;">Learn DataStage&nbsp;by Tekslate - Fastest growing sector in the industry.
Explore Online <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage Training</a> and course is aligned with industry needs &amp; developed by industry veterans.
Tekslate will turn you into DataStage&nbsp;Expert.</pre>
<ul>
<li>In ETL, Extraction has to be done is an specified time.</li>
<li>Loading has also to be done is an Specified time.</li>
<li>If Extraction has to be done between 10pm &ndash; 11 to 5 pm and loading to be done</li>
</ul>
Between 12:00am to 2:00am - - &gt;&nbsp;Extract Window: Specified time&nbsp; given by the client to hit source and Extract&nbsp; the data is known&nbsp; as Extract Window - - &gt;&nbsp;Load Window: Specified time&nbsp; given by the client to hit the target and load the data is known as Load Window. - - &gt; After Extraction the data collected should be stored in an area called as Stage Area. <img class="alignnone wp-image-6927" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_101-460x335.png" alt="Screenshot_10" width="443" height="322" /> &nbsp; After Loading the data in to warehouse the permanent data will be deleted Known as flush.

ETL Project Phases in Data Stage - Tekslate

ETL Project Phases in Data Stage

etl-project-phases-data-stage

&nbsp; Repository :- / Work Area / Project It is a work Area with collection of different Components.
<ul>
<li>Different Repository Components are</li>
<li>Table Definition</li>
<li>Shared Contains</li>
<li>Routines ..etc</li>
<li>Jobs</li>
</ul>
&nbsp; It is an Environment for developing ETL jobs using the components.
Engine :- /Data stage Server
<ul>
<li>Responsible for Executing Data stage jobs.</li>
<li>Engine is also Responsible for partitioning, Re- partitioning and Reverse partitioning.</li>
</ul>
Plugins are used for communication with external sources or target
<h2><img class="alignnone size-full wp-image-6970" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_351.png" alt="Screenshot_35" width="377" height="75" /></h2>
&nbsp;
<ul>
<li>Though interface, the local of oracle are added to the Data stage</li>
</ul>
<ul>
<li>Plug &ndash; ins will reside in package Installer</li>
</ul>
Packs We need to install Data stage packages using packes of package Installer Client Components 
<ul>
<li>Designer</li>
<li>Director</li>
<li>Manager</li>
<li>Admin</li>
</ul>
&nbsp; Server
<ul>
<li>Repository</li>
<li>Engine</li>
<li>Package Installer</li>
<li>&nbsp;client</li>
</ul>
<ul>
<li>Designer</li>
<li>Director</li>
<li>Manager</li>
<li>Admin</li>
</ul>
&nbsp; 8.0.1: Common user interface :- / unified users:
<ul>
<li>Designer</li>
<li>Director</li>
<li>Web console</li>
<li>Admin</li>
<li>Information Analyzer</li>
</ul>
&nbsp; &nbsp;
<ol>
<li>Common repository : / Meta Data Server </li>
</ol>
Common Environment&nbsp; to store w. r. t. Data Quality Data profiling and data Transformation.
<ol start="2">
<li>Common Engine :-</li>
</ol>
Responsible for information Analysis , Data Quality and data Transformation.
<ol start="3">
<li>Common Connectivity:-</li>
</ol>
Responsible for plug ins&nbsp; and packs.
<ol start="4">
<li>Common Shared Services :--</li>
</ol>
Acts as interface between all components
<img class="alignnone wp-image-6971" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_361-460x420.png" alt="Screenshot_36" width="486" height="442" />
<h2>&nbsp;</h2>
<h2>&nbsp;Architecture</h2>
<h3>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2 &ndash; layer Architecture </h3>
<img class="alignnone wp-image-6932" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_113-460x201.png" alt="Screenshot_11" width="524" height="229" /> &nbsp; &nbsp; - - &gt;&nbsp;&nbsp;
W &ndash; H Inman is known as the faster of warehouse . - - &gt;&nbsp;Top &ndash; bottom approach was given by W &ndash; H Inman. - - &gt;Bottom &ndash; up approach was given by Ralph &ndash; Kimball. &nbsp;
<img class="alignnone wp-image-6933" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_122-460x329.png" alt="Screenshot_12" width="465" height="332" />
<pre style="text-align: center;">Aspired to become an DataStage?
Explore the post to discover the know-hows on <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage Training</a>.</pre>
<pre style="text-align: center;">&nbsp;</pre>
L4 &nbsp;Project Architecture
<img class="alignnone wp-image-6934" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_131-460x254.png" alt="Screenshot_13" width="464" height="255" /> &nbsp;
Different format of .txt - - &gt;&nbsp;&nbsp;Fixed width flat files - - &gt;&nbsp;&nbsp;&nbsp;Comma separated values - - &gt;&nbsp;&nbsp;Variable length files - - &gt;&nbsp;&nbsp;&nbsp;Head and tail files &nbsp;
<img class="alignnone size-full wp-image-6935" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_141.png" alt="Screenshot_14" width="180" height="371" />
For indepth understanding of DataStage click on
<ul>
<li class="title"><a href="https://tekslate.com/introduction-datastage/" target="_blank" rel="noopener">Introduction to DataStage</a></li>
<li class="title"><a href="https://tekslate.com/architecture-data-stage/" target="_blank" rel="noopener">Architecture of Data Stage</a></li>
<li><a href="https://tekslate.com/oracle-enterprise-data-stage/" target="_blank" rel="noopener">Oracle Enterprise in Data Stage</a></li>
<li><a href="https://tekslate.com/scd-slow-changing-dimension-data-stage/" target="_blank" rel="noopener">SCD(Slow changing Dimension) in Data Stage</a></li>
<li><a href="https://tekslate.com/tutorials/datastage-tutorials/" target="_blank" rel="noopener">DataStage Tutorials</a></li>
</ul>
<div class="tags">&nbsp;</div>
&nbsp;

Architecture of Data Stage

architecture-data-stage

<h2>Roles and Responsibilities of ETL Developer in Data Stage</h2>
ETL Developer roles and responsibilities are:
-Understand Requirements.
-Prepare Questions.
-Logical design (flow chart)
-Physical design (coding)
-Unit test
-Performance Tuning
-Peer view &ndash; if it is ok, proceed to next step, if else start again from 3 step
-Design Turnover Document (DTD)/ <a href="https://en.wikipedia.org/wiki/Technical_documentation" target="_blank" rel="noopener">Technical Design Document</a> (TDD)/ Detailed Design Document (DDD)&nbsp; -- &gt; produces job Report
-Back ups
-Job sequencing
<h3>General flow of a project:</h3>
<img class="aligncenter wp-image-6938" title="Roles and Responsibilities of ETL Developer in Data Stage" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_151-460x159.png" alt="Roles and Responsibilities of ETL Developer in Data Stage" width="615" height="211" /> &nbsp; &nbsp;
<h3>Ware house modeling</h3>
- - &gt;&nbsp;&nbsp;Architecture
- - &gt;&nbsp;&nbsp;Dimensional modeling(schemas)
- - &gt;&nbsp;&nbsp;Dimensional characteristics
- - &gt;&nbsp;&nbsp;Fact characteristics
- - &gt;&nbsp;for a developer if he is given by Document he needs to, follow all the 10 rules and regulations.
<h3>Document</h3>
__ One Record for each Kindle (Multiple Records&nbsp; for multiple address and Dummy Records for joint Accounts)
<ul>
<li>In a Banking sector, for each and every customer, he may have many accounts such as</li>
</ul>
&nbsp; <img class="aligncenter wp-image-6939" title="Roles and Responsibilities of ETL Developer in Data Stage-document" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_161.png" alt="Roles and Responsibilities of ETL Developer in Data Stage" width="272" height="169" /> &nbsp; - - &gt;So, one record should be maintained for every customer
<pre style="text-align: center;">Desired to gain proficiency on DataStage?
Explore the blog post on <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage training</a> to become a pro in DataStage.</pre>
Format of Mapping Document&nbsp; <img class="aligncenter wp-image-6940 size-full" title="Roles and Responsibilities of ETL Developer in Data Stage-document" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_171.png" alt="Roles and Responsibilities of ETL Developer in Data Stage" width="340" height="142" /> &nbsp; - - &gt;&nbsp;We have two formats of Mapping documents
<ol>
<li>
<h4>Word</h4>
</li>
<li>
<h4>Excel sheet</h4>
</li>
</ol>
- - &gt;&nbsp;we may also get job templates as input for designing jobs. Format
<table style="height: 86px;" width="644">
<tbody>
<tr>
<td width="62">&nbsp;S NO</td>
<td width="65">&nbsp;Load order</td>
<td width="66">Target entity</td>
<td width="70">Target Attribute</td>
<td width="66">SRC entity</td>
<td width="70">SRC Attribute</td>
<td width="62">TRF</td>
<td width="61">EH</td>
<td width="67">Cornet</td>
</tr>
<tr>
<td width="62">&nbsp;</td>
<td width="65">&nbsp;</td>
<td width="66">&nbsp;</td>
<td width="70">&nbsp;</td>
<td width="66">&nbsp;</td>
<td width="70">&nbsp;</td>
<td width="62">&nbsp;</td>
<td width="61">&nbsp;</td>
<td width="67">&nbsp;</td>
</tr>
</tbody>
</table>
&nbsp;
<table style="height: 107px;" width="704">
<tbody>
<tr>
<td width="71">S no</td>
<td width="71">Load order</td>
<td width="71">Target entity</td>
<td width="71">Target Attribute</td>
<td width="71">SRC entity</td>
<td width="71">SRC Attribute</td>
<td width="71">TRF</td>
<td width="71">EH</td>
<td width="71">Cornet</td>
</tr>
<tr>
<td width="71">&nbsp;</td>
<td width="71">&nbsp;</td>
<td width="71">&nbsp;</td>
<td width="71">&nbsp;</td>
<td width="71">&nbsp;</td>
<td width="71">&nbsp;</td>
<td width="71">&nbsp;</td>
<td width="71">&nbsp;</td>
<td width="71">&nbsp;</td>
</tr>
</tbody>
</table>
 Example
EXP _ EMP &agrave; hire date &agrave; EMP
Dept Name &agrave; DNO &agrave;DEFT
First Name
Last Name
Middle Name.
- - &gt;&nbsp;&nbsp;from these 5 details a Mapping document is text prepared and given to the developer
Exp _ EMP :-
To find out the experience of an employee we need to&nbsp; know the hire date&nbsp; from the EMP table
So, EMP is one of the Source entity, and its Source attributes will be ENO, ENAME, Hire date, DNO .
DEPT Name :-
To Know the Dept Name, We need to Know the DNO from DEPT table
So, DEPT is another Source Entity with DNO, DANME as Source attributes.
- - &gt;first, last and Middle Names Will be taken from the EName of EMP Table.
- - &gt; Now the Target Attribute, that we need to find out are, ENO, First Name, Middle Name, Last Name, EXP_EMP, Dname
Target Entity or Target&nbsp; Table Will be EXP_ EMP Conventions:-
Set of rules to be followed
There are 4 types of conventions that are to be followed while naming :
-Job Naming
-Stage Naming
-Variable Naming
-Parameters Naming
&nbsp;

Roles and Responsibilities of ETL Developer in Data Stage

roles-responsibilities-etl-developer-data-stage

<h2 style="text-align: justify;">&nbsp;Data Formats</h2>
<ul style="text-align: justify;">
<li>Data formats are also of different and are need to converted as of our required data</li>
<li>r/m/d</li>
<li>d/m/y</li>
<li>m/d/y</li>
<li>PQ 0102</li>
</ul>
That is &nbsp;&nbsp;010102
2nd &nbsp;jan 2001
Where&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; P =0&nbsp; Q =1&nbsp; R = 2&nbsp; S =&nbsp; 3
<ul style="text-align: justify;">
<li>Transformations may take place in different ways from Source to target.</li>
</ul>
- - &gt;&nbsp;We may load the Source attribute ENO as it is to the Target &nbsp;Attribute , or may make Same &nbsp;&nbsp;attribute.
<h2 style="text-align: justify;">Attributes</h2>
The data-* attributes is used to store custom data private to the page or application.
The data-* attributes gives us the ability to embed custom data attributes on all HTML elements.
The stored (custom) data can then be used in the page's JavaScript to create a more engaging user experience (without any Ajax calls or server-side database queries).
The data-* attributes consist of two parts:
-The attribute name should not contain any uppercase letters, and must be at least one character long after the prefix "data-"
-The attribute value can be any string
Note: Custom attributes prefixed with "data-" will be completely ignored by the user agent.
Ex: We have ENO as N 1234, N 3567
-The transformation we need to make is change All Ns to the value 8.
-Now, ENO Will be 81234,83563
-Both the source entities may have on <a class="zem_slink" title="Referential integrity" href="http://en.wikipedia.org/wiki/Referential_integrity" target="_blank" rel="noopener wikipedia">Referential Integrity</a> i.e they are related With each other to make same transformations.
-The main 4 jobs involved in doing an project are :
-Designing
-Performance
-Testing
-Conventions
<img class="alignnone wp-image-6946" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_181-460x117.png" alt="Screenshot_18" width="667" height="169" />
- - &gt;&nbsp;Both the Source Entities , are related with each other through joins, known as
Horizontal Combining in <a class="zem_slink" title="Data warehouse" href="http://en.wikipedia.org/wiki/Data_warehouse" target="_blank" rel="noopener wikipedia">data Warehousing</a>.
<a class="zem_slink" title="Dimensional modeling" href="http://en.wikipedia.org/wiki/Dimensional_modeling" target="_blank" rel="noopener wikipedia">Dimensional modeling</a> / ER Modeling:
- - &gt;&nbsp;Modeling will never happen at <a class="zem_slink" title="Database" href="http://en.wikipedia.org/wiki/Database" target="_blank" rel="noopener wikipedia">data base</a> i.e&nbsp; Oracle
<img class="alignnone size-full wp-image-6947" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_191.png" alt="Screenshot_19" width="368" height="143" />
&nbsp;- - &gt;&nbsp;&nbsp;Source system when modeled will always have only <a class="zem_slink" title="Entity&ndash;relationship model" href="http://en.wikipedia.org/wiki/Entity%E2%80%93relationship_model" target="_blank" rel="noopener wikipedia">E- R Model</a>
&nbsp;- - &gt;&nbsp;Target system when modeled may have either E- R / Dimensional model
<pre style="text-align: center;">Desired to gain proficiency on DataStage?
Explore the blog post on <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage&nbsp;training</a> to become a pro in DataStage.</pre>
<h2 style="text-align: justify;">Modeling</h2>
-Logical representation of Source system or Target system w. r. t client requirements
-r .t customer perspective
-Logical representation is nothing but pictorial representation.
-Modeling popular Modeling Tools Modeling Tools.
-Most popular Modeling Tools are
-ERWIN
2 ERSTUDIO
Based on <a class="zem_slink" title="Business requirements" href="http://en.wikipedia.org/wiki/Business_requirements" target="_blank" rel="noopener wikipedia">Business Requirements</a> Document (BRB), Modeling Will be done by Data Modeler&nbsp; /DBA/Team leader (TL)/ Project loader (PL)
- - &gt;To create it in a database,
<ul style="text-align: justify;">
<li>using the same Modeling Tools, We can also Create <a class="zem_slink" title="Physical model" href="http://en.wikipedia.org/wiki/Physical_model" target="_blank" rel="noopener wikipedia">physical Model</a>(A code/ query)</li>
<li>Once the Queries are created , click on RUN,&nbsp; data will be created to data base.</li>
</ul>
<img class="alignnone wp-image-6948" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_201-460x74.png" alt="Screenshot_20" width="544" height="87" />
<h3 style="text-align: justify;">Forward engineering</h3>
Designed from Scratch (from beginning)
<a class="zem_slink" title="Reverse engineering" href="http://en.wikipedia.org/wiki/Reverse_engineering" target="_blank" rel="noopener wikipedia">Reverse&nbsp; engineering</a>
Making some attributes to the already existing Model
why do we called Target system as a Dimension Model ?
<ul style="text-align: justify;">
<li>A client can see the information in different dimensions.</li>
<li>Will always maintain lowest level of information i.e Detailed</li>
<li>Will have Meaningful, descriptive , detailed info.</li>
</ul>
(Ex: Suppose we want exp of emp as target info, Target system will products
Emp&nbsp; of Emp in different ways , that is in years, months, days, Hours, Seconds, nano Seconds )

Data Formats Attributes & Modeling  in Data Stage

data-formats-attributes-data-stage

Scheduling jobs in DataStage | Types of links In DataStage - Updated 2022

Scheduling Jobs in DataStage

scheduling-jobs-in-datastage

<h3>Filter</h3>
Example
<table style="height: 360px;" width="534">
<tbody>
<tr>
<td width="109">Eno</td>
<td width="109">Ename</td>
<td width="109">Dno</td>
</tr>
<tr>
<td width="109">100</td>
<td width="109">Moon</td>
<td width="109">12</td>
</tr>
<tr>
<td width="109">200</td>
<td width="109">Moon soon</td>
<td width="109">13</td>
</tr>
<tr>
<td width="109">300</td>
<td width="109">Mani</td>
<td width="109">14</td>
</tr>
<tr>
<td width="109">400</td>
<td width="109">Mani kumar</td>
<td width="109">16</td>
</tr>
<tr>
<td width="109">500</td>
<td width="109">Mani</td>
<td width="109">18</td>
</tr>
<tr>
<td width="109">600</td>
<td width="109">Moon</td>
<td width="109">20</td>
</tr>
<tr>
<td width="109">700</td>
<td width="109">Def</td>
<td width="109">22</td>
</tr>
<tr>
<td width="109">800</td>
<td width="109">stu</td>
<td width="109">24</td>
</tr>
</tbody>
</table>
&nbsp; &nbsp;
<ul>
<li>Grep &ldquo;moon&rdquo;</li>
</ul>
<table style="height: 138px;" width="428">
<tbody>
<tr>
<td width="102">Eno</td>
<td width="102">Ename</td>
<td width="102">Dno</td>
</tr>
<tr>
<td width="102">100</td>
<td width="102">Moon</td>
<td width="102">12</td>
</tr>
<tr>
<td width="102">200</td>
<td width="102">Moon Soon</td>
<td width="102">13</td>
</tr>
</tbody>
</table>
&nbsp;
<ul>
<li>Grep - I &ldquo;moon&rdquo;</li>
</ul>
<table style="height: 184px;" width="425">
<tbody>
<tr>
<td width="67">Eno</td>
<td width="67">Ename</td>
<td width="67">Dno</td>
</tr>
<tr>
<td width="67">100</td>
<td width="67">Moon</td>
<td width="67">12</td>
</tr>
<tr>
<td width="67">200</td>
<td width="67">Monsoon</td>
<td width="67">13</td>
</tr>
<tr>
<td width="67">600</td>
<td width="67">moon</td>
<td width="67">20</td>
</tr>
</tbody>
</table>
&nbsp;
<ul>
<li>Grep - w &ldquo;moon&rdquo;</li>
</ul>
<table style="height: 93px;" width="399">
<tbody>
<tr>
<td width="61">Eno</td>
<td width="61">Ename</td>
<td width="61">Dno</td>
</tr>
<tr>
<td width="61">100</td>
<td width="61">Moon</td>
<td width="61">12</td>
</tr>
</tbody>
</table>
&nbsp;
<ul>
<li>Grep - v &ldquo;moon&rdquo;</li>
</ul>
<table style="height: 320px;" width="529">
<tbody>
<tr>
<td width="89">Eno</td>
<td width="89">Ename</td>
<td width="89">Dno</td>
</tr>
<tr>
<td width="89">300</td>
<td width="89">Mani</td>
<td width="89">14</td>
</tr>
<tr>
<td width="89">400</td>
<td width="89">Mani Kumar</td>
<td width="89">16</td>
</tr>
<tr>
<td width="89">500</td>
<td width="89">Mani</td>
<td width="89">18</td>
</tr>
<tr>
<td width="89">600</td>
<td width="89">Moon</td>
<td width="89">20</td>
</tr>
<tr>
<td width="89">700</td>
<td width="89">Def</td>
<td width="89">22</td>
</tr>
<tr>
<td width="89">800</td>
<td width="89">stu</td>
<td width="89">24</td>
</tr>
</tbody>
</table>
&nbsp; Data set :-&nbsp; (.ds Extension) limitations of Sequential file Stage:-
<ol>
<li>Sequential file stage by default process the data</li>
</ol>
&nbsp; SRC &nbsp; &nbsp; &nbsp;<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp; &nbsp; &nbsp;Staging Area Extraction&nbsp; <img class="alignnone size-full wp-image-6983" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_421.png" alt="Screenshot_42" width="402" height="117" /> &nbsp;
<ul>
<li>Here data gets Stored in a local Server (or) local Repository (or) Data base</li>
<li>Using Sequential file , data gets stored in local server . bet file format.</li>
</ul>
&nbsp;
<h4>Sequential file stage has an Memory limit</h4>
<img class="alignnone wp-image-6984" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_43-460x99.png" alt="Screenshot_43" width="473" height="101" />
<pre style="text-align: center;">Learn DataStage by Tekslate - Fastest growing sector in the industry.
Explore Online <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage Training</a> and course is aligned with industry needs &amp; developed by industry veterans.
 Tekslate will turn you into DataStage Expert.</pre>
<h4>Conversion takes place , when we use Sequential file stage for staging the data</h4>
&nbsp;
<ul>
<li>A stage is an operator prebuilt in component</li>
<li>Sequential file is associated with import and export operators</li>
</ul>
<img class="alignnone wp-image-6985" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_44-460x429.png" alt="Screenshot_44" width="451" height="419" /> &nbsp;
<h4>&nbsp;Data lands outside the Data stage</h4>

Filter's & Data Set in Data Stage

filters-data-stage

To overcome the limitations of sequential file, we use Data set
<ol>
<li>Data set is a parallel processing Stage which is used for staging the data, when we design dependent jobs</li>
</ol>
(That is Transformation jobs are dependent on extraction jobs.&nbsp;&nbsp;And loading jobs are dependent on Transformation jobs ).
<ol start="2">
<li>Data set can accommodate more than 2 GB</li>
<li>Data set stores data in Native format, so no conversion is required</li>
<li>Data lands inside the data stage repository</li>
</ol>
<h3>Types of Data Set</h3>
Virtual :-&nbsp; Data moving through the link is virtual, (temporary) Persistent :- Data &nbsp;created with Data set is persistent,&nbsp; (permanent) In target file,&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; C: /data/output. DS &agrave;persistent &agrave;permanent
<h3>Alias of Data set</h3>
<ol>
<li>Orchestrate file</li>
<li>Operating System file</li>
</ol>
<h3>Data set files</h3>
Data set is not a single file, but has multiple files
<ul>
<li>Descriptor file</li>
<li>Data file</li>
<li>Control file</li>
<li>header file</li>
</ul>
<ol>
<li>Descriptor file : -</li>
</ol>
Schema details and address of data &agrave; Structure () table definition C:/data/file. ds
<ol start="2">
<li>Data file :-</li>
</ol>
Contains data in Native format C:/IBM/Information Server / Server/data set/ file. Ds
<ol start="3">
<li>Control file (or) header file :-</li>
</ol>
Resides in operating system.
<h3>How to create and recues Data set</h3>
Extraction: <img class="alignnone wp-image-6988" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_45-460x117.png" alt="Screenshot_45" width="403" height="102" />
<ul>
<li>After compiling &agrave; Run</li>
<li>Now, copy the Target file path that is .ds file path</li>
<li>Go to columns&agrave;Save&agrave;Table Definer&agrave;Ok&agrave;Select the file&agrave;Ok</li>
</ul>
Transformation <img class="alignnone size-full wp-image-6989" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_46.png" alt="Screenshot_46" width="188" height="101" /> &nbsp; Here are&nbsp;properties&nbsp; Paste the Extraction Target path<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Columns &agrave;Load
<ul>
<li>Data is the Extraction Target gets into Transformation Data set.</li>
</ul>
To view Data set outside the job&nbsp;
<ul>
<li>Data set cannot be directly copied or deleted because, it has multiple files</li>
</ul>
<ul>
<li>Organize Data set :- (View, copy, delete)</li>
</ul>
<ul>
<li>We use Data set utilities for organizing Data.</li>
</ul>
&nbsp; Dataset utilities
<ol>
<li>GUI- Data set Management &ndash; wind one</li>
<li>CMD - $ orch admin &ndash; UNIX<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp; &nbsp; &nbsp;Command line</li>
</ol>
&nbsp; Tools <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;Data set Management <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Shows the list of files <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Select the file <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />ok <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;Data set Management window opens <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Here we can view data, copy data, delete data. &nbsp; Data Set version
<ul>
<li>By default 4.1</li>
<li>We can create Dataset with whatever version we want</li>
<li>Version control can be done using an Environmental variable.</li>
</ul>
&nbsp; <img class="alignnone size-full wp-image-6990" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_47.png" alt="Screenshot_47" width="363" height="106" /> &nbsp; Step 1: Select the job properties Symbol <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;parameters <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Add&nbsp; Environmental variable <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;Select APT &ndash; WPITE &ndash;DS-VERSION &ndash;ok<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Now Compile and Run <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />While RUN, We can Select the version &nbsp; Data set operators It does not have an operator generally but uses copy operator To see the operators of each stage : Job properties <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;Generated ASH How to flush Data in the Dataset ? File set :-&nbsp; (.fs extension) File set is a file stage, which is used for staging the data when we design jobs. <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Similarities between Dataset and file set
<ul>
<li>S &gt; 2GB PX</li>
<li>S &gt;2GB PX</li>
</ul>
Differences D.S
<ol>
<li>No reject</li>
<li>Internal use</li>
<li>Native format</li>
<li>Ds</li>
<li>Copy operator</li>
<li>Data is organized as segments.</li>
</ol>
&nbsp; F.S&nbsp;
<ol>
<li>Rejects</li>
<li>External Applications</li>
<li>Binary format</li>
<li>.fs</li>
<li>Import / export operator</li>
<li>Multiple files</li>
</ol>
&nbsp; Internal use&nbsp; Data that is created with Dataset can be used only for internal use. That is, .ds format is the only w. r. t Dataset <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />file set Creation is same as Dataset creations Sequential Stage Target properties&nbsp; <img class="alignnone wp-image-6992" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_48-460x135.png" alt="Screenshot_48" width="450" height="132" /> Target properties File =? File update mode:
<ol>
<li>a) overwrite</li>
<li>b) Append</li>
<li>c) Create (error if exists)</li>
</ol>
Create the file, if the target file does not exit Options Clean up on failure = True First line is column name&nbsp; = False Reject mode = Continue Step 1 Select the job properties Symbol <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;parameters <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Add&nbsp; Environmental variable <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Select APT &ndash; CLOBER &ndash; OUTPUT <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Now compile and Full <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />At Run time APT &ndash; CLOBER &ndash; OUTPUT&nbsp; = False <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;Aborts APT &ndash; CLOBER &ndash; OUTPUT&nbsp; = True <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;create file Clean up on failure :- (Works with Append mode) If true = automatically clears partially loaded records, if the job is failed for any restart from the point where it has drooped. Development and Debug Stage&nbsp; Divided into three groups
<ol>
<li>Stages that generate sample data
<ol>
<li>row generator</li>
<li>Column generator</li>
</ol>
</li>
</ol>
&nbsp; Row generator &nbsp; It is a development Stage, Which generates Sample data (sys/user defined) And Supports only one output. Column&nbsp; generator &nbsp;&nbsp; It is a development Stage, Which generates columns with Sample data (sys/user defined) And Supports input and&nbsp; output.
<ol start="2">
<li>Satges that pick Sample data</li>
</ol>
<ul>
<li>Head</li>
<li>Tail</li>
<li>Sample</li>
</ul>
<ol start="3">
<li>Stage that helps in debugging</li>
</ol>
<ul>
<li>Peek</li>
</ul>
<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Development Stage creates Sample data, Suppose if the Client does not give data, we create the Structure and sample data. &nbsp;Row generator :-&nbsp; (one input) <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />System Generated&nbsp; <img class="alignnone size-full wp-image-6993" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_49.png" alt="Screenshot_49" width="458" height="130" /> Right click on Row generator <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />properties <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />No. of Records = 100 <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />ok&agrave;column <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />load <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Select the file <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;ok <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;view data User-defined Properties <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp; &nbsp;No. of Records <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;columns <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;load <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;click on Serial file&agrave;Edit column Meta data window opens <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;Generator <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;type&nbsp; (cycle / random)
<ul>
<li>Limit</li>
</ul>
Var char&nbsp; Generator Algorithm&nbsp; = cycle Value = ? (Here values takes n, no. of name) Algorithm&nbsp; =&nbsp; Alphabet String = ? (String data only 1 name, and display each Alphabet ) That is string = abc Output is :&nbsp; a b c Integer&nbsp;
<ul>
<li>Generator</li>
<li>Type = cycle <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Increment initial value limit <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />part part count</li>
</ul>
<pre style="text-align: center;">Desired to gain proficiency on DataStage? 
Explore the blog post on <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage training</a> to become a pro in DataStage.</pre>
Type&nbsp; = Random <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;limit seed signed Data&nbsp;
<ul>
<li>Generator</li>
</ul>
<ul>
<li>Eparch</li>
</ul>
<ul>
<li>Percent invalid</li>
</ul>
<ul>
<li>Type = cycle Random</li>
</ul>
<ul>
<li>Use current date.</li>
</ul>
&nbsp; Column Generator :-&nbsp; (only 1 input /output) <img class="alignnone wp-image-6994" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_50-460x164.png" alt="Screenshot_50" width="487" height="172" /> &nbsp; Column Generator is associated with
<ol>
<li>Aggregate</li>
<li>Tunneling</li>
<li>Supports</li>
</ol>
Based on this we can group all Employee into 1 group Right click on column Generator
<h2>&nbsp; &nbsp; &nbsp;&nbsp;&darr;</h2>
Properties
<h2>&nbsp; &nbsp;&nbsp;&darr;</h2>
With in options
<h2>&nbsp; &nbsp;&nbsp;&darr;</h2>
Column method = Explicit
<h2>&nbsp; &nbsp;&nbsp;&darr;</h2>
Column to Generator = COMPANEY
<h2>&nbsp; &nbsp;&nbsp;&darr;</h2>
Column to Generator = COUNTRY
<h2>&nbsp; &nbsp;&nbsp;&darr;</h2>
Click, on output options
<h2>&nbsp; &nbsp;&nbsp;&darr;</h2>
&nbsp; Left hold the mouse, and drag it and perform Mapping
<h2>&nbsp; &nbsp;&nbsp;&darr;</h2>
Ok View data [Mapping is needed for column Generator, because it has both input and output] User-defined&nbsp; C.G
<h2>&nbsp; &nbsp;&nbsp;&darr;</h2>
Properties
<h2>&nbsp; &nbsp;&nbsp;&darr;</h2>
Output
<h2>&nbsp; &darr;</h2>
Columns
<h2>&nbsp; &darr;</h2>
Double click on country
<h2>&nbsp; &darr;</h2>
Generator
<h2>&nbsp; &darr;</h2>
Algorithm = cycle
<h2>&nbsp; &darr;</h2>
Value = IBM
<h2>&nbsp; &darr;</h2>
Ok

Types of Data Sets in Data Stage

types-of-data-sets-in-data-stage

Head Tile and Sample in Datastage | Updated 2023 - Tekslate

Head Tail and Sample in DataStage

head-tail-sample-data-stage

Configuring Oracle Database in DataStage - Updated 2023

Configuring Oracle Database in DataStage

configuring-oracle-database-in-datastage

CSV Files in Data Stage - Tekslate

CSV Files in Data Stage

csv-files-data-stage

String functions in Datastage

String Functions in DataStage

string-functions-datastage

Sorting Techniques in Datastage

Sorting Techniques in DataStage

sorting-techniques-data-stage

Horizontal Sorting in Data Stage

horizontal-sorting-combining-data-stage

Funnel in Datastage

Funnel in DataStage

funnel-requirements-data-stage

Joins in Data Stage - Tekslate

Joins in Data Stage

joins-data-stage

<h2>SCD(Slow changing Dimension)</h2>
EX:- Suppose we have an customer Table, we have some fields which are frequently, ofliny,&nbsp; slowly, Rarely, Rapidly changed &nbsp; <img class="alignnone wp-image-7100" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_70-460x102.png" alt="Screenshot_70" width="468" height="103" /> &nbsp;
<h3>Different types of loadings</h3>
&nbsp;
<ol>
<li>Initial</li>
<li>&nbsp;incremental</li>
</ol>
&nbsp;
<ul>
<li>Initial data (source) :-</li>
</ul>
C ID &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; CN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Add 11 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; A&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; HYD&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; stored in Achieve Data base or OLTP or Source 22 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; B &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; SEC 33 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; C &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;DEL &nbsp; The information is should from 2000 to 29th dec 2010 This data is stored in Target &nbsp;<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" /> Ware House &nbsp;<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" /> Before Data
<ul>
<li>Source (After data) :-</li>
</ul>
CID&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Add &nbsp; <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />value column 11 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;A&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HYD &nbsp;<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Copy 22 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;B&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GNT &nbsp; <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Edit 33 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; E&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;VZD &nbsp; <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />insert &nbsp;
<ul>
<li>Refresh the target data with source data based on type 1, type 2, type 3</li>
</ul>
<ul>
<li>Differences :-</li>
</ul>
Compare Before Data with After Data Extract&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Extract Target&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Source Data&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Data &nbsp;
<ul>
<li>If there are any changes in key column<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />Insert</li>
<li>If there are any changes in value column<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;edit</li>
</ul>
&nbsp; Output :- <img class="alignnone wp-image-7101" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_711-460x169.png" alt="Screenshot_71" width="532" height="195" /> &nbsp; &nbsp; Table 1 :- EX: &nbsp;Before Data(Target) CID &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; CN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Add 11 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; A&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; HYD 22 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; B&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;GNT 33 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; C&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;DEL 32 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; D&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; VZD &nbsp; After Data: (Source) &nbsp; CID &nbsp; &nbsp; &nbsp; CN &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Add 11 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; A &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;ASM <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;Edit 22 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;B &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;GNT <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;copy 44 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;G &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;PUNE&nbsp;<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" /> insert 32 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;E &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Cal&nbsp;<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />edit &nbsp;
<ul>
<li>Type 1 will not maintain historical data, maintains only current data .</li>
</ul>
<pre style="text-align: center;">Get through the interview bar with our selected interview questions for <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage</a> enthusiasts</pre>
Output :- CID &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; CN &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Add 11 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; A &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; ASM 22 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;B &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;GNT 33 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;C &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; DEL 44 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; G &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;PUNE 32 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;E &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; CAL &nbsp;
<ul>
<li>Before Data will not Maintain history</li>
</ul>
&nbsp;
<h3>Type 1 implementation will always&nbsp; happen in ODS </h3>
<img class="alignnone wp-image-7102" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_721-460x159.png" alt="Screenshot_72" width="492" height="169" /> &nbsp; &nbsp; Dimensional&nbsp; Tables<img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" /> CUST id,&nbsp;&nbsp; C NAME, C ADD , Gender Fact Tables <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;C id, BAL, AREA, Trane Type , Data <img class="alignnone size-full wp-image-6279" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/712.png" alt="7" width="55" height="13" />&nbsp;Maintained History &nbsp;
<h3>Type 2&nbsp; :-&nbsp; (Maintain History)</h3>
&nbsp; After Data :- (Source)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Before&nbsp; Data :- (Target) &nbsp; CID&nbsp;&nbsp; CN&nbsp;&nbsp;&nbsp; ADD&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; CID&nbsp;&nbsp; CN&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;ADD 11&nbsp;&nbsp;&nbsp; A&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; HYD&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;11&nbsp;&nbsp;&nbsp; A&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; HYD 22&nbsp;&nbsp; B&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GNT&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;22&nbsp;&nbsp; &nbsp;&nbsp;B&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;SEC 44&nbsp;&nbsp;&nbsp; E&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PUN&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;33 &nbsp;&nbsp;&nbsp;C&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;DEL &nbsp; Output :- &nbsp; CID&nbsp;&nbsp; CN&nbsp;&nbsp;&nbsp; ADD 11&nbsp;&nbsp;&nbsp; A&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; HYD 22&nbsp;&nbsp; &nbsp;&nbsp;B&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;GNT 33&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; C&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;DEL ----------------------------------- 22&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; B&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; GNT 44&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; E&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; PUN &nbsp;
<ul>
<li>Here the key column is repeating, so we insert an new constraint</li>
<li>Surrogate key</li>
</ul>
For indepth understanding of DataStage click on
<ul>
<li class="title"><a href="https://tekslate.com/introduction-datastage/" target="_blank" rel="noopener">Introduction to DataStage</a></li>
<li class="title"><a href="https://tekslate.com/architecture-data-stage/" target="_blank" rel="noopener">Architecture of Data Stage</a></li>
<li><a href="https://tekslate.com/oracle-enterprise-data-stage/" target="_blank" rel="noopener">Oracle Enterprise in Data Stage</a></li>
<li><a href="https://tekslate.com/scd-slow-changing-dimension-data-stage/" target="_blank" rel="noopener">SCD(Slow changing Dimension) in Data Stage</a></li>
<li><a href="https://tekslate.com/tutorials/datastage-tutorials/" target="_blank" rel="noopener">DataStage Tutorials</a></li>
</ul>

SCD(Slow changing Dimension) in Data Stage

scd-slow-changing-dimension-data-stage

<h3>Director</h3>
<ol>
<li>Schedule</li>
<li>Views</li>
<li>unclock job</li>
<li>monitor</li>
<li>batch jobs</li>
<li>message handling</li>
</ol>
Tool <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Run Director
<pre style="text-align: center;">Learn the core features of <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage</a>&nbsp;and become master with our expertise tutorials.</pre>
<h3>Job Report -&nbsp; / DTD</h3>
DTD &ndash; design Turnover&nbsp; Document File <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Generate report <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />click the link&agrave;Generates the report <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;report for parallel job opens
<h3>Views</h3>
Tool <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Run Director &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> the page we&nbsp; see here is Job view<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> click on any Job&agrave;We can have Status view &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />click on logs <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> we can have log view We can filter the logs&nbsp;&nbsp; (Tell some warning ? how can you handle warning ?

Director in Data Stage

director-data-stage

<h2>Schedule</h2>
A job which is not compiled can&rsquo;t be Schedule
Right-click a job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />add to Schedule &agrave; Set the time
<h3>To see Scheduled jobs</h3>
<ul>
<li>Click on the Schedule symbol</li>
<li>We can Schedule Multiple jobs at some time</li>
<li>We can Reschedule, unschedule the jobs</li>
</ul>
<h2>Real-time</h2>
In Real-time, Scheduling will not be done in Director
Some Tools
<ol>
<li>auto sys</li>
<li>control M</li>
<li>Cron TAB &ndash; utility in UNIX</li>
</ol>
<h3>If a job is found in this way</h3>
Job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />finished&nbsp; (see log) <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />finished with warnings
<h3>Warning</h3>
For example:-&nbsp; when binding the input interface field &ldquo;Dept no&rdquo; to field &ldquo;Dept no&rdquo; converting a nullable source result; a fatal run-time error could occur;&nbsp; use a modified operator, so specify the value to which the null should be converted.
<h3>Message Handling</h3>
To eliminate Warnings&nbsp; in logs job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Right-click &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Add rule to messages Handle <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Add rule
<h3>Monitor</h3>
Tools <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;New Monitor<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Monitoring the job at stage level
when we are loading with less&nbsp; no. of records, Data set in not a good performance
&nbsp; <img class="alignnone wp-image-7128" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_811-460x42.png" alt="Monitor" width="555" height="50" />
<h2>Batch Jobs</h2>
Executing Set of jobs in Sequence
Note:
<ol>
<li>can control the order of execution of <a title="Datastage Jobs" href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">datastage jobs </a>in no. of ways</li>
<li>Batch Jobs</li>
<li>JCL Scripting &agrave; job control language</li>
<li>Job Sequencing</li>
<li>&nbsp;M or Cron Tab</li>
</ol>
<blockquote>
Inclined to build a profession as Datastage Developer? Then here is the blog post on, explore <a title="Datastage Training" href="datastage-training" target="_blank" rel="noopener">Datastage Training</a>
</blockquote>
Tools <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Batch<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />New <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Save as new Batch &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Browse and select 1 job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Add job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;we can see JCL Scripting <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Select 1 MCL Job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Add
<ul>
<li>2 Jobs are running in a sequence</li>
<li>Batch successfully compiled with no errors</li>
</ul>
<h2><img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Unlock Jobs</h2>
Q:&nbsp;&nbsp; How to about / terminate a Job while it is Running? &nbsp;
Job name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; status
Peek 1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Running <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;while in running condition &nbsp;
Click on STOP (or) Unlock the jobs
<h3>Director</h3>
<ul>
<li>Clean up resources</li>
<li>The developer will Access the director</li>
<li>Clean up resources to get enabled in director</li>
<li>Administrator &agrave;projects &agrave; enable job admin in director</li>
</ul>
<h3>Clean up resources</h3>
Job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Clean up resources <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;PID <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;process In &rsquo;s&nbsp; show all <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Select
<h3>Release all (or) In Command line </h3>
$&nbsp; PS <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />to list p id&lsquo;s
$ Kill cmd <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;to unlock Jobs
 EX: $ kill peek 1
<h3>How to run a Job through the command line? </h3>
DS Job &ndash; run &ndash; project name- job name-parameter
<h4>If parameters are not there </h4>
DS job &ndash;run &ndash; project name &ndash; job name
<h2>For an in-depth understanding of DataStage click on</h2>
<ul>
<li class="title"><a href="https://tekslate.com/introduction-datastage/" target="_blank" rel="noopener">Introduction to DataStage</a></li>
<li class="title"><a href="https://tekslate.com/architecture-data-stage/" target="_blank" rel="noopener">Architecture of Data Stage</a></li>
<li><a href="https://tekslate.com/oracle-enterprise-data-stage/" target="_blank" rel="noopener">Oracle Enterprise in Data Stage</a></li>
<li><a href="https://tekslate.com/scd-slow-changing-dimension-data-stage/" target="_blank" rel="noopener">SCD(Slow changing Dimension) in Data Stage</a></li>
<li><a href="https://tekslate.com/tutorials/datastage-tutorials/" target="_blank" rel="noopener">DataStage Tutorials</a></li>
</ul>

Viewing scheduled jobs in Datastage

Viewing Scheduled Jobs in DataStage

viewing-scheduled-jobs-in-datastage

<h3>Designer</h3>
(a) Import/ Export (b)Node configuration (c)Advanced find &nbsp;
<h3>&nbsp;(a) Import/ Export</h3>
We can import and export entre project or individual components of repository <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />To more jobs from 1 project to other <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />job backups &nbsp;
<h3>Export :- (Project &ndash; India /DWS)</h3>
<img class="alignnone wp-image-7134" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_821-460x209.png" alt="Screenshot_82" width="410" height="186" /> &nbsp; &nbsp; C :/ Document and setting / admin/ desktop/Backup. new &agrave;Job components to export [Export job designs with executables&nbsp; read only items] Type of export&nbsp;&nbsp;&nbsp; DSX&nbsp; (platform dependent)&nbsp;&nbsp; XML (independent)&nbsp; &agrave;Export&agrave;Export complete
<h3>To add a new file to already existing one</h3>
<ul>
<li>Append to existing file</li>
</ul>
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />We are Storing the back up file in the Desktop Desktop <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;click on Backup. New &nbsp; &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />open<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />you can view an basic Script
<h3>We can also view through command line prompt</h3>
DS export DS import Open a Designer client &ndash; with new project :- Import :-&nbsp;&nbsp; (project &ndash; India/ Analyzer&nbsp; project) Data stage&nbsp; components &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> import from file (Browse the file that we saves in export)
<ul>
<li>Import Selected</li>
<li>performance analysis <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;ok &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /></li>
</ul>
&nbsp;
<table style="height: 229px;" width="596">
<tbody>
<tr>
<td width="223">Data Stage Import</td>
<td width="217">&nbsp;</td>
</tr>
<tr>
<td width="223">Type 1</td>
<td width="217">Name&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Status</td>
</tr>
<tr>
<td width="223">Table definition &hellip;&hellip;&hellip;&hellip;&hellip;&hellip;&hellip;.</td>
<td width="217">&nbsp;</td>
</tr>
<tr>
<td width="223">Job&nbsp; definition &hellip;&hellip;&hellip;&hellip;&hellip;&hellip;</td>
<td width="217">&nbsp;</td>
</tr>
<tr>
<td width="223">Job definition &hellip;&hellip;&hellip;&hellip;&hellip;&hellip;..</td>
<td width="217">&nbsp;</td>
</tr>
</tbody>
</table>
<h3>Multiple&nbsp; Job Compiler</h3>
To compile&nbsp; n no. of Jobs&nbsp;&nbsp; Tools <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Multiple Job Compile <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;parallel <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Next <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Remove all And select required Jobs <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Next <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Next <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Start Compile&nbsp; show compile report&nbsp;&nbsp; when finished Opens a Note pad
<ul>
<li>Data stage Compilation Report</li>
</ul>
&nbsp; Server : India Project :dws Started at : Finished at : Time Taken : 3items processed 2 items compiled Successfully 1 items&nbsp; failed compilation Compile information &nbsp; <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />list <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />compiled ok Can Multiple users run the same job at some time ? Sol:- when 2&nbsp; or more people run the same job at same time with different parameters, use an option / enable the job option Allow Multiple instances Open any job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;job properties<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> parameters Can 2 persons open the job at same time ? Sol :- &nbsp; yes 1st person will have Read, write options 2nd&nbsp;&nbsp; person will have only Read option Note:- <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />w can&rsquo;t run the Multiple jobs at same time in Designer <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />we can run Multiple jobs at same time in director <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;To run Multiple Jobs at same time in director <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;It asks for Invocation Id
<h3>Invocation Id</h3>
A user defined identification to run a job for that instance It is creating an instance In the output, different users should give different output extensions &nbsp; Example:-&nbsp; &nbsp;abc&nbsp; (1.txt)&nbsp; abc&nbsp; (2.txt) Tool &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Run Director <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;click on the job&agrave;In vocation id<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />1000<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Any number &nbsp;
<h3>Scheduling</h3>
Click on any Job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Right click <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Add to schedule &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Set the time &nbsp; (b) Node Configuration <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Its not our Role, DSA will do it <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />For Sometimes we need to run a job on particular&nbsp; node Configuration, then we use Node configuration. &nbsp; Example A Data set&nbsp; is created on 4 nodes abut we created a job on 2 nodes &agrave;while we run, it aborts &agrave; SO, we need to make Sure that it run on 4 nodes using Node Configuration. &nbsp;
<h3>Questions based on Node Configuration&nbsp;</h3>
What is a Node ? Sol :- logical CPU What is Node Configuration ? Sol :- Technique of creating logical CPU&rsquo;s How to See Nodes ? Sol :-&nbsp; 3 ways to see Nodes :-
<ol>
<li>check at logs</li>
<li>configuration file</li>
<li>Job Score</li>
</ol>
<pre style="text-align: center;">Learn DataStage by Tekslate - Fastest growing sector in the industry.
Explore Online <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage&nbsp;Training </a>and course is aligned with industry needs &amp; developed by industry veterans.
Tekslate will turn you into DataStage Expert.</pre>
<pre style="text-align: center;">&nbsp;</pre>
<h4>(a) check at logs</h4>
Tools <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;run director <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;click on log <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Select&nbsp; Main program&nbsp; : APT&nbsp; configure file <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Node details are displayed
<h4>(b) configuration file</h4>
(Real time it is disabled) Tools <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />configuration <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;default <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;node details are displayed
<h4>(c) Job Score</h4>
Job properties <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;parameters <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Add environment variables <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Reporting<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Report Score <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Report Score = True <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Report comes in log What are Node Components Sol :-
<ul>
<li>Node Name - logical CPU name</li>
<li>fast Name - Server name or system name</li>
<li>Pools &ndash; logical area where stages are executed</li>
<li>resource &ndash; memory associated with node</li>
</ul>
&nbsp; Disk &ndash; permanent &nbsp; &ldquo;C :/ IBM / Information Server / Server/ Datasets&rdquo; &nbsp; Scratch disk&nbsp; - temporary &nbsp; &ldquo;C :/ IBM / Information Server / Server/ Scratch&rdquo; What is default config file name ? Sol :- every job is exe on default. Apt&nbsp;&nbsp; (actual parameter) How many nodes in your project ? Sol :- 8 or 16&nbsp; (2^0, 2^1)&hellip;&hellip;&hellip;&hellip;. How to create a new Node Configuration file ? Sol :-&nbsp;&nbsp; Tools <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;configurations <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;copy the code <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Select New &nbsp; <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />part the Code 3 or 4&nbsp; times &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> change the node Names&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Node 1 Node2 Node 3&hellip;&hellip;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Save configuration <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />check <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;configuration configured &nbsp; <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;&lsquo;0&rsquo; errors detected How to run a job on a Specific Config file ? Sol :-&nbsp; $apt- config- file<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Job properties <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;parameters <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Add&nbsp; Environment variables <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Parameter name <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;$ apt &ndash; config &ndash; file &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> ok &nbsp;
<h3>Compile and RUN</h3>
We can browse and change the path if uni processing System with 1 CPU needs minimum 1 node&nbsp; to run a job, then for&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; SMP with 4 CPU needs How many minimum nodes ? Sol :-&nbsp;&nbsp;&nbsp; 1 Node

Designer Implementation in Data Stage

designer-implementation-data-stage

<h3>Container in DataStage</h3>
<ul>
<li>It is a reusable Component</li>
</ul>
Container are of Two types
<ol>
<li>Local container <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;job Specific</li>
<li>Shared container <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;project Specific</li>
</ol>
<h3>Local Container </h3>
Used for minimizing the complexity and can only&nbsp; be used in a job.
<ul>
<li>Select Some part in the job <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Click edit<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Construct &ndash; container<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Local</li>
<li>It gets packed in to a single box called container</li>
</ul>
&nbsp; <img class="alignnone size-full wp-image-7137" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_83.png" alt="Screenshot_83" width="375" height="109" /> &nbsp;
Container <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Data set Container <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Data set &nbsp; Right Click <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;De construct&nbsp;&nbsp; (gives original pattern back)
<h3>&nbsp;</h3>
<h3>Shared Container</h3>
<ul>
<li>These are reusable components used with in the project</li>
</ul>
&nbsp; Oracle <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Transformer <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Data set<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />We want to use this output is n no. of jobs
<ul>
<li>Select the whole job<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Edit<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Construct container<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Shared<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Save it</li>
</ul>
&nbsp; To use this output, in other job Open <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Container <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Select the shared Container <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Drag and drop
&nbsp; <img class="alignnone size-full wp-image-7138" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_84.png" alt="Screenshot_84" width="410" height="134" />
<pre style="text-align: center;">Interested in mastering DataStage?Learn more about <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage Tutorial</a> in this blog post.</pre>
&nbsp;
Oracle <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Transformer <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Container output<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Res <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;properties <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;output <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;select out<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Load <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;reconcile&nbsp; from container link &nbsp;
Shared container used in an Surrogate key
When we have Surrogate key Generator in different jobs, we replace it using an shared container. &nbsp;
&nbsp;
<img class="alignnone wp-image-7139" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_85-460x135.png" alt="Screenshot_85" width="494" height="145" />
<img class="alignnone wp-image-7140" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_86-460x122.png" alt="Screenshot_86" width="490" height="129" /> &nbsp;
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />
Both Jobs are loading output in to same Target Table <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Rather taking multiple Surrogate keys, Select it and make it as container
&nbsp; <img class="alignnone wp-image-7141" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_87-460x164.png" alt="Screenshot_87" width="475" height="168" /> &nbsp;
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />
There will be Co-ordination between&nbsp; all Surrogate keys <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />No need to write the functionality again and again

Container in DataStage

container-data-stage

<h2>Miscellaneous Stage</h2>
Staging area &ndash; Retains data from 7 days to 1m other.
<h3> Compress And Expand </h3>
Both are interdependent stages
Used for doing Data Backup
<h3>Compress Stage&nbsp; </h3>
Compress the data using COMPESS&nbsp;&nbsp; (or) GZIP Commands.
&nbsp; <img class="alignnone wp-image-7144" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_88-460x127.png" alt="Compress stage" width="454" height="124" /> &nbsp;
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />8 records are compressed to 1 record.
<h3>Expand Stage </h3>
Expand the Compressed data using UNCOMPRESS or GUNZIP.
&nbsp; &nbsp;<img class="alignnone wp-image-7145" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_90-460x90.png" alt="Expand Stage" width="471" height="91" />
<h3>Encode and Decode&nbsp; </h3>
Encrypts and Compresses
<img class="alignnone wp-image-7146" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_911-460x135.png" alt="Encode and Decode" width="484" height="142" />
<h3>Generic </h3>
It is a processing&nbsp; Stage used to cell the operators on to the Data stage
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />The stage&nbsp;is an operator &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />prebuilt component
&nbsp; <img class="alignnone wp-image-7147" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_921-460x73.png" alt="Generic" width="498" height="78" />
<h3>Functionality </h3>
IBM developed a tool called X &ndash; Migratory <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Converts Server jobs to parallel jobs <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Conversion rate = 70%.
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />X &ndash; Migratory when it is migratory / converting Server jobs to parallel jobs it is going to substitute those stages which are not these in a parallel environment with the generic stage call the functionality of the stage with its operator.
<h3> pivot </h3>
Reverse pivot <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;converting columns to rows
Pivot <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />converting rows to columns
Ex:-&nbsp;&nbsp;&nbsp; &nbsp;input:-
11 &nbsp;&nbsp;A&nbsp; &nbsp;&nbsp;HYD
22 &nbsp;&nbsp;B &nbsp;&nbsp;SEC
33 &nbsp;C&nbsp; &nbsp;&nbsp;DEL
Output
<table style="height: 411px;" width="581">
<tbody>
<tr>
<td width="44">11</td>
<td width="44">&nbsp;</td>
<td width="44">A</td>
<td width="44">11</td>
<td width="44">&nbsp;</td>
<td width="44">11</td>
<td width="44">HYD</td>
</tr>
<tr>
<td width="44">A</td>
<td width="44">&nbsp;</td>
<td width="44">HYD</td>
<td width="44">11</td>
<td width="44">&nbsp;</td>
<td width="44">A</td>
<td width="44">HYD</td>
</tr>
<tr>
<td width="44">HYD</td>
<td width="44">&nbsp;</td>
<td width="44">B</td>
<td width="44">22</td>
<td width="44">&nbsp;</td>
<td width="44">22</td>
<td width="44">Sec</td>
</tr>
<tr>
<td width="44">22</td>
<td width="44">&nbsp;</td>
<td width="44">SEC</td>
<td width="44">22</td>
<td width="44">&nbsp;</td>
<td width="44">B</td>
<td width="44">Sec</td>
</tr>
<tr>
<td width="44">B</td>
<td width="44">(or)</td>
<td width="44">C</td>
<td width="44">33</td>
<td width="44">(or)</td>
<td width="44">33</td>
<td width="44">Del</td>
</tr>
<tr>
<td width="44">Sec</td>
<td width="44">&nbsp;</td>
<td width="44">Del</td>
<td width="44">33</td>
<td width="44">&nbsp;</td>
<td width="44">C</td>
<td width="44">Del</td>
</tr>
<tr>
<td width="44">33</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
</tr>
<tr>
<td width="44">C</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
</tr>
<tr>
<td width="44">Del</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
<td width="44">&nbsp;</td>
</tr>
</tbody>
</table>
Sequential file <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />pivot<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Data set &nbsp;&nbsp;&nbsp;&nbsp; 
Primary.txt &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;properties<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />output 
Columns&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Derivation &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;length
COL1 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; E ID, E NAME, STATE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 100
COL2 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; STATE &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10 &nbsp;
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Ok
<blockquote>
Inclined to build a profession as Datastage Developer? Then here is the blog post on, explore <a title="Datastage Training" href="datastage-training" target="_blank" rel="noopener">Datastage Training</a>
</blockquote>
<h2>XML </h2>
(a)&nbsp; oracle &ndash; XML format
(b)&nbsp;&nbsp;&nbsp; XML &ndash; file format
<ul>
<li>oracle enterprise <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />XML output</li>
</ul>
Columns <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Description <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;/EMP/ EMP No&nbsp;&nbsp; /EMP/ EMP NAME&nbsp;&nbsp; /EMP/ JOB&nbsp;/EMP/ MGR
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Transformation setting&nbsp;&nbsp;&nbsp; single row<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />option&agrave;write output to a file <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />new folder<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Data XML
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />XML OUT.XML<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />E :/Data XML/XML OUT.XML<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> ok.
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />We can see the XML files in the folder that we created.&nbsp; That is Data XML
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />each and every record is created in an XML format.
<h3>Sequential file &agrave;XML input&agrave;Sequential file </h3>
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> File pattern &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Dataxml.xml <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />load with single column rec varchar 100
Input <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />XML Source column &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />xml document <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />output<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> columns <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;load <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;import xml Table Definition &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />open <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Data XML.
<h2>For an in-depth understanding of DataStage click on</h2>
<ul>
<li class="title"><a href="https://tekslate.com/introduction-datastage/" target="_blank" rel="noopener">Introduction to DataStage</a></li>
<li class="title"><a href="https://tekslate.com/architecture-data-stage/" target="_blank" rel="noopener">Architecture of Data Stage</a></li>
<li><a href="https://tekslate.com/oracle-enterprise-data-stage/" target="_blank" rel="noopener">Oracle Enterprise in Data Stage</a></li>
<li><a href="https://tekslate.com/scd-slow-changing-dimension-data-stage/" target="_blank" rel="noopener">SCD(Slow changing Dimension) in Data Stage</a></li>
<li><a href="https://tekslate.com/tutorials/datastage-tutorials/" target="_blank" rel="noopener">DataStage Tutorials</a></li>
</ul>

Staging area in Datastage

Staging Area in DataStage

staging-area-in-datastage

<h2 style="text-align: justify;">Job Sequencing</h2>
It is Controlling the order of execution.
Master Job
Can run multiple jobs with the same name
<img class="alignnone wp-image-7152" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_931-460x204.png" alt="Screenshot_93" width="426" height="189" />
Trigger&nbsp;
Checking the compilation of job 1,&nbsp; if it is compiled without errors, then job 2 is compiled.
Error handling&nbsp;
This situation arises when Job is Aborted
Notification
When everything is successful, we should get a confirmation mail that it is Successful.
Wait for file&nbsp;
Waits for the file, until it is searched and gets loaded into the source.
Checkpoints
If a job is aborted in between by any chance, check point make sure that the job starts from the point where it has failed.
Parameter Mapping&nbsp;
We cannot use a parameter defined in 1 job to another job, TO do this we use parameter Mapping.
Job sequencing Stages are categorized into 4 groups&nbsp;
<ol style="text-align: justify;">
<li>Run stages</li>
<li>Flow control stages</li>
<li>Error handling stages</li>
<li>Other stages</li>
</ol>
New <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Select Sequence job<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />click on view &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />click on palette option <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />palette opens
Designing a Master Job&nbsp;
To run a set of jobs in specific sequence &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> click view &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Repository <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Select Jobs <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Drag and Drop Required jobs.
<img class="alignnone size-full wp-image-7153" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_94.png" alt="Screenshot_94" width="457" height="125" />
Drag&nbsp; the Job Activity&nbsp; from the palette <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Right click <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Browse the job&nbsp;
Job Activity
Supports multiple outputs but only 1 i/p
Trigger
<img class="alignnone wp-image-7154" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_95-460x275.png" alt="Screenshot_95" width="498" height="297" />
Sequencer
Multiple&nbsp; inputs and multiple outputs
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />here we are executing Trigger Stage that is job finished with no errors/ Warnings.
Error handling
If job&nbsp; aborts, we use Terminator
<img class="alignnone wp-image-7155" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_96-460x180.png" alt="Screenshot_96" width="441" height="172" />
Master page
<img class="alignnone wp-image-7156" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1112-460x355.png" alt="111" width="490" height="377" />
<pre style="text-align: center;">Interested in mastering <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage&nbsp;Training</a>?
Enroll now for a FREE demo on DataStage Training.</pre>
Exception handling <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Notification Activity <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Terminator
If Server is failed <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Exception handling
<h3 style="text-align: justify;">Job properties</h3>
Automatically handle Activities that fail <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />If this is enabled, then only exceptions will perform and continues the job
Notification Activity&nbsp;
Job finishes &ndash; mail has to be delivered
Aborts &ndash; mail
Server down &ndash; mail
Wait for file Activity
Properties <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />browse the record file<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> wait for file to appear do not timeout
Check points
Job properties
Add checkpoints to sequence is restartable on failure
Click on Job Activity
We can see the option<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Do not check point run
Parameter Mapping
Jobs <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Insert parameter <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Job properties <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Add parameters
Oracle&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; oracle parameter&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parameter set
D no &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;deptn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Integer
&nbsp;

Job Sequencing in Data Stage - Tekslate

Job Sequencing in Data Stage

job-sequencing-data-stage

<h2 style="text-align: justify;">Partitioning Technique With Performance Tuning</h2>
Partitioning is the process of dividing an input data set into multiple segments, or partitions. Each processing node in your system then performs an operation on an individual partition of the data set rather than on the entire data set.
&nbsp;
<h2>Data Stage basically allows 2 types of partitioning</h2>
<ul>
<li style="text-align: justify;"><a href="#key-based-technique">Key Based Partitioning</a></li>
<li style="text-align: justify;"><a style="color: #236fa1;" href="#keyless-techniques">Keyless Partitioning</a></li>
</ul>
How to tune the jobs using the partitioning Technique?
&nbsp;
<h3 style="text-align: justify;"><a id="key-based-technique"></a>Key-based Technique</h3>
<ul>
<li>Hash</li>
<li>Modulus</li>
<li>Range</li>
<li>DB/2</li>
</ul>
<table style="border-collapse: collapse; width: 100%; height: 26px; background-color: #ecf0f1;" border="1px" cellpadding="10">
<tbody>
<tr style="height: 26px;">
<td style="width: 98.3496%; height: 26px; text-align: center;">Want to acquire industry skills and gain complete knowledge of Datastage? Enroll in Instructor-Led live <a style="color: #236fa1;" href="datastage-training" target="_blank">Datastage Training</a> to become Job Ready!</td>
</tr>
</tbody>
</table>
<h4>Hash</h4>
Rows with the same key column (or multiple columns) go to the same partition. <a title="Hash Maps in TeraData" href="https://tekslate.com/hash-map-teradata/" target="_blank" rel="noopener">Hash</a> is very often used and sometimes improves performance, however it is important to have in mind that hash partitioning does not guarantee load balance and misuse may lead to skew data and poor performance.
<h4>Modulus</h4>
Data is partitioned&nbsp;on one specified numeric field by calculating modulus against the number of partitions. Not used very often.
<h4>Range</h4>
An expensive refinement to hash partitioning. It is similar to hash but partition mapping is user-determined and partitions are ordered. Rows are distributed according to the values in one or more key fields, using a range map (the 'Write Range Map' stage needs to be used to create it). Range partitioning requires processing the data twice which makes it hard to find a reason for using it.
&nbsp;
<h3 style="text-align: justify;"><a id="keyless-techniques"></a>Keyless Techniques</h3>
<ul>
<li style="text-align: justify;">Same</li>
<li style="text-align: justify;">Entire</li>
<li style="text-align: justify;">Round Robin</li>
<li style="text-align: justify;">Random</li>
</ul>
<h4>Entire</h4>
All rows from a dataset are distributed to each partition. Duplicated rows are stored and the data volume is significantly increased.
<h4>Same</h4>
Existing partitioning remains unchanged. No data is moved between nodes.
<h4>Round robin</h4>
Rows are alternated evenly across partitions. This partitioning method guarantees an exact load balance (the same number of rows processed) between nodes and is very fast.
<h4>Random</h4>
Rows are randomly distributed across partitions
All key-based stages, by default, are associated with Hash as a Key-based Technique.
<h3>Hash Technique</h3>
Hash Partitioning is one of the most popular and frequently used techniques in the Data Stage. Under this part, we send data with the Same Key Colum to the same partition.
Example
<img class="aligncenter wp-image-7159" title="Partitioning Technique in DataStage-Hash Technique" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_97-460x219.png" alt="Partitioning Technique in DataStage" width="486" height="230" />
Same Key Column Values are Given to the Same Node.
Hash partitioning Technique can be Selected into 2 cases
<ul style="text-align: justify;">
<ul style="text-align: justify;">
<li>If Key Column &gt; 1</li>
<li>If key column = 1, other than Integer</li>
</ul>
</ul>
<table style="border-collapse: collapse; width: 100%; height: 29px; background-color: #ecf0f1;" border="1px" cellpadding="10">
<tbody>
<tr style="height: 29px;">
<td style="width: 98.3496%; height: 29px; text-align: center;">Check out our tutorial on the <a style="color: #236fa1;" href="https://tekslate.com/datastage-tutorials" target="_blank">Datastage</a>!</td>
</tr>
</tbody>
</table>
<h3 style="text-align: left;">&nbsp;Why Modulus?</h3>
<ul style="text-align: justify;">
<li>Modules are having good performance when compared to the hash.</li>
<li>In modules, it distributes the data by calculating MOC Value</li>
</ul>
<a href="https://tekslate.com/tutorials/datastage-tutorials/"><img class="aligncenter wp-image-7160 size-full" title="DataStage Tutorials-Partitioning Technique" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_98.png" alt="MOD" width="263" height="215" /></a>
Note:&nbsp;Modules is Having Good performance than Hash
MOD is selected, when it has only 1 key column, and it is an integer.
Join <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Dept no, E no
We use Hash,&nbsp; As key column &gt; 1
That is dept no, E no (we have 2 key column values)
<img class="aligncenter wp-image-7161" title="DataStage tutorials-Partitioning Technique in DataStage" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_99-460x227.png" alt="Partitioning Technique in DataStage" width="438" height="216" />
because, we want the same distribution from joining to Aggregator, and key column value is also the same, so we use SAM <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;SAME is Key less
<img class="aligncenter wp-image-7162 size-full" title="DataStage tutorials-Partitioning Technique in DataStage" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_100.png" alt="Partitioning Technique" width="386" height="213" />
We cant use SAME, here, as join has 2 key column values D no, loc, if we use SAME, We don&rsquo;t Know that what data is getting from joining to Aggregator.
Modulus &agrave; because of only 1 key column (D NO) and Integer.
<img class="aligncenter wp-image-7163" title="DataStage tutorials-Partitioning Technique in DataStage" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_1011-460x163.png" alt="Partitioning Technique in DataStage" width="419" height="148" />
For an in-depth understanding of DataStage click on
<ul>
<li class="title"><a style="color: #236fa1;" title="Introduction to DataStage" href="https://tekslate.com/introduction-datastage/" target="_blank" rel="noopener">Introduction to DataStage</a></li>
<li class="title"><a style="color: #236fa1;" title="Architecture of Data Stage" href="https://tekslate.com/architecture-data-stage/" target="_blank" rel="noopener">Architecture of Data Stage</a></li>
<li><a style="color: #236fa1;" title="Oracle Enterprise in Data Stage" href="https://tekslate.com/oracle-enterprise-data-stage/" target="_blank" rel="noopener">Oracle Enterprise in Data Stage</a></li>
<li><a style="color: #236fa1;" title="Slow changing Dimension in Data Stage" href="https://tekslate.com/scd-slow-changing-dimension-data-stage/" target="_blank" rel="noopener">SCD(Slow changing Dimension) in Data Stage</a></li>
<li><a style="color: #236fa1;" title="DataStage Tutorials" href="https://tekslate.com/tutorials/datastage-tutorials/" target="_blank" rel="noopener">DataStage Tutorials</a></li>
</ul>

Partitioning Technique in DataStage

partitioning-technique-in-datastage

<h3>Key less Technique</h3>
SAME To more previously staged partitioning to next corresponding stage with&nbsp;&nbsp;&nbsp; same&nbsp; key column values. ENTIRE Can only be used in&nbsp; lookup stage with reference&nbsp; link when it is &ldquo;normal lookup&rdquo; <img class="alignnone size-full wp-image-7166" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_1021.png" alt="Screenshot_102" width="269" height="126" /> &nbsp; In Entire what ever the records that are processing on N1, the same set of records are processed on the rest of the node. <img class="alignnone size-full wp-image-7167" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_103.png" alt="Screenshot_103" width="330" height="124" />
<pre style="text-align: center;">At TekSlate, we offer resources that help you in learning various IT courses.
We avail both written material and demo video tutorials.
To gain in-depth knowledge and be on par with &nbsp;practical experience, then explore<a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">&nbsp;DataStage Training Videos</a>.</pre>
RANDOM - Default / Auto Round Robin Generates the next number with sequence Note :- All the Generating stages gives the best performance when they are associated with Round Robin.
<h3>Generating &nbsp;Stages</h3>
column generator , Row generator, Surrogate Key stage RANGE&nbsp; It is an Expensive partitioning Technique than&nbsp; Mod and Hash . This technique&nbsp; needs data pre processing. Note :- Stage&nbsp; :- Write Range Map (Development/ Debug stage) <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;To create the range if we want to use Range partitioning Web Console log in to &ldquo;IBM information Server&rdquo; with Same designer ID and Password&agrave;Administration <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />users and groups <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />users <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />user <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />open
<ul>
<li>To create New user <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;New</li>
<li>Do main Management <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;license up gradations</li>
<li>Session Management <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />login time</li>
<li>Users and groups</li>
<li>Log Management <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Access all logic from Director</li>
<li>Scheduling Management <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Scheduling vices</li>
</ul>
Reports We&nbsp; can view and generate reports Web console Admin<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> DS Admin<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Project level Administration&nbsp; (or)&nbsp; Repository level
<h2>Brief on&nbsp;Surrogate key in DataStage</h2>
Surrogate key
<ul>
<li>A kind of Alternative primary key</li>
<li>Generate sequence number</li>
</ul>
Output
<table style="height: 275px;" width="569">
<tbody>
<tr>
<td width="97">Sid</td>
<td width="97">Cid</td>
<td width="97">CN</td>
<td width="97">Add</td>
</tr>
<tr>
<td width="97">1</td>
<td width="97">11</td>
<td width="97">A</td>
<td width="97">HYD</td>
</tr>
<tr>
<td width="97">2</td>
<td width="97">22</td>
<td width="97">B</td>
<td width="97">SEC</td>
</tr>
<tr>
<td width="97">3</td>
<td width="97">33</td>
<td width="97">C</td>
<td width="97">Del</td>
</tr>
<tr>
<td width="97">4</td>
<td width="97">22</td>
<td width="97">B</td>
<td width="97">GNT</td>
</tr>
<tr>
<td width="97">5</td>
<td width="97">44</td>
<td width="97">E</td>
<td width="97">PUN</td>
</tr>
</tbody>
</table>
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />If we don&rsquo;t take surrogate key <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />it maintains a new column active flag(indicate current address) o/p:
<table>
<tbody>
<tr>
<td width="97">CID</td>
<td width="97">CN</td>
<td width="97">ADD</td>
<td width="97">A.F</td>
</tr>
<tr>
<td width="97">11</td>
<td width="97">A</td>
<td width="97">Hyd</td>
<td width="97">Y</td>
</tr>
<tr>
<td width="97">22</td>
<td width="97">B</td>
<td width="97">SEC</td>
<td width="97">Y/N</td>
</tr>
<tr>
<td width="97">&nbsp;33</td>
<td width="97">C</td>
<td width="97">Del</td>
<td width="97">Y</td>
</tr>
<tr>
<td width="97">&nbsp;22</td>
<td width="97">B</td>
<td width="97">GNT</td>
<td width="97">Y</td>
</tr>
<tr>
<td width="97">44</td>
<td width="97">E</td>
<td width="97">PUN</td>
<td width="97">Y</td>
</tr>
</tbody>
</table>
Ex: After data;(source)
<table style="height: 184px;" width="300">
<tbody>
<tr>
<td width="52">CID</td>
<td width="52">CN</td>
<td width="52">ADD</td>
</tr>
<tr>
<td width="52">11</td>
<td width="52">A</td>
<td width="52">HYD</td>
</tr>
<tr>
<td width="52">22</td>
<td width="52">B</td>
<td width="52">Cal</td>
</tr>
<tr>
<td width="52">66</td>
<td width="52">G</td>
<td width="52">ASm</td>
</tr>
</tbody>
</table>
Before Data(Target)
<table>
<tbody>
<tr>
<td width="54">CID</td>
<td width="54">CN</td>
<td width="54">ADD</td>
<td width="54">A.F</td>
</tr>
<tr>
<td width="54">11</td>
<td width="54">A</td>
<td width="54">Hyd</td>
<td width="54">Y</td>
</tr>
<tr>
<td width="54">22</td>
<td width="54">B</td>
<td width="54">SEC</td>
<td width="54">Y</td>
</tr>
<tr>
<td width="54">&nbsp;33</td>
<td width="54">C</td>
<td width="54">Del</td>
<td width="54">Y</td>
</tr>
</tbody>
</table>
Output Before data(Target)
<table>
<tbody>
<tr>
<td width="97">CID</td>
<td width="97">CN</td>
<td width="97">ADD</td>
<td width="97">A.F</td>
</tr>
<tr>
<td width="97">11</td>
<td width="97">A</td>
<td width="97">Hyd</td>
<td width="97">Y</td>
</tr>
<tr>
<td width="97">22</td>
<td width="97">B</td>
<td width="97">SEC</td>
<td width="97">Y/N</td>
</tr>
<tr>
<td width="97">&nbsp;33</td>
<td width="97">C</td>
<td width="97">Del</td>
<td width="97">Y</td>
</tr>
<tr>
<td width="97">&nbsp;22</td>
<td width="97">B</td>
<td width="97">GNT</td>
<td width="97">Y</td>
</tr>
<tr>
<td width="97">66</td>
<td width="97">G</td>
<td width="97">ASM</td>
<td width="97">Y</td>
</tr>
</tbody>
</table>
After Data: (Source)
<table>
<tbody>
<tr>
<td width="52">CID</td>
<td width="52">CN</td>
<td width="52">ADD</td>
</tr>
<tr>
<td width="52">11</td>
<td width="52">A</td>
<td width="52">GNT</td>
</tr>
<tr>
<td width="52">22</td>
<td width="52">B</td>
<td width="52">DEL</td>
</tr>
<tr>
<td width="52">55</td>
<td width="52">H</td>
<td width="52">MP</td>
</tr>
</tbody>
</table>
Output
<table>
<tbody>
<tr>
<td width="93">CID</td>
<td width="93">CN</td>
<td width="93">Add</td>
<td width="93">A.F</td>
<td width="93">Effective start date</td>
<td width="93">Effective end date</td>
</tr>
<tr>
<td width="93">11</td>
<td width="93">A</td>
<td width="93">Hyd</td>
<td width="93">Y/N</td>
<td width="93">3-6-06</td>
<td width="93">(9999-12-31) 26-10-10</td>
</tr>
<tr>
<td width="93">22</td>
<td width="93">B</td>
<td width="93">SEC</td>
<td width="93">Y/N</td>
<td width="93">3-6-06</td>
<td width="93">(99-12-31) 27-9-07</td>
</tr>
<tr>
<td width="93">33</td>
<td width="93">C</td>
<td width="93">Del</td>
<td width="93">Y</td>
<td width="93">3-6-06</td>
<td width="93">99-12-31</td>
</tr>
<tr>
<td width="93">22</td>
<td width="93">B</td>
<td width="93">Cal</td>
<td width="93">Y/N</td>
<td width="93">28-9-07</td>
<td width="93">(99-12-31) 26-10-10</td>
</tr>
<tr>
<td width="93">6</td>
<td width="93">G</td>
<td width="93">ASm</td>
<td width="93">Y</td>
<td width="93">28-9-07</td>
<td width="93">99-12-31</td>
</tr>
<tr>
<td width="93">11</td>
<td width="93">A</td>
<td width="93">GNT</td>
<td width="93">Y</td>
<td width="93">27-10-10</td>
<td width="93">99-12-31</td>
</tr>
<tr>
<td width="93">22</td>
<td width="93">B</td>
<td width="93">DEL</td>
<td width="93">Y</td>
<td width="93">27-10-10</td>
<td width="93">99-12-31</td>
</tr>
<tr>
<td width="93">55</td>
<td width="93">H</td>
<td width="93">MP</td>
<td width="93">Y</td>
<td width="93">27-10-10</td>
<td width="93">99-12-31</td>
</tr>
</tbody>
</table>
&nbsp;Active Tag :- (A.F) <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />New&nbsp; records will maintain flag &lsquo;Y&rsquo; <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Old records &lsquo;N&rsquo; <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Tables will data maintain effective start data and effective end data <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />If data is not displayed, then we get a new column record version , unique id
<pre style="text-align: center;">&nbsp;</pre>
Record version, unique id <img class="alignnone wp-image-7109" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_731-460x215.png" alt="Screenshot_73" width="512" height="239" /> SCD&nbsp; - Type&nbsp; Table 1 Create table SRC&nbsp;&nbsp; (SNO number (3), S name&nbsp;&nbsp; Varchar 2 (25));<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Create the table and reject values Table 2 Create table CIM&nbsp;&nbsp; (SNO number (2), S name&nbsp;&nbsp; Varchar 3 Sname varchar2 (25));<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Just create the table only 5 steps for SCD &ndash; Type 1&nbsp;
S1:- Select output link as Fact link
S2: Map the lookup key SNO
S3: Source type&nbsp; = flat file
= Source Name&nbsp;&nbsp;&nbsp;&nbsp; = (Empty file Name)
S4 : identifying the purpose of each field
S5: Map required fields to fact table
<h3>Load properties</h3>
S1: Set Data set properties S2: Set oracle properties Table name = dim 1 Write method = upset&nbsp; (update and insert) Upset order = update then insert lab :-&nbsp; SCD &ndash; 1 Oracle enterprise 0 load&nbsp; dim 1 Oracle enterprise 1&nbsp; Load &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;src Sno&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Sname 111&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;shilpa 222&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; repeka 333&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Archama SCD
<ol>
<li>Select output link as fact</li>
<li>Map the look up key sno</li>
<li>Source type = flat file</li>
</ol>
Source&nbsp;&nbsp; name&nbsp; = E :/shilpa/scd1.txt&nbsp;&nbsp;&nbsp;&nbsp; empty file <img class="alignnone wp-image-7110" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_75-460x135.png" alt="Screenshot_75" width="443" height="130" />
<ol>
<li>Map required fields to fact&nbsp; table</li>
</ol>
Compile and RUN (b) Data set ----------------------&agrave;oracle enterprise &agrave; In table&nbsp; (a) , save the columns is Data set 1 and also copy the path of div 1 data set Now, is this job Data set<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />paste the path is the file <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;columns <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />load <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> (SCD)&nbsp;

Key less Technique in Data Stage

key-less-technique-data-stage

<h3>Administrator</h3>
For a Specific project Project Add create protected project Ok In, if we don&rsquo;t have web console, In 8 we have wed console, and same permissions are given here Add check points. Tracing&nbsp; Reporting the activities&nbsp; that are requested by&nbsp; the client at server . How can you delete logs in Repository ? Sol :- Auto purge&nbsp; of Job logs. RCP:-&nbsp; Job properties&nbsp; :-&nbsp; Entire RCP
<ul>
<li>Enable Runtime column propagation <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Violating Mapping constraints According to Requirements</li>
</ul>
Data Quality <img class="alignnone wp-image-7171" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_104-460x373.png" alt="Screenshot_104" width="414" height="335" /> &nbsp; <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />1 person data is stored in 3 different ways such a data is called Dirty Data. <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />for,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Credit :&nbsp; 100 cores&nbsp; * 100 customers&nbsp; = 10, 000 cores But here out of 100 %, only 60 % are unique customers
<ul>
<li>100 = 6000 cores</li>
</ul>
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />4000 cores is the un necessary cost used. &agrave;Here IBM comes in to picture&nbsp; and says that pay me 1000 cores, using Data quality we eliminate remaining 40 % <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />So, here using Data Quality , we are eliminating the Duplicate records.
<pre style="text-align: center;">Inclined to build a profession as DataStage Developer? Then here is the blog post on <a href="https://tekslate.com/datastage-training/" target="_blank" rel="noopener">DataStage&nbsp;Training</a>.</pre>
<h3>Reasons&nbsp; for Dirty data</h3>
<ol>
<li>lock of information Standards</li>
<li>1 field is used for Multiple</li>
</ol>
Data in a data base is after misplaced. Data Redundancy Data is required in to Systems, because the data entry mechanism is not aware that the original&nbsp; record is already there. To perform Data Quality , we have 4 stages :- &nbsp; <img class="alignnone wp-image-7172" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_105-460x163.png" alt="Screenshot_105" width="481" height="170" /> &nbsp; &nbsp;
<h3>Survive Stage :-&nbsp; Consolidate</h3>
<img class="alignnone wp-image-7173" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_106-460x224.png" alt="Screenshot_106" width="503" height="245" /> &nbsp;
<h3>Power of Data Quality in Result set</h3>
Rules set<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />For name For address For Social Security EX :- for Name :- &nbsp; <img class="alignnone wp-image-7174" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_107-460x119.png" alt="Screenshot_107" width="397" height="102" /> &nbsp; Designer&nbsp; View <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Repository <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;Standardization&nbsp;&nbsp;&nbsp; Rules<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />USA <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />US NAME &nbsp; EX :- <img class="alignnone wp-image-7175" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_108-460x337.png" alt="Screenshot_108" width="546" height="399" /> &nbsp; &nbsp; (c) Container -&nbsp; Standardize &nbsp; <img class="alignnone wp-image-7176" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_109-460x182.png" alt="Screenshot_109" width="511" height="201" />

Administrator in Data Stage

administrator-data-stage

<h3>RESUME Preparation Tips :-</h3>
&nbsp; Resume&nbsp; and&nbsp; C.V both mean the Same
<h2>&nbsp; &nbsp; &darr; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&darr;</h2>
British&nbsp; English &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;American English &nbsp;
<h4>Standard format :-</h4>
Arial or Times New Roman Font size&nbsp; = 12
<ul>
<li>Let the Entire 1st page be your professional Summary.</li>
<li>What technologies you know, how good you are ?</li>
<li>All Technologies in Descending</li>
</ul>
&nbsp; Total Exp:- Relevant EXP :- Current location :- Current company :- Notice period : Name : E mail &ndash; id&nbsp; : Mobile :- &nbsp;
<h4>Professional Summary :-</h4>
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> &nbsp; Technical Skills :- ETL Tools :- Data stage : Language :- Academics : Project : Title : Client : Tools : Description : &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;

RESUME Preparation Tips  in Data Stage

resume-preparation-tips-data-stage

In general, we have 4 &nbsp;Architecture
<ul>
<li>Simple</li>
<li>ODS&nbsp; - Operations Data source</li>
<li>NDS &ndash; Normal&nbsp; Data source</li>
<li>Integrity &nbsp;</li>
</ul>
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> we can implement this Architecture either in Inmon Model or Kimball Model
Top to bottom&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Bottom to top
<img class="alignnone size-full wp-image-7180" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_1101.png" alt="Architecture" width="348" height="126" /> &nbsp;
Universally&nbsp; -&nbsp;&nbsp; 8 Architecture
Inmon &ndash; Simple, ODS,&nbsp; NDS, Integrity
Kimbell - Simple, ODS,&nbsp; NDS, Integrity.
<h3>Simple</h3>
SRC ---- WH
<img class="alignnone size-full wp-image-7181" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_1111.png" alt="Simple Architecture" width="153" height="142" />
<h3>&nbsp;ODS</h3>
<img class="alignnone size-full wp-image-7184" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_1121.png" alt="ODS Architecture" width="354" height="205" /> &nbsp;
<ul>
<li>Inmon Models are preferred when Requirements are very clear.</li>
<li>Kim bell Model is preferred when&nbsp;&nbsp; Requirements are not clear. &nbsp;</li>
<li>For the project, we deal with</li>
</ul>
<ol>
<li>ODS --&agrave;80% ODS Model</li>
<li>NDS</li>
</ol>
<h3>WHY ODS?</h3>
Sol:- when&nbsp; Business Requirements are defined, click/customer requests the team&nbsp; what we want exactly
. &nbsp; <img class="alignnone wp-image-7185" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_1131-460x406.png" alt="ODS" width="388" height="341" /> &nbsp;
<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Project Architecture depends on no. of operational and Analysis Reports.
<ul>
<li>More priority is given to operational and Analysis Reports</li>
<li>Are we have more operational Reports we use ODS.</li>
<li>Less operational Reports and more Analytical Reports, we use NDS.</li>
<li>In NDS, we can&rsquo;t See ODS, But in ODS, we can find warehouse and Data marts (NDS).</li>
<li>ODS gets refreshed every 10 seconds.</li>
<li>ODS dim Table will always be type 1 (No History)</li>
</ul>
&nbsp; <img class="alignnone size-full wp-image-7186" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_1141.png" alt="ODS" width="431" height="383" /> &nbsp; &nbsp;
<h2>Architecture</h2>
<img class="alignnone wp-image-7187" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_1151-460x306.png" alt="Architecture" width="454" height="301" /> &nbsp;
<h2>Data Acquisition&nbsp;</h2>
Pull &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> You get&nbsp; (take) the data
Push <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />he sends the data
DRA:-&nbsp; Data Repository Area
CSA:- Common Staging Ares
DRA:- Will get only files&nbsp; -&nbsp;&nbsp;&nbsp; Memory which is accessed by different departments.
Data Query:-&nbsp;&nbsp; When the&nbsp; data is being such&nbsp; what the data should contain
Quality Gates:-&nbsp; Requirements .
Every file should have a specific file name:-
<img class="alignnone size-full wp-image-7188" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_116.png" alt="Control info" width="388" height="128" />
<h3>Script&nbsp;</h3>
To validate the files according to Quality&nbsp; Gates.
<img class="alignnone wp-image-7189" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_117-460x327.png" alt="Script" width="482" height="342" />
<blockquote>
Inclined to build a profession as DataStage Developer? Then here is the blog post on, explore <a title="Datastage Training" href="datastage-training" target="_blank" rel="noopener">Datastage Training</a>
</blockquote>
<h3>Loading </h3>
Loading, Stage, and warehouse is a Single Data stage.
Loading<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Staging&nbsp; ODS <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" /> Warehousing &nbsp;<img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Box size of 100 Tear loading &nbsp;
10 TB <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />&nbsp;loading
30 TB <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Staging
60 TB <img class="alignnone size-full wp-image-7108" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/1111.png" alt="11" width="37" height="12" />Warehouse. &nbsp;
Dumping the data from DRA to loading &agrave; a copy of data
&nbsp; <img class="alignnone wp-image-7190" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2014/12/Screenshot_118-460x457.png" alt="Loading" width="459" height="456" /> &nbsp;
When we load Data to Target data, We maintain all details, that is sequence no;
<h2>For an in-depth understanding of DataStage click on</h2>
<ul>
<li class="title"><a href="https://tekslate.com/introduction-datastage/" target="_blank" rel="noopener">Introduction to DataStage</a></li>
<li class="title"><a href="https://tekslate.com/architecture-data-stage/" target="_blank" rel="noopener">Architecture of Data Stage</a></li>
<li><a href="https://tekslate.com/oracle-enterprise-data-stage/" target="_blank" rel="noopener">Oracle Enterprise in Data Stage</a></li>
<li><a href="https://tekslate.com/scd-slow-changing-dimension-data-stage/" target="_blank" rel="noopener">SCD(Slow changing Dimension) in Data Stage</a></li>
<li><a href="https://tekslate.com/tutorials/datastage-tutorials/" target="_blank" rel="noopener">DataStage Tutorials</a></li>
</ul>

Project Architecture of Datastage

Project Architecture in DataStage

project-architecture-data-stage

Performance Tuning in DataStage

performance-tuning

<h2 style="text-align: justify;">Types of partition </h2>
<ul>
<li style="text-align: justify;">Hash</li>
<li style="text-align: justify;">Modulus</li>
<li style="text-align: justify;">DB2</li>
<li style="text-align: justify;">Auto</li>
<li style="text-align: justify;">Random</li>
<li style="text-align: justify;">Range</li>
<li style="text-align: justify;">Round Robin</li>
<li style="text-align: justify;">Entire</li>
<li style="text-align: justify;">Same</li>
</ul>
<h2 style="text-align: justify;">Key-based partition</h2>
<ul style="text-align: justify;">
<li>Hash</li>
<li>Modulus</li>
<li>DB2</li>
</ul>
<h3 style="text-align: justify;">Hash</h3>
Determines partition based on key-value(s). One or more keys with different data types are supported.
Example: Key is State. All &ldquo;CA&rdquo; rows go into one partition; all &ldquo;MA&rdquo; rows go into one partition. Two rows of the same state never go into different partitions.
<h3 style="text-align: justify;">Modulus</h3>
Partition based on the modulus of key divided by the number of partitions. The Key is an Integer type.
Example: Key is OrderNumber (Integer type). Rows with the same order number will all go into the same partition.
<h3 style="text-align: justify;">DB2</h3>
Matches DB2 EEE partitioning
Example: This partition is used when loading data into the DB2 table. It takes the partition key from the loading DB2 table and inserts the records effectively. If the partition key is defined in the DB2 database then it takes that Partition key otherwise it defaults to the primary key.
This figure gives a clear view of Key-based Partitioning and repartitioning.
<img class=" wp-image-20285 aligncenter" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2016/05/Capture.843.png" alt="DB2" width="941" height="203" />
Ascential parallel technology operates by a divide-and-conquer technique, splitting the largest integration jobs into subsets ("partition parallelism") and flowing these subsets concurrently across all available processors ("pipeline parallelism"). This combination of pipeline and partition parallelism delivers true linear scalability (defined as an increase in performance proportional to the number of processors) and makes hardware the only mitigating factor to performance. However, downstream processes may need data partitioned differently. Consider a transformation that is based on customer last name, but the enriching needs to occur on zip code - for house-holding purposes - with loading into the warehouse based on customer credit card number (more on parallel database interfaces below). With dynamic data re-partitioning, data is re-partitioned on-the-fly between processes - without landing the data to disk-based on the downstream process data partitioning needs.
<h3 style="text-align: justify;">Keyless partition</h3>
<ul>
<li style="text-align: justify;">Round Robin</li>
<li style="text-align: justify;">Entire</li>
<li style="text-align: justify;">Same</li>
</ul>
<h3 style="text-align: justify;">Round Robin</h3>
The first record goes to the first processing node, the second to the second processing node, and so on. When DataStage reaches the last processing node in the system, it starts over. This method is useful for resizing partitions of an input data set that are not equal in size. The round-robin method always creates approximately equal-sized partitions. This method is the one normally used when DataStage initially partitions data.
Example: Assume 8 nodes are allocated to store the records then 1st record will go into the first node and the 2nd record will go into the second node &hellip;&hellip;8th record will go into the eighth node and the 9th record will go into the first node and so on&hellip;.
<h3 style="text-align: justify;">Entire</h3>
Send all rows down all partitions
Example: Assume 8 nodes are allocated, then in all the 8 nodes all the records will be passed.
<h3 style="text-align: justify;">Same</h3>
Preserve the same partitioning.
Example: Two stages in a job (Sort and Dataset). In the sort stage, you have done &ldquo;Hash&rdquo; partition and in the dataset, you have given the &ldquo;Same&rdquo; partition. In the dataset, the data will be preserved with the hash partition.
Application Execution:
<ul>
<li style="text-align: justify;">Parallel jobs can be executed in two ways</li>
</ul>
<ol style="list-style-type: lower-alpha;">
<li style="text-align: justify;">Sequential</li>
<li style="text-align: justify;">Parallel</li>
</ol>
In the first slot of the below figure sequential execution is shown. A parallel job can be executed in two processing, SMP and MPP, in the second and third slots of the below figure it is shown.
<img class=" wp-image-20286 aligncenter" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2016/05/Capture.844.png" alt="Application Execution" width="683" height="371" />
<h2 style="line-height: normal; text-align: justify;">Best allocation</h2>
<h3 style="margin-left: 18pt; text-indent: -18pt; text-align: justify;">Best allocation of Partitions in DataStage for storage area</h3>
<table>
<tbody>
<tr>
<td width="60">S. No</td>
<td width="125">No of Ways</td>
<td width="109">Volume of Data</td>
<td width="127">Best way of Partition</td>
<td width="170">Allocation of Configuration File (Node)</td>
</tr>
<tr>
<td width="60">&nbsp;</td>
<td width="125">DB2 EEE&nbsp; extraction in serial</td>
<td width="109">Low</td>
<td width="127">-</td>
<td width="170">1</td>
</tr>
<tr>
<td width="60">2.</td>
<td width="125">DB2 EEE extraction in parallel</td>
<td width="109">High</td>
<td width="127">Node number = current node (key)</td>
<td width="170">64 (Depends on how many nodes are allocated)</td>
</tr>
<tr>
<td width="60">&nbsp;3.</td>
<td width="125">Partition or Repartition in the Stages of DataStage</td>
<td width="109">Any</td>
<td width="127">Modulus (It should be a single key that to integer) Hash (Any number of keys with different data type)</td>
<td width="170">8 (Depends on how many nodes are allocated for the job)</td>
</tr>
<tr>
<td width="60">&nbsp;4.</td>
<td width="125">Writing into DB2</td>
<td width="109">Any</td>
<td width="127">DB2</td>
<td width="170">-</td>
</tr>
<tr>
<td width="60">&nbsp;5.</td>
<td width="125">Writing into Dataset</td>
<td width="109">Any</td>
<td width="127">Same</td>
<td width="170">1,2,4,8,16,32,64 etc&hellip; (Based on the incoming records it writes into it.)</td>
</tr>
<tr>
<td width="60">&nbsp;6.</td>
<td width="125">Writing into Sequential File</td>
<td width="109">Low</td>
<td width="127">-</td>
<td width="170">1</td>
</tr>
</tbody>
</table>
<h3 style="text-align: justify;">Best allocation of Partitions in DataStage for each stage</h3>
<table>
<tbody>
<tr>
<td width="148">S. No</td>
<td width="148">Stage</td>
<td width="148">Best way of Partition</td>
<td width="148">Important points</td>
</tr>
<tr>
<td width="148">&nbsp;1.</td>
<td width="148">Join</td>
<td width="148">Left and Right link: Hash or Modulus</td>
<td width="148">All the input links should be sorted based on the joining key and partitioned with higher key order.</td>
</tr>
<tr>
<td width="148">&nbsp;2.</td>
<td width="148">Lookup</td>
<td width="148">Main link: Hash or same Reference link: Entire</td>
<td width="148">Both the links need not be in the sorted order</td>
</tr>
<tr>
<td width="148">&nbsp;3.</td>
<td width="148">Merge</td>
<td width="148">Master and update link: Hash or Modulus</td>
<td width="148">All the input links should be sorted based on the merging key and partitioned with higher key order. Pre-sort makes merge &ldquo;lightweight&rdquo; for memory.</td>
</tr>
<tr>
<td width="148">&nbsp;4.</td>
<td width="148">Remove Duplicate, Aggregator</td>
<td width="148">Hash or Modulus</td>
<td width="148">If the input link is in sorted order based on the key it will perform better.</td>
</tr>
<tr>
<td width="148">&nbsp;5.</td>
<td width="148">Sort</td>
<td width="148">Hash or Modulus</td>
<td width="148">Sorting happens after partitioning</td>
</tr>
<tr>
<td width="148">&nbsp;6.</td>
<td width="148">Transformer, Funnel, Copy, Filter</td>
<td width="148">Same</td>
<td width="148">None</td>
</tr>
<tr>
<td width="148">&nbsp;7.</td>
<td width="148">Change Capture</td>
<td width="148">Left and Right link: Hash or Modulus</td>
<td width="148">Both the input links should be in the sorted order based on the key and partitioned with higher key order.</td>
</tr>
</tbody>
</table>
<blockquote>
Inclined to build a profession as Datastage Developer? Then here is the blog post on, explore <a title="Datastage Training" href="datastage-training" target="_blank" rel="noopener">Datastage Training</a>
</blockquote>
<h2 style="text-align: justify;">Appropriate Ways</h2>
<h3 style="text-align: justify;">Step 1: (Serial extraction with proper partition)</h3>
In this job, extraction is made serial in both the DB2 stages. All the records will reside in 1 partition. As input link1 of the Sort stage has been &ldquo;hash&rdquo; partitioned, 1 partition will split into 8 partitions based on the given key(s), and records are also sorted based on the joining key(s). In the Join stage for link2, the &ldquo;same&rdquo; partition is given, and on the other side for link3 with the same key(s) &ldquo;hash&rdquo; partition and internal sorting is done so that records from 1 partition will split into 8 partitions and also sorted within the partition. Now joining will be done effectively and the records are captured in the dataset with the &ldquo;same&rdquo; partition, which will maintain the records in 8 partitions. For 1 partition, 1 page will be created. So, for the entire job, 32 pages will be created.
<img class=" wp-image-20287 aligncenter" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2016/05/Capture.845.png" alt="Serial extraction" width="626" height="163" />
<h4 style="text-align: justify;">Suggestion:</h4>
This will be one of the best ways to design the job for a huge volume of data. But, extracting the records in serial makes performance to go down. Multiple instances can be enabled in this job and can be called in parallel to attain parallelism.
<h3 style="text-align: justify;">Step 2: (Parallel extraction with proper partition)</h3>
This job is the same as the previous job but extraction alone made in parallel. In extracting the records in the DB2 side, 64 partitions will be created.
<img class=" wp-image-20288 aligncenter" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2016/05/Capture.846.png" alt="Parallel extraction" width="637" height="252" />
<h4 style="text-align: justify;">Suggestion:</h4>
This will be one of the best ways to design jobs for a huge volume of data. But, the number of database connections been established is huge.
<h4 style="text-align: justify;">Step 1:</h4>
As the extraction is done in a serial mode in both the DB2 stages, the allocated node will be 1 that is all the records will be in one partition and in all the remaining stages have the &ldquo;same&rdquo; partition, so the number of partition is retained as 1 in all the stages. So, the page created for this job will be 4.
<img class=" wp-image-20289 aligncenter" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2016/05/Capture.847.png" alt="step 1" width="675" height="234" />
Suggestion:
This job will work fine for the low records. But for the huge records, it will hang. Since the entire job is in serial mode.
<h4 style="text-align: justify;">Step 2:</h4>
Here the extraction is made parallel in both the DB2 stages. So, the allocated node will be 64 and in all the other stage the partition are &ldquo;same&rdquo; so for every stage the allocated node will be 64. So the page created for this job will be 256.
<img class=" wp-image-20290 aligncenter" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2016/05/Capture.848.png" alt="Step 2" width="626" height="209" />
Suggestion:
For less or huge records this job will work very fast but it will occupy maximum space in the system. For this job wastage of memory (Paging Issue may come up) happens because 256 pages will be created and sometimes this kind of job will make the entire system to come down.
<h4 style="text-align: justify;">Step 3:</h4>
An unpredictable thing may happen here because it all depends on the stages it will decide on its own since it is an auto partition.<img class=" wp-image-20293 aligncenter" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2016/05/Capture.849.png" alt="Step 3" width="635" height="261" />
Suggestion
Never use auto partition in the job.
<h4 style="text-align: justify;">Step 4:</h4>
In this case performance wise no issues but will lose the data on the Join stage because on the Link2 it is coming as 8 nodes and on the other side Link3 in 64 nodes so DataStage cannot join the data accurately since Join Stage is depended on the master link. In this case, Link2 is the master link so it will join only with the 8 nodes and it will leave the other 56 nodes which is coming from the Link3. Therefore data coming from that 56 nodes will not be considered.
<img class=" wp-image-20294 aligncenter" src="https://newtekslateassets.s3.us-west-2.amazonaws.com/wp-content/uploads/2016/05/Capture.850.png" alt="Step 4" width="644" height="257" />
Suggestion:
Never join the link without having the same partitioning key on both sides otherwise data loss may occur and also sort the records before joining.
<h2>For an in-depth Knowledge on DataStage, click on below</h2>
<ul>
<li><a href="https://tekslate.com/introduction-datastage/" target="_blank" rel="noopener">Introduction to DataStage</a></li>
<li><a href="https://tekslate.com/scheduling-jobs-in-datastage/" target="_blank" rel="noopener">Scheduling Jobs in DataStage</a></li>
<li><a href="https://tekslate.com/string-functions-datastage/" target="_blank" rel="noopener">String Functions in DataStage</a></li>
<li><a href="https://tekslate.com/partitioning-technique-in-datastage/" target="_blank" rel="noopener">Partitioning Technique in DataStage</a></li>
<li><a href="https://tekslate.com/datastage-interview-questions-answers" target="_blank" rel="noopener">DataStage Interview Questions &amp; Answers</a></li>
</ul>

DataStage - Types of Partition | TekSlate DataStage Tutorials

Types of Partition

different-types-of-partition-datastage