Showing posts with label Indexing. Show all posts
Showing posts with label Indexing. Show all posts

Sunday 6 August 2017

Implementing Partial Indexing in Endeca

Hi all,
    After Spending more time with the Partial Indexing, I am Writing this posts, so it could be useful for the other people trying to implement the Partial Indexing for their Projects.

Usually all the Projects, would have Configured the Baseline Indexing in the Scheduled Manner, If We have got the Requirement of Indexing the Inventory From Endeca , Will think of Having the Partial Indexing Configured for the Scheduled Manner, the Constraint is that it cannot pick up the Changes, Since the Inventory Comes or updates as part of the Feed Process.

Before pitching in to the Discussion of that, let’s see what are all can be achieved with the Partial Indexing.

When the Deployment is made from BCC by Creating a new Proj will update/add the Properties , if the Properties Belong to the Same Repository.ie Adding the Promotion will not reflect after the Deployment and the  Partial Indexing , the Reason is that the Promotion belongs to the Different Repository not the Product Catalog.

So , How Does the Flow Goes or what is Behind the Scenes of the Partial Indexing ? .
Before Knowing the Flow, we all should aware of the Basics of Partial Indexing.


Basics of Partial Indexing

There is a Repository called the Incremental Item Queue that is Responsible for Triggering the Partial Indexing. We have a Property called the Monitored OutputConfigs that should have our Config paths eg: ProductCatalogOutputConfig , Article OutputConfig.

There Will be Generation Id Associated With the Each OutputConfig Paths. So During Deployment the Sku or the Product, or Article Will be Added to this Incremental Item Queue Repository along with the Generation Id and the Item Descriptor . During the Partial Indexing this table will be Flushed out by submitting those to the Indexing . If the Products are not loading as part of the Partial Indexing after knowing that Items are avalaible in Incremental Item Queue Repository, then the Problem with miss match of the GenerationId.

You are also should be aware if the Tables from the DB Where these Operations take place.

srch_update_queue: this Table Where will be having the
config_repo_id will be the Config id of the output config path .
item_desc_name will be the Item Descriptor name .
repository_id is the Id of the updated product / Sku .
generation is the value of the Current id .
change_type is the value of the Change Type .
please find the Below Example .

values ('14500010','product','product101','2279','0');

srch_config this table is the Place where will be getting the Config Repoid and Generation for the Particular Config Path.

So now you are aware of the Flow of Partial Indexing Flow. If your Requirement to Indexing comes as part of the Feed then you have to identify the Changed Items and Write explicitly in to the Incremental Item Queue and Scheduled the Indexing of the Partial then it will take.

If your Property Belongs to the Different ItemDescriptior or Repository , make sure that you are making the Linking to any of the property in the Product Catalog  after linking only it will be taken as part of the Indexing otherwise will not be taken.
Only those Properties that will not be added as part of the BCC Deployment , We have to Write Explicitly in to the Incremental Item Queue. Others not Required .

So Keep this Information or the Constraint in mind Before Start Implementing to your Application .


Happy Learning.!!!!! 

Tuesday 11 July 2017

Same Dimension Value Across all the Environments

Hi All,
    Most of us , When having more Environments will face issues regarding the Dimension Values Generation. I Have Faced a lot , though am not hard-coding the Dimension Values, faced issues relating to recreating the refinements  if any Dimension Change occur . Now we are going to see how to Handle this .

When you have Indexing Successful in any environment, then Import its Dimension Values Using Below Command / Script

When the App is Created using the CRS Deployment Template, go to the Location /opt/endeca/apps/ATGen/test_data/ You Will Find a file called initial_dval_id_mappings.csv
Delete it and Proceed to Execte the Command, this Command has to be Executed from the CAS bin.

sh cas-cmd.sh exportDimensionValueIdMappings -m appName-dimension-value-id-manager -f /opt/endeca/apps/ATGen/test_data/initial_dval_id_mappings.csv 

Don't Forget to Create the Output File with the Following name initial_dval_id_mappings.csv , if you are Changing it you have to Change in the Script as well . It is better to use the Same .

Once this File is Created Check it out , you will have all the Dimensions Indexed as part of your Indexing. This File Has to be Moved Across all the Environment and Initialize_services.sh has to be called for the First Time. Once it is Called Going Forward all the Dimension values will remain same as the Lower Environment .

initial_dval_id_mappings.csv this file  will be called as part of the InitialSetup during the Initialize_Services.sh .

<script id="InitialSetup">
<bean-shell-script>
<![CDATA[
IFCR.provisionSite();
CAS.importDimensionValueIdMappings("TAen-dimension-value-id-manager",
InitialSetup.getWorkingDir() + "/test_data/initial_dval_id_mappings.csv");
]]>
</bean-shell-script>
</script>

Apart from this Approach we can also do the ImportDimensionValueIdMappings Script. Using the File Will make the Dimension Value not to Change for the Longer Run.


Friday 24 March 2017

Weblogic cxf logging services

When you face the below logs in web logic console and the console stopped unexpectedly or hangs or stepped unexpectedly , then you have to follow the below steps for resolving it.

Mar 24, 2017 6:44:55 PM org.apache.cxf.services.RecordStoreService.RecordStorePort.RecordStore
INFO: Outbound Message
---------------------------
ID: 1
Address: http://localhost:8500/ATGen-dimvals/
Encoding: UTF-8
Content-Type: text/xml
Headers: {Accept=[application/fastinfoset, */*], Accept-Encoding=[gzip;q=1.0, identity; q=0.5, *;q=0], SOAPAction=[""]}
Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><startTransaction xmlns="http://recordstore.itl.endeca.com/" xmlns:ns2="http://record.itl.endeca.com/"><transactionType>READ_WR
--------------------------------------
Mar 24, 2017 6:44:55 PM org.apache.cxf.services.RecordStoreService.RecordStorePort.RecordStore
INFO: Inbound Message
----------------------------
ID: 1
Response-Code: 200
Encoding: ISO-8859-1
Content-Type: application/fastinfoset
Headers: {Content-Length=[196], content-type=[application/fastinfoset], Date=[Fri, 24 Mar 2017 13:14:55 GMT], Server=[Jetty(6.1.26)]}
Payload: α ☺ 8╧♥soap(http://schemas.xmlsoap.org/soap/envelope/≡???Envelope???♥Body8╧☻ns2∟http://record.itl.endeca.com/═!http://recordstore.itl.endeca.com/≡=?↨startTransactionResponse=?♣return=?☺id?26   ≡

Step:1

Go to $WL_HOME/../user_projects/domains/base_domain/config/
And add the following In config.xml

<log-filter>
    <name>CXFFilter</name>
    <filter-expression>((SUBSYSTEM = 'org.apache.cxf.interceptor.LoggingOutInterceptor') OR (SUBSYSTEM = 'org.apache.cxf.interceptor.LoggingInInterceptor')) AND (SEVERITY = 'WARNING')</filter-expression>
  </log-filter>

 <server>
   <name>Prod</name>
   <log>
     <log-file-filter>CXFFilter</log-file-filter>
     <stdout-filter>CXFFilter</stdout-filter>
     <memory-buffer-severity>Debug</memory-buffer-severity>
   </log>
   <listen-port>7103</listen-port>
   <web-server>
     <web-server-log>
       <number-of-files-limited>false</number-of-files-limited>
     </web-server-log>
   </web-server>
   <listen-address></listen-address>
 </server>

For the server which you were using should be updated . with this filter .

You have to define for the server and after that restart the weblogic  admin and managed server .

You can also define this from WebLogic console which is very easy when compared to this.

<YOUR_DOMAIN>/logfilters/


create the new filter termed  CXFFilter for the expression give  the below 


((SUBSYSTEM = 'org.apache.cxf.interceptor.LoggingOutInterceptor') OR (SUBSYSTEM = 'org.apache.cxf.interceptor.LoggingInInterceptor')) AND (SEVERITY = 'WARNING')


then go to your corresponding server and select logging, for the LogFile and Standard out give the filter as CXFFilter , save and restart the server .



Step:2

Go to <JAVA_HOME>\jre\lib\logging.properties  and set the following properties

org.apache.cxf.interceptor.LoggingInInterceptor.level = WARNING
org.apache.cxf.interceptor.LoggingOutInterceptor.level = WARNING


After Setting up this then you have to Start the Server and start indexing . Happy Indexing !!!!!!

Thursday 2 March 2017

Triggering AutoIndexing in Stage Environment


Hi Guys , today I Gonna Share the WorkFlow for the Staging indexing , this tutorials will be very interesting with respect to the other functionalities because almost all is OOTB, you need to place right things In place.

ATG does not provide the automatic indexing enable for the staging env, you need to enable it .let’s find the detailed discussion below.
Before proceeding to this tutorial you need to make sure that you have the following modules built as part of your build.

BCC: DAF.Endeca.Index.Versioned,DCS. Endeca.Index.Versioned these two components are responsible for the Componets required for indexing in Staging env.

Staging: DAF.Endeca.Index,DCS.Endeca.Index

Once you are done with the above modules, you can proceed further.

1) Make sure to add the following outputconfig in /atg/search/repository/IndexingDeploymentListener

indexingOutputConfigs+=\
  /atg/commerce/search/ProductCatalogOutputConfig_staging

2) Make sure to update the staging details in following component /atg/search/SynchronizationInvoker_staging

host=10.20.30.70
port=1072

where 
host nameof staging
port  is staging rmi port

3)make sure that the /atg/commerce/search/IndexedItemsGroup_staging points to your productCatalog_Staging Repository to do that update the ProductCatalogOutputConfig_staging by following values

repositoryItemGroup=/atg/commerce/search/IndexedItemsGroup_staging/
repository^=/atg/commerce/search/IndexedItemsGroup_staging.repository

Only these changes are required to be done in the BCC Side.


We will see the Changes in the Staging side now

1) update the CategoryTreeService, RepositoryTypeDimensionExporter, SchemaExporter with the below changes

indexingOutputConfig=/atg/commerce/search/ProductCatalogOutputConfig_staging

2)update the ProductCatalogSimpleIndexingAdmin to points to /atg/commerce/search/ProductCatalogOutputConfig_staging during indexing .

phaseToPrioritiesAndTasks=\
  PreIndexing=5:CategoryTreeService,\
  RepositoryExport=10:\
    SchemaExporter;\
    CategoryToDimensionOutputConfig_staging;\
    RepositoryTypeDimensionExporter;\
    /atg/commerce/search/ProductCatalogOutputConfig_staging,\
  EndecaIndexing=15:EndecaScriptService

That’s it you are done with the Staging Indexing workflow .So after this changes once you deploy the project indexing will be triggered in both staging and production.

For Indexing only in Production you can refer my previous posts.


Happy Day

Saturday 17 December 2016

RuleSet in ATG / Indexed Item Group

HI all, I feel all are good and doing their best. Today we are going to see some interesting topic on the indexing part of the endeca. We cannot claim this topic to be fully endeca . Its ATG topic that we are utilizing for the indexing as well. It is  considered to be the simpler one but I am writing this blog to ease most of the logic we write during the accessor time.Its very effective and easy.

Ruleset can be defined as the couple of condition that is written for some specific purpose or does some specific purpose.As of my knowledge on endeca there is no mechsanism where we can restrict the products or the skus to be restricted from the search , apart from the recordfilter property constraints. If we implement the above constraint also that’s going to be the highly performance driven one. If that does not require more than reverting the code is also a complex one.
ProductCatalog outputconfig is a component that is responsible for providing the product for indexing . in the above component we can define the Repository Item group property to be pointing to the IndexedItem Group component where we will be defining the ruleset.

Rule set has two types of the condition.
1) Accepts rule.
2) Rejects rule.
These rules were called during the indexing process and corresponding filtration is done.  You can find the oracle document briefly describing about this. I am going to describe in terms of  Integration with the ATG and some of the problems we face during the development of the endeca.

Case:1
To Remove the Inactive products during the Indexing. We can define this type of condition in both accepts and reject clause.
 rules=\
<ruleset>\
<accepts>\
                <rule op\=eq>\
                                <valueof target\="isActive">\
                                <valueof constant\="true">\
                 </rule>\
</accepts>\
 </ruleset>

Case: 2
To remove only the specific set of items during the indexing. Where item may be product or some other repo item. Were id is the repo id.
rules=\
<ruleset>\
<rejects>\
                <rule op\=includes>\
   <valueof constant\="[item1,item2]">\
  <valueof target\="id">\
    </rule>\
    </rejects>\
</ruleset>

How to define in that component .

/atg/commerce/search/IndexedItemsGroup.properties

Define the ruleset in the string format is must. we cannot define the ruleset for the skulevel . Since the product-sku mapping has defined the sku as map .



Thursday 27 October 2016

Scheduled indexing in endeca

In Endeca Scheduled Indexing can be made by following the Below steps.

Step :1

Go to ProductCatalogSimpleIndexingAdmin

and Enable the Following property enableScheduledIndexing to true .

Step :2

Set the Calendar event in the Below Property

baselineSchedule=calendar * * 7 * 23 30

where 7 is the Week,23 30 is  it will trigger indexing daily by 23:30

 to trigger indexing for every one hour 

set the Following property to

baselineSchedule=calendar * * * * * 59

It will trigger for every one hour periodically.


Triggering Automatic mail After Indexing


Hi All Many of you where working in Endeca, will enable the Schedule Indexing, automatic Indexing and Waiting to See the Status of the indexing is Successful or not, will though to Trigger a Mail also after the Indexing. This tutorial will give you a Detailed Information how to check the Status of the Indexing and Trigger Mail after that.

I am Writing This Blog Assuming that you are intersted to trigger a mail after every Scheduled Indexing .


Step 1:

Extend the Class SimpleIndexingAdmin

Step 2:

Override the Method performScheduledTask

@Override
public void performScheduledTask(Scheduler pScheduler, ScheduledJob pJob)
  {
  long startTime = System.currentTimeMillis();
  long nextTime = pJob.getSchedule().getNextJobTime(startTime);
  boolean status=Boolean.FALSE;
  try {
  if (pJob == this.mBaselineScheduledJob) {
  status=indexBaseline();
 
  }
  else if (pJob == this.mPartialScheduledJob) {
  status=indexPartial();
  }
  else
  vlogError("Failed to find indexing job corresponding to {0}", new Object[] { pJob });
  }
  catch (IndexingException e) {
  vlogError(e, "Error while executing scheduled job {0}", new Object[] { pJob.getJobName() });

  long currentTime = System.currentTimeMillis();
  boolean enoughTime = nextTime - currentTime - this.mRetryInMs > 2L * (currentTime - startTime);

  if ((this.mRetryInMs > 0L) && (enoughTime)) {
  try {
  Thread.sleep(this.mRetryInMs);
  vlogInfo(e, "Retrying scheduled job {0} after a failure", new Object[] { pJob.getJobName() });

  if (pJob == this.mBaselineScheduledJob){
  status=indexBaseline();
 
  }
  else if (pJob == this.mPartialScheduledJob){
  status=indexPartial();
  }
  }
  catch (IndexingException ee) {
  vlogError(e, "Error while retrying scheduled job {0}", new Object[] { pJob.getJobName() });
  }
  catch (InterruptedException te) {
  vlogError(e, "InterruptedException while while retrying scheduled job {0}", new Object[] { pJob.getJobName() });
  }
  }
  else if ((this.mRetryInMs > 0L) && (!(enoughTime)))
  vlogInfo(e, "Not retrying scheduled job {0} - too little time till the next scheduled run", new Object[] { pJob.getJobName() });
  }
  if (getEmailConfiguration().isEnableEmailServices()) {
  @Override
public void performScheduledTask(Scheduler pScheduler, ScheduledJob pJob)
  {
  long startTime = System.currentTimeMillis();
  long nextTime = pJob.getSchedule().getNextJobTime(startTime);
  boolean status=Boolean.FALSE;
  try {
  if (pJob == this.mBaselineScheduledJob) {
  status=indexBaseline();
 
  }
  else if (pJob == this.mPartialScheduledJob) {
  status=indexPartial();
  }
  else
  vlogError("Failed to find indexing job corresponding to {0}", new Object[] { pJob });
  }
  catch (IndexingException e) {
  vlogError(e, "Error while executing scheduled job {0}", new Object[] { pJob.getJobName() });

  long currentTime = System.currentTimeMillis();
  boolean enoughTime = nextTime - currentTime - this.mRetryInMs > 2L * (currentTime - startTime);

  if ((this.mRetryInMs > 0L) && (enoughTime)) {
  try {
  Thread.sleep(this.mRetryInMs);
  vlogInfo(e, "Retrying scheduled job {0} after a failure", new Object[] { pJob.getJobName() });

  if (pJob == this.mBaselineScheduledJob){
  status=indexBaseline();
 
  }
  else if (pJob == this.mPartialScheduledJob){
  status=indexPartial();
  }
  }
  catch (IndexingException ee) {
  vlogError(e, "Error while retrying scheduled job {0}", new Object[] { pJob.getJobName() });
  }
  catch (InterruptedException te) {
  vlogError(e, "InterruptedException while while retrying scheduled job {0}", new Object[] { pJob.getJobName() });
  }
  }
  else if ((this.mRetryInMs > 0L) && (!(enoughTime)))
  vlogInfo(e, "Not retrying scheduled job {0} - too little time till the next scheduled run", new Object[] { pJob.getJobName() });
  }
  if (getEmailConfiguration().isEnableEmailServices()) {
  getEmailConfiguration().sendAutomatedEmail(status);
}
  }

Where getEmailConfiguration() is a GenericService class that will be acting as a Email Sender.

Step 3:

For this Tutorial am using the Template Email Sender that is coming as part of the ATG for sending the email.

Befor Going into Coding Part let us under stand the Basic Concepts.

TemplateEmailInfoImpl

/atg/scenario/DefaultTemplateEmailInfo  is a component responsible for holding the Content of the Mail, Of class TemplateEmailInfoImpl


 TemplateEmailSender

/atg/userprofiling/email/TemplateEmailSender is a component responsible for the Sending the Emails.
you have to inject the component and call its 

getTemplateEmailSender().sendEmailMessage(pTemplateEmailInfo, emailTo, true, false);

This method will send the email on the Successful trigger of Indexing .


public synchronized void sendAutomatedEmail(boolean pScuccess) {
if (isLoggingDebug()) {
logDebug("BEGIN:::sendEmail Method. ::");
}
if (pSuccess) {
pTemplateEmailInfo.setMessageTo(getEmailMessageTo());
emailTo[0] = pTemplateEmailInfo.getMessageTo();
pTemplateEmailInfo
.setMessageSubject(getSuccessSubjectMsg());
else {
pTemplateEmailInfo.setMessageTo(getEmailMessageTo());
emailTo[0] = pTemplateEmailInfo.getMessageTo();
pTemplateEmailInfo.setMessageSubject(getFailureSubjectMsg());
if (isLoggingDebug()) {
vlogDebug("Email sending Failure {0}",emailTo[0]);
}
}

try {

getTemplateEmailSender().sendEmailMessage(pTemplateEmailInfo, emailTo, true, false);
}catch(TemplateEmailException exp){
if (isLoggingError()) {
vlogError("Error in sending email from sendEndecaBaselineIndexingEmail", exp);
}
}

if (isLoggingDebug()) {
logDebug("END:::sendEmail Method. ::");
}
}

in the DefaultTemplateEmailInfo Define the Following for the Temaplate.It is Mandatory to Define the template Url. This is the JSP where we used to define the Content used to send.



Step 4:

$class=atg.userprofiling.email.TemplateEmailInfoImpl
contentProcessor=/atg/userprofiling/email/HtmlContentProcessor
templateURL=/email/mailtemplate.jsp
mailingName=Service-Email
fillFromTemplate=true

mailtemplate.jsp

<%@ taglib uri="/dspTaglib" prefix="dsp"%>
<%@ taglib uri="/dspELTaglib" prefix="dspel" %>
<%@ taglib uri="c" prefix="c"%>
<%@ taglib uri="fmt" prefix="fmt" %>
<dsp:page>
<div id="main">
Indexing process has completed.Please check the Status for more details.
</div>
</dsp:page>

Once if all is done then the Email will be triggered automatically.



Sunday 16 October 2016

Triggering Baseline Indexing & Checking the Indexing Status

Triggering Baseline Indexing


Step 1:

Navigate to Dyn/Admin of the instance where Indexing has to be triggered and Select the Component /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/







Concurrent Write Exception


 <CAS-Installed Dir>/CAS/11.1.0/bin

Then Execute the Following Script.

1)      sh cas-service-shutdown.sh

2)      sh cas-service.sh &


Once The above two steps are done then Trigger the Indexing again. It must succeed now.


 Checking the Indexing Status

 To check Indexing is working as Expected Follow the Below Steps.


Step 1:

Navigate to Dyn/Admin of the  

select the Component /atg/commerce/endeca/index/ProductCatalogSimpleIndexingAdmin/

Step 2:

Select the Indexing History.

This will show the list of indexing status, whether it is Succeed or failed.  In case it is failing then you have to follow the Triggering Baseline Indexing steps to start the Indexing again. 


Thursday 13 October 2016

Defining Accessors

How to DefineAccessors during Indexing 


Open the product-sku-output-config.xml and you can define by the Following way.

<property name="activeProperty" type="string" output-name="activeProperty"
property-accessor="/atg/commerce/endeca/index/accessor/activePropertyAccessor" is-non-repository-property="true"
            is-dimension="true" autogen-dimension-values="false" text-searchable="true" wildcard-searchable="true" />


You have to define the is-non-repository-property="true" for defining it as a Accessor Property. Then we have to define accessor for the Component path where we have written our accessor.



PropertyAccessors

Types of Property Accessors in Endeca



Property Accessor play a vital Role in Processing the Data during the Endeca Indexing . Almost all the applications will not be without the Accessors written to it.

There are two types of Accessors which is widely used or it can be defined as we can write accessors by extending the following two classes.

These can be defined as the based on the Type of usage we need.

1)       Extending  GenerativePropertyAccessor


This type of class can be used only we have multiple data to be returned during the Indexing. When we are extending this class then we have to implement the Below method or Override it to make it work Functional. The Return type of this accessor should be map , so it is MultiValued .

package com.endeca.index;

import java.util.HashMap;
import java.util.List;
import java.util.Map;

import atg.repository.RepositoryItem;
import atg.repository.search.indexing.Context;
import atg.repository.search.indexing.GenerativePropertyAccessor;
import atg.repository.search.indexing.IndexingOutputConfig;
import atg.repository.search.indexing.specifier.OutputProperty;
import atg.repository.search.indexing.specifier.PropertyTypeEnum;

public class ActivePropertyAccessor extends GenerativePropertyAccessor {

protected Map<String, Object> getPropertyNamesAndValues(Context pContext, RepositoryItem pItem, String pPropertyName, PropertyTypeEnum pType,
                                                boolean pIsMeta) {
                                Map<String, Object> res = new HashMap<String, Object>();
                 
// Business Logic for Processing the data

// pItem-Repository item of the Type  where we have defined (ie Sku  item or product item)

// pPropertyName – property name


                                return res;
                }


 2)       Extending PropertyAccessorImpl


This class can be used when the single value is used during Indexing.

package com.endeca.index;

import java.util.HashMap;
import java.util.Map;

import atg.repository.RepositoryException;
import atg.repository.RepositoryItem;
import atg.repository.search.indexing.Context;
import atg.repository.search.indexing.PropertyAccessorImpl;
import atg.repository.search.indexing.specifier.PropertyTypeEnum;

public class ActivePropertyAccessor extends PropertyAccessorImpl {

               
protected Boolean getTextOrMetaPropertyValue(Context pContext, RepositoryItem pItem, String pPropertyName,
                                                PropertyTypeEnum pType) {

                                // business logic for parsing data

                                return queryPropVal;
                }