Implementing Partial Indexing in Endeca

Implementing Partial Indexing in Endeca

Indexing Syed Ghouse Habib   August 06, 2017

Hi all,

    After Spending more time with the Partial Indexing, I am Writing this posts, so it could be useful for the other people trying to implement the Partial Indexing for their Projects.

Usually all the Projects, would have Configured the Baseline Indexing in the Scheduled Manner, If We have got the Requirement of Indexing the Inventory From Endeca , Will think of Having the Partial Indexing Configured for the Scheduled Manner, the Constraint is that it cannot pick up the Changes, Since the Inventory Comes or updates as part of the Feed Process.

Before pitching in to the Discussion of that, let’s see what are all can be achieved with the Partial Indexing.

When the Deployment is made from BCC by Creating a new Proj will update/add the Properties , if the Properties Belong to the Same Repository.ie Adding the Promotion will not reflect after the Deployment and the  Partial Indexing , the Reason is that the Promotion belongs to the Different Repository not the Product Catalog.

So , How Does the Flow Goes or what is Behind the Scenes of the Partial Indexing ? .

Before Knowing the Flow, we all should aware of the Basics of Partial Indexing.

Basics of Partial Indexing

There is a Repository called the Incremental Item Queue that is Responsible for Triggering the Partial Indexing. We have a Property called the Monitored OutputConfigs that should have our Config paths eg: ProductCatalogOutputConfig , Article OutputConfig.

There Will be Generation Id Associated With the Each OutputConfig Paths. So During Deployment the Sku or the Product, or Article Will be Added to this Incremental Item Queue Repository along with the Generation Id and the Item Descriptor . During the Partial Indexing this table will be Flushed out by submitting those to the Indexing . If the Products are not loading as part of the Partial Indexing after knowing that Items are avalaible in Incremental Item Queue Repository, then the Problem with miss match of the GenerationId.

You are also should be aware if the Tables from the DB Where these Operations take place.

srch_update_queue: this Table Where will be having the

config_repo_id will be the Config id of the output config path .

item_desc_name will be the Item Descriptor name .

repository_id is the Id of the updated product / Sku .

generation is the value of the Current id .

change_type is the value of the Change Type .

please find the Below Example .

values ('14500010','product','product101','2279','0');

srch_config this table is the Place where will be getting the Config Repoid and Generation for the Particular Config Path.

So now you are aware of the Flow of Partial Indexing Flow. If your Requirement to Indexing comes as part of the Feed then you have to identify the Changed Items and Write explicitly in to the Incremental Item Queue and Scheduled the Indexing of the Partial then it will take.

If your Property Belongs to the Different ItemDescriptior or Repository , make sure that you are making the Linking to any of the property in the Product Catalog  after linking only it will be taken as part of the Indexing otherwise will not be taken.

Only those Properties that will not be added as part of the BCC Deployment , We have to Write Explicitly in to the Incremental Item Queue. Others not Required .

So Keep this Information or the Constraint in mind Before Start Implementing to your Application .

Happy Learning.!!!!! 

Same Dimension Value Across all the Environments

Same Dimension Value Across all the Environments

Indexing Syed Ghouse Habib   July 11, 2017

Hi All,
    Most of us , When having more Environments will face issues regarding the Dimension Values Generation. I Have Faced a lot , though am not hard-coding the Dimension Values, faced issues relating to recreating the refinements  if any Dimension Change occur . Now we are going to see how to Handle this .

When you have Indexing Successful in any environment, then Import its Dimension Values Using Below Command / Script

When the App is Created using the CRS Deployment Template, go to the Location /opt/endeca/apps/ATGen/test_data/ You Will Find a file called initial_dval_id_mappings.csv
Delete it and Proceed to Execte the Command, this Command has to be Executed from the CAS bin.

sh cas-cmd.sh exportDimensionValueIdMappings -m appName-dimension-value-id-manager -f /opt/endeca/apps/ATGen/test_data/initial_dval_id_mappings.csv 

Don't Forget to Create the Output File with the Following name initial_dval_id_mappings.csv , if you are Changing it you have to Change in the Script as well . It is better to use the Same .

Once this File is Created Check it out , you will have all the Dimensions Indexed as part of your Indexing. This File Has to be Moved Across all the Environment and Initialize_services.sh has to be called for the First Time. Once it is Called Going Forward all the Dimension values will remain same as the Lower Environment .

initial_dval_id_mappings.csv this file  will be called as part of the InitialSetup during the Initialize_Services.sh .

<script id="InitialSetup">
<bean-shell-script>
<![CDATA[
IFCR.provisionSite();
CAS.importDimensionValueIdMappings("TAen-dimension-value-id-manager",
InitialSetup.getWorkingDir() + "/test_data/initial_dval_id_mappings.csv");
]]>
</bean-shell-script>
</script>

Apart from this Approach we can also do the ImportDimensionValueIdMappings Script. Using the File Will make the Dimension Value not to Change for the Longer Run.

Weblogic cxf logging services

Weblogic cxf logging services

Weblogic Syed Ghouse Habib   March 24, 2017

When you face the below logs in web logic console and the console stopped unexpectedly or hangs or stepped unexpectedly , then you have to follow the below steps for resolving it.

Mar 24, 2017 6:44:55 PM org.apache.cxf.services.RecordStoreService.RecordStorePort.RecordStore

INFO: Outbound Message

---------------------------

ID: 1

Address: http://localhost:8500/ATGen-dimvals/

Encoding: UTF-8

Content-Type: text/xml

Headers: {Accept=[application/fastinfoset, */*], Accept-Encoding=[gzip;q=1.0, identity; q=0.5, *;q=0], SOAPAction=[""]}

Payload: <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><startTransaction xmlns="http://recordstore.itl.endeca.com/" xmlns:ns2="http://record.itl.endeca.com/"><transactionType>READ_WR

--------------------------------------

Mar 24, 2017 6:44:55 PM org.apache.cxf.services.RecordStoreService.RecordStorePort.RecordStore

INFO: Inbound Message

----------------------------

ID: 1

Response-Code: 200

Encoding: ISO-8859-1

Content-Type: application/fastinfoset

Headers: {Content-Length=[196], content-type=[application/fastinfoset], Date=[Fri, 24 Mar 2017 13:14:55 GMT], Server=[Jetty(6.1.26)]}

Payload: α ☺ 8╧♥soap(http://schemas.xmlsoap.org/soap/envelope/≡???Envelope???♥Body8╧☻ns2∟http://record.itl.endeca.com/═!http://recordstore.itl.endeca.com/≡=?↨startTransactionResponse=?♣return=?☺id?26   ≡

Step:1

Go to $WL_HOME/../user_projects/domains/base_domain/config/

And add the following In config.xml

<log-filter>
    <name>CXFFilter</name>
    <filter-expression>((SUBSYSTEM = 'org.apache.cxf.interceptor.LoggingOutInterceptor') OR (SUBSYSTEM = 'org.apache.cxf.interceptor.LoggingInInterceptor')) AND (SEVERITY = 'WARNING')</filter-expression>
  </log-filter>

 <server>

   <name>Prod</name>

   <log>

     <log-file-filter>CXFFilter</log-file-filter>

     <stdout-filter>CXFFilter</stdout-filter>

     <memory-buffer-severity>Debug</memory-buffer-severity>

   </log>

   <listen-port>7103</listen-port>

   <web-server>

     <web-server-log>

       <number-of-files-limited>false</number-of-files-limited>

     </web-server-log>

   </web-server>

   <listen-address></listen-address>

 </server>

For the server which you were using should be updated . with this filter .

You have to define for the server and after that restart the weblogic  admin and managed server .

You can also define this from WebLogic console which is very easy when compared to this.

<YOUR_DOMAIN>/logfilters/

create the new filter termed  CXFFilter for the expression give  the below 

((SUBSYSTEM = 'org.apache.cxf.interceptor.LoggingOutInterceptor') OR (SUBSYSTEM = 'org.apache.cxf.interceptor.LoggingInInterceptor')) AND (SEVERITY = 'WARNING')

then go to your corresponding server and select logging, for the LogFile and Standard out give the filter as CXFFilter , save and restart the server .


Step:2

Go to <JAVA_HOME>\jre\lib\logging.properties  and set the following properties

org.apache.cxf.interceptor.LoggingInInterceptor.level = WARNING

org.apache.cxf.interceptor.LoggingOutInterceptor.level = WARNING

After Setting up this then you have to Start the Server and start indexing . Happy Indexing !!!!!!

Triggering AutoIndexing in Stage Environment

Triggering AutoIndexing in Stage Environment

Indexing Syed Ghouse Habib   March 02, 2017

Hi Guys , today I Gonna Share the WorkFlow for the Staging indexing , this tutorials will be very interesting with respect to the other functionalities because almost all is OOTB, you need to place right things In place.

ATG does not provide the automatic indexing enable for the staging env, you need to enable it .let’s find the detailed discussion below.

Before proceeding to this tutorial you need to make sure that you have the following modules built as part of your build.

BCC: DAF.Endeca.Index.Versioned,DCS. Endeca.Index.Versioned these two components are responsible for the Componets required for indexing in Staging env.

Staging: DAF.Endeca.Index,DCS.Endeca.Index

Once you are done with the above modules, you can proceed further.

1) Make sure to add the following outputconfig in /atg/search/repository/IndexingDeploymentListener

indexingOutputConfigs+=\

  /atg/commerce/search/ProductCatalogOutputConfig_staging

2) Make sure to update the staging details in following component /atg/search/SynchronizationInvoker_staging

host=10.20.30.70

port=1072

where 

host nameof staging

port  is staging rmi port

3)make sure that the /atg/commerce/search/IndexedItemsGroup_staging points to your productCatalog_Staging Repository to do that update the ProductCatalogOutputConfig_staging by following values

repositoryItemGroup=/atg/commerce/search/IndexedItemsGroup_staging/

repository^=/atg/commerce/search/IndexedItemsGroup_staging.repository

Only these changes are required to be done in the BCC Side.

We will see the Changes in the Staging side now

1) update the CategoryTreeService, RepositoryTypeDimensionExporter, SchemaExporter with the below changes

indexingOutputConfig=/atg/commerce/search/ProductCatalogOutputConfig_staging

2)update the ProductCatalogSimpleIndexingAdmin to points to /atg/commerce/search/ProductCatalogOutputConfig_staging during indexing .

phaseToPrioritiesAndTasks=\

  PreIndexing=5:CategoryTreeService,\

  RepositoryExport=10:\

    SchemaExporter;\

    CategoryToDimensionOutputConfig_staging;\

    RepositoryTypeDimensionExporter;\

    /atg/commerce/search/ProductCatalogOutputConfig_staging,\

  EndecaIndexing=15:EndecaScriptService

That’s it you are done with the Staging Indexing workflow .So after this changes once you deploy the project indexing will be triggered in both staging and production.

For Indexing only in Production you can refer my previous posts.

Happy Day

RuleSet in ATG / Indexed Item Group

RuleSet in ATG / Indexed Item Group

Indexing Syed Ghouse Habib   December 17, 2016

HI all, I feel all are good and doing their best. Today we are going to see some interesting topic on the indexing part of the endeca. We cannot claim this topic to be fully endeca . Its ATG topic that we are utilizing for the indexing as well. It is  considered to be the simpler one but I am writing this blog to ease most of the logic we write during the accessor time.Its very effective and easy.

Ruleset can be defined as the couple of condition that is written for some specific purpose or does some specific purpose.As of my knowledge on endeca there is no mechsanism where we can restrict the products or the skus to be restricted from the search , apart from the recordfilter property constraints. If we implement the above constraint also that’s going to be the highly performance driven one. If that does not require more than reverting the code is also a complex one.

ProductCatalog outputconfig is a component that is responsible for providing the product for indexing . in the above component we can define the Repository Item group property to be pointing to the IndexedItem Group component where we will be defining the ruleset.

Rule set has two types of the condition.

1) Accepts rule.

2) Rejects rule.

These rules were called during the indexing process and corresponding filtration is done.  You can find the oracle document briefly describing about this. I am going to describe in terms of  Integration with the ATG and some of the problems we face during the development of the endeca.

Case:1

To Remove the Inactive products during the Indexing. We can define this type of condition in both accepts and reject clause.

 rules=\

<ruleset>\

<accepts>\

                <rule op\=eq>\

                                <valueof target\="isActive">\

                                <valueof constant\="true">\

                 </rule>\

</accepts>\

 </ruleset>

Case: 2

To remove only the specific set of items during the indexing. Where item may be product or some other repo item. Were id is the repo id.

rules=\

<ruleset>\

<rejects>\

                <rule op\=includes>\

   <valueof constant\="[item1,item2]">\

  <valueof target\="id">\

    </rule>\

    </rejects>\

</ruleset>

How to define in that component .

/atg/commerce/search/IndexedItemsGroup.properties

Define the ruleset in the string format is must. we cannot define the ruleset for the skulevel . Since the product-sku mapping has defined the sku as map .

Scheduled indexing in endeca

Scheduled indexing in endeca

Indexing Syed Ghouse Habib   October 27, 2016

In Endeca Scheduled Indexing can be made by following the Below steps.

Step :1

Go to ProductCatalogSimpleIndexingAdmin

and Enable the Following property enableScheduledIndexing to true .

Step :2

Set the Calendar event in the Below Property

baselineSchedule=calendar * * 7 * 23 30

where 7 is the Week,23 30 is  it will trigger indexing daily by 23:30

 to trigger indexing for every one hour 

set the Following property to

baselineSchedule=calendar * * * * * 59

It will trigger for every one hour periodically.