Thursday, June 20, 2019

Workflow Monitoring - On Demand

The out of the box Workflow Monitoring level specified when deploying a version of a workflow is a great tool but it requires it having been explicitly turned on in order to be of value. I have found that while this was turned on for all users in a production environment, we would have a massive surplus of log data that would fill up the logging tables unnecessarily, some users logins may be setup to only do troubleshooting. Alternatively, a specific user could have logging turned up for so that all workflows are tracked rather than having to know all the workflow process names where an error is going to occur as the user may not know before hand which workflow is causing the problem. What is needed is to add logic to the 'Workflow Process Manager' business service.  The PreInvoke needs the following script added:

function Service_PreInvokeMethod (MethodName, Inputs, Outputs) {
  if (MethodName.indexOf("Run") >=0) {
    TheApplication().Utility.logRaiseWF(Inputs, MethodName);
  }
  return (ContinueOperation);
}

The Invoke needs the following script added:

function Service_InvokeMethod (MethodName, Inputs, Outputs) {
  if (MethodName.indexOf("Run") >=0) {
    TheApplication().Utility.logResetWF(Inputs, MethodName);
  }
}

The Utility service is enabled in the Application start event. The 'XXX Utilities' business service uses two additional methods.

function logRaiseWF(Inputs, callingMethod) {
// Method called from Workflow Process Manager, PreInvoke method for methods having 'Run' in the method name if the
// Utilities Service is enabled for the OM Component the session is running within.

  //If The Troubleshoot process property is passed (when invoking from a Named Method user prop) or as a child prop (when
  //invoked from a WF as a subprocess) or the user's log level is 3 or higher, temporarily set WF monitoring for the WF
  //process to detail.  logResetWF is called immediately after invokation to turn WF monitoring off.
  if (gCurrentLogLvl >= 3 || 
   Inputs.GetProperty("Troubleshoot") == "Y" || 
   (Inputs.GetChildCount() > 0 && Inputs.GetChild(0).GetProperty("Troubleshoot") == "Y")) { 
    if (TheApplication().GetProfileAttr("IsStandaloneWebClient") == "TRUE") {
      logStep("PPT Utilities.logRaiseWF.callingMethod: "+callingMethod+" - ProcessName: "+Inputs.GetProperty("ProcessName")); 
      logPS(Inputs);
    } else {
      logSetWFLevel(Inputs, "3 - Detail");
    }
  }

  //The first WF Called in a stack uses the RunProcess method.  If subprocess is called from a WF, the _ method is used. 
  //All payloads captured after the initial RunProcess call are stored in an array and dumped if an error occurs.  If 
  //system preference 'PPT Hold Buffer Max' is greater than 0 then the array is instead always kept at that count rather
  //than resetting on WF start
  if (gsTraceIntfaceReqResp != "FALSE" && callingMethod.indexOf("Run")== 0 && gHoldBufferMax == 0) gHoldReqResp = [];
}

function logResetWF(Inputs, callingMethod) {
// Method called from Workflow Process Manager, Invoke method for methods having 'Run' in the method name if the
// Utilities Service is enabled for the OM Component the session is running within.

  //Conditions controlling whether to reset WF monitoring level must mirror those in the logRaiseWF function
  if (gCurrentLogLvl >= 3 || 
   Inputs.GetProperty("Troubleshoot") == "Y" || 
   (Inputs.GetChildCount() > 0 && Inputs.GetChild(0).GetProperty("Troubleshoot") == "Y")) { 
    if (TheApplication().GetProfileAttr("IsStandaloneWebClient") == "TRUE") {
      logStep("PPT Utilities.logResetWF.callingMethod: "+callingMethod+" - ProcessName: "+Inputs.GetProperty("ProcessName")); 
    } else {
      logSetWFLevel(Inputs, "0 - None");
    }
  }
}

Finally, these wrapper functions call a function to actually set and reset the monitoring level on a deployed workflow:

function logSetWFLevel (Inputs, logLevel) {
//Use:  
//Returns: 
  var boWF:BusObject = TheApplication().GetBusObject("PPT Workflow Process Deployment");
  var bcWF:BusComp = boWF.GetBusComp("Workflow Process Deployment");
  var found:Boolean = false;
  var propName = Inputs.GetFirstProperty();

  while (propName != "") {
    if (propName.indexOf("ProcessName") == 0) {
      with (bcWF) {
        ActivateField("Monitoring Level");
        ClearToQuery();
        SetSearchSpec("Name", Inputs.GetProperty(propName));
        SetSearchSpec("Deployment Status", "Active");
        ExecuteQuery(ForwardOnly);
        found = FirstRecord();
        if (found) {
          SetFieldValue("Monitoring Level", logLevel);
          WriteRecord();
        }
      }     
    }
    propName = Inputs.GetNextProperty();
  }
}

To minimize the times where logging is actually raised, one of the following conditions must be true:

  • An input process property called 'Troubleshoot' has a 'Y' value.  I use this process prop across most of my workflows and only pass the 'Y' value when passed as a command/Named Method BC User property from a button on an admin view or an applet gear menu option.  It is useful to expose manually triggering a workflow this way to replicate a process that is normally triggered by the system in some way.
  • Users log level is 3 or higher

Note that if using the thick client, logging is NOT turned on so minimize SQL in the siebel.log file and since this information can easily be set through the SIEBEL_LOG_EVENTS system variable along with the Step and Process execution events.

XML Logger - EAI Data Transformation Engine

If you are already capturing the XML payloads of a web service using the XML Logger, then extending it to further troubleshoot how you might have ended up with that payload might be useful since integration workflows frequently undergo multiple transformations. The existing logRequest and logResponse as invoked by the PreInvoke method of the 'XXX Utilities' business service are central to the actual logic.  What is needed now is to add logic to the 'EAI Data Transformation Engine' business service.  The PreInvoke needs the following script added:

function Service_PreInvokeMethod (MethodName, Inputs, Outputs){
  if (MethodName=="Execute") {
    TheApplication().Utility.InvokeMethod("logTransformRequest", Inputs, TheApplication().NewPropertySet());
  } 
  return (ContinueOperation);
}

The Invoke needs the following script added:

function Service_InvokeMethod (MethodName, Inputs, Outputs) {
  if (MethodName=="Execute") {
    TheApplication().Utility.InvokeMethod("logTransformResponse", Outputs, TheApplication().NewPropertySet());
  } 
}

The logRequest and logResponse methods set the direction attribute to 'EAI Transform' to distinguish it and uses the MapName attribute as the functional name.  The Record Id is assumed to be an element called 'Id' in the first top level container.

I made a personal decision that logging these payloads was not universally necessary and therefore wanted to make logging these records conditional.  In my case I made it run in two scenarios:
  • If the executing user's log level is 5 and payload logging is enabled
  • If the payload logging is not generally enabled but an error occurred subsequently in the executing workflow of a subprocess it called.

Get Process and Thread Ids

It is sometimes useful to know the PID or Thread.  For instance if an error occurs and I want to get the specific server OM Log file I can find that file if I knew the Thread and PID.  I am not aware of any way to get these values directly though it is obvious the Siebel application OM has these values internally.  The TraceOn function allows these values to be used while opening and naming the trace file:
TheApplication().TraceOn("TraceFile_$p_$t.log", "Allocation", "All");
results in name like
TraceFile_7382_8188.log
So the trick is to create this file then read the relevant values out of the name.  To do so, I create the file with a unique name that will be known to the script creating it:
var unique = TheApplication().LoginName()+"-"+TimeStamp("DateTimeMilli");
TheApplication().TraceOn(path+"Trace-"+unique+"_$p_$t.log", "Allocation", "All");
To get the values I am interested in, I need to output the directory listing of only this known file to a log I can then open and read, then use format of the name of this file to extract the two values I am interested in:

function SetThreadPID() {
  var pid = TheApplication().GetProfileAttr("XXX OS PID");
  var threadId = TheApplication().GetProfileAttr("PPT OS Thread ID");
  var line, pidThread, path, command, outs;

  try {
    if (threadId == "") {
      if (TheApplication().GetProfileAttr("IsStandaloneWebClient") == "TRUE") {
        path = gsLogPath;
      } else {
        path = TheApplication().GetProfileAttr("Syspref Error Trace Temp Loc");
      }

      if (path != "" && path.toUpperCase() != "FALSE") {
        var unique = TheApplication().LoginName()+"-"+TimeStamp("DateTimeMilli");
        TheApplication().TraceOn(path+"Trace-"+unique+"_$p_$t.log", "Allocation", "All");
        TheApplication().Trace("TEST");
        TheApplication().TraceOff();  
    
        command = "dir "+path+"Trace-"+unique+"_*.log > "+path+"Trace-"+unique+".log";
        outs = Clib.system(command);
        var fp:File = Clib.fopen(path+"Trace-"+unique+".log","r");
        if (fp != null){
          while(Clib.feof(fp) == 0){
            line = Clib.fgets(fp);
            if (line.length > 0){
              if (line.indexOf(unique)>=0) {
                pidThread = line.substring(line.indexOf(unique)+unique.length+1, line.length - 5)
                pid = pidThread.substring(0, pidThread.indexOf("_"));
                threadId = pidThread.substring(pidThread.indexOf("_")+1);
                TheApplication().SetProfileAttr("PPT OS PID", pid);
                TheApplication().SetProfileAttr("PPT OS Thread ID", threadId);
                Clib.fclose(fp);
                outs = Clib.remove(path+"Trace-"+unique+"_"+pid+"_"+threadId+".log")
                outs = Clib.remove(path+"Trace-"+unique+".log")
                break;
              }
            }
          }
        }
      }
    }
  } catch(e) {
    RaiseError(e);
  } finally {
    fp = null;
  }
}

Once the PID and Thread are stored in profile attributes they are available to the business layer to for instance set a PID or thread column on a custom error table.

Note that 'Syspref Error Trace Temp Loc' is a custom field added to the 'Personalization Profile' BC which has the calculation:
SystemPreference("PPT Error Trace Temp Loc")
This is then set to a directory that the Siebel application server has access to (the Siebel temp directory can generally be used safely).

Use Open UI to Dynamiclly Manipulate Detail Tabs

In a screen with many view tabs it may be useful for process automation to minimize clicking on detail tabs if user does not need to navigate there when no records are present.  Open UI allows changing the view tab labels to provide indicators to signal to the user information about that tab.  In order to do so, join fields or calculations based on MV fields relevant to the child BC need to exist and be exposed as controls on the parent BC applet (they can be hidden).

One additional feature is to hide the view tabs not requiring navigation for UI optimization but keeping them available in case the user needs them.  Siebel uses a UI dropdown widget when there are too many views to fit horizontally across.  We can leverage this widget to conditionally place additional views based on the values in BC fields.


The following script is attached to a navigation manifest event:



if(typeof(SiebelAppFacade.pptCustomNavigationPR) === "undefined"){ 
  SiebelJS.Namespace("SiebelAppFacade.pptCustomNavigationPR"); 
  define ("siebel/custom/pptCustomNavigationPR", ["siebel/accnavigationphyrender"], function () { 
    SiebelAppFacade.pptCustomNavigationPR = (function(){ 
      var PM; 
      var PRName = ""; 
      function pptCustomNavigationPR(pm){ 
      SiebelAppFacade.pptCustomNavigationPR.superclass.constructor.apply(this,arguments);} 
      SiebelJS.Extend(pptCustomNavigationPR, SiebelAppFacade.AccNavigationPhyRenderer); 
            
      pptCustomNavigationPR.prototype.Init = function() { 
        SiebelAppFacade.pptCustomNavigationPR.superclass.Init.apply(this, arguments); 
        PM = this.GetPM(); 
        PRName = PM.GetPMName(); 
      }; 
      /*pptCustomNavigationPR.prototype.ShowUI = function(){ 
        SiebelAppFacade.pptCustomNavigationPR.superclass.ShowUI.apply(this, arguments); 
        //implement ShowUI method here 
      }; 
      pptCustomNavigationPR.prototype.BindEvents = function(){ 
        SiebelAppFacade.pptCustomNavigationPR.superclass.BindEvents.apply(this, arguments); 
        //implement BindEvents method here 
      };*/
            
      pptCustomNavigationPR.prototype.BindData = function(bRefresh){ 
        SiebelAppFacade.pptCustomNavigationPR.superclass.BindData.call(this, bRefresh); 
                
        //Prototype for child record counter on view tabs 
        if (PRName == "NavigationDetailObject_PM"){ 
          //the framework is processing detail navigation, this is a good place for code that manipulates view tabs. Get applet, control, value and properties 
          //Code assumes the top form applet has a control that exposes a count 
          var oView = SiebelApp.S_App.GetActiveView();
                                  
          if (typeof(oView) != 'undefined' && oView != null && oView.hasOwnProperty("GetName") == true) {
            var sViewName = oView.GetName();
                 
            //Limit execution of this script to only views matching a naming convention as it must have a parent form applet containing the hidden calculated fields
            if ( sViewName.indexOf("XXX Search Text") >= 0 ){
              var suppressTabs = true;

              //This applet must have controls containing the BC fields that will be used in the array
              var applet = oView.GetAppletMap()["XXX Parent Form Applet"];

              //Declared Array where index is UI display name of the detail tab and value is either BC Field Name or '-'.  If field name, non 0/non null value indicate tab should
              //be displayed.  '-' indicates tab should always be hidden.  If tab should always be displayed, do not put it in the array
              var tabList = {"Contacts":"XXX Contact Count","Activities":"XXX Activity Count","Service Requests":"XXX SR Count","Notes":"XXX Notes Flag","Fees":"XXX Fees Flag","Audit Trail":"-"};
              var hitCount, fieldName;
              var tabIndex = 0;
              var tabs = [];
              var tabScreens = [];
              var showWidget = false;
              var lastTab;
      
              //loop through each visible detail tab... 
              $(".siebui-subview-navs .siebui-nav-tabScreen .ui-tabs-nav a").each(function(index){ 
                //get the current tab label text 
                var currentLabel = $(this).text(); 
                  
                //check if tab is in array of labels that need modification
                fieldName = tabList[currentLabel];
                if (typeof(fieldName)!='undefined' && fieldName != ""){  //we found the tab 
                  if (fieldName == "-") hitCount = "";
                  else hitCount = applet.GetBusComp().GetFieldValue(fieldName);

                  //Either modify the label if an indicator needs to be appended or if tab is to be suppressed, add to an array of labels to appear in the option list
                  if (hitCount != "" && hitCount != "0") {
                  //now change the text 
                    $(this).text(currentLabel + " (" + hitCount + ")"); 
                    lastTab = $(this);
                  } else if (suppressTabs) {
                  //If this tab should be generally suppressed, there are no indicators needing to be displayed, and it is not currently selected
                    if (fieldName != "" && (hitCount == "" || hitCount == 0) && $(this).parent().attr("tabindex")!= "0") {
                      showWidget = true;
                      tabs[tabIndex] = currentLabel;
                      tabScreens[tabIndex++] = $(this).attr("data-tabindex").substring(9);
                      $(this).remove();
                    } else {
                      lastTab = $(this);
                    }
                  }   
                }
              }); 
              
              //If any tabs need to be suppressed, display a drop down at the end of the detail tab row with list of view tabs that have been suppressed
              if (showWidget == true) {
                var j=0;
                var htmlstring = lastTab.parent().parent().html();
                var append = '<li><select aria-atomic="true" aria-label="Third" bar="" class="siebui-nav-links siebui-nav-viewlist" id="j_s_vctrl_div_tabScreen" level="" role="combo" view="">==$0<option hidden="" value=""></option>';</select></li>
                while (j < tabIndex) {
                  append = append + '<option value="tabScreen'+tabScreens[j]+'">'+tabs[j++]+'</option>';
                }
                append = append + '</select></li>';
                lastTab.parent().parent().html(htmlstring+append);
              }
            }
          }
        }        
      }; 
              
      return pptCustomNavigationPR; 
    }()); 
    return "SiebelAppFacade.pptCustomNavigationPR"; 
  }); 
}

Tuesday, March 8, 2016

Thick Client Event Logging

There are surprisingly few blog posts out there about vanilla options for logging in the thick client.  Perhaps this is because everyone knows how to do it and if so feel free to ignore this.  But perhaps it is because most people just struggle through using inefficient methodologies.

Here is a simple tip for troubleshooting when using the thick client.  There is an OS environment variable called SIEBEL_LOG_EVENTS (if it does not exist you can create it). Many developers know how to set this to an integer between 0 and 5, but values of 4 and 5 where good detail is provided create a file that unreasonably large and hard to parse.  When troubleshooting on the thin client you can set individual event log levels from the Administration - Server Configuration screen, component events view.  You can do the same when using a thick client though you need to do it using the SIEBEL_LOG_EVENTS variable.  You can use any combination of event aliases and levels, but the value I have found useful is the following:
StpExec=4,PrcExec=4,ObjMgrSqlLog=4,SQLParseAndExecute=4,ObjMgrBusServiceLog=4,EventContext=4,ProcessRequest=4,ObjMgrDBConnLog=5,SecAdpLog=5,ObjMgrSessionLog=5,ObjMgrBusCompLog=2
Basically you can enter any comma separated list of event aliases.

One coda is that if I were trying to troubleshoot a WF issue, I could open this log and do a find for the word 'Instantiating'.  The first instance I would find is the Start step of the WF Process followed by the values of the process properties being set by that step:
PrcExec Create 4 00000002569f1a98:0 2016-01-20 15:31:33 Instantiating process definition 'PPT Passport History Response Integration'.
PrcExec PropSet 4 00000002569f1a98:0 2016-01-20 15:31:33 Setting runtime value of property 'Namespace: 'USER' Name: 'ConfigItem' Datatype: 'String'' to:
PrcExec PropSet 4 00000002569f1a98:0 2016-01-20 15:31:33 Start
PrcExec PropSet 4 00000002569f1a98:0 2016-01-20 15:31:33 Setting runtime value of property 'Namespace: 'USER' Name: 'ObjectName' Datatype: 'String'' to:
PrcExec PropSet 4 00000002569f1a98:0 2016-01-20 15:31:33 Workflow - PPT Test Error Process
PrcExec PropSet 4 00000002569f1a98:0 2016-01-20 15:31:33 Setting runtime value of property 'Namespace: 'USER' Name: 'CurrentStep' Datatype: 'String'' to:
PrcExec PropSet 4 00000002569f1a98:0 2016-01-20 15:31:33 Convert Siebel Message PPH
Subsequent occurrences look like this:
StpExec Create 4 00000002569f1a98:0 2016-01-20 15:31:33 Instantiating step definition 'Start'.
StpExec End 4 00000002569f1a98:0 2016-01-20 15:31:33 Stopping step instance of 'Start' with a 'Completed' status.
In this way you can step through the WF.  The advantage of this logging level over say looking at the WF Instance Monitor or only using WF Simulator, is you will be able to see the SQL executed and the bind variables used, what BCs were instantiated along the way, and what BS methods might have been called.

Tuesday, February 9, 2016

EAI Integration Map expressions

While there are many posts I have seen that talk about expressions supported by 'EAI Data Transformation Engine', I have never seen an attempt to compile a list of supported functions and examples of there uses.  So this will be a humble beginning that will hopefully grow over time.  Note that these functions are mostly VB so if trying out one that is not listed, start with what is supported in VB.  They can also be found in Siebel Bookshelf as Siebel Query Language expressions

Do not include XML element (use System type)
IfNull([Middle Name], [Conflict Id])

Transforming Dates
(from 'YYYYMMDD' to 'MM/DD/YYYY'):
Right(Left([Source Field Name],7),2) +"/"+Right([Source Field Name], 2)+"/"+Left([Source Field Name], 4)

(from Siebel Date to externally recognized format):
ToChar([Birth Date], 'YYYY-MM-DD')
ToChar([Completion Date], 'YYYY-MM-DDThh:mm:ss')

Conditional Logic:
IIF([Source Field Name] = "false", "N", "Y")

SSN formatting (Strip hyphens):
IIF(InStr([PPT Social Security Number], "-") > 0, Left([PPT Social Security Number], 3)+Mid([PPT Social Security Number], 5, 2)+Right([PPT Social Security Number], 4), [PPT Social Security Number])

EAI Lookup for an Inbound Interface:
  • EAILookupSiebel("XXX",[Source Field Name])
    • XXX is the Type in the EAI Lookup table.  This needs also needs to be setup as a value under the EAI_LOOKUP_MAP_TYPE LOV type.
  • IIF([Source Field Name] IS NULL, "", EAILookupSiebel("XXX",[Source Field Name]))
    • EAILookupSiebel fails if no value is found so minimize this possibility unless an exception is desired
Extract the file name from a File Path:
Mid([Source Field Name], InStr([Source Field Name], "/", -1) + 1)

Thursday, October 22, 2015

Interesting Web Service approach

I was recently working on a client where all the integration used HTTP Web Services but were not implemented using what I think most Siebel Developers would think of as the "Best Practice".  Basically, the payloads were created using XSLT and the actual call to the Web Service was invoked using 'EAI HTTP Transport' rather than a custom WS Proxy BS created using the Tools wizard. 

The reasoning provided to me behind using XSLT to create the payload is that it was more flexible.  What do I mean by that?  Well assume an outbound interface is needed where the "Best Practice" alternative would be to use 'EAI Siebel Adapter' to query an Integration Objects , then to use 'EAI Data Transformation Engine' to transform the payload into an IO (that was initially created by consuming a WSDL) recognizable by the Web Service.  The limitation in this approach is around a couple pieces that Siebel Tools wizards have trouble with.  The first is that some modern standards compliant  WSDL definitions that use recursive data types cannot be imported at all.  The second is that the SOAP Header follows a standard that is somewhat outdated, is not configurable, and basically requires scripting to create a custom header anyway.  So the question is whether it is better to create the custom header using a scripted Filter Service, or to create the payload using XSLT.  Once you go down the path of using XSLT, you basically cant use a Proxy Service anymore (since the proxy would be adding the SOAP envelope) so 'EAI HTTP Transport' is used instead.

I can think of a number of downsides to this approach:
  • Deployment Complexity increases
    • XSLT files must be deployed to the File System and kept in sync across however many app servers (and failover servers) are used by EAI in the respective environments
    • No WSDL is actually consumed in this approach so web service end points must be stored somewhere which will likely be different in each environment
  • More Steps in the WF
    • 'EAI XSLT Service'  uses UTF-16 input so it is likely Encoding will need to occur both to and from using 'Transcode Service'
    • Reading the XSL File from the file system
  • Mainainability
    • Siebel resources that know XSLT are presumably more rare than Siebel resources familiar with more "Best Practice" approaches
  • Performance
    • Calls to the File System to get the XSLT file might add significant load to a high volume interface
  • Data Integrity
    • The integrity of the outbound message data structure is not really enforceable in Siebel. Using XSLT requires the developer to create a payload that is correct as only the external system would be able to validate it.  This is perhaps debateable because ultimately the developer will probably need to resolve this one way or the other during development.  While I personally believe it is easier to troubleshoot problems that are actually identified within Siebel due to the strict defintion of the messages maintained in Siebel, I will concede that might be personal preference.
I am curious what others think and whether developers that use this approach can defend it better than I can.  Ultimately, I think a scripted filter service is a better solution to the custom Soap Header issue, though I think this approach seems reasonable if the WSDL cannot be consumed, and modifying it is not possible.